Patent Publication Number: US-11651763-B2

Title: Multi-speaker neural text-to-speech

Description:
CROSS-REFERENCE To RELATED APPLICATIONS 
     This patent application is a divisional of and claims the priority benefit of co-pending and commonly-owned U.S. patent application Ser. No. 15/974,397 (Docket No. 28888-2144), filed on 8 May 2018, entitled “SYSTEMS AND METHODS FOR MULTI-SPEAKER NEURAL TEXT-TO-SPEECH,” and listing Sercan O. Arik, Gregory Diamos, Andrew Gibiansky, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou as inventors, which claims the priority benefit under 35 USC § 119(e) to U.S. Provisional Patent Application No. 62/508,579 (Docket No. 28888-2144P), filed on 19 May 2017, entitled “SYSTEMS AND METHODS FOR MULTI-SPEAKER NEURAL TEXT-TO-SPEECH,” and listing Andrew Gibiansky, Sercan O. Arik, John Miller, Jonathan Raiman, Kainan Peng, Wei Ping, Gregory Diamos, and Yanqi Zhou as inventors. Each of the aforementioned patent documents is incorporated by reference herein in its entirety and for all purposes. 
    
    
     BACKGROUND 
     Technical Field 
     The present disclosure relates generally to systems and methods for machine learning that can provide improved computer performance, features, and uses. More particularly, the present disclosure relates to systems and methods for improved automation of text-to-speech conversion. 
     Background 
     Artificial speech synthesis, commonly known as text-to-speech (TTS), has a variety of applications in technology interfaces, accessibility, media, and entertainment. Fundamentally, it allows, among other uses, human-technology interactions without requiring visual interfaces. 
     Due to its complexity, developing TTS systems can be very labor intensive and difficult. Most TTS systems are built with a single speaker voice, and multiple speaker voices are provided by having distinct speech databases or model parameters. As a result, developing a TTS system with support for multiple voices requires much more data and development effort than a system which only supports a single voice. 
     Accordingly, what is needed are improved systems and methods that create high quality text-to-speech systems using neural speech synthesis models that can learn effectively from small amounts of data spread among hundreds of different speakers. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       References will be made to embodiments of the invention, examples of which may be illustrated in the accompanying figures. These figures are intended to be illustrative, not limiting. Although the invention is generally described in the context of these embodiments, it should be understood that it is not intended to limit the scope of the invention to these particular embodiments. Items in the figures are not to scale. 
       FIG. (“FIG.”)  1  depicts a diagram of an example inference system according to embodiments of the present document. 
         FIG.  2 A  graphically depicts a diagram for training procedure for a single-speaker Deep Voice 2 embodiment, according to embodiments of the present document. 
         FIG.  2 B  graphically depicts a diagram for inference/deployment a single-speaker Deep Voice 2 embodiment, according to embodiments of the present document. 
         FIG.  3 A  graphically depicts a diagram for training procedure for a multi-speaker Deep Voice 2 embodiment, according to embodiments of the present document 
         FIG.  3 B  graphically depicts a diagram for inference/deployment of a multi-speaker Deep Voice 2 embodiment, according to embodiments of the present document. 
         FIG.  4    graphically depict an architecture embodiment for a multi-speaker segmentation model, according to embodiments of the present document. 
         FIG.  5    depicts graphically depicts a simplified diagram for training of a multi-speaker segmentation model, according to embodiments of the present document. 
         FIG.  6    depict a method for training a multi-speaker segmentation model, according to embodiments of the present document. 
         FIG.  7    graphically depict an architecture embodiment for a multi-speaker duration model, according to embodiments of the present document. 
         FIG.  8 A  depicts graphically depicts a simplified diagram for training of a multi-speaker duration model, according to embodiments of the present document. 
         FIG.  8 B  is a simplified graphical depiction of a trained multi-speaker duration model&#39;s role during deployment/inference, according to embodiments of the present document. 
         FIG.  9    depict a method for training a multi-speaker duration model, according to embodiments of the present document. 
         FIG.  10    graphically depict an architecture embodiment for a multi-speaker frequency model, according to embodiments of the present document. 
         FIG.  11 A  depicts graphically depicts a simplified diagram for training of a multi-speaker frequency model, according to embodiments of the present document. 
         FIG.  11 B  is a simplified graphical depiction of a trained multi-speaker frequency model&#39;s role during deployment/inference, according to embodiments of the present document. 
         FIG.  12    depict a method for training a multi-speaker frequency model, according to embodiments of the present document. 
         FIG.  13    depicts a vocal model architecture, according to embodiments of the present document. 
         FIG.  14 A  depicts graphically depicts a simplified diagram for training of a multi-speaker vocal model, according to embodiments of the present document. 
         FIG.  14 B  is a simplified graphical depiction of a trained multi-speaker vocal model&#39;s role during deployment/inference, according to embodiments of the present document. 
         FIG.  15    depict a method for training a multi-speaker vocal model, according to embodiments of the present document. 
         FIG.  16    depicts a method for using a multi-speaker text-to-speech system, according to embodiments of the present document. 
         FIG.  17    depicts a Tacotron-WaveNet model with speaker conditioning in the Encoder CBHG module and decoder with two ways to convert spectrogram to audio: Griffin-Lim or a speaker-conditioned Vocal model, according to embodiments of the present document. 
         FIG.  18    depicts principal components of the learned speaker embeddings of an 80-layer vocal model embodiment and for a character-to-spectrogram model embodiment for VCTK dataset, according to embodiments of the present document. 
         FIG.  19    depicts a simplified block diagram of a computing device/information handling system, in accordance with embodiments of the present document. 
         FIG.  20    depicts time-dependent fundamental frequency profiles (collectively the F 0  and voicedness) generated by a model embodiment when the inputs and embeddings correspond to two speakers (Speaker S 6  and S 24 ), according to embodiments of the present document. 
         FIG.  21    presents attention plots for three speakers who speak at different speeds, according to embodiments of the present document. 
         FIG.  22    graphically depicts an architecture for speaker discrimination, according to embodiments of the present document. 
     
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS 
     In the following description, for purposes of explanation, specific details are set forth in order to provide an understanding of the invention. It will be apparent, however, to one skilled in the art that the invention can be practiced without these details. Furthermore, one skilled in the art will recognize that embodiments of the present invention, described below, may be implemented in a variety of ways, such as a process, an apparatus, a system, a device, or a method on a tangible computer-readable medium. 
     Components, or modules, shown in diagrams are illustrative of exemplary embodiments of the invention and are meant to avoid obscuring the invention. It shall also be understood that throughout this discussion that components may be described as separate functional units, which may comprise sub-units, but those skilled in the art will recognize that various components, or portions thereof, may be divided into separate components or may be integrated together, including integrated within a single system or component. It should be noted that functions or operations discussed herein may be implemented as components. Components may be implemented in software, hardware, or a combination thereof. 
     Furthermore, connections between components or systems within the figures are not intended to be limited to direct connections. Rather, data between these components may be modified, re-formatted, or otherwise changed by intermediary components. Also, additional or fewer connections may be used. It shall also be noted that the terms “coupled,” “connected,” or “communicatively coupled” shall be understood to include direct connections, indirect connections through one or more intermediary devices, and wireless connections. 
     Reference in the specification to “one embodiment,” “preferred embodiment,” “an embodiment,” or “embodiments” means that a particular feature, structure, characteristic, or function described in connection with the embodiment is included in at least one embodiment of the invention and may be in more than one embodiment. Also, the appearances of the above-noted phrases in various places in the specification are not necessarily all referring to the same embodiment or embodiments. 
     The use of certain terms in various places in the specification is for illustration and should not be construed as limiting. A service, function, or resource is not limited to a single service, function, or resource; usage of these terms may refer to a grouping of related services, functions, or resources, which may be distributed or aggregated. 
     The terms “include,” “including,” “comprise,” and “comprising” shall be understood to be open terms and any lists the follow are examples and not meant to be limited to the listed items. Any headings used herein are for organizational purposes only and shall not be used to limit the scope of the description or the claims. Each reference mentioned in this patent document is incorporate by reference herein in its entirety. 
     It shall be noted that these experiments and results are provided by way of illustration and were performed under specific conditions using a specific embodiment or embodiments; accordingly, neither these experiments nor their results shall be used to limit the scope of the disclosure of the current patent document. 
     Furthermore, one skilled in the art shall recognize that: (1) certain steps may optionally be performed; (2) steps may not be limited to the specific order set forth herein; (3) certain steps may be performed in different orders; and (4) certain steps may be done concurrently. 
     1. Introduction 
     Artificial speech synthesis, commonly known as text-to-speech (TTS), has a variety of applications in technology interfaces, accessibility, media, and entertainment. Most TTS systems are built with a single speaker voice, and multiple speaker voices are provided by having distinct speech databases or model parameters. As a result, developing a TTS system with support for multiple voices requires much more data and development effort than a system which only supports a single voice. 
     Presented herein, it is shown that all-neural multi-speaker TTS systems, which share the vast majority of parameters between different voices, can be built. It is also shown that not only can embodiments of a single model generate speech from multiple different voices, but also that significantly less data is required per speaker than when training single-speaker systems. 
     Presented herein are architecture embodiments (which may be generally referred to herein for convenience as Deep Voice 2 or DV2) and which may be considered as novel architectures related to embodiments disclosed in commonly-assigned U.S. patent application Ser. No. 15/882,926 (Docket No. 28888-2105), filed on 29 Jan. 2018, entitled “SYSTEMS AND METHODS FOR REAL-TIME NEURAL TEXT-TO-SPEECH,” and U.S. Prov. Pat. App. No. 62/463,482 (Docket No. 28888-2105P), filed on 24 Feb. 2017, entitled “SYSTEMS AND METHODS FOR REAL-TIME NEURAL TEXT-TO-SPEECH,” each of the aforementioned patent documents is incorporated by reference herein in its entirety (which disclosures may be referred to, for convenience, as “Deep Voice 1”). In one or more embodiments, a WaveNet-based (van der Oord, Aaron; Dieleman, Sander; Zen, Heiga; Simonyan, Karen; Vinyals, Oriol; Graves, Alex; Kalchbrenner, Nal; Senior, Andrew; Kavukcuoglu, Koray. “WaveNet: A Generative Model for Raw Audio,” arXiv:1609.03499, 2016, which is available at arxiv.org/pdf/1609.03499.pdf (2016) and is incorporated by reference herein in its entirety) spectrogram-to-audio neural vocoder is introduced; and, in one or more embodiments, it may be used with Tacotron (Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, et al. Tacotron: Towards end-to-end speech synthesis. In Interspeech, 2017, which is incorporated by reference herein in its entirety) as a replacement for Griffin-Lim audio generation. Using these two single-speaker models as a baseline, embodiments of multi-speaker neural speech synthesis were created by incorporating trainable speaker embeddings into embodiments of Deep Voice 2 and Tacotron. 
     Section 2 discusses related work and some of the aspects in the current patent document that are distinct from prior work. In Section 3, certain Deep Voice 2 embodiments are presented and some of the novel differences from Deep Voice 1 are highlighted. Section 4 explains embodiments of the speaker embedding technique for neural TTS models and shows multi-speaker variants of the Deep Voice 2 and Tacotron architectures. Section 5.A quantifies the improvement for single speaker TTS through a mean opinion score (MOS) evaluation and Section 5.B presents the synthesized audio quality of embodiments of multi-speaker Deep Voice 2 and Tacotron via both MOS evaluation and a multi-speaker discriminator accuracy metric. Section 6 provides a discussion of some of the results. 
     2. Related Work 
     Discussed herein are related work, starting from single-speaker neural speech synthesis and moving on to multi-speaker speech synthesis and metrics for generative model quality. 
     With regards to single-speaker speech synthesis, deep learning has been used for a variety of subcomponents, including duration prediction, fundamental frequency prediction, acoustic modeling, and more recently autoregressive sample-by-sample audio waveform generation. Embodiments of the present document build upon recent work in entirely neural TTS systems, including Deep Voice 1, Tacotron, and Char2Wav (J. Sotelo, S. Mehri, K. Kumar, J. F. Santos, K. Kastner, A. Courville, and Y. Bengio, “Char2Wav: End-to-End Speech Synthesis.” In ICLR2017 workshop submission (2017), which is incorporated by reference herein in its entirety). While these works focus on building single-speaker TTS systems, embodiment presented herein focus on extending neural TTS systems to handle multiple speakers with less data per speaker. 
     Others have attempted a multi-speaker TTS system. For instance, in traditional Hidden Markov Model (HMM)-based TTS synthesis, an average voice model is trained using multiple speakers&#39; data, and then is adapted to different speakers. Deep Neural Network (DNN)-based systems have also been used to build average voice models, with i-vectors representing the speakers as additional inputs and separate output layers for each target speaker. Similarly, some used a shared hidden representation among different speakers with speaker-dependent output layers predicting vocoder parameters (e.g., line spectral pairs, aperiodicity parameters, etc.). For further context, others empirically studied DNN-based multi-speaker modeling. More recently, speaker adaptation has been tackled with generative adversarial networks (GANs), but further work is necessary before adversarial training methods lead to acceptable quality samples. 
     Unlike these prior approaches, embodiments herein instead use trainable speaker embeddings for multi-speaker TTS, which is a novel technique. Unlike prior work which depends on i-vectors, the speaker embeddings used in one or more embodiments are trained jointly with the rest of the model from scratch using backpropagation, and thus can directly learn the features relevant to the speech synthesis task. In addition, embodiments do not rely on per-speaker output layers or average voice modeling, which leads to higher-quality synthesized samples and lower data requirements, as there are fewer unique parameters per speaker to learn. 
     To evaluate the distinctiveness of the samples in an automated way, the classification accuracy of a speaker discriminator was used. Similar metrics such as an “Inception score” have been used for quantitative quality evaluations of GANs for image synthesis. Speaker discriminator models have been studied with both traditional GMM-based methods and more recently with deep learning approaches. 
     3. Single-Speaker Deep Voice 2 Embodiments 
     In this section, embodiments of a neural TTS system, which may be generally referred to as Deep Voice 2, are presented. In one or more embodiments, the general overall structure of embodiments of the Deep Voice 1 system (referenced above) were kept, as depicted in  FIG.  1   . 
       FIG.  1    depicts a diagram of an example inference system  100 , according to embodiments of the present document. As shown in the embodiment depicted in  FIG.  1   , the stages are: first, text-phonemes conversion; second, predict phoneme durations; third, upsample and generate fundamental frequency (F 0 ) profile; finally, feed F 0  and phonemes to a vocal model to a signal representing a synthesized speech of the input text. Also depicted in  FIG.  1    is a speaker identifier  105  which is used to provide multi-speaker capability. In one or more embodiments, the speaker identifier may be a vector  105  that comprises three speaker identifier embedding vectors  110 ,  115 , and  120 , which are used with the duration model, frequency model, and vocal model, respectively. In one or more embodiments, the speaker identifier embedding vectors are trained separately. One of the motivations for presenting an improved single-speaker model is to use it as a starting point for high-quality multi-speaker model embodiments. 
     One major difference between embodiments of Deep Voice 2 and embodiments of Deep Voice 1 is the separation of the phoneme duration and frequency models. A typical Deep Voice 1 embodiment has a single model to jointly predict phoneme duration and frequency profile (voicedness and time-dependent fundamental frequency, F 0 ). In one or more Deep Voice 2 embodiments, the phoneme durations are predicted first and then are used as inputs to the frequency model. 
     For convenience,  FIG.  2 A  graphically depicts a system diagram for training procedure for a single-speaker Deep Voice 2 embodiment, according to embodiments of the present document, and  FIG.  2 B  graphically depicts a system diagram for inference/deployment of a single-speaker Deep Voice 2 embodiment, according to embodiments of the present document. 
     As shown in  FIG.  2 A , training audio  210  and its corresponding transcription  210  are inputs used to train the various component models of the system  200 A (e.g., frequency model  225 , segmentation model  245 , duration model  240 , and vocal model  255 ). From the text  210 , phonemes  235  are extracted using, in one or more embodiments, a pronunciation dictionary  215 . In one or more embodiments, a pronunciation dictionary  215  may be substituted by a grapheme-to-phoneme model as in some Deep Voice 1 embodiments, to adapt for unseen words in the pronunciation dictionary. In one or more embodiments, the phonemes include phoneme with stresses, when applicable to the phoneme. 
     In one or more embodiments, for frequency extraction  220  during training, a fundamental frequency and voicedness estimation algorithm, such as Praat (a free computer software package for the scientific analysis of speech in phonetics, which was designed and developed by Paul Boersma and David Weenink of the Institute of Phonetic Sciences—University of Amsterdam), may be used, although it should be noted that other fundamental frequency and voicedness estimation algorithm or algorithms may be used. In one or more embodiments, the output is a frequency profile  230  for the audio. In one or more embodiments, the input audio  205 , frequency profile  230 , and phonemes  235  are used to train a frequency model  225  that produces frequencies profiles. 
     In one or more embodiments, the phonemes  235  and audio  205  are used to train a segmentation model  245  that produces segmented utterance  250 . In one or more embodiments, the phonemes  235  and segmented utterances  250  are used to train a duration model  240  that indicates the duration of the phonemes. In one or more embodiments, the vocal model  255  is trained using the input audio  205  and the corresponding frequency profile  203 , phonemes  235 , and segmented utterances  250  to produce a signal representing synthesized human speech of the written text  210 . 
       FIG.  2 B  graphically depicts a system diagram  200 B for inference/deployment a single-speaker Deep Voice 2 embodiment, according to embodiments of the present document. In the text-to-speech (TTS) system  200 B synthesizes human speech from an input text  210  by converting the input text to phonemes  235  and inputting the phonemes into a trained duration model  240  and into a trained fundamental frequency model  225  that outputs phoneme durations and fundamental frequency profiles, respectively. In one or more embodiments, the trained vocal model  255  uses the phonemes  235 , phoneme durations  260 , and frequency profiles  230  to generate a signal representing synthesized human speech of the input text  210 . The intermediate evaluation metrics for Deep Voice 2 model embodiments are compared to their Deep Voice 1 embodiment counterparts in Table 1. 
     
       
         
           
               
             
               
                 TABLE 1 
               
             
            
               
                   
               
               
                 Comparison of segmentation, duration, and frequency model 
               
               
                 embodiments for Deep Voice 1 embodiments and Deep Voice 
               
               
                 2 embodiments, both trained with the same single-speaker 
               
               
                 data set and evaluated as described in Deep Voice 1. 
               
            
           
           
               
               
               
               
            
               
                   
                   
                 Deep 
                 Deep 
               
               
                 Model 
                 Evaluation metric 
                 Voice 1 
                 Voice 2 
               
               
                   
               
               
                 Segmentation 
                 Phoneme pair error rate 
                 7% 
                 0.1% 
               
               
                 Duration 
                 Mean absolute error 
                 38.0 ms 
                 17.5 ms 
               
               
                 Frequency 
                 Mean absolute error 
                 29.4 Hz 
                 24.5 Hz 
               
               
                   
               
            
           
         
       
     
     In the following subsequent subsections, embodiments of the models used in Deep Voice 2 are presented. In one or more embodiments, all models are trained separately, and the hyperparameters are specified in Appendix A. A quantitative comparison of Deep Voice 1 embodiments and Deep Voice 2 embodiments is also presented herein. 
     A. Segmentation Model Embodiments 
     Estimation of phoneme locations may be treated as an unsupervised learning problem in Deep Voice 2 embodiments. In one or more embodiments, the segmentation model is a convolutional-recurrent architecture with connectionist temporal classification (CTC) loss applied to classify phoneme pairs, which are then used to extract the boundaries between them. In one or more embodiments, some architecture changes in Deep Voice 2 embodiments are the addition of batch normalization and residual connections in the convolutional layers. Specifically, a Deep Voice 1 segmentation model embodiment computes the output of each layer as:
 
 h   (l) =relu( W   (l)   *h   (l−1)   +b   (l) )   (1)
 
     where h (l)  is the output of the lth layer, W (l)  are the convolution filterbank, b (l)  is the bias vector, and * is the convolution operator. In contrast, in one or more embodiments, the Deep Voice 2 segmentation model layers instead compute:
 
 h   (l) =relu( h   (l−1) +BN( W   (l)   *h   (l−1) ))   (2)
 
     where BN represents batch normalization. In addition, it was found that the segmentation model may make mistakes for boundaries between silence phonemes and other phonemes, which can significantly reduce the segmentation accuracy on some datasets. Accordingly, in one or more embodiments, a post-processing step is introduced to correct these potential mistakes: whenever the segmentation model decodes a silence boundary, the location of the boundary is adjusted with a silence detection heuristic. For example, in one or more embodiments, the smoothed normalized audio power is computed as p[n]=(x[n] 2 /x max   2 )*g[n], where x[n] is the audio signal, g[n] is the impulse response of a Gaussian filter, x max  is the maximum value of x[n], and * is a one-dimensional convolution operation. In one or more embodiments, the silence phoneme boundaries are assigned when p[n] exceeds a fixed threshold. In one or more embodiments, the optimal parameter values for the Gaussian filter and the threshold depend on the dataset and audio sampling rate. 
     B. Duration Model Embodiments 
     In one or more Deep Voice 2 embodiments, instead of predicting a continuous-valued duration, duration prediction is formulated as a sequence labeling problem. The phoneme duration may be discretized into log-scaled buckets, and each input phoneme may be assigned to the bucket label corresponding to its duration. In one or more embodiments, the sequence is modelled with a conditional random field (CRF) loss function with pairwise potentials at output layer. In one or more embodiments, during inference, discretized durations may be decoded from the CRF using the Viterbi forward-backward algorithm. In one or more embodiments, it was found that quantizing the duration prediction and introducing the pairwise dependence implied by the CRF improved synthesis quality. 
     C. Frequency Model Embodiments 
     In one or more embodiments, after decoding from the duration model, the predicted phoneme durations are upsampled from a per-phoneme input to a per-frame input for the frequency model. In one or more embodiments, each frame is ensured to be 10 milliseconds. For instance, if a phoneme lasts 20 milliseconds, the input features corresponding to that phoneme will be repeated in two frames. In one or more embodiments, phonemes that last less than 10 milliseconds are extended to be a single frame. 
     Deep Voice 2 frequency model embodiments comprise multiple layers: firstly, bidirectional gated recurrent unit (GRU) layers generate hidden states from the input features. From these hidden states, an affine projection followed by a sigmoid nonlinearity may be used to produce the probability that each frame is voiced. In addition, the hidden states may be used to make two separate normalized fundamental frequency (F 0 ) predictions. In one or more embodiments, the first prediction, f GRU , is made with a single-layer bidirectional GRU followed by an affine projection. In one or more embodiments, the second prediction, f conv , is made by adding up the contributions of multiple convolutions with varying convolution widths and a single output channel. Finally, the hidden state may be used with an affine projection and a sigmoid nonlinearity to predict a mixture ratio ω, which may be used to weigh the two normalized frequency predictions and combine them into:
 
 f=ω·f   GRU +(1−ω)· f   conv    (3)
 
     The normalized prediction f may then be converted to the true frequency F 0  prediction via:
 
 F   0 =μ F     0   +σ F     0     ·f    (4)
 
     where μ F     0    and σ F     0    are, respectively, the mean and standard deviation of F 0  for the speaker the model is trained on. In one or more embodiments, it was found that predicting the fundamental frequency F 0  with a mixture of convolutions and a recurrent layer performs better than predicting with either one individually. This may be attributed to the hypothesis that including the wide convolutions reduces the burden for the recurrent layers to maintain state over a large number of input frames, while processing the entire context information efficiently. 
     D. Vocal Model Embodiments 
     In one or more embodiments, a Deep Voice 2 vocal model may be based on a WaveNet architecture with a two-layer bidirectional quasi-recurrent neural network (QRNN) conditioning network, similar to embodiments of Deep Voice 1. However, in one or more embodiments, the 1×1 convolution between the gated tanh nonlinearity and the residual connection is removed. In addition, the same conditioner bias may be used for every layer of the WaveNet, instead of generating a separate bias for every layer as was done in one or more embodiments of Deep Voice 1. It was found that these changes reduce model size drastically (e.g., by a factor of ˜7) and speed up inference significantly (e.g., by ˜25%), while yielding no perceptual change in quality. 
     4. Multi-Speaker Model Embodiments with Trainable Speaker Embeddings 
     To synthesize speech from multiple speakers, in one or more embodiments, each of the models were augmented with a single low-dimensional speaker embedding vector per speaker. Unlike previous work, embodiments do not rely on per-speaker weight matrices or layers. Speaker-dependent parameters may be stored in a very low-dimensional vector and thus there is near-complete weight sharing between speakers. Embodiments use speaker embeddings to produce recurrent neural network (RNN) initial states, nonlinearity biases, and multiplicative gating factors, used throughout the networks. 
     In one or more embodiments, speaker embeddings are initialized randomly with a uniform distribution over [−0.1, 0.1] and trained jointly via backpropagation; each model has its own speaker embeddings. 
     To encourage each speaker&#39;s unique voice signature to influence the model, the speaker embeddings may be incorporated into multiple portions of the model. Empirically, it was found that, for embodiments, simply providing the speaker embeddings to the input layers does not work as well for any of the presented models besides the vocal model, possibly due to the high degree of residual connections present in the WaveNet and due to the difficulty of learning high-quality speaker embeddings. It was observed, in embodiments, that several patterns tend to yield high performance: 
     Site-Specific Speaker Embeddings: For every use site in the model architecture, transform the shared speaker embedding to the appropriate dimension and form through an affine projection and a nonlinearity. 
     Recurrent Initialization: Initialize the hidden states of recurrent layers with site-specific speaker embeddings. 
     Input Augmentation: Concatenate a site-specific speaker embedding to the input at every timestep of a recurrent layer. 
     Feature Gating: Multiply layer activations elementwise with a site-specific speaker embedding to render adaptable information flow. The feature gating may allow a model to learn the union of all necessary features while allowing speaker embeddings to determine which features are used for each speaker and how much influence these features will have on the activations. 
       FIG.  3 A  graphically depicts a diagram  300 A for training procedure for a multi-speaker Deep Voice 2 embodiment, according to embodiments of the present document, and  FIG.  3 B  graphically depicts a diagram  300 B for inference/deployment of a multi-speaker Deep Voice 2 embodiment, according to embodiments of the present document. Note that for each diagram  300 A and  300 B, the addition of speaker identity  305  allows for multi-speaker input or capabilities for the duration model  340 , the frequency model  325 , the segmentation model  345 , and the vocal model  355 . 
     How speaker embeddings are used in each model embodiments is described next. 
     A. Multi-Speaker Deep Voice 2 Embodiments 
     In one or more embodiments, Deep Voice 2 models have separate speaker embeddings for each model; yet, they can be viewed as chunks of a larger speaker embedding, which are trained independently. 
     a) Segmentation Model Embodiments 
       FIG.  4    graphically depict an architecture embodiment for a multi-speaker segmentation model, according to embodiments of the present document. Note that in the embodiment  400  depicted in  FIG.  4   , speaker embedding  405  is provided as an input to the segmentation model  400 . 
     In multi-speaker segmentation model embodiments, feature gating may be used in the residual connections of the convolution layers. Instead of Eq. 2, the batch-normalized activations may be multiplied by a site-specific speaker embedding:
 
 h   (l) =relu( h   (l−1) +BN( W*h   (l−1)   ·g   s )   (5)
 
     where g s  is a site-specific speaker embedding. In one or more embodiments, the same site-specific embedding may be shared for all the convolutional layers. In addition, in one or more embodiments, each of the recurrent layers is initialized with a second site specific embedding. Similarly, each layer may use the same site-specific embedding, rather than having a separate embedding per layer. 
       FIG.  5    depicts graphically depicts a simplified diagram for training of a multi-speaker segmentation model  345 , according to embodiments of the present document. It should be noted that, in one or more embodiments, the trained segmentation model  345  is used to produce segmented utterances  250 , which are used to train other models, such as the duration model  340 , the frequency model  325 , and the vocal model  355 . However, in embodiments, the trained segmentation model  345  is not used during deployment/inference, such as depicted in  FIG.  3 B . 
       FIG.  6    depict a method for training a multi-speaker segmentation model, according to embodiments of the present document. In one or more embodiments, a phoneme dictionary (or trained grapheme-to-phoneme model) is used ( 605 ) to convert training written text  210 , which is a transcription corresponding to audio of training speech utterances  205 , to training phonemes  235  corresponding to the written text  210  and the training speech utterances audio  205 . In one or more embodiments, the training speech utterance  205 , a speaker identifier  305  that identifies a speaker for each utterance, and the corresponding training phonemes  235  are used ( 610 ) to train a segmentation model  345  to output the segmented speech utterances  250  that are segmented by phoneme boundaries. As previously noted, estimation of phoneme locations may be treated as an unsupervised learning problem. In one or more embodiments, phoneme pairs are classified using CTC loss, and CTC scores are used to determine boundaries imposing the overlap condition. 
     b) Duration Model Embodiments 
       FIG.  7    graphically depict an architecture embodiment for a multi-speaker duration model, according to embodiments of the present document. Note that in the embodiment  700  depicted in  FIG.  7   , speaker embedding  705  is provided as an input to the duration model  700 . 
     In one or more embodiments, the multi-speaker duration model may use speaker-dependent recurrent initialization and input augmentation. A site-specific embedding may be used to initialize RNN hidden states, and another site-specific embedding is provided as input to the first RNN layer by concatenating it to the feature vectors. 
       FIG.  8 A  depicts graphically depicts a simplified diagram for training of a multi-speaker duration model  340 , according to embodiments of the present document. It should be noted that, in one or more embodiments, the trained duration model  340  is used to produce phoneme durations  260 . 
       FIG.  9    depict a method for training a multi-speaker duration model, according to embodiments of the present document. In one or more embodiments, the training phonemes  235  corresponding to the input text  210 , the speaker identifier  305  that identifies a speaker for each utterance, and the segmented speech utterances  250  from the trained segmentation model  345  are used ( 905 ) to train a multi-speaker duration model  340  to output phoneme durations  260 . 
       FIG.  8 B  is a simplified graphical depiction of a trained multi-speaker duration model&#39;s  340  role during deployment/inference, according to embodiments of the present document. Note that during inference, the trained duration model  340  receives the phonemes  235  of the input text  210  and also receives as an input speaker embedding  305  identifying the speaker and outputs corresponding phoneme durations  260 , which are used elsewhere in the deployed system (see, for example,  FIG.  3 B ). 
     c) Frequency Model Embodiments 
       FIG.  10    graphically depict an architecture embodiment for a multi-speaker frequency model, according to embodiments of the present document. Note that in the embodiment  1000  depicted in  FIG.  10   , speaker embedding  1005  is provided as an input to the frequency model  1000 . 
     In one or more embodiments, the multi-speaker frequency model may use recurrent initialization, which initializes the recurrent layers (except for the recurrent output layer) with a single site-specific speaker-embedding. 
     As described in Section 3.C., the recurrent and convolutional output layers in the single-speaker frequency model predict a normalized frequency, which is then converted into the true fundamental frequency F 0  by a fixed linear transformation. In one or more embodiments, the linear transformation depends on the mean and standard deviation of observed fundamental frequency for the speaker. These values vary greatly between speakers: male speakers, for instance, tend to have a much lower mean fundamental frequency F 0 . To better adapt to these variations, in one or more embodiments, the mean and standard deviation are made trainable model parameters and are multiplied by scaling terms which depend on the speaker embeddings. Specifically, instead of Eq. 4, the F 0  prediction may be computed as:
 
 F   0 =μ F     0   ·(1+softsign( V   μ   T   g   f ))+σ F     0   ·(1+softsign( V   σ   T   g   f ))· f    (6)
 
     where g f  is a site-specific speaker embedding, μ F     0    and σ F     0    are trainable scalar parameters initialized to the F 0  mean and standard deviation on the dataset, and V μ  and V σ  are trainable parameter vectors. 
       FIG.  11 A  depicts graphically depicts a simplified diagram for training of a multi-speaker frequency model  325 , according to embodiments of the present document. It should be noted that, in one or more embodiments, the trained frequency model  325  is used to produce frequency profiles  230  of the phonemes  235 . 
       FIG.  12    depict a method for training a multi-speaker frequency model, according to embodiments of the present document. In one or more embodiments, frequency profiles for the training speech utterances audio  205  are extracted ( 1205 ), in which a frequency profile may comprise voicedness and time-dependent fundamental frequency, F 0 . As discussed previously, a frequency extractor  220  may utilize a fundamental frequency and voicedness estimator, such as Praat software. The training phonemes  235 , the audio of training speech utterances  205 , the speaker identifier  305  that identifies the speaker for each utterance, and the extracted frequency profiles  230  for the training speech utterances  205  may then be used ( 1210 ) to train a frequency model  325  to output frequency profiles akin to the frequency profile  230  from the frequency extractor and with a capability to be adjusted based upon a speaker identifier  305 . 
       FIG.  11 B  is a simplified graphical depiction of a trained multi-speaker frequency model&#39;s  325  role during deployment/inference, according to embodiments of the present document. Note that during inference, the trained frequency model  325  receives the phonemes  235  of the input text  210 , the phoneme durations  260  from the trained duration model  340 , and an input speaker embedding  305  identifying the speaker and outputs corresponding frequency profiles  230  for the phonemes durations, which are used by the vocal model in synthesizing the speech (see, for example,  FIG.  3 B ). 
     d) Vocal Model Embodiments 
       FIG.  13    depicts a vocal model architecture, according to embodiments of the present document. In  FIG.  13   , components, by function, as follows: inputs are  1302  and  1320 ; convolutions and QRNNs are  1304 ,  1306 ,  1310 ,  1312 ,  1322 ,  1332 ,  1348 ,  1350 ,  1364 , and  1370 ; unary operations are  1324 ,  1342 ,  1346 ,  1362 , and  1368 ; softmax is  1372 ; binary operations are  1338 ,  1340 ,  1344 ,  1352 , and  1360 ; and reshapes, transposes, and slices are  1308 ,  1314 ,  1316 , and  1318 ; and split channels are  1334  and  1336 . In one or more embodiments, the site-specific speaker embeddings are concatenated onto each input frame of the conditioner  1302 , besides phoneme and frequency features. It shall be noted that the example values (e.g., 256 Hertz, 16 Hertz, etc. are provided by way of illustration and not limitation; one skilled in the art shall recognize that these specific values are not critical). 
     In one or more embodiments, the multi-speaker vocal model may use only input augmentation, with the site-specific speaker embedding concatenated onto each input frame of the conditioner. This differs from the global conditioning suggested in Oord et al. (2016) for the WaveNet architecture and allows the speaker embedding to influence the local conditioning network as well. 
     Without speaker embeddings, in one or more embodiments, the vocal model is still able to generate somewhat distinct-sounding voices because of the distinctive features provided by the frequency and duration models. However, having speaker embeddings in the vocal model increases the audio quality. It was observed in one or more embodiments that the embeddings converge to a meaningful latent space. 
       FIG.  14 A  depicts graphically depicts a simplified diagram for training of a multi-speaker vocal model  355 , according to embodiments of the present document. 
       FIG.  15    depict a method for training a multi-speaker vocal model, according to embodiments of the present document. In one or more embodiments, the training phonemes  235 , the audio of training speech utterances  205 , the speaker identifier  305  that identifies the speaker for each utterance, one or more frequency profiles  230 , and the segmented utterances  250  are used to train ( 1505 ) a vocal model  325  to output a signal representing synthesized human speech of the input text  210 . 
       FIG.  14 B  is a simplified graphical depiction of a trained multi-speaker vocal model&#39;s  355  role during deployment/inference, according to embodiments of the present document. Note that during inference, the trained vocal model  355  receives the phonemes  235  of the input text  210 , the phoneme durations  260  from the trained duration model  340 , the frequency profiles  230  from the trained frequency model  325 , and an input speaker embedding  305  identifying the speaker and outputs a signal representing synthesized human speech of the input text  210  that has audio characteristics corresponding to the speaker identity identified in the speaker identity indicator  305 . Thus, a different speaker identity indicator  305  will cause the trained vocal model  355  to generate the same words but with different audio characteristics corresponding to the different speaker identity. 
     e) Deployed Embodiments 
       FIG.  16    depicts a method for using a multi-speaker text-to-speech system, according to embodiments of the present document. In one or more embodiments, the input text (e.g.,  210  in  FIG.  3 B ) is converted ( 1605 ) to phonemes  235 . A trained duration model (e.g.,  340  in  FIG.  3   ) receives as inputs the phonemes  235  and a speaker identifier  305  and outputs ( 1610 ) phoneme durations  260  for the phonemes  235  relative to the speaker identifier. A trained frequency model (e.g.,  325  in  FIG.  3   ) receive as inputs the phonemes, the speaker identifier, and the phoneme durations and outputs ( 1615 ) frequency profiles for the phonemes relative to the speaker identifier. Finally, in one or more embodiments, a trained vocal model (e.g.,  355  in  FIG.  3   ) receives as input the speaker identifier, the phonemes, the phoneme durations, the frequency profiles for the phonemes (e.g., a frequency profile for a phoneme is the fundamental frequency profile and a probability of whether it is voiced) to synthesize ( 1620 ) a signal representing synthesized speech of the written text  210 , in which the synthesized audio has audio characteristics corresponding to the speaker identity. Thus, if the speaker identifier were changed, the audio characteristics would change to correspond to the identified speaker. For example, if the speaker identifier was a female with an English accent, the synthesized audio would sound like a woman with an English accent, but if the speaker identifier was changed to a male with a North American accent, the synthesized audio, while still saying the same words, would sound quite different. 
     B. Multi-Speaker Tacotron Embodiments 
     In addition to extending Deep Voice 2 embodiments with speaker embeddings, Tacotron (Wang et al., 2017), a sequence-to-sequence character-to-waveform model, was also modified as discussed herein to include speaker embeddings. When training multi-speaker Tacotron variants, it was found that model performance is highly dependent on model hyperparameters, and that some models often fail to learn attention mechanisms for a small subset of speakers. It was also found in one or more embodiments that if the speech in each audio clip does not start at the same timestep, the models are much less likely to converge to a meaningful attention curve and recognizable speech; thus, all initial and final silence in each audio clip may be trimmed. Due to the sensitivity of the model to hyperparameters and data preprocessing, additional tuning may be necessary to obtain maximal quality. Thus, one of the focuses herein is on demonstrating that Tacotron, like Deep Voice 2 embodiments, is capable of handling multiple speakers through speaker embeddings, rather than comparing the quality of the two architectures. 
     a) Character-to-Spectrogram Model Embodiments 
     In one or more embodiments, the Tacotron character-to-spectrogram architecture comprises a convolution-bank-highway-GRU (CBHG) encoder, an attentional decoder, and a CBHG post-processing network. Modifications to the architecture are discussed below. 
     It was found that incorporating speaker embeddings into the CBHG post-processing network degrades output quality, whereas incorporating speaker embeddings into the character encoder is beneficial. Without a speaker-dependent CBHG encoder, such a model embodiment is incapable of learning its attention mechanism and cannot generate meaningful output (see Appendix C.2 for speaker-dependent attention visualizations). To condition the encoder on the speaker, one site-specific embedding may be used as an extra input to each highway layer at each timestep and initialize the CBHG RNN state with a second site-specific embedding. 
     In one or more embodiments, it is beneficial to augment the decoder with speaker embeddings. One site-specific embedding may be used as an extra input to the decoder pre-net, one extra site-specific embedding as the initial attention context vector for the attentional RNN, one site-specific embedding as the initial decoder GRU hidden state, and one site-specific embedding as a bias to the tanh in the content-based attention mechanism. 
     b) Spectrogram-to-Waveform Model Embodiments 
     The original Tacotron implementation in Wang et al. uses the Griffin-Lim algorithm to convert spectrograms to time-domain audio waveform by iteratively estimating the unknown phases. Estimation of the unknown phases may be done by repeatedly converting between frequency and time domain representations of the signal using the short-time Fourier transform and its inverse, substituting the magnitude of each frequency component to the predicted magnitude at each step. Minor noise in the input spectrogram can cause noticeable estimation errors in the Griffin-Lim algorithm and the generated audio quality is degraded. To produce higher quality audio from Tacotron, instead of using Griffin-Lim, in one or more embodiments, a WaveNet-based neural vocoder may be trained to convert from linear spectrograms to audio waveforms. In one or more embodiments, a model similar to a Deep Voice 2 vocal model embodiment may be used, but takes linear-scaled log-magnitude spectrograms instead of phoneme identity and F 0  as input. A combined Tacotron-WaveNet model embodiment is shown in  FIG.  17   .  FIG.  17    depicts a Tacotron-WaveNet model with speaker conditioning in the Encoder CBHG module and decoder with two ways to convert spectrogram to audio: Griffin-Lim or a speaker-conditioned Vocal model, according to embodiments of the present document. As will be shown in Section 5.A. (below), WaveNet-based neural vocoder indeed significantly improves single-speaker Tacotron as well. 
     5. Results 
     In this section, the results on both single-speaker and multi-speaker speech synthesis using the described architectures are presented. All model hyperparameters are presented in Appendix A. It shall be noted that these experiments and results are provided by way of illustration and were performed under specific conditions using a specific embodiment or embodiments; accordingly, neither these experiments nor their results shall be used to limit the scope of the disclosure of the current patent document. 
     A. Single-Speaker Speech Synthesis Results 
     Embodiments of Deep Voice 1, Deep Voice 2, and Tacotron were trained on an internal English speech database containing approximately 20 hours of single-speaker data. The intermediate evaluations of models in Deep Voice 1 and Deep Voice 2 can be found in Table 1 (above). A Mean Opinion Score (MOS) evaluation using a crowdMOS framework was run to compare the quality of samples and the results are shown in Table 2. The results show conclusively that the architecture improvements in Deep Voice 2 yield significant gains in quality over the Deep Voice 1 embodiment. They also demonstrate that converting Tacotron-generated spectrograms to audio using WaveNet is preferable to using the iterative Griffin-Lim algorithm. 
     
       
         
           
               
             
               
                 TABLE 2 
               
             
            
               
                   
               
               
                 Mean Opinion Score (MOS) evaluations with 95% confidence intervals 
               
               
                 of Deep Voice 1, Deep Voice 2, and Tacotron embodiments. 
               
               
                 Using the crowdMOS toolkit, batches of samples from these 
               
               
                 models were presented to raters on Mechanical Turk. Since 
               
               
                 batches contained samples from all models, the experiment 
               
               
                 naturally induces a comparison between the models. 
               
            
           
           
               
               
               
               
            
               
                   
                 Model 
                 Sample Freq. 
                 MOS 
               
               
                   
                   
               
               
                   
                 Deep Voice 1 
                 16 KHz 
                 2.05 ± 0.24 
               
               
                   
                 Deep Voice 2 
                 16 KHz 
                 2.96 ± 0.38 
               
               
                   
                 Tacotron (Griffin-Lim) 
                 24 KHz 
                 2.57 ± 0.28 
               
               
                   
                 Tacotron (WaveNet) 
                 24 KHz 
                 4.17 ± 0.18 
               
               
                   
                   
               
            
           
         
       
     
     B. Multi-Speaker Speech Synthesis Results 
     In one or more embodiments, the aforementioned models were trained on the VCTK dataset with 44 hours of speech, which contains 108 speakers with 400 utterances each. The model embodiments were also trained on an internal dataset of audiobooks, which contains  477  speakers with  30  minutes of audio each (for a total of ˜238 hours of audio). The consistent sample quality observed from the inventive models disclosed herein indicates that these inventive architecture embodiments can easily learn hundreds of distinct voices with a variety of different accents and cadences. It was also observed that the learned embeddings lie in a meaningful latent space (see  FIG.  18    as an example and Appendix C for more details).  FIG.  18    depicts principal components of the learned speaker embeddings for an 80-layer vocal model ( 1805 ) and for a character-to-spectrogram model for VCTK dataset ( 1810 ), according to embodiments of the present document. 
     In order to evaluate the quality of the synthesized audio, MOS evaluations were performed using the crowdMOS framework; the results are presented in Table 3 (below). Ground truth samples were purposely included in the set being evaluated, because the accents in the VCTK dataset are likely to be unfamiliar to North American crowdsourced raters and will thus be rated poorly due to the accent rather than due to the model quality. By including ground truth samples, the MOS of the models can be compared with the ground truth MOS and thus the model quality can be evaluated rather than the data quality; however, the resulting MOS may be lower, due to the implicit comparison with the ground truth samples. Overall, it was observed that the Deep Voice 2 model can approach an MOS value that is close to the ground truth, when low sampling rate and companding/expanding taken into account. 
     A multi-speaker TTS system with high sample quality but indistinguishable voices would result in high MOS but fail to meet the desired objective of reproducing the input voices accurately. To demonstrate that the model embodiments of the present document not only generate high quality samples, but also generate easily distinguishable voices, the classification accuracy of a speaker discriminative model embodiment was also measured on the generated samples. The speaker discriminative was a convolutional neural architecture trained to classify utterances based on their speaker, trained on the same dataset as the TTS systems themselves. If the voices were indistinguishable (or the audio quality was low), the classification accuracy would be much lower for synthesized samples than it is for the ground truth samples. As shown in Table 3, classification accuracy demonstrates that samples generated from model embodiments of the present document are as distinguishable as the ground truth samples (see Appendix D). The classification accuracy is significantly lower for Tacotron with WaveNet, and the generation errors in the spectrogram are likely exacerbated by the WaveNet, as it is trained with ground truth spectrograms. 
     
       
         
           
               
             
               
                 TABLE 3 
               
             
            
               
                   
               
               
                 MOS and classification accuracy for multi-speaker model embodiments. To obtain 
               
               
                 MOS, the crowdMOS toolkit was used as detailed in Table 2 (above). Batches of 
               
               
                 synthesized samples from model embodiments as well as ground truth samples were 
               
               
                 presented to raters on Mechanical Turk. Since, in embodiments, batches contained 
               
               
                 samples from all sources, the experiment naturally induces a comparison between 
               
               
                 the models and ground truth samples. The classification accuracies of the speaker 
               
               
                 discriminative models (see Appendix D for details) on the samples, showing that 
               
               
                 the synthesized voices are as distinguishable as ground truth audio. 
               
            
           
           
               
               
               
               
               
            
               
                   
                   
                 Sample 
                   
                 Classification 
               
               
                 Dataset 
                 Multi-Speaker Model 
                 Freq. 
                 MOS 
                 Accuracy 
               
               
                   
               
            
           
           
               
               
               
               
               
               
            
               
                 VCTK 
                 Deep Voice 2 (20-layer WaveNet) 
                 16 
                 KHz 
                 2.87 ± 0.13 
                 99.9% 
               
               
                 VCTK 
                 Deep Voice 2 (40-layer WaveNet) 
                 16 
                 KHz 
                 3.21 ± 0.13 
                  100% 
               
               
                 VCTK 
                 Deep Voice 2 (60-layer WaveNet) 
                 16 
                 KHz 
                 3.42 ± 0.12 
                 99.7% 
               
               
                 VCTK 
                 Deep Voice 2 (80-layer WaveNet) 
                 16 
                 KHz 
                 3.53 ± 0.12 
                 99.9% 
               
               
                 VCTK 
                 Tacotron (Griffin-Lim) 
                 24 
                 KHz 
                 1.68 ± 0.12 
                 99.4% 
               
               
                 VCTK 
                 Tacotron (20-layer WaveNet) 
                 24 
                 KHz 
                 2.51 ± 0.12 
                 60.9% 
               
               
                 VCTK 
                 Ground Truth Data 
                 48 
                 KHz 
                 4.65 ± 0.06 
                 99.7% 
               
               
                 Audiobooks 
                 Deep Voice 2 (80-layer WaveNet) 
                 16 
                 KHz 
                 2.97 ± 0.17 
                 97.4% 
               
               
                 Audiobooks 
                 Tacotron (Griffin-Lim) 
                 24 
                 KHz 
                 1.73 ± 0.22 
                 93.9% 
               
               
                 Audiobooks 
                 Tacotron (20-layer WaveNet) 
                 24 
                 KHz 
                 2.11 ± 0.20 
                 66.5% 
               
               
                 Audiobooks 
                 Ground Truth Data 
                 44.1 
                 KHz 
                 4.63 ± 0.04 
                 98.8% 
               
               
                   
               
            
           
         
       
     
     6. Some Conclusions 
     It was shown herein how entirely-neural speech synthesis pipelines may be extended to multi-speaker text-to-speech via low-dimensional trainable speaker embeddings. Improved single-speaker model embodiments were developed to use as a starting point for multi-speaker embodiments. Next, the applicability of technique embodiments discussed herein were demonstrated by training both multi-speaker Deep Voice 2 models and multi-speaker Tacotron models, and their quality were evaluated through MOS. Speaker embedding technique embodiments were used to create high quality text-to-speech systems and conclusively show that neural speech synthesis models can learn effectively from small amounts of data spread among hundreds of different speakers. 
     One skilled in the art shall recognize that alternative systems and/or methods may be generated, such as adding new speakers to a system by fixing model parameters and solely training new speaker embeddings, and using the speaker embeddings as a meaningful vector space, as is possible with word embeddings. 
     7. Computing System Embodiments 
     In one or more embodiments, aspects of the present patent document may be directed to or implemented on information handling systems/computing systems. For purposes of this disclosure, a computing system may include any instrumentality or aggregate of instrumentalities operable to compute, calculate, determine, classify, process, transmit, receive, retrieve, originate, route, switch, store, display, communicate, manifest, detect, record, reproduce, handle, or utilize any form of information, intelligence, or data for business, scientific, control, or other purposes. For example, a computing system may be a personal computer (e.g., laptop), tablet computer, phablet, personal digital assistant (PDA), smart phone, smart watch, smart package, server (e.g., blade server or rack server), a network storage device, or any other suitable device and may vary in size, shape, performance, functionality, and price. The computing system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, ROM, and/or other types of memory. Additional components of the computing system may include one or more disk drives, one or more network ports for communicating with external devices as well as various input and output (I/O) devices, such as a keyboard, a mouse, touchscreen and/or a video display. The computing system may also include one or more buses operable to transmit communications between the various hardware components. 
       FIG.  19    depicts a simplified block diagram of a computing device/information handling system (or computing system) according to embodiments of the present disclosure. It will be understood that the functionalities shown for system  1900  may operate to support various embodiments of an information handling system—although it shall be understood that an information handling system may be differently configured and include different components. 
     As illustrated in  FIG.  19   , system  1900  includes one or more central processing units (CPU)  1901  that provides computing resources and controls the computer. CPU  1901  may be implemented with a microprocessor or the like, and may also include one or more graphics processing units (GPU)  1917  and/or a floating point coprocessor for mathematical computations. System  1900  may also include a system memory  1902 , which may be in the form of random-access memory (RAM), read-only memory (ROM), or both. 
     A number of controllers and peripheral devices may also be provided, as shown in  FIG.  19   . An input controller  1903  represents an interface to various input device(s)  1904 , such as a keyboard, mouse, or stylus. There may also be a scanner controller  1905 , which communicates with a scanner  1906 . System  1900  may also include a storage controller  1907  for interfacing with one or more storage devices  1908  each of which includes a storage medium such as magnetic tape or disk, or an optical medium that might be used to record programs of instructions for operating systems, utilities, and applications, which may include embodiments of programs that implement various aspects of the present invention. Storage device(s)  1908  may also be used to store processed data or data to be processed in accordance with the invention. System  1900  may also include a display controller  1909  for providing an interface to a display device  1911 , which may be a cathode ray tube (CRT), a thin film transistor (TFT) display, or other type of display. The computing system  1900  may also include a printer controller  1912  for communicating with a printer  1913 . A communications controller  1914  may interface with one or more communication devices  1915 , which enables system  1900  to connect to remote devices through any of a variety of networks including the Internet, a cloud resource (e.g., an Ethernet cloud, an Fiber Channel over Ethernet (FCoE)/Data Center Bridging (DCB) cloud, etc.), a local area network (LAN), a wide area network (WAN), a storage area network (SAN) or through any suitable electromagnetic carrier signals including infrared signals. 
     In the illustrated system, all major system components may connect to a bus  1916 , which may represent more than one physical bus. However, various system components may or may not be in physical proximity to one another. For example, input data and/or output data may be remotely transmitted from one physical location to another. In addition, programs that implement various aspects of this invention may be accessed from a remote location (e.g., a server) over a network. Such data and/or programs may be conveyed through any of a variety of machine-readable medium including, but are not limited to: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices that are specially configured to store or to store and execute program code, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, and ROM and RAM devices. 
     Embodiments of the present invention may be encoded upon one or more non-transitory computer-readable media with instructions for one or more processors or processing units to cause steps to be performed. It shall be noted that the one or more non-transitory computer-readable media shall include volatile and non-volatile memory. It shall be noted that alternative implementations are possible, including a hardware implementation or a software/hardware implementation. Hardware-implemented functions may be realized using ASIC(s), programmable arrays, digital signal processing circuitry, or the like. Accordingly, the “means” terms in any claims are intended to cover both software and hardware implementations. Similarly, the term “computer-readable medium or media” as used herein includes software and/or hardware having a program of instructions embodied thereon, or a combination thereof. With these implementation alternatives in mind, it is to be understood that the figures and accompanying description provide the functional information one skilled in the art would require to write program code (i.e., software) and/or to fabricate circuits (i.e., hardware) to perform the processing required. 
     It shall be noted that embodiments of the present invention may further relate to computer products with a non-transitory, tangible computer-readable medium that have computer code thereon for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind known or available to those having skill in the relevant arts. Examples of tangible computer-readable media include, but are not limited to: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices that are specially configured to store or to store and execute program code, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, and ROM and RAM devices. Examples of computer code include machine code, such as produced by a compiler, and files containing higher level code that are executed by a computer using an interpreter. Embodiments of the present invention may be implemented in whole or in part as machine-executable instructions that may be in program modules that are executed by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In distributed computing environments, program modules may be physically located in settings that are local, remote, or both. 
     One skilled in the art will recognize no computing system or programming language is critical to the practice of the present invention. One skilled in the art will also recognize that a number of the elements described above may be physically and/or functionally separated into sub-modules or combined together. 
     It will be appreciated to those skilled in the art that the preceding examples and embodiments are exemplary and not limiting to the scope of the present disclosure. It is intended that all permutations, enhancements, equivalents, combinations, and improvements thereto that are apparent to those skilled in the art upon a reading of the specification and a study of the drawings are included within the true spirit and scope of the present disclosure. It shall also be noted that elements of any claims may be arranged differently including having multiple dependencies, configurations, and combinations. 
     8. Appendix A—Model Hyperparameters 
     The hyperparameters of the model embodiments used herein are provided in Table  4  (below). 
     In one or more embodiments, to speed up the training of character-to-spectrogram model in Tacotron in the experiments, a penalty term in the form CTC loss (obtained from the attention hidden states) was added to the overall loss function. Faster convergence was observed in some cases. 
     Learning rate is presented as a triple  -r-s, which means that the initial learning rate of   was decayed by a factor of r every s iterations. All experiment model embodiments used the Adam optimization technique with β 1 =0.9, β 2 =0.9 9, and ε= 10   −8.    
     In embodiments, convolutional layers are presented as l×, o, h×w filters, which means that there are l convolutional layers, and o (output channels) filters in each layer. The filter size is h×w, where height h is in frequency bins and width w is in time frames. Any unspecified details or hyperparameters for Deep Voice 2 are identical to those from the best models used in the original implementation of Deep Voice 1, which is also discussed in S. O. Arik, M. Chrzanowski, A. Coates, G. Diamos, A. Gibiansky, Y. Kang, X. Li, J. Miller, J. Raiman, S. Sengupta, and M. Shoeybi. Deep voice: Real-time neural text-to-speech. In ICML, 2017, which is also available at arxiv.org/pdf/1702.07825.pdf, each document is incorporated in its entirety herein by reference. Similarly, any unspecified details or hyperparameters for our Tacotron implementation are identical to those from the best models used in the original implementation of Tacotron (Wang et al., 2017). 
     
       
         
           
               
             
               
                 TABLE 4 
               
             
            
               
                   
               
               
                 Model hyperparameters for speech synthesis models 
               
            
           
           
               
               
               
               
            
               
                   
                 Single-Speaker 
                 VCTK 
                 Audiobooks 
               
               
                   
                   
               
            
           
           
               
               
               
               
               
            
               
                 Segmentation 
                 Number of MFCCs 
                 40 
                 40 
                 40 
               
               
                   
                 Convolutional layers 
                 4×, 1289 × 5 
                 4×, 64 9 × 5 
                 5×, 128 9 × 5 
               
               
                   
                   
                 filters 
                 filters 
                 filters 
               
               
                   
                 Recurrent layers (Bi-GRU) 
                 4×, 512-wide 
                 4×, 1024-wide 
                 4×, 1024-wide 
               
               
                   
                 Dropout keep probability 
                 0.95 
                 0.85 
                 0.85 
               
               
                   
                 Learning rate 
                 10 −3  − 0.95 − 400 
                 2 · 10 −4  − 0.95 − 
                 2 · 10 −4  − 0.95 − 
               
               
                   
                   
                   
                 1000 
                 2000 
               
               
                   
                 Silence threshold 
                 0.1 
                 0.1 
                 0.1 
               
               
                   
                 Gaussian width 
                 0.25 ms 
                 0.25 ms 
                 0.25 ms 
               
               
                   
                 Batch size 
                 8 
                 8 
                 8 
               
               
                   
                 Speaker embedding size 
                 N/A 
                 16 
                 32 
               
               
                 Duration 
                 Fully-connected 
                 2×, 256 units 
                 4×, 256 units 
                 4×, 256 units 
               
               
                   
                 Recurrent layers (Bi-GRU) 
                 4×, 256-wide 
                 4×, 512-wide 
                 4×, 512-wide 
               
               
                   
                 Dropout keep probability 
                 0.8 
                 0.85 
                 0.85 
               
               
                   
                 Output buckets 
                 100 
                 250 
                 300 
               
               
                   
                 Learning rate 
                 3 · 10 −4  − 0.9 − 
                 6 · 10 −4  − 0.9 − 
                 3 · 10 −4  − 0.9 − 
               
               
                   
                   
                 300 
                 400 
                 800 
               
               
                   
                 Batch size 
                 128 
                 32 
                 32 
               
               
                   
                 Speaker embedding size 
                 N/A 
                 16 
                 32 
               
               
                 Frequency 
                 Hidden layers (Bi-GRU) 
                 3×, 256-wide 
                 3×, 512-wide 
                 3×, 512-wide 
               
               
                   
                 Output dimension 
                 32 
                 32 
                 64 
               
               
                   
                 Convolution widths 
                 5, 10, 20 
                 3, 6, 15, 30 
                 6, 9, 18, 35 
               
               
                   
                 Learning rate 
                 10 −3  − 0.9 − 300 
                 4 · 10 −4  − 0.9 − 
                 4 · 10 −4  − 0.9 − 
               
               
                   
                   
                   
                 300 
                 300 
               
               
                   
                 Batch size 
                 32 
                 32 
                 32 
               
               
                   
                 Speaker embedding size 
                 N/A 
                 16 
                 16 
               
               
                 Vocal 
                 Layers 
                 60 
                 20/40/60/80 
                 80 
               
               
                   
                 Learning rate 
                 10 −3  − 0.9886 − 
                 10 −3  − 0.9886 − 
                 10 −3  − 0.9886 − 
               
               
                   
                   
                 1000 
                 1000 
                 1000 
               
               
                   
                 Batch size 
                 8 
                 8 
                 8 
               
               
                   
                 Speaker embedding size 
                 N/A 
                 16 
                 16 
               
               
                 Character- 
                 Enc.-CBHG bank size 
                 16 
                 16 
                 16 
               
               
                 to- 
                 Enc.-CBHG channels 
                 128 
                 128 
                 128 
               
               
                 Spectrogram 
                 Enc.-CBHG recurrent size 
                 128 
                 128 
                 128 
               
               
                   
                 Enc.-CBHG highway layers 
                 4 
                 4 
                 4 
               
               
                   
                 Enc.-CBHG maxpool width 
                 2 
                 2 
                 2 
               
               
                   
                 Enc.-CBHG proj. sizes 
                 128, 128 
                 128, 128 
                 128, 128 
               
               
                   
                 Enc.-CBHG proj. width 
                 3 
                 3 
                 3 
               
               
                   
                 Decoder layers 
                 3 
                 3 
                 3 
               
               
                   
                 Dropout keep probability 
                 0.5 
                 0.8 
                 0.8 
               
               
                   
                 Attention size 
                 128 
                 256 
                 512 
               
               
                   
                 Attention state size 
                 256 
                 256 
                 256 
               
               
                   
                 Decoder prenet sizes 
                 256, 128 
                 256, 128 
                 256, 128, 64 
               
               
                   
                 Post-CBHG bank size 
                 8 
                 8 
                 8 
               
               
                   
                 Post-CBHG channels 
                 128 
                 512 
                 512 
               
               
                   
                 Post-CBHG conv. widths 
                 3 
                 3 
                 3 
               
               
                   
                 Post-CBHG recurrent size 
                 128 
                 256 
                 256 
               
               
                   
                 Post-CBHG highway layers 
                 4 
                 4 
                 4 
               
               
                   
                 Post-CBHG maxpool width 
                 2 
                 2 
                 2 
               
               
                   
                 Reduction factor 
                 4 
                 4 
                 4 
               
               
                   
                 CTC loss coefficient 
                 0.01 
                 0.01 
                 0.01 
               
               
                   
                 Learning rate 
                 10 −3  − 1 − N/A 
                 10 −3  − 0.95 − 3000 
                 10 −3  − 0.95 − 3000 
               
               
                   
                 Batch size 
                 16 
                 16 
                 16 
               
               
                   
                 Speaker embedding size 
                 N/A 
                 32 
                 32 
               
               
                   
               
            
           
         
       
     
     9. Appendix B—Training Time for Multi-Speaker Model Embodiments 
     The details of training time for multi-speaker model embodiments are presented in Table 5. In one or more embodiments, the same optimized WaveNet inference kernels described in Deep Voice 1 were used. 
     
       
         
           
               
             
               
                 TABLE 5 
               
             
            
               
                   
               
               
                 Training time details of multi-speaker Deep 
               
               
                 Voice 2 and Tacotron models on VCTK dataset. 
               
            
           
           
               
               
               
               
               
            
               
                   
                   
                   
                 Number 
                   
               
               
                   
                   
                 Time per 
                 of itera- 
                 Total 
               
               
                 Model 
                 Hardware 
                 iteration 
                 tions 
                 time 
               
               
                   
               
            
           
           
               
               
               
               
               
               
               
            
               
                 Segmentation 
                 1 Titan X GPU 
                 1200 
                 ms 
                  90k 
                 30 
                 hours 
               
               
                 Duration 
                 1 Tesla K40 
                 320 
                 ms 
                  60k 
                 4.5 
                 hours 
               
               
                   
                 GPU 
               
               
                 Frequency 
                 1 Tesla K40 
                 1100 
                 ms 
                  25k 
                 7.5 
                 hours 
               
               
                   
                 GPU 
               
               
                 Vocal (20 layer) 
                 8 Titan X GPUs 
                 280 
                 ms 
                 350k 
                 27 
                 hours 
               
               
                 Vocal (40 layer) 
                 8 Titan X GPUs 
                 450 
                 ms 
                 350k 
                 44 
                 hours 
               
               
                 Vocal (60 layer) 
                 8 Titan X GPUs 
                 500 
                 ms 
                 500k 
                 83 
                 hours 
               
               
                 Vocal (80 layer) 
                 8 Titan X GPUs 
                 780 
                 ms 
                 500k 
                 108 
                 hours 
               
               
                 Character-to- 
                 1 TitanX GPU 
                 450 
                 ms 
                 200k 
                 25 
                 hours 
               
               
                 spectrogram 
               
               
                   
               
            
           
         
       
     
     10. Appendix C—Interpretation of Learned Embeddings 
     In this section, the consequences of speaker-dependent models on intermediate model outputs, model activations, and the distributions of the learned embeddings are discussed. 
     A. Appendix C.1—Speaker-Dependent Fundamental Frequency Profiles 
     To demonstrate the significance of the speaker embeddings, inference was performed for a frequency model embodiment with the speaker embedding vectors corresponding to the actual speaker and a different speaker. As shown in  FIG.  20   , while the input phoneme features are dominant in determining the overall shape of the fundamental frequency profiles, the actual values are highly speaker dependent. For example, when the speaker embedding vector of a male is substituted with the speaker embedding vector of a female, the output frequency profile would cause generation of a female voice despite all the other input features correspond to the correct male speaker. 
       FIG.  20    depicts time-dependent fundamental frequency profiles  2040  in graphs  2005 ,  2010 ,  2015 , and  2020  (collectively the F 0  and voicedness) generated by a model embodiment when the inputs and embeddings correspond to two speakers (Speaker S 6  and S 24 ), according to embodiments of the present document. Speaker S 6  is a 23-year-old female with a Southern England accent, and Speaker S 24  is a 24-year-old male with an Indian accent. The pronounced sentence was “Six spoons of fresh snow peas five thick slabs of blue cheese and maybe a snack for her brother Bob.” Spikes in the ground truth  2030  are caused by the estimation errors using the Praat software, and it was observed that the frequency model embodiment can compensate them in some cases, learning from the other correct ground truth samples. 
     B. Appendix C.2—Speaker-Dependent Attention Plots 
       FIG.  21    presents attention plots for three speakers who speak at different speeds, according to embodiments of the present document. Speaker S 25  is a 22-year-old male with a Scottish accent (graph  2105 ); Speaker S 4  is a 23-year-old female with a Southern England accent (graph  2110 ); and Speaker S 16  is a 22-year-old male with a London English accent (graph  2115 ). The pronounced sentence is “Six spoons of fresh snow peas five thick slabs of blue cheese and maybe a snack for her brother Bob.”  FIG.  21    demonstrates that the modifications in an embodiment of the encoder architecture are highly effective in making the attention model speaker dependent such that different portions of the input text can be focused depending on the speaking features of the speaker. 
     C. Appendix C.3—Principal Components of the Embeddings 
     The latent space of learned speaker embeddings was explored by visualizing them in a lower dimensional space.  FIG.  18    shows the first two principal components of the learned embeddings of a vocal model embodiment ( 1805 ) and a character-to-spectrogram model embodiment ( 1810 ), according to embodiments of the present document. Plots  1805  and  1810  depict the principal components of learned embeddings for the model embodiments, shown with the genders of the speakers. Also explored were the principal components of English language accents. The regions were determined according to https://en.wikipedia.org/wiki/Regional_accents_of_English. Although they were initialized randomly and completely trained based on a loss function related to the generative quality, discriminative patterns in the learned embeddings can be observed. Gender of the speaker was the most apparent discriminative pattern in the learned embeddings that even a linear classifier fit on the shown two-dimensional space can classify the gender with a very high accuracy. There are also apparent discriminative patterns for the region of the speaker. In a two-dimensional space, Great Britain and North American accent regions especially seem highly separable. 
     11. Appendix D—Speaker Discriminative Model Embodiments 
     To compute multi-speaker discriminator accuracy, a speaker discriminator model trained on the ground truth data set of multiple speakers was used. Although using another discriminator model, such as Deep Speaker or other methods, would also suffice, a new deep-learning-based discriminative model was created. It is noted that the accuracy results on the test set are on par with the state-of-the-art speaker classification methods in the literature.  FIG.  22    graphically depicts an architecture for speaker discrimination, according to embodiments of the present document. In one or more embodiments, a mel-frequency cepstral coefficients (MFCCs) computed after resampling the input to a constant sampling frequency was used. Then, in one or more embodiments, two-dimensional convolutional layers convolving over both time and cepstral frequency bands, with a rectified linear unit (relu) nonlinearity clipped to a maximum of six after each convolutional layer were used. In one or more embodiments, the last convolutional layer is followed by max-pooling layer. The model  2400  then mean-pools over time for all utterance timesteps and applies a fully connected layer with a relu nonlinearity followed by a fully connected output layer with a softmax nonlinearity and cross-entropy classification loss. In one or more embodiments, to avoid overfitting to the dataset, a dropout is applied after every relu nonlinearity. 
     To demonstrate that the classification results are not sensitive to the choice of the hyperparameters of the discriminative model, the classification accuracy for other choices are demonstrated in this section. Hyperparameters for all the discriminator models are available in Table 7. Only the results for the models, D3 and D8, are presented in Table 3, as they yielded the highest validation set accuracy. 
     
       
         
           
               
             
               
                 TABLE 6 
               
             
            
               
                   
               
               
                 Hyperparameters for speaker discriminative models for VCTK dataset. 
               
            
           
           
               
               
               
               
               
            
               
                 Parameter 
                 D1 
                 D2 
                 D3 (in Table 3) 
                 D4 
               
               
                   
               
               
                 Audio resampling freq. 
                 16 KHz 
                 16 KHz 
                 16 KHz 
                 16 KHz 
               
               
                 Number of MFCCs 
                 20 
                 20 
                 80 
                 80 
               
               
                 Hop length 
                 160 
                 160 
                 160 
                 160 
               
               
                 Convolution Layers 
                 5×, 32 2 × 10 
                 5×, 32 9 × 5 
                 5×, 32 2 × 20 
                 5×, 32 9 × 5 
               
               
                   
                 filters 
                 filters 
                 filters 
                 filters 
               
               
                 Maxpool width &amp; stride 
                 2 
                 2 
                 2 
                 2 
               
               
                 Fully connected size 
                 16 
                 16 
                 32 
                 32 
               
               
                 Dropout keep 
                 0.75 
                 0.75 
                 0.75 
                 0.75 
               
               
                 probability 
               
               
                 Learning rate 
                 10 −3  − 0.95 − 
                 10 −3  − 0.95 − 
                 10 −3  − 0.95 − 
                 10 −3  − 0.95 − 
               
               
                   
                 1000 
                 1000 
                 1000 
                 1000 
               
               
                   
               
            
           
         
       
     
     
       
         
           
               
             
               
                 TABLE 7 
               
             
            
               
                   
               
               
                 Hyperparameters for speaker discriminative models for Audiobook dataset. 
               
            
           
           
               
               
               
               
               
            
               
                 Parameter 
                 D5 
                 D6 
                 D7 
                 D8 (in Table 3) 
               
               
                   
               
               
                 Audio resampling freq. 
                 16 KHz 
                 16 KHz 
                 16 KHz 
                 16 KHz 
               
               
                 Number of MFCCs 
                 20 
                 20 
                 80 
                 80 
               
               
                 Hop length 
                 160 
                 160 
                 160 
                 160 
               
               
                 Convolution layers 
                 3×, 32 9 × 5 
                 5×, 32 2 × 10 
                 7×, 32 9 × 5 
                 5×, 32 2 × 10 
               
               
                   
                 filters 
               
               
                 Maxpool width &amp; stride 
                 2 
                 2 
                 2 
                 2 
               
               
                 Fully connected size 
                 32 
                 32 
                 32 
                 32 
               
               
                 Dropout keep 
                 0.75 
                 0.75 
                 0.75 
                 0.75 
               
               
                 probability 
               
               
                 Learning rate 
                 10−3 − 0.95 − 
                 10−3 − 0.99 − 
                 10−3 − 0.95 − 
                 10−3 − 0.99 − 
               
               
                   
                 1000 
                 2000 
                 1000 
                 2000 
               
               
                   
               
            
           
         
       
     
     
       
         
           
               
             
               
                 TABLE 8 
               
             
            
               
                   
               
               
                 Classification accuracy using speaker discriminative models D1, D2, D3, 
               
               
                 and D4. The corresponding model hyperparameters are given in Table 6. 
               
            
           
           
               
               
               
               
               
               
            
               
                 Dataset 
                 Multi-Speaker Model 
                 D1 
                 D2 
                 D3 (in Table 3) 
                 D4 
               
               
                   
               
            
           
           
               
               
               
               
               
               
            
               
                 VCTK 
                 Deep Voice 2 (20-layer WaveNet) 
                 97.87% 
                 97.60% 
                 99.92% 
                 99.84% 
               
               
                 VCTK 
                 Deep Voice 2 (40-layer WaveNet) 
                 98.56% 
                 98.68% 
                 100.00% 
                 100.00% 
               
               
                 VCTK 
                 Deep Voice 2 (60-layer WaveNet) 
                 98.56% 
                 98.44% 
                 99.68% 
                 99.80% 
               
               
                 VCTK 
                 Deep Voice 2 (80-layer WaveNet) 
                 99.06% 
                 99.21% 
                 99.96% 
                 99.96% 
               
               
                 VCTK 
                 Tacotron (Griffin-Lim) 
                 95.89% 
                 96.24% 
                 99.37% 
                 99.60% 
               
               
                 VCTK 
                 Tacotron (20-layer WaveNet) 
                 87.93% 
                 85.19% 
                 60.87% 
                 65.43% 
               
               
                 VCTK 
                 Ground Truth Data (16 kHz) 
                 98.07% 
                 98.00% 
                 99.74% 
                 99.66% 
               
               
                   
               
            
           
         
       
     
     
       
         
           
               
             
               
                 TABLE 9 
               
             
            
               
                   
               
               
                 Classification accuracy using speaker discriminative models D5, D6, D7, 
               
               
                 and D8. The corresponding model hyperparameters are given in Table 7. 
               
            
           
           
               
               
               
               
               
               
            
               
                 Dataset 
                 Multi-Speaker Model 
                 D5 
                 D6 
                 D7 
                 D8 (in Table 3) 
               
               
                   
               
            
           
           
               
               
               
               
               
               
            
               
                 Audiobooks 
                 Deep Voice 2 (80-layer WaveNet) 
                 98.24% 
                 95.66% 
                 96.80% 
                 97.42% 
               
               
                 Audiobooks 
                 Tacotron (Griffin-Lim) 
                 96.93% 
                 94.89% 
                 92.24% 
                 93.87% 
               
               
                 Audiobooks 
                 Tacotron (20-layer WaveNet) 
                 83.36% 
                 80.81% 
                 60.00% 
                 66.53% 
               
               
                 Audiobooks 
                 Ground Truth Data (16 kHz) 
                 96.30% 
                 97.49% 
                 98.39% 
                 98.80%