Title: DEEP VOICE 3: SCALING TEXT-TO-SPEECH WITH CONVOLUTIONAL SEQUENCE LEARNING

Abstract: We present Deep Voice 3, a fully-convolutional attention-based neural textto-speech (TTS) system. Deep Voice 3 matches state-of-the-art neural speech synthesis systems in naturalness while training an order of magnitude faster. We scale Deep Voice 3 to dataset sizes unprecedented for TTS, training on more than eight hundred hours of audio from over two thousand speakers. In addition, we identify common error modes of attention-based speech synthesis networks, demonstrate how to mitigate them, and compare several different waveform synthesis methods. We also describe how to scale inference to ten million queries per day on a single GPU server.

1. INTRODUCTION:

Text-to-speech (TTS) systems convert written language into human speech. TTS systems are used in a variety of applications, such as human-technology interfaces, accessibility for the visuallyimpaired, media and entertainment. Traditional TTS systems are based on complex multi-stage hand-engineered pipelines (Taylor, 2009) . Typically, these systems first transform text into a compact audio representation, and then convert this representation into audio using an audio waveform synthesis method called a vocoder.

Recent work on neural TTS has demonstrated impressive results, yielding pipelines with simpler features, fewer components, and higher quality synthesized speech. There is not yet a consensus on the optimal neural network architecture for TTS. However, sequence-to-sequence models (Wang et al., 2017; Sotelo et al., 2017; Arık et al., 2017) have shown promising results.

In this paper, we propose a novel, fully-convolutional architecture for speech synthesis, scale it to very large audio data sets, and address several real-world issues that arise when attempting to deploy an attention-based TTS system. Specifically, we make the following contributions:

1. We propose a fully-convolutional character-to-spectrogram architecture, which enables fully parallel computation and trains an order of magnitude faster than analogous architectures using recurrent cells (e.g., Wang et al., 2017) . 2. We show that our architecture trains quickly and scales to the LibriSpeech ASR dataset (Panayotov et al., 2015) , which consists of 820 hours of audio data from 2484 speakers. 3. We demonstrate that we can generate monotonic attention behavior, avoiding error modes commonly affecting sequence-to-sequence models. 4. We compare the quality of several waveform synthesis methods, including WORLD (Morise et al., 2016) , Griffin-Lim (Griffin & Lim, 1984), and WaveNet (Oord et al., 2016) .

Automatic speech recognition (ASR) datasets are often much larger than traditional TTS corpora but tend to be less clean, as they typically involve multiple microphones and background noise.

Although prior work has applied TTS methods to ASR datasets (Yamagishi et al., 2010) , Deep Voice 3 is, to the best of our knowledge, the first TTS system to scale to thousands of speakers with a single model.

Sequence-to-sequence models (Sutskever et al., 2014; Cho et al., 2014) encode a variable-length input into hidden states, which are then processed by a decoder to produce a target sequence. An attention mechanism allows a decoder to adaptively select encoder hidden states to focus on while generating the target sequence (Bahdanau et al., 2015) . Attention-based sequence-to-sequence models are widely applied in machine translation (Bahdanau et al., 2015) , speech recognition (Chorowski et al., 2015) , and text summarization (Rush et al., 2015) . Recent improvements in attention mechanisms relevant to Deep Voice 3 include enforced-monotonic attention during training (Raffel et al., 2017) , fully-attentional non-recurrent architectures (Vaswani et al., 2017) , and convolutional sequenceto-sequence models (Gehring et al., 2017) . Deep Voice 3 demonstrates the utility of monotonic attention during training in TTS, a new domain where monotonicity is expected. Alternatively, we show that with a simple heuristic to only enforce monotonicity during inference, a standard attention mechanism can work just as well or even better. Deep Voice 3 also builds upon the convolutional sequence-to-sequence architecture from Gehring et al. (2017) by introducing a positional encoding similar to that used in Vaswani et al. (2017) , augmented with a rate adjustment to account for the mismatch between input and output domain lengths.

3. MODEL ARCHITECTURE:

In this section, we present our fully-convolutional sequence-to-sequence architecture for TTS (see Fig. 1 ). Our architecture is capable of converting a variety of textual features (e.g. characters, phonemes, stresses) into a variety of vocoder parameters, e.g. mel-band spectrograms, linear-scale log magnitude spectrograms, fundamental frequency, spectral envelope, and aperiodicity parameters. These vocoder parameters can be used as inputs for audio waveform synthesis models. The Deep Voice 3 architecture consists of three components:

• Encoder: A fully-convolutional encoder, which converts textual features to an internal learned representation. • Decoder: A fully-convolutional causal decoder, which decodes the learned representation with a multi-hop convolutional attention mechanism into a low-dimensional audio representation (mel-scale spectrograms) in an autoregressive manner. • Converter: A fully-convolutional post-processing network, which predicts final vocoder parameters (depending on the vocoder choice) from the decoder hidden states. Unlike the decoder, the converter is non-causal and can thus depend on future context information.

The overall objective function to be optimized is a linear combination of the losses from the decoder (Section 3.5) and the converter (Section 3.7). We separate decoder and converter and apply multi-task training, because it makes attention learning easier in practice. To be specific, the loss for mel-spectrogram prediction guides training of the attention mechanism, because the attention is trained with the gradients from mel-spectrogram prediction besides vocoder parameter prediction.

In multi-speaker scenario, trainable speaker embeddings as in Arık et al. (2017) are used across encoder, decoder and converter. Next, we describe each of these components and the data preprocessing in detail. Model hyperparameters are available in Table 4 within Appendix C.

3.1. TEXT PREPROCESSING:

Text preprocessing is crucial for good performance. Feeding raw text (characters with spacing and punctuation) yields acceptable performance on many utterances. However, some utterances may have mispronunciations of rare words, or may yield skipped words and repeated words. We alleviate these issues by normalizing the input text as follows:

1. We uppercase all characters in the input text. 2. We remove all intermediate punctuation marks.

3. We end every utterance with a period or question mark. 4. We replace spaces between words with special separator characters which indicate the duration of pauses inserted by the speaker between words. We use four different word separators, indicating (i) slurred-together words, (ii) standard pronunciation and space characters, (iii) a short pause between words, and (iv) a long pause between words. For example, the sentence "Either way, you should shoot very slowly," with a long pause after "way" and a short pause after "shoot", would be written as "Either way%you should shoot/very slowly%." with % representing a long pause and / representing a short pause for encoding convenience.2 

3.2. JOINT REPRESENTATION OF CHARACTERS AND PHONEMES:

Deployed TTS systems (e.g., Capes et al., 2017; Gonzalvo et al., 2016) should include a way to modify pronunciations to correct common mistakes (which typically involve proper nouns, foreign words, and domain-specific jargon). A conventional way to do this is to maintain a dictionary to map words to their phonetic representations.

Our model can directly convert characters (including punctuation and spacing) to acoustic features, and hence learns an implicit grapheme-to-phoneme model. This implicit conversion is difficult to correct when the model makes mistakes. Thus, in addition to character models, we also train phoneme-only models and mixed character-and-phoneme models by allowing phoneme input option explicitly. These models are identical to character-only models, except that the input layer of the encoder sometimes receives phoneme and phoneme stress embeddings instead of character embeddings.

A phoneme-only model requires a preprocessing step to convert words to their phoneme representations (by using an external phoneme dictionary or a separately trained grapheme-to-phoneme model)3 . A mixed character-and-phoneme model requires a similar preprocessing step, except for words not in the phoneme dictionary. These out-of-vocabulary words are input as characters, allowing the model to use its implicitly learned grapheme-to-phoneme model. While training a mixed character-and-phoneme model, every word is replaced with its phoneme representation with some fixed probability at each training iteration. We find that this improves pronunciation accuracy and minimizes attention errors, especially when generalizing to utterances longer than those seen during training. More importantly, models that support phoneme representation allow correcting mispronunciations using a phoneme dictionary, a desirable feature of deployed systems. By providing a sufficiently large receptive field, stacked convolutional layers can utilize long-term context information in sequences without introducing any sequential dependency in computation.

3.3. CONVOLUTION BLOCKS FOR SEQUENTIAL PROCESSING:

We use the convolution block depicted in Fig. 2 as the main sequential processing unit to encode hidden representations of text and audio. The convolution block consists of a 1-D convolution filter, a gated-linear unit as a learnable nonlinearity (Dauphin et al., 2017) , a residual connection to the input, and a scaling factor of √ 0.5 4 . The gated linear unit provides a linear path for the gradient flow, which alleviates the vanishing gradient issue for stacked convolution blocks while retaining non-linearity. To introduce speaker-dependent control, a speaker-dependent embedding is added as a bias to the convolution filter output, after a softsign function. We use the softsign nonlinearity because it limits the range of the output while also avoiding the saturation problem that exponentialbased nonlinearities sometimes exhibit. We initialize the convolution filter weights with zero-mean and unit-variance activations throughout the entire network.

The convolutions in the architecture can be either non-causal (e.g. in encoder and converter) or causal (e.g. in decoder). To preserve the sequence length, inputs are padded with k -1 timesteps of zeros on the left for causal convolutions and (k -1)/2 timesteps of zeros on the left and on the right for non-causal convolutions, where k is an odd convolution filter width. 5 Dropout is applied to the inputs prior to the convolution for regularization.

3.4. ENCODER:

The encoder network (depicted in Fig. 1 ) begins with an embedding layer, which converts characters or phonemes into trainable vector representations, h e . These embeddings h e are first projected via a fully-connected layer from the embedding dimension to a target dimensionality. Then, they are processed through a series of convolution blocks described in Section 3.3 to extract time-dependent text information. Lastly, they are projected back to the embedding dimension to create the attention key vectors h k . The attention value vectors are computed from attention key vectors and text embeddings, h v = √ 0.5(h k + h e ), to jointly consider the local information in h e and the long-term context information in h k . The key vectors h k are used by each attention block to compute attention weights, whereas the final context vector is computed as a weighted average over the value vectors h v (see Section 3.6).

3.5. DECODER:

The decoder (depicted in Fig. 1 ) generates audio in an autoregressive manner by predicting a group of r future audio frames conditioned on the past audio frames. Since the decoder is autoregressive, it must use causal convolution blocks. We choose mel-band log-magnitude spectrogram as the compact low-dimensional audio frame representation. Similar to Wang et al. (2017) , we empirically observed that decoding multiple frames together (i.e. having r > 1) yields better audio quality.

The decoder network starts with multiple fully-connected layers with rectified linear unit (ReLU) nonlinearities to preprocess input mel-spectrograms (denoted as "PreNet" in Fig. 1 ). Then, it is followed by a series of causal convolution and attention blocks. These convolution blocks generate the queries used to attend over the encoder's hidden states (see Section 3.6). Lastly, a fully-connected layer output the next group of r audio frames and also a binary "final frame" prediction (indicating whether the last frame of the utterance has been synthesized). Dropout is applied before each fully-connected layer prior to the attention blocks, except for the first one. An L1 loss6 is computed using the output mel-spectrograms and a binary cross-entropy loss is computed using the final-frame prediction.

3.6. ATTENTION BLOCK:

We use a dot-product attention mechanism (depicted in Fig. 3 ) similar to Vaswani et al. (2017) . The attention mechanism uses a query vector (the hidden states of the decoder) and the per-timestep key vectors from the encoder to compute attention weights, and then outputs a context vector computed as the weighted average of the value vectors.

We observe empirical benefits from introducing a inductive bias where the attention follows a monotonic progression in time. Thus, we add a positional encoding to both the key and the query vectors. These positional encodings h p are chosen as h p (i) = sin ( ωsi /10000 k /d ) (for even i) or cos ( ωsi /10000 k /d ) (for odd i), where i is the timestep index, k is the channel index in the positional encoding, d is the total number of channels in the positional encoding, and ω s is the position rate of the encoding. The position rate dictates the average slope of the line in the attention distribution, roughly corresponding to speed of speech. For a single speaker, ω s is set to one for the query, and be fixed for the key to the ratio of output timesteps to input timesteps (computed across the entire dataset). For multi-speaker datasets, ω s is computed for both the key and query from the speaker embedding for each speaker (depicted in Fig. 3 ). As sine and cosine functions form an orthonormal basis, this initialization yields an attention distribution in the form of a diagonal line (see Fig. 4 (a) ).

We initialize the fully-connected layer weights used to compute hidden attention vectors to the same values for the query projection and the key projection. Positional encodings are used in all attention blocks. We use context normalization as in Gehring et al. (2017) . A fully-connected layer is applied to the context vector to generate the output of the attention block. Overall, positional encodings improve the convolutional attention mechanism.

Production-quality TTS systems have very low tolerance for attention errors. Hence, besides positional encodings, we consider additional strategies to eliminate the cases of repeating or skipping words. One approach is to substitute the canonical attention mechanism with the monotonic attention mechanism introduced in Raffel et al. ( 2017), which approximates hard-monotonic stochastic decoding with soft-monotonic attention by training in expectation. 7 Despite the improved monotonicity, this strategy may yield a more diffused attention distribution. In some cases, several char-acters are attended at the same time and high quality speech couldn't be obtained. We attribute this to the unnormalized attention coefficients of the soft alignment, potentially resulting in weak signal from the encoder. Thus, we propose an alternative strategy of constraining attention weights only at inference to be monotonic, preserving the training procedure without any constraints. Instead of computing the softmax over the entire input, we instead compute the softmax only over a fixed window starting at the last attended-to position and going forward several timesteps8 . The initial position is set to zero and is later computed as the index of the highest attention weight within the current window. This strategy also enforces monotonic attention at inference as shown in Fig. 4 , and yields superior speech quality.

3.7. CONVERTER:

The converter network takes as inputs the activations from the last hidden layer of the decoder, applies several non-causal convolution blocks, and then predicts parameters for downstream vocoders. Unlike the decoder, the converter is non-causal and non-autoregressive, so it can use future context from the decoder to predict its outputs.

The loss function of the converter network depends on the type of the vocoder used:

1. Griffin-Lim vocoder: Griffin-Lim algorithm converts spectrograms to time-domain audio waveforms by iteratively estimating the unknown phases. We find raising the spectrogram to a power parametrized by a sharpening factor before waveform synthesis is helpful for improved audio quality, as suggested in Wang et al. (2017) . L1 loss is used for prediction of linear-scale log-magnitude spectrograms. 

2.. WORLD vocoder::

The WORLD vocoder is based on (Morise et al., 2016) . As vocoder parameters, we predict a boolean value (whether the current frame is voiced or unvoiced), an F0 value (if the frame is voiced), the spectral envelope, and the aperiodicity parameters. We use a cross-entropy loss for the voiced-unvoiced prediction, and L1 losses for all other predictions (see Fig. 5 ),

3.. WaveNet vocoder::

We separately train a WaveNet to be used as a vocoder treating melscale log-magnitude spectrograms as vocoder parameters. These vocoder parameters are input as external conditioners to the network. The WaveNet is trained using ground-truth mel-spectragrams and audio waveforms. The architecture besides the conditioner is similar to the WaveNet described in Arık et al. (2017) . While the WaveNet in Arık et al. ( 2017) is conditioned with linear-scale log-magnitude spectrograms, we observed better performance with mel-scale spectrograms, which corresponds to a more compact representation of audio. In addition to L1 loss on mel-scale spectrograms at decode, L1 loss on linear-scale spectrogram is also applied as Griffin-Lim vocoder. 

4. RESULTS:

In this section, we present several different experiments and metrics to evaluate our speech synthesis system. We quantify the performance of our system and compare it to other recently published neural TTS systems.

Data: For single-speaker synthesis, we use an internal English speech dataset containing approximately 20 hours of audio with a sample rate of 48 kHz. For multi-speaker synthesis, we use the VCTK (Yamagishi et al., 2009) and LibriSpeech (Panayotov et al., 2015) datasets. The VCTK dataset consists of audios for 108 speakers, with a total duration of ∼44 hours. The LibriSpeech dataset consists of audios for 2484 speakers, with a total duration of ∼820 hours. The sample rate is 48 kHz for VCTK and 16 kHz for LibriSpeech.

Fast Training: We compare Deep Voice 3 to Tacotron, a recently published attention-based TTS system. For our system on single-speaker data, the average training iteration time (for batch size 4) is 0.06 seconds using one GPU as opposed to 0.59 seconds for Tacotron, indicating a ten-fold increase in training speed. In addition, Deep Voice 3 converges after ∼ 500K iterations for all three datasets in our experiment, while Tacotron requires ∼ 2M iterations as suggested in Wang et al. (2017) . This significant speedup is due to the fully-convolutional architecture of Deep Voice 3, which exploits the parallelism of a GPU during training.

Attention Error Modes: Attention-based neural TTS systems may run into several error modes that can reduce synthesis quality -including (i) repeated words, (ii) mispronunciations, and (iii) skipped words.9 One reason for (i) and (iii) is that the attention-based model does not impose a monotonically progressing mechanism. In order to track the occurrence of attention errors, we construct a custom 100-sentence test set (see Appendix E) that includes particularly-challenging cases from deployed TTS systems (e.g. dates, acronyms, URLs, repeated words, proper nouns, foreign words etc.) Attention error counts are listed in Table 1 and indicate that the model with joint representation of characters and phonemes, trained with standard attention mechanism but enforced the monotonic constraint at inference, largely outperforms other approaches.

Naturalness::

We demonstrate that choice of waveform synthesis matters for naturalness ratings and compare it to other published neural TTS systems. Results in Table 2 indicate that WaveNet, a neural vocoder, achieves the highest MOS of 3.78, followed by WORLD and Griffin-Lim at 3.63 and 3.62, respectively. Thus, we show that the most natural waveform synthesis can be done with a neural vocoder, and that basic spectrogram inversion techniques can match advanced vocoders with high quality single speaker data. The WaveNet vocoder sounds more natural as the WORLD vocoder introduces various noticeable artifacts. Yet, lower inference latency may render the WORLD vocoder preferable: the heavily engineered WaveNet implementation runs at 3X realtime per CPU core (Arık et al., 2017) , while WORLD runs up to 40X realtime per CPU core (see the subsection below).

Model Mean Opinion Score (MOS) Deep Voice 3 (Griffin-Lim) 3.62 ± 0.31 Deep Voice 3 (WORLD)

3.63 ± 0.27 Deep Voice 3 (WaveNet)

3.78 ± 0.30 Tacotron (WaveNet)

3.78 ± 0.34 Deep Voice 2 (WaveNet)

2.74 ± 0.35

Table 2 : Mean Opinion Score (MOS) ratings with 95% confidence intervals using different waveform synthesis methods. We use the crowdMOS toolkit (Ribeiro et al., 2011) ; batches of samples from these models were presented to raters on Mechanical Turk. Since batches contained samples from all models, the experiment naturally induces a comparison between the models.

Model MOS (VCTK) MOS (LibriSpeech) Deep Voice 3 (Griffin-Lim) 3.01 ± 0.29 2.37 ± 0.24 Deep Voice 3 (WORLD)

3.44 ± 0.32 2.89 ± 0.38 Deep Voice 2 (WaveNet)

3.69 ± 0.23 -Tacotron (Griffin-Lim) 2.07 ± 0.31 -Ground truth 4.69 ± 0.04 4.51 ± 0.18

Table 3 : MOS ratings with 95% confidence intervals for audio clips from neural TTS systems on multi-speaker datasets. We also use crowdMOS toolkit; batches of samples including ground truth were presented to human raters. Multi-speaker Tacotron implementation and hyperparameters are based on Arık et al. ( 2017), which is a proof-of-concept implementation. Deep Voice 2 and Tacotron systems were not trained for the LibriSpeech dataset due to prohibitively long time required to optimize hyperparameters.

Multi-Speaker Synthesis::

To demonstrate that our model is capable of handling multi-speaker speech synthesis effectively, we train our models on the VCTK and LibriSpeech data sets. For Lib-riSpeech (an ASR dataset), we apply a preprocessing step of standard denoising (using SoX (Bagwell, 2017)) and splitting long utterances into multiple at pause locations (which are determined by Gentle (Ochshorn & Hawkins, 2017) ). Results are presented in Table 3 . We purposefully include ground-truth samples in the set being evaluated, because the accents in datasets are likely to be unfamiliar to our North American crowdsourced raters. Our model with the WORLD vocoder achieves a comparable MOS of 3.44 on VCTK in contrast to 3.69 from Deep Voice 2, which is the state-of-theart multi-speaker neural TTS system using WaveNet as vocoder and seperately optimized phoneme duration and fundamental frequency prediction models. We expect further improvement by using WaveNet for multi-speaker synthesis, although it may substantially slow down inference. The MOS on LibriSpeech is lower compared to VCTK, which we mainly attribute to the lower quality of the training dataset due to the various recording conditions and noticeable background noise. 10 In the literature, Yamagishi et al. (2010) also observes worse performance, when apply parametric TTS method to different ASR datasets with hundreds of speakers. Lastly, we find that the learned speaker embeddings lie in a meaningful latent space (see Fig. 7 in Appendix D).

Optimizing Inference for Deployment: In order to deploy a neural TTS system in a cost-effective manner, the system must be able to handle as much traffic as alternative systems on a comparable amount of hardware. To do so, we target a throughput of ten million queries per day or 116 queries per second (QPS)11 on a single-GPU server with twenty CPU cores, which we find is comparable in cost to commercially deployed TTS systems. By implementing custom GPU kernels for the Deep Voice 3 architecture and parallelizing WORLD synthesis across CPUs, we demonstrate that our model can handle ten million queries per day. We provide more details on the implementation in Appendix B.

5. CONCLUSION:

We introduce Deep Voice 3, a neural text-to-speech system based on a novel fully-convolutional sequence-to-sequence acoustic model with a position-augmented attention mechanism. We describe common error modes in sequence-to-sequence speech synthesis models and show that we successfully avoid these common error modes with Deep Voice 3. We show that our model is agnostic of the waveform synthesis method, and adapt it for Griffin-Lim spectrogram inversion, WaveNet, and WORLD vocoder synthesis. We demonstrate also that our architecture is capable of multispeaker speech synthesis by augmenting it with trainable speaker embeddings, a technique described in Deep Voice 2. Finally, we describe the production-ready Deep Voice 3 system in full including text normalization and performance characteristics, and demonstrate state-of-the-art quality through extensive MOS evaluations. Future work will involve improving the implicitly learned grapheme-tophoneme model, jointly training with a neural vocoder, and training on cleaner and larger datasets to scale to model the full variability of human voices and accents from hundreds of thousands of speakers.

Appendices:

A DETAILED MODEL ARCHITECTURE OF DEEP VOICE 3

The detailed model architecture in depicted in Fig. 6 . ) The hidden state of the decoder then gets fed to a converter network to output linear spectrograms for Griffin-Lim or parameters for WORLD, which can be used to synthesize the final waveform. Weight normalization (Salimans & Kingma, 2016) is applied to all convolution filters and fully-connected layer weight matrices in the model.

B OPTIMIZING DEEP VOICE 3 FOR DEPLOYMENT:

Running inference with a TensorFlow graph turns out to be prohibitively expensive, averaging approximately 1 QPS12 . Instead, we implement custom GPU kernels for Deep Voice 3 inference. Due to the complexity of the model and the large number of output timesteps, launching individual kernels for different operations in the graph (convolutions, matrix multiplications, unary and binary operations etc.) is impractical: the overhead of launch a CUDA kernel is approximately 50 µs, which, when aggregated across all operations in the model and all output timesteps, limits throughput to approximately 10 QPS. Thus, we implement a single kernel for the entire model, which avoids the overhead of launching many CUDA kernels. Finally, instead of batching computation in the kernel, our kernel operates on a single utterance and we launch as many concurrent streams as there are Streaming Multiprocessors (SMs) on the GPU. Every kernel is launched with one block, so we expect the GPU to schedule one block per SM, allowing us to scale inference speed linearly with the number of SMs.

On a single Nvidia Tesla P100 GPU with 56 SMs, we achieve an inference speed of 115 QPS, which corresponds to our target ten million queries per day. We parallelize WORLD synthesis across all 20 CPUs on the server, permanently pinning threads to CPUs in order to maximize cache performance.

In this setup, GPU inference is the bottleneck, as WORLD synthesis on 20 cores is faster than 115 QPS.

We believe that inference can be made significantly faster through more optimized kernels, smaller models, and fixed-precision arithmetic; we leave these aspects to future work.

C MODEL HYPERPARAMETERS:

All hyperparameters of the models used in this paper are shown in Table 4 . 2017), we apply principal component analysis to the learned speaker embeddings and analyze the speakers based on their ground truth genders. Fig. 7 shows the genders of the speakers in the space spanned by the first two principal components. We observe a very clear separation between male and female genders, suggesting the low-dimensional speaker embeddings constitute a meaningful latent space. 

E 100-SENTENCE TEST SET:

The 100 sentences used to quantify the results in Table 1 are listed below (note that % symbol corresponds to pause):

:

RNNs introduce sequential dependencies that limit model parallelism during training.

The pause durations can be obtained through either manual labeling or by estimated by a text-audio aligner such as Gentle(Ochshorn & Hawkins, 2017). Our single-speaker dataset is labeled by hand and our multispeaker datasets are annotated using Gentle.

We use CMUDict 0.6b.

The scaling factor ensures that we preserve the input variance early in training.

We restrict to odd convolution widths to simplify the convolution arithmetic.

We choose L1 loss since it yields the best result empirically. Other loss such as L2 may suffer from outlier spectral features, which may correspond to non-speech noise.

The paperRaffel et al. (2017) also proposes hard monotonic attention process by sampling. It aims to improve the inference speed by only attending over states that are selected via sampling, and thus avoiding compute over future states. In our work, we did not benefit from such speedup, and we observed poor attention behavior in some cases, e.g. being stuck on the first or last character.

We use a window size of 3 in our experiments.

As an example, consider the phrase "DOMINANT VEGETARIAN", which should be pronounced with phonemes "D AA M AH N AH N T . V EH JH AH T EH R IY AH N ." The following are example errors for the above three error modes: (i) "D AA M AH N AH N T . V EH JH AH T EH T EH R IY AH N .", (ii) "D AE M AH N AE N T . V EH JH AH T EH R IY AH N .", (iii) "D AH N T . V EH JH AH T EH R IY AH N ."

We test Deep Voice 3 on a subsampled LibriSpeech with only 108 speakers (same as VCTK) and observe worse quality of generated samples than VCTK.

A query is defined as synthesizing the audio for a one second utterance.

The poor TensorFlow performance is due to the overhead of running the graph evaluator over hundreds of nodes and hundreds of timesteps. Using a technology such as XLA with TensorFlow could speed up evaluation but is unlikely to match the performance of a hand-written kernel.

Figures and Tables:

Figure 1: Deep Voice 3 uses residual convolutional layers to encode text into per-timestep key and value vectors for an attention-based decoder. The decoder uses these to predict the mel-scale log magnitude spectrograms that correspond to the output audio. (Light blue dotted arrows depict the autoregressive process during inference.) The hidden states of the decoder are then fed to a converter network to predict the vocoder parameters for waveform synthesis. See Appendix A for more details.

Figure 2: The convolution block consists of a 1-D convolution with a gated linear unit and a residual connection. Here c denotes the dimensionality of the input. The convolution output of size 2 • c is split into equal-sized portions: the gate vector and the input vector.

Figure3: Positional encodings are added to both keys and query vectors, with rates of ω key and ω query respectively. Forced monotonocity can be applied at inference by adding a mask of large negative values to the logits. One of two possible attention schemes is used: softmax or monotonic attention fromRaffel et al. (2017). During training, attention weights are dropped out.

Figure 5: Generated WORLD vocoder parameters with fully connected (FC) layers.

Figure6: Deep Voice 3 uses a deep residual convolutional network to encode text and/or phonemes into per-timestep key and value vectors for an attentional decoder. The decoder uses these to predict the mel-band log magnitude spectrograms that correspond to the output audio. (Light blue dotted arrows depict the autoregressive synthesis process during inference.) The hidden state of the decoder then gets fed to a converter network to output linear spectrograms for Griffin-Lim or parameters for WORLD, which can be used to synthesize the final waveform. Weight normalization(Salimans & Kingma, 2016) is applied to all convolution filters and fully-connected layer weight matrices in the model.

Figure 7: The first two principal components of the learned embeddings for (a) VCTK dataset (108 speakers) and (b) LibriSpeech dataset (2484 speakers).

Attention error counts for single-speaker Deep Voice 3 models on the 100-sentence test set, given in Appendix E. One or more mispronunciations, skips, and repeats count as a single mistake per utterance. "Phonemes & Characters" refers to the model trained with a joint character and phoneme representation, as discussed in Section 3.2. We did not include phoneme-only models because the test set contains out-of-vocabulary words. All models use Griffin-Lim as their vocoder.

Table content: <table><tr><td>Text Input</td><td colspan="5">Attention Inference constraint Repeat Mispronounce Skip</td></tr><tr><td>Characters-only</td><td>Dot-Product</td><td>Yes</td><td>3</td><td>35</td><td>19</td></tr><tr><td>Phonemes &amp; Characters</td><td>Dot-Product</td><td>No</td><td>12</td><td>10</td><td>15</td></tr><tr><td colspan="2">Phonemes &amp; Characters Dot-Product</td><td>Yes</td><td>1</td><td>4</td><td>3</td></tr><tr><td>Phonemes &amp; Characters</td><td>Monotonic</td><td>No</td><td>5</td><td>9</td><td>11</td></tr></table>

Hyperparameters used for best models for the three datasets used in the paper.

Table content: <table><tr><td colspan="2">Parameter Single-Speaker</td><td>VCTK</td><td>LibriSpeech</td></tr><tr><td>FFT Size</td><td>4096</td><td>4096</td><td>4096</td></tr><tr><td>FFT Window Size / Shift</td><td>2400 / 600</td><td>2400 / 600</td><td>1600 / 400</td></tr><tr><td>Audio Sample Rate</td><td>48000</td><td>48000</td><td>16000</td></tr><tr><td>Reduction Factor r</td><td>4</td><td>4</td><td>4</td></tr><tr><td>Mel Bands</td><td>80</td><td>80</td><td>80</td></tr><tr><td>Sharpening Factor</td><td>1.4</td><td>1.4</td><td>1.4</td></tr><tr><td>Character Embedding Dim.</td><td>256</td><td>256</td><td>256</td></tr><tr><td>Encoder Layers / Conv. Width / Channels</td><td>7 / 5 / 64</td><td>7 / 5 / 128</td><td>7 / 5 / 256</td></tr><tr><td>Decoder Affine Size</td><td>128, 256</td><td>128, 256</td><td>128, 256</td></tr><tr><td>Decoder Layers / Conv. Width</td><td>4 / 5</td><td>6 / 5</td><td>8 / 5</td></tr><tr><td>Attention Hidden Size</td><td>128</td><td>256</td><td>256</td></tr><tr><td>Position Weight / Initial Rate</td><td>1.0 / 6.3</td><td>0.1 / 7.6</td><td>0.1 / 2.6</td></tr><tr><td>Converter Layers / Conv. Width / Channels</td><td>5 / 5 / 256</td><td>6 / 5 / 256</td><td>8 / 5 / 256</td></tr><tr><td>Dropout Keep Probability</td><td>0.95</td><td>0.95</td><td>0.99</td></tr><tr><td>Number of Speakers</td><td>1</td><td>108</td><td>2484</td></tr><tr><td>Speaker Embedding Dim.</td><td>-</td><td>16</td><td>512</td></tr><tr><td>ADAM Learning Rate</td><td>0.001</td><td>0.0005</td><td>0.0005</td></tr><tr><td>Anneal Rate / Anneal Interval</td><td>-</td><td colspan="2">0.98 / 30000 0.95 / 30000</td></tr><tr><td>Batch Size</td><td>16</td><td>16</td><td>16</td></tr><tr><td>Max Gradient Norm</td><td>100</td><td>100</td><td>50.0</td></tr><tr><td>Gradient Clipping Max. Value</td><td>5</td><td>5</td><td>5</td></tr></table>

Peer review (iclr2018):
 This paper discusses a text-to-speech system which is based on a convolutional attentive seq2seq architecture. It covers experiments on a few datasets, testing the model*s ability to handle increasing numbers of speakers. By and large, this is a "system" paper - it mostly describes the successful application of many different existing ideas to an important problem (with some exceptions, e.g. the novel method of enforcing monotonic alignments during inference). In this type of paper, I typically am most interested in hearing about *why* a particular design choice was made, what alternatives were tried, and how different ideas worked. This paper is lacking in this regard - I frequently was left looking for more insight into the particular system that was designed. Beyond that, I think more detailed description of the system would be necessary in order to reimplement it suitably (another important potential takeaway for a "system" paper). Separately, I the thousands-of-speakers results are just not that impressive - a MOS of 2 is not really useable in the real-world. For that reason, I think it*s a bit disingenuous to sell this system as "2000-Speaker Neural Text-to-Speech". For the above reasons, I*m giving the paper a "marginally above" rating. If the authors provide improved insight, discussion of system specifics, and experiments, I*d be open to raising my review. Below, I give some specific questions and suggestions that could be addressed in future drafts. - It might be worth giving a sentence or two defining the TTS problem - the paper is written assuming background knowledge about the problem setting, including different possible input sources, what a vocoder is, etc. The ICLR community at large may not have this domain-specific knowledge. - Why "softsign" and not tanh? Seems like an unusual choice. - What do the "c" and "2c" in Figure 2a denote? - Why scale (h_k + h_e) by .sqrt{0.5} when computing the attention value vectors? - "An L1 loss is computed using the output spectrograms" I assume you mean the predicted and target spectrograms are compared via an L1 loss. Why L1? - In Vaswani et al., it was shown that a learned positional encoding worked about as well as the sinusoidal position encodings despite being potentially more flexible/less "hand-designed" for machine translation. Did you also try this for TTS? Any insight? - Some questions about monotonic attention: Did you use the training-time "soft" monotonic attention algorithm from Raffel et al. during training and inference, or did you use the "hard" monotonic attention at inference time? IIUC the "soft" algorithm doesn*t actually force strict monotonicity. You wrote "monotonic attention results in the model frequently mumbling words", can you provide evidence/examples of this? Why do you think this happens? The monotonic attention approach seems more principled than post-hoc limiting softmax attention to be monotonic, why do you think it didn*t work as well? - I can*t find an actual reference to what you mean by a "wavenet vocoder". The original wavenet paper describes an autoregressive model for waveform generation. In order to use it as a vocoder, you*d have to do conditioning in some way. How? What was the structure of the wavenet you used? Why? These details appear to be missing. All you write is the sentence (which seems to end without a period) "In the WaveNet vocoder, we use mel-scale spectrograms from the decoder to condition a Wavenet, which was trained separated". - Can you provide examples of the mispronunciations etc. which were measured for Table 1? Was the evaluation of each attention mechanism done blindly? - The 2.07 MOS figure produced for tacotron seems extremely low, and seems to indicate that something went wrong or that insufficient care was taken to report this baseline. How did you adapt tacotron (which as I understand is a single-speaker model) to the multi-speaker setting? - Table 3 begs the question of whether Deep Voice 3 can outperform Deep Voice 2 when using a wavenet vocoder on VCTK (or improve upon the poor 2.09 MOS score reported). Why wasn*t this experiment run? - The paragraph and appendix about deploying at scale is interesting and impressive, but seems a bit out of place - it probably makes more sense to include this information in a separate "systems" paper.