End-to-end Adversarial Text-to-Speech
Authors: Jeff Donahue
Sander Dieleman
Mikolaj Binkowski
Erich Elsen
Karen Simonyan
Venue: ICLR 2021 Oral (2021)
Keywords: text-to-speech
speech synthesis
adversarial
GAN
end-to-end
feed-forward
generative model

Abstract: Modern text-to-speech synthesis pipelines typically involve multiple processing stages, each of which is designed or learnt independently from the rest. In this work, we take on the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Our proposed generator is feed-forward and thus efficient for both training and inference, using a differentiable alignment scheme based on token length prediction. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses constraining the generated audio to roughly match the ground truth in terms of its total duration and mel-spectrogram. To allow the model to capture temporal variation in the generated audio, we employ soft dynamic time warping in the spectrogram-based prediction loss. The resulting model achieves a mean opinion score exceeding 4 on a 5 point scale, which is comparable to the state-of-the-art models relying on multi-stage training and additional supervision.

Published as a conference paper at ICLR 2021
END-TO-END ADVERSARIAL TEXT-TO-SPEECH
Jeff Donahue; Sander Dieleman," Mikotaj Binkkwski, Erich Elsen, Karen Simonyan" DeepMind
(jeffdonahue, sediellm, binek, eriche, simonyan] (ammonnyn)ngooglee.e.eooo @google com
ABSTRACT
Modern text-to-speech synthesis pipelines typically involve multiple processing stages, each of which is designed or learnt independently from the rest. In this work, we take on the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Our proposed generator is feed-forward and thus efficient for both training and inference, using a differentiable alignment scheme based on token length prediction. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses constraining the generated audio to roughly match the ground truth in terms of its total duration and mel-spectrogram. To allow the model to capture temporal variation in the generated audio, we employ soft dynamic time warping in the spectrogram ppectrogram-based prediction loss. The resulting model achieves a mean opinion score exceeding 4 on a 5 point scale, which is comparable to the state-of-the-ard models relying multi-stage training and additional supervision!
INTRODUCTION
A text-to-speech (TTS) system processes natural language text inputs to produce synthetic human- like speech outputs. Typical TTS pipelines consist of number of stages trained or designed independently -e.g. text normalisation, aligned linguistic featurisation, mel-spectrogram synthesis, and raw audio waveform synthesis (Tallor 2009). Although these pipelines have proven capable of realistic and high-fidelity speech synthesis and enjoy wide real-world use today, these modular approaches come with a number of drawbacks. They often require supervision at each stage, in some cases necessitating expensive "ground truth" annotations guide the outputs of each stage, and sequential training of the stages. Further, they are unable to reap the full potential rewards of data-driven "end-to-end" learning widely observed in number of prediction and synthesis task domains across machine learning.
In this work, aim simplify the TTS pipeline and take the challenging task of synthesising speech from text or phonemes in an end-to-end manner. We propose EATS annd-to-end Adversarial Text-to-Speech- generative models for TTS trained adversarially (Goodfellow et al.] (014) that operate on either pure text or raw (temporally unaligned) phoneme input sequences, and produce raw speech waveforms output. These models eliminate the typical intermediate bottlenecks present in most state-of-ther of-the-rrt TTS engines by maintaining learnt intermediate feature representations throughout the network.
Our speech synthesis models composed of two high-level submodules, detailed in Section 2 An aligner processes the raw input sequence and produces relatively low-frequency (200 Hz) aligned features in its own learnt, abstract feature space. The features output by the aligner may be thought of as taking the place of the earlier stages of typical TTS pipelines e.g., temporally aligned mel- spectrograms or linguistic features. These features are then input to the decoder which upsamples the features from the aligner by ID convolutions to produce 24 kHz audio waveforms.
By carefully designing the aligner and guiding training by combination of adversarial feedback and domain-specific loss functions, demonstrate that TTS system can be learnt nearly end-to-end,
Equal contribution. First author determined by coin toss. Listen to our model reading this abstract at: publ cat oos/oonn//ddd/ee Knd Aewendd-Avvearrraaa Text Speech

---

Published as a conference paper at ICLR 2021
resulting in high-fidelityy natural- soundinng speech approaching the sate-of-the-aar TTS systems. Our main contributions include:
A fully differentiable and efficient feed-forward aligner architecture that predicts the duration of each input token and produces an aalined representation.
The use of flexible dynamic time warping-base baandd prediction losses to enforce alignment with input conditioning while allowing the model to capture the variability of timing in human speech.
An overall system achieving mean opinion score of 4.083, approaching the state of the art from models trained using richer supervisory signals.
METHOD
Our goal is to learn a neural network (the generator) which maps an input sequence of characters or phonemes raw audio at 24 kHz. Beyond the vastly different lengths of the input and output signals, this task is also challenging because the input and output are not aligned, i.e. it is not known beforehand which output tokens each input token will correspond to. To address these challenges, we divide the generator into two blocks: (i) the aligner, which maps the unaligned input sequence to a representation which is aligned with the output, but has a lower sample rate of 200 Hz; and (ii) the decoder, which upsamples the aligner's output to the full audio frequency. The entire generator architecture is differentiable, and is trained end to end. Importantly, is also feed-forward convolutional network, which makes it well-suited for applications where fast batched inference is important: our EATS implementation generates speech at a speed of 200 realtime on a single NVIDIA V100 GPU (see Appendix AAaad Table for details). It is illustrated in Figure [I]
The generator is inspired by GAN-TTS (Binkowski et al. 2020). text-to-speech generative ad- versarial network operating on aligned linguistic features. We employ the GAN-TTS generator as the decoder in our model, but instead of upsampling pre-computed linguistic features, its input comes from the aligner block. We make speaker-conditionn by feeding in a speaker embedding alongside the latent vector z, enable training larger dataset with recordings from multiple speakers. We also adopt the multiple random window discriminators (RWDs) from GAN-TTS, which have been proven effective for adversarial raw waveform modelling, and we preprocess real audio input by applying a simple p-law transform. Hence, the generator is trained to produce audio in the μ-law domain and we apply the inverse transformation to its outputs when sampling.
The loss function use train the generator is as follows:
LG LG + Apred f'pred + Aength Llength:
where LG.adv is the adversaria loss, linear in the discriminators' outputs, paired with the hinge loss (Lim & Ye 2017: Tran et al.) 2017) used as the discriminators' objective, as used in GAN- TTS (Binkowski et al. (020). The of adversarial (Goodfellow et al., (014)) loss is advantage of our approach, as this setup allows for efficient feed-forward training and inference, and such losses tend to be beckins in practice, a useful behaviour in a strongly conditioned setting where realism is an important design goal, as in the case of text-to-speede In the remainder of this section, we describe the aligner network and the auxiliary prediction (Chrea) and length (Clength) losses in detail, and recap the components which were adopted from GAN-TTS.
2.1 ALIGNER
Given a token sequence x = (x1,..., x of length N, we first compute token representations h f(x, z, s). where ff is stack of dilated convolutions (van den Oord et al. 2016) interspersed with batch normalisation (loffe & Szegedy 2013) and ReLU activations. The latents and speaker embedding modulate the scale and shift parameters of the batch normalisation layers (Dumoulin et al. 2017 De Vries et al., 2017). We then predict the length for each input token individually: Ln = g ((,,, 2, s), where g is an MLP. We use a ReLU nonlinearity at the output to ensure that the predicted lengths are non-negative. We can then find the predicted token end positions a cumulative sum of the token lengths: e, = Σm=1 Ims and the token centre positions as C, = Cn - 1/1.. Based on these predicted positions, we can interpolate the token representations into representation at 200 Hz, a = (a)..... as). where S [ex] is the total number of output time steps. To compute dt, we obtain interpolation weights for the token representations h, using softmax over
audio-aligned
2

---

Published as a conference paper at ICLR 2021
the squared distance between t and Cys scaled by temperature parameter o², which set to 10.0
(i.e. a Gaussian kernel);
exp (-0 (1 cn)
(2)
Σm=1 exp (-0-211 cm))2)
Using these weights, we can then compute at = Σm= wof hy, which amounts to non-uniform interpolation. By predicting token lengths and obtaining positions using cumulative summation, instead of predicting positions directly, we implicitly enforce monotonicity of the alignment. Note that tokens which have non-monttonic effect prosody, such as punctuation, entire utterance thanks to the stack of dilated convolutions f. whose receptive field is large enough to allow for propagation of information across the entire token sequence. The convolutions also ensure generalisation across different sequence lengths. Appendix B includes pseudocode for the aligner.
still affect the
2.2 WINDOWED GENERATOR TRAINING
Decoder
GAN-TTS
Training examples vary widely in length, from about to 20 seconds. We cannot pad all se- quences to maximal length during training, as this would be wasteful and prohibitively expen- sive: 20 seconds of audio at kHz correspond to 480,000 timesteps, which results in high mem- ory requirements. Instead, we randomly extract second window from each example, which we will refer to as a training window, by uni- formly sampling random offset 17. The aligner produces a 200 Hz audio-aligned representation for this window, which is then fed to the de- coder (see Figure D]. Note that we only need to compute a, for time steps t that fall within the sampled window, but we do have to com- pute the predicted token lengths L, for the entire input sequence. During evaluation, we simply produce the audio-aligned representation for the full utterance and run the decoder on it, which is possible because fully convolutional.
Embed Concat
Generator
Linear
BN, ReLU
Transpose
Softmax along the horizortal aws
logits grid
Aligner
7
nomp oo-oftct
5.9
token centres
C. - L -1,/2
token lengths
Linear
BN, ReLU, Conv
Linear
BN, ReLU, Conv 256
Concat
Dilated Conv Block
Embed
Conv
2.3 ADVERSARIAL DISCRIMINATORS
kaet s de meet
Phonemizer
Random window discriminators. We ensemble of random window discriminators (RWDs) adopted from GAN-TTS. Each RWD operates on audio fragments of different lengths, randomly sampled from the training window. We use five RWDs with window sizes 240, 480, 960, 1920 and 3600. This enables each RWD to operate at different resolution. Note that 3600 samples at 24 kHz corresponds to 150 ms of audio, all RWDs operate on short timescales. All RWDs in our model are unconditional with respect to text: they cannot access the text quence or the aligner output. (GAN-TTS uses 10 RWDs, including 5 conditioned on linguistic features which we omit.) They are, however, conditioned on the speaker, via projection embedding (Miyato & Koyama 2018
Speaker ID
Cat sat on the mat.
Figure 1: A diagram of the generator, including the monotonic interpolation-base aligner. = and ch denote the latent Gaussian vector and the number of output channels, respectively. During training, audio windows have fixed length of 2 seconds and are generated from the conditioning text using random offsets 1 and predicted phoneme lengths; the shaded areas in the logits grid and waveform are not synthesised. For inference (sampling), we set 0. In the No Phonemes ablation, the phone- mizer skipped and the character sequence fed directly into the aligner.
Spectrogram discriminator. We additional discriminator which operates the full training window in the spectrogram domain. We extract log-scaled mel-spectrsgrams from the audio signals and use the BigGAN- architecture (Brock et al. 2018), essentially treating the spectrograms as
3

---

Published as a conference paper at ICLR 2021
images. The spectrogram discriminator also uses speaker identity through projection embedding. Details on the spectrogram discriminator architecture are included in Appendix
2.4 SPECTROGRAM PREDICTION LOSS
In preliminary experiments, discovered that adversarial feedback is insufficient to learn alignment. At the start of training. the aligner does not produce an accurate alignment, so the information in the input tokens is incorrectly temporally distributed. This encourages the decoder to ignore the aligner output. The unconditional discriminators provide no useful learning signal to correct this. If we want use conditional discriminators instead, we face a different problem: we do not have aligned ground truth. Conditional discriminators also need an aligner module, which cannot function correctly at the start of training, effectively turning them into unconditional discriminators. Although it should be possible in theory to train the discriminators aligner modules adversarially, we find that this does not work in practice, and training gets stuck.
Instead, we propose to guide learning by using an explicit prediction loss in the spectrogram domain: we minimise the L1 loss between the log-scaled -pestpectss of the generator output, and the corresponding ground truth training window. This helps training to take off, and renders conditional discriminators unnecessary, simplifying the model. Let Sgen be the spectrogram of the generated audio, Sgt the spectrogram of the corresponding ground truth, and S Sit. f) the log-scaled magnitude at time step t and mel-frequency bin
f. Then the prediction loss is:
pred = + > E/=1 |Seen [t, f] 1t, fll-
T and F are the total number of time steps and mel-frequency bins respectively. Computing the prediction loss in the spectrogram domain, rather than the time domain, has the advantage of increased invariance phase differences between the generated and ground truth signals, which are not perceptually salient. Seeing as the spectrogram extraction operation has several hyperparameters and its implementation is not standardised, we provide the code used for this in Appendix [ We applied a small amount of jitter (by up to +-60 samples at 24 kHz) to the ground truth waveform before computing Sgt. which helped reduce artifacts in the generated audio.
The inability to learn alignment from adversarial feedback alone is worth expanding on: likelihood- based autoregressive models have no issues learning alignment, because they are able to benefit from teacher forcing (Williams & Zipser 1989) during training: the model is trained next step prediction on each sequence step, given the preceding ground truth, and it is expected to infer alignment only one step at time. This is not compatible with feed-forward adversarial models however, so the prediction loss is necessary to bootstrap alignment learning for our model.
perform
Note that although we make use of mel-spectrograms for training in Lpred (and to compute the inputs for the spectrogram discriminator, Section 2.3), the generator itself does not produce spectrograms as part of the generation process. Rather, its outputs are raw waveforms, and we convert these waveforms to spectrograms only for training (backpropagating gradients through the waveform to mel-spectrogram conversion operation).
2.5 DYNAMIC TIME WARPING
The spectrogram prediction loss incorrectly assumes that token lengths are deterministic. We can relax the requirement that the generated and ground truth spectrograms are exactly aligned, by incorporating dynamic time warping (DTW) (Sakoe, 1971; Sakoe & Chiba 1978). We calculate the prediction loss by iteratively finding a minimal-cost alignment path p between the generated and target spectrograms, Sgen and Sg. We start at the first time step in both spectrograms: Pgen.1 = Pgt.1 1. At each iteration k. we take one of three possible actions:
1. go the next time step in both Sgen Sgei Pgen.A+1 = Pgen, + 1, Pgt,k+1 Pat.k + 1;
2. go the next time step in Sgt only: Pgenk+11 Pgen,ks Pgt,t+1 Pgt,k
3. go to the next time step in Sgen only: PgenA Pgen, + 1, Pgt.R+1 Pgt.k-
The resulting path is p (pgen, 1, Pgt.1)...., (Pgen,Kp: Pet,KK))), where Kp is the length. Each action is assigned a cost based on the L, distance between Sgen [ggen.k] and Sgt Pgt..l and a warp penalty w which is incurred if we choose not to advance both spectrograms in lockstep (i.e. we
4

---

Published as a conference paper at ICLR 2021
warping the spectrogram by taking action 2 or 3; use 1.0). The warp penalty thus encourages alignment paths that do not deviate too far from the identity aligmment. Let óg be an indicator which 1 for iterations where warping occurs, and 0 otherwise. Then the total path cost Cp is:
Cp = Σ w Ög + Eド=1 SgenlPgen, k. fl SgelPge.ke fll)
(4)
K depends the degree of warping (T S K s 27 1). The DTW prediction loss is then:
pid min Cp: pEP
(5)
where P is the set of all valid paths. p E P only when Pzen.1 = Pst.1 = 1 and Pgen,K,, = Pgt.K, T. aligned. To find the minimum, we use dynamic
ic. the first and last timesteps of the spectrograms programming. Figure 2 shows a diagram of an optimal alignment path between two sequences.
DTW is differentiable, but the minimum across all paths makes op- timisation difficult, because the gradient is propagated only through the minimal path. We use a soft version of DTW instead (Cuturi & Blondel 2017), which replaces the minimum with the soft minimum: Spre - log EpEp exp (-
D
D
(6)
C
where 0.01 is temperature parameter and the loss scale factor
Apred = 1.0. Note that the minimum operation is recovered by letting - 0. The resulting loss is weighted aggregated cost across all paths, enabling gradient propagation through all feasible paths. This creates trade-off: higher t makes optimisation easier, but the resulting loss less accurately reflects the minimal path cost. Pscudocode for the soft DTW procedure is provided in Appendix
A
A
A
S
Figure 2: Dynamic time warp- ing between two sequences finds minimal-cost align- ment path. Positions where warping occurs are marked with a border.
By relaxing alignment in the prediction loss, the generator can pro- duce waveforms that are not exactly aligned, without being heavily penalised for it. This creates a synergy with the adversarial loss: instead of working against each other because of the rigidity of the prediction loss, the losses now cooperate to reward realistic audio generation with stochastic alignment. Note that the prediction loss computed on training window, and not on full length utterances, so we still assume that the start and end points of the windows are exactly aligned. While this might be incorrect, it does not seem to be much of problem in practice.
2.6 ALIGNER LENGTH LOSS
To ensure that the model produces realistic token length predictions, we add loss which encourages the predicted utterance length to be close to the ground truth length. This length is found by summing all token length predictions. Let L. be the the number of time steps in the training utterance at 200 Hz, 1, the predicted length of the nihh token, and N the number of tokens, then the length loss is:
(L (L- -A- (4
Ciength
We use scale factor Atength 0.1. Note that we cannot match the predicted lengths I, to the ground truth lengths individually, because the latter are not available.
2.7 TEXT PRE-PROCESSING
Although our model works well with character input, we find that sample quality improves signifi- cantly using phoneme input instead. This is not too surprising. given the heterogeneous way which spellings map to phonemes, particularly in the English language. Many character sequences also have special pronunciations, such as numbers, dates, units of measurement and website domains, and very large training dataset would be required for the model to learn to pronounce these correctly. Text normalisation (Zhang et al. 2019) can be applied beforehand to spell out these sequences as they are typically pronounced (e.g. g.. 1976 could become nineteen seventy six). potentially followed by conversion to phonemes. We use an open source tool, phonemizer (Bernard 2020), which performs partial normalisation and phonemisation (see Appendix FF). Finally, whether we train on text or phoneme input sequences, we pre- and post-pad the sequence with a special silence token (for training and inference), allow the aligner to account for silence the beginning and end of each utterance.
5

---

we compare Published as a conference paper at ICLR 2021
3
RELATED WORK
Speech generation saw significant quality improvements once treating it as a generative modelling problem became the norm (Zen et al., 2009 van den Oord et al. 2016). Likelihood- approaches dominate, but generative adversarial networks (GANs) (Goodiellow et al., 20014) have been making significant inroads recently. A common thread through most of the literature is separation of the speech generation process into multiple stages: Ooarse-graing temporally aligned intermediate representations, such as mel- specttrtrogss,ss are used to divide the task into more manageable sub- problems. Many works focus exclusively on either spectrogram generation or vocoding (generating waveform from a spectrogram). Our work is different in this respect, and we will point out which stages of the generation process are addressed by each model. In Appendix []] Table 6 these methods in terms of the inputs and outputs to each stage of their pipelines.
Initially, most likelihood- basse models for TTS were autoregressive (van den Oord et al. 2016: 6 Mehri et al.] 2017] Arik et al., (017), which means that there is sequential dependency between subsequent time steps of the produced output signal. That makes these models impractical for real-time use, although this can be addressed with careful engineering (Kalchbrenner et al. 2018 Valin & Skoglund 2019). More recently, flow-based models (Papamakarios et al., 2019) have been explored as a ford-forward alternative that enables fast inference (without sequential dependencies). These can
either be trained directly using maximum likelihood (Prenger et al. 2219 Kim et al. 2019 Ping et al.
2019b), through distillation from an autoregressive model (van den Oord et al.] 2018 Ping et al.
2019a). All of these models produce waveforms conditioned on an intermediate representation: either spectrograms or "linguistic features", which contain remporally-aligne high-level information about the speech signal. Spectrogram-condiitonn
waveform models are often referred to as vocoders.
A growing body of work has applied GAN (Goodfellow et al,, 2014) variants to speech synthe- sis (Donahue et al.] 2019). An important advantage of adversarial losses for TTS is a focus on realism over diversity; the latter is less important in this setting. This enables more efficient use of capacity compared to models trained with maximum likelihood. MelGAN (Kumar et al., 2019) and
Parallel WaveGAN YYamamoto et al. (2020) are adversarial vocoders, producing raw waveforms from mel-spectroggamss Neekhara et al. (2019) predict magnitude spectrograms from mel- mel- seesssectsograms. Most directly related to our work is GAN-TTS (Binkowski et al. (020), which produces waveforms conditioned on aligned linguistic features, and we build upon that work.
Another important line of work covers spectrogram generation from text. Such models rely on a vocoder to convert the spectrograms into waveforms (for which one of the previously mentioned models could be used, or a traditional spectrogram inversion technique (Griffin & Lim 1984)) Tacotron 1 & 2 (Wang et al. 2017 Shhnn et al. 2018), Deep Voice 2 & 3 (Gibiansky et | ct al. 2018), TransformerTTS (Li et al. 2019). ). Flowtron (Valle et al. 2020), and VoiceLoop (Taigman et al. 2017) are autoregressive models that generate spectrograms or vocoder features frame by frame. Guo et al. (2019) suggest using an adversarial loss to reduce exposure bias (Bengio et al. 2015] Ranzato al.) 2016) in such models. MelNet (Vasquez & Lewis 2019) is autoregressive over both time and frequency. ParaNet (Peng et al., 2019) and FastSpeech (Ren et al. 2019) are non- autoregressive, but they require distillation (Hinton et 2015) from an autoregressive model. Recent flow-based approaches Flow-TTS (Miao et al.) 2020) and Glow-TTS (Kim et al. 2020) are feed- forward without requiring distillation. Most spectrogram generation models require training of custom vocoder model on generated spectrograms, because their predictions are imperfect and the vocoder needs to be able to compensate for thi Note that some of these works also propose new vocoder architectures in tandem with spectrogram generation models.
Unlike all of the aforementioned methods, as highlighted in Appendix Table our model is single beed-forward neural network, trained end-to-end in single stage. which produces waveforms given character or phoneme sequences, and learns to align without additional supervision from auxiliary sources (e.g. temporally aligned linguistic features from an external model) or teacher forcing. This simplifies the training process considerably. Char2wav (Sotelo et al., 2017) is finetuned end-to-end in the same fashion, but requires a pre-training stage with vocoder features used for intermediate supervision.
Spectrogram prediction losses have been used extensively for feed-forward audio prediction mod-
els Yamamoto al. 2019 2020 Yang et al. 2020 Arik et al. 2018 Engel et al. 2020 Wang et al.
This also implies that the spectrogram generation model and the vocoder have to be trained sequentially.
6

---

Published as a conference paper at ICLR 2021
2019; Défossez et al. 2018). We note that the L1 loss (along with (Défossez al. 2018 )). is comparatively simple, as spectrogram losses in the literature tend to have separate terms penalising magnitudes. log- magnitudes and phase components, each with their own scaling factors, and often across multiple resolutions. Dynamic time warping on spectrograms is a component of many speech recognition systems (Sakoe 1971. Sakoe & Chiba 1978), and has also been used for evaluation of TTS systems (Sailor& & Patil 2014 Chevelu et al. 2015). Cuturi & Blondel (2017) proposed the soft version of DTW we use in this work differentiable loss function for time series models. Kim et al. (020) propose Monotonic Alignment Search (MAS), which relates to DTW in that both use dynamic programming implicitly align sequences for TTS. However, they have different goals: MAS finds the optimal alignment between the text and latent representation, whereas we use DTW to relax the constraints imposed by our spectrogram prediction loss term, Several mechanisms have been proposed to exploit monotonicity in tasks that require sequence alignment, including attention mechanisms (Graves, 2013 Zhang et al., 2018: Vasquez Lewis, 2019: He ct al. 2019; Raffel et al., 2017] Chiu & Raffel, 2018). loss functions (Graves et al.) 2006; Graves 2012) and search-based approaches (Kim et al.) 2020). For TTS, incorporating this constraint has been found to help generalisation long sequences (Battenberg al. (220). We incorporate monotonicity by using an interpolation mechanism, which is cheap to compute because it is not recurrent (unlike many monotonic attention mechanisms).
EVALUATION
In this section we discuss the setup and results of our empirical evaluation, describing the hyperparam- eter settings used for training and validating the architectural decisions and loss function components detailed in Section [ Our primary metric used to evaluate speech quality is the Mean Opinion Score (MOS) given by human raters, computed by taking the mean of 1-5 naturalness ratings given across 1000 held-out conditioning sequences. In Appendix [ we also report the Fréchet DeepSpeech Distance (FDSD), proposed by BBnkkowsi et al. (2020) as a speech synthesis quality metric. Appendix A reports training and evaluation hyperparameters we used for all experiments.
4.1 MULTI-SPEAKER DATASET
We train all models a private dataset that consists of high-quality recordings of human speech performed by professional voice actors, and corresponding text. The voice pool consists of 69 female and male voices of North American English speakers, while the audio clips contain full sentences of lengths varying from less than 1 to 20 seconds at 24 kHz frequency. Individual voices are unevenly distributed, accounting for from 15 minutes to over 51 hours of recorded speech, totalling 260.49 hours. At training time, we sample 2 second windows from the individual clips, post- post-padding those shorter than seconds with silence. For evaluation, we focus on the single most prolific speaker in our dataset, with all our main MOS results reported with the model conditioned on that speaker ID, but also report MOS results for each of the top four speakers using our main multi-speaker model.
4.2 RESULTS
In Table I we present quantitative results for our EATS model described in Section 2 as well as several ablations of the different model and learning signal components. The architecture and training setup of each ablation is identical to our base EATS model except in terms of the differences described by the columns in Table Each ablation is "subtractive", representing the full EATS system minus one particular feature. Our main result achieved by the base multi-speaker model is mean opinion score (MOS) of 4. 4.083. Although is difficult to compare directly with prior results from the literature due to dataset differences, we nonetheless include MOS results from prior works (Binkowski et al., 2020; van den Oord et al.) [0016] 20188), with MOS in the 4.2 to 4.4+ range. Compared to these prior models, which rely aligned linguistic features, EATS uses substantially less supervision.
The No RWDs, No MelSpecD, and No Discriminators ablations all achieved substantially worse MOS results than our proposed model, demonstrating the importance of adversarial feedback. In particular, the No RWDs ablation, with an MOS of 2.526, demonstrates the importance of the raw audio feedback, and removing RWDs significantly degrades the high frequency components. No MelSpecD causes intermittent artifacts and distortion, and removing all discriminators results in audio that sounds robotic and distorted throughout. The No Liength and No Epred ablations result in model that does not train at all. Comparing model with No DTW (MOS 3.559), the temporal
7

---

Published as a conference paper at ICLR 2021
Model
Data
Inputs
RWD
MSD
Liength
Pped
Align
MOS
Natural Speech
4.55 0.075
GAN-TTS (Binkowski ct al. 2020
4.213 + 0.046
WaveNet (van den Oord et al 2016
4.41 0.069
Par. WaveNer Van den Oord al. 2018
4.41 0.078
Tacotron (Shen ct al, 2018
4.526 + 0.066
No Clengsh
MS
Ph
4
V
MI
[does not train]
No Cpred
MS
Ph
V
V
x
MI
[does not train]
No Discriminators
MS
Ph
X
grd
M1
1.407 + 0.040
RWDs
MS
grod
MT
2.526 + 0.060
No Phonemes
MS
Ch
V
Bred
MI
3.423 1: 0.073
MelSpeeD
Brod
MI
3.525 + 0.057
Int.
Attn
3.551 + 0.073
DTW
Epred
MI
3.559 0.065
Single Speaker
pred
MI
3.829 0.055
EATS (Ours)
MS
Ph
Epred
MI
4.083 + 0.049
Table 1: Mean Opinion Scores (MOS) for our final EATS model and the ablations described in Section 4 sorted by MOS. The middle columns indicate which components our final model are enabled or ablated. Data describes the training set as Multispeaker (MS) or Single Speaker (SS). Inputs describes the inputs as raw characters (Ch) or phonemes (Ph) produced by Phonemizer. RWD (Random Window Discriminators), MSD (Mel-spectrogram Discriminator). and Liength (length prediction loss) indicate the presence (V) or absence (x) of each of these training components described in Section 2 Spred indicates which spectrogram prediction loss used: with DTW (L predd Eq. 6). without DTW (Epred. Eq. 3), or absent (x), Align describes the architecture of the aligner monotonic interpolation (MI) artentionn- (Attn). We also compare against recent -tate-of-tae faate-of-the-arte the-art approaches from the literature which are trained on aligned linguistic features (unlike our models). Our MOS evaluation set matches that of GAN-TTS (Bińkowski et al. (020) (and our "Single Speaker" training subset matches the GAN-TTS training set); the other approaches are not directly comparable due dataset differences.
Speaker
#2
#3
Speaking Time (Hours)
51.68 4.083 + 0.049
31.21
20.68
10.32
MOS
3.828 + 0.051
4.149 + 0.045
3.761 + 0.052
Table 2: Mean Opinion Scores (MOS) for the top four speakers with the most data in our training set. All evaluations done using our single multi-speaker EATS model.
flexibility provided by dynamic time warping significantly improves fidelity: removing it causes warbling and unnatural phoneme lengths. No Phonemes is trained with raw character inputs and attains MOS 3.423, due to occasional mispronunciations and unusual stress patterns. No Mon. Int. uses aligner with transformer- beaseerree--baeedddddded attention mechanism (described in Appendix G in place of our monotonic interpolation architecture, which turns out to generalise poorly to long utterances (yielding MOS 3.551). Finally, comparing against training with only Single Speaker (MOS 3.829) shows that our EATS model benefits from a much larger multi-speaker dataset, even though MOS evaluated only on this same single speaker on which the ablation was solely trained. Samples from each ablation are available at tttps:
com wttpcm//eeeeaacecepeeeeseeeeeeee
We demonstrate that the aligner learns use the latent vector z to vary the predicted token lengths in Appendix H In Table 2 we present additional MOS results from our main multi-speaker EATS model for the four most prolific speakers in training dataf) MOS generally improves with more training data, although the correlation is imperfect (c.g., Speaker #3 achieves the highest MOS with only the third most training data).
AAll of the MOS results in Table are on samples from single speaker, referred Speaker #1 Table
8

---

Published as a conference paper at ICLR 2021
5 DISCUSSION
We have presented adversarial approach to text-to-speech synthesis which can learn from relatively weak supervisory signal normalised text or phonemes paired with corresponding speech audio. The speech generated by proposed model matches the given conditioning texts and generalises to unobserved texts, with naturalness judged by human raters approaching state-of-the-art systems with multi-stage training pipelines or additional supervision. The proposed system described in Section is efficient in both training and inference. In particular, does not rely on autoregressive sampling teacher forcing, avoiding issues like exposure bias (Bengio al. 2015 Ranzato et al., 2016) and reduced parallelism at inference time, or the complexities introduced by distillation to more efficient ferd-fo model after the fact (van den Oord et al. 2018 Ping et al. 2019ap.
While there remains a gap between the fidelity of the speech produced by our method and the state- of-the-art systems, we nonetheless believe that the end-to-end problem setup is a promising avenue for future advancements and research in text-to-speech. End-to-end learning enables the system as whole to benefit from large amounts of training data, freeing models to optimise their intermediate representations for the task at hand, rather than constraining them to work with the typical bottlenecks (e.g., mellspectroggamss, aligned linguistic features) imposed by most TTS pipelines today. We see some evidence this occurring in the comparison between our main result, trained using data from 69 speakers, against the Single Speaker ablation: the former is trained using roughly four times the data and synthesises more natural speech in the single voice on which the latter is trained.
Notably, our current approach does not attempt to address the text normalisation and phonemisation problems, relying
separate, fixed system for these aspects, while a fully end-to-end TTS system could operate on unnormalised raw text. We believe that fully data-driven approach could ultimately prevail even in this setup given sufficient training data and model capacity.
ACKNOWLEDGMENTS
The authors would like to thank Norman Casagrande, Yutian Chen, Aidan Clark, Kazuya Kawakami, Pauline Luc, and many other colleagues DeepMind for valuable discussions and input.

---

Published as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogencous systems. arXiv:1603.04461 2015.
Sercan Ô Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, Shubho Sengupta, and Mohammad Shoeybi. Deep Voice: Real-time neural text-to-speech. In ICML,
Sercan Ô Arık, Heewoo Jun, and Gregory Diamos. Fast spectrogram inversion using multi-head convolutional neural networks. IEEE Signal Processing Letters, 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. 107.0::607 2016.
Eric Battenberg, RJ Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby. Location-relative attention mechanisms for robust long-form speech synthesis. In ICASSP, 2020.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In NeurlPS, 2015.
Mathieu Bernard. Phonemizer. cattps://gite http://ww..cco/ooooopoo///h/eeee
2020.
Mikolaj Binkkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks, In ICLR, 2020.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2018
Peter Buchlovsky, David Budden, Dominik Grewe, Chris Jones, John Aslanides, Frederic Besse, Andy Brock, Aidan Clark, Sergio Gómez Colmenarejo, Aedan Pope, Fabio Viola, and Dan Belov, TF-Replicator: Distributed machine learning for researchers. arXiv preprint arXiv:1902.0m 1902000405. 2019.
Jonathan Chevelu, Damien Lolive, Sébastien Le Maguer, and David Guennec. How to compare TTS systems: A new subjective evaluation methodology focused on differences. In International Speech Communication Association, 2015.
Chung-Cheng Chiu and Colin Raffel. Monotonic chunkwise attention. In ICLR, 2018.
Marco Cuturi and Mathieu Blondel. Soft-DTW: a differentiable loss function for time-series. In ICML., 2017.
Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In NeurIPS, 2017.
Alexandre Défossez, Neil Zeghidour, Nicolas Usunier, Léon Bottou, and Francis Bach. SING: Symbol-to-instrument neural generator. In NeurlPS, 2018.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In ICLR, 2019.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In ICLR, 2017.
Jesse Engel, Lamtharn (Hanoi) Hantrakul, Chenjie Gu, and Adam Roberts. DDSP: Differentiable digital signal processing. In ICLR, 2020.
10

---

Published as a conference paper at ICLR 2021
Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. Deep Voice 2: Multi-speaker neural text-to-speehh. NewrIPS, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
Alex Graves. Sequence transduction with recurrent neural networks. 2012. 2012.
Alex Graves. Generating sequences with recurrent neural networks. 1308.0::300.0555 2013.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, 2006.
Daniel Griffin and Jae Lim. Signal estimation from modified short-time Fourier transform. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1984.
Haohan Guo, Frank K Soong, Lei He, and Lei Xie. A new GAN- end-to-end TTS training algorithm. In Interspeech, 2019.
Mutian He, Yan Deng, and Lei He. Robust seguence-to-sequenee acoustic modeling with stepwise monotonic attention for neural TTS. In Interspeech, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local nash equilibrium. In NeurlPS, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NeurlPS Deep Learning and Representation Learning Workshop, 2015.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JCML, 2015.
Fumitada Itakura. Minimum prediction residual principle applied to speech recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1975.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aäron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In ICML, 2018
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-TTS: A generative flow for text-to-speech via monotonic alignment search. arXiv:2005.11129, 2020.
Sungwon Kim, Sang-Gh Gil Lee, Jongyoon Song, Jachyeon Kim, and Sungroh Yoon. FloWaveNet: A generative flow for raw audio. In ICML, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, and Aaron Courville. MelGAN: Generative adversarial networks for conditional waveform synthesis. In NeurlPS, 2019.
Karol Kurach, Mario Lučié, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. A large-scale study on regularization and normalization in GANs. In ICML, 2019.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In AAAI, 2019.
Jae Hyun Lim and Jong Chul Ye. Geometric GAN. arXiv: 1705:0 2017.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. SampleRNN: An unconditional end-to-end neural audio generation model. In ICLR, 2017.
11

---

Published as a conference paper at ICLR 2021
Chenfeng Miao, Shuang Liang, Minchuan Chen, Jun Ma, Shaojun Wang, and Jing Xiao. Flow-TTS: A non-autoregressive network for text to speech based on flow. In ICASSP, 2020.
Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. ICLR, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICIR, 2018.
Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. WORLD: A vocoder-based high-quality speech synthesis system for real-time applications. JEICE Transactions on Information and Systems, 2016.
Paarth Neekhara, Chris Donahue, Miller Puckette, Shlomo Dubnov, and Julian McAuley, Expediting TTS synthesis with adversarial vocoding. In Interspeech, 2019.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji
Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. ar2iv:191..0266.. 2019.
Kainan Peng, Wei Ping. Zhao Song, and Kexin Zhao. 1905.00900 2019.
Parallel neural text-to-speech.
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep Voice 3: 2000-speaker neural text-to-speech. In ICLR, 2018.
Wei Ping, Kainan Peng, and Jitong Chen. ClariNet: Parallel wave generation in end-to-end text-to- speech. ICLR, 2019a.
Wei Ping, Kainan Peng. Kexin Zhao, and Zhao Song. WaveFlow: A compact flow-based model for raw audio. 191iv:1912.01229, 2019b.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. WaveGlow: A flow-based generative network for speech synthesis. In ICASSP, 2019.
Colin Raffel, ThangTThng Luong, Peter J Liu, Ron J Weiss, and Douglas Eck. Online and linear-ting time attention by enforcing monotonic alignments. In ICML, 2017.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In ICLR, 2016.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. FastSpeech: Fast, robust and controllable text speech. In NeurlPS, 2019.
Malcolm Reynolds, Gabriel Barth-Maron, Frederic Besse, Diego de Las Casas, Andreas Fidjeland, Tim Green, Adrià Puigdomenech, Sébastien Racanière, Jack Rae, and Fabio Viola. Open sourcing Somnet new library for constructing neural networks. https: /ttpp:://eepmiiiiea /ttp///eemd..co////// 2017.
sore-s-suciing-ooneee
Hardik B Sailor and Hemant A Patil. Fusion of magnitude and phase-based features for objective evaluation of TTS voice. In International Symposium on Chinese Spoken Language Processing. 2014.
Hiroaki Sakoe. Dynamic-programminn programming approach to continuous speech recognition. In International Congress of Acoustics, 1971.
Hiroaki Sakoe and Seibi Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1978.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang. RJ Ekerrv-Ryam Rif A. Saurous, Yannis Agiomyrgiannakis, and Yonghui Wu. Natural TTS synthesis by conditioning WaveNet on Mel spectrogram predictions. In ICASSP, 2018.
Jose Sotelo, Soroush Mchri, Kundan Kumar, Joao Felipe Santos, Kyle Kastner, Aaron Courville, and Yoshua Bengio. Char2Wav: End-to-end speech synthesis. In ICIR, 2017.
12

---

Published as a conference paper at ICLR 2021
Yaniv Taigman, Lior Wolf, Adam Polyak, and Eliya Nachmani. VoiceLoop: Voice fitting and synthesis via a phonological loop. 705.0322 2017.
Paul Taylor. Text-to-specch synthesis. Cambridge University Press, 2009,
Dustin Tran, Rajesh Ranganath, and David M. Blci. Hierarchical implicit models and likelihood-fri variational inference. In NeurIPS, 2017.
Jean-Mare Valin and Jan Skoglund. LPCNet: Improving neural speech synthesis through linear prediction. In ICASSP, 2019.
Rafael Valle, Kevin Shih, Ryan Prenger, and Bryan Catanzaro. Flowtron: autoregressive flow-based generative network for text-to-speech synthesis. MxXiv:205500995 2020.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukeuoglu. WaveNet: A generative model for raw audio. 1009 2016.
Aäron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. Parallel WaveNet: Fast high-fidelity speech synthesis. In ICML, 2018.
Sean Vasquez and Mike Lewis. MelNet: A generative model for audio in the frequency domain. arXiv:1906.01083, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
Xin Wang, Shinji Takaki, and Junichi Yamagishi. Neural Nource-filter-bast waveform model for statistical parametric speech synthesis. In ICASSP, 2019.
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang. Ying Xiao, Zhifeng Chen, Samy Bengio, Quoe Le. Yannis Agiomyygiannakis. Rob Clark, and Rif Saurous. Tacotron: Towards end-to-end speech synthesis. In Interspeech, 2017.
Ronald Williams and David Zipser. learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1989.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Probability density distillation with generative adversarial networks for high-quality parallel waveform generation. In Interspecch, 2019.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel WaveGAN: A fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In
Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, and Lei Xie. Multi-band MelGAN: Faster waveform generation for high-quality text-to-speech. arXiv:2005.09 10660, 2020.
Heiga Zen, Keiichi Tokuda, and Alan W Black. Statistical parametric speech synthesis. Speech Communication, 2009.
Hao Zhang, Richard Sproat, Axel H Ng, Felix Stahlberg, Xiaochang Peng, Kyle Gorman, and Brian Roark. Neural models of text normalization for speech applications. Computational Linguistics, 2019.
Jing-Xuan Zhang, Zhen-Hua Ling, and Li-Rong Dai. Forward attention sequence-to-sequeece acoustic modeling for speech synthesis. In ICASSP, 2018.
13

---

Published as a conference paper at ICLR 2021
Utt. Batch
# Batch /
# Utt.
Length /
Length
Med. Run Time (s)
Realtime
Hardware
Run
Run
Utt. (s)
Run (s)
Factor
TPU v3 (1 chip)
16
10
160
30
4800
30.53
157.2x
V100 GPU (1) Xeon CPU (6 cores)
8
10
80
30
2400
11.60
206.8x
2
10
20
30
600
70.42
8.520
Table 3: EATS batched inference benchmarks, timing inference (speech generation) on Google Cloud TPU v3 (1 chip with 2 cores), a single NVIDIA V100 GPU, or an Intel Xeon E5-1650 v4 CPU at 3.60 GHz (6 physical cores). We use batch size of 2, 8, 16 utterances (Utt.), each 30 seconds long (input length of 600 phoneme tokens, padded if necessary). One "run" consists of 10 consecutive forward passes at the given batch size. We perform 101 such runs and report the median run time (Med. Run Time (s)) and the resulting Realtime Factor, the ratio of the total duration of the generated speech (Length Run (s)) to the run time. (Note: GPU benchmarking is done using single precision (IEEE FP32) floating point; switching to half precision (IEEE FP16) could yield further speedups.)
A HYPERPARAMETERS AND OTHER DETAILS
Our models are trained for 5 10 steps, where a single step consists of one discriminator update followed by one generator update, each using a minibatch size of 1024, with batches sampled indepen- dently in each of these two updates. Both updates are computed using the Adam optimizer (Kingma & Ba 2015) with 31 = and 32 = 0.999, and a learning rate of 10 with a cosine decay (Loshchilov & Hutter 2017) schedule used such that the learning rate is 0 at step 500K. We apply spectral normalisation (Miyato et al., 2018) to the weights of the generator's decoder module and to the discriminators (but not the generator's aligner module). Parameters are initialised orthogonally and off-diagonal orthogonal regularisation with weight 10-1 is applied to the generator, following BigGAN (Brock al.) 2018). Minibatches are split over 64 or 128 cores (32 or 64 chips) of Google Cloud TPU v3 Pods, which allows training of a single model within up to 58 hours. We use cross- replica BatchNorm (Toffe & Szegedy 2015) compute batch statistics aggregated across all devices. Like in GAN-TTS (Binkowski et al. 2020), our trained generator requires computation of standing statistics before sampling: i.e., accumulating batch norm statistics from 200 forward passes. As in GAN-TTS (Binkkwssi et al. (020) and BigGAN (Brock et al.] 2018) 8), we use an exponential moving average the generator weights for inference, with a decay of 0.9999. Although GANs are known to exhibit stability issues sometimes, found that EATS model training consistently converges.
Models were implemented using TensorFlow (Abadi et al. 2015) vl framework and the Son- net (Reynolds et al.) (017) neural network library. We used the TF-Replicator (Buchlovsky et al.. 2019) library for data parallel training over TPUs.
Inference speed. In Table 3 report benchmarks for EATS batched inference on two modern hardware platforms (Google Cloud TPU v3, NVIDIA V100 GPU, Intel Xeon E5-1650 CPU), We find that EATS generate speech two orders of magnitude faster than realtime GPU or TPU, demonstrating the efficiency of our feed-forward model. On the GPU, generating 2400 seconds (40 minutes) of speech (80 utterances of 30 seconds each) takes 11. 11.60 seconds average (median), for realtime factor of 206.8x. On TPU we observe a realtime factor of 157.2x per chip (2 cores), or 78.6x per core. On a CPU, inference runs 8.52 realtime.
14

---

Published as a conference paper at ICLR 2021
B ALIGNER PSEUDOCODE
In Figure 3 we present pseudocode for the EATS aligner described in Section 2.1
C SPECTROGRAM DISCRIMINATOR ARCHITECTURE
In this Appendix we present details of the architecture of the spectrogram discriminator (Section 2.3 The discriminator's inputs are 47 80 x 1 images, produced by adding channel dimension to the 47 x 80 output of the mel-spectrogram computation (Appendix D) from the length 48000 input waveforms (2 seconds of audio 24 kHz).
Then, the architecture like that of the BigGAN-deep (Brock al. 2018 discriminator for 128 128 images (listed in BigGAN (Brock et al. 2018) Appendix B, Table 7 (b)), but removing the first two "ResBlocks" and the LNoa-Local Block (self-attention) rows 2-4 in the architecture table (keeping row I, the input convolution, and rows 5+ afterwards, as is). This removes one 2 x 2 downsampling step as the resolution of the spectrogram inputs smaller than the 128 128 images for which the BigGAN-deep architecture was designed. We set the channel width multiplier referenced in the table to ch 64.
15

---

Published as a conference paper at ICLR 2021
def ignae (Loken_sequeenees Loken_vooakl sive, lungtha, speakes_ nom speakers, noise, out ottsstte out wew.eeeeeeeeeeeeeeeee Jergeeeeeerg 3iqma2
Reccuess audio al alinndd festarex and lengths for the given Input sequantes. *x" denotes the patch size chroughour the contente.
Arysz
token sequeneez:: batch Loken sequences indicating the ID each token, padoed to a fixed naximom requence length 1490 for craining, 660 for sampling). tokens may cither corzespond to zow characters or phonenes (as curput by Phonenizwc), Each sequence should begin and aid with special aflence token (assuted to have aiready beep added to the inputs). edrypeeint, chaps-(N, te se.peeence lengthw600))
taken_ pocaa_ sesier int indicsting the number of tokens, (Ai3 values in tokersequenies should be
sokwn size) cet_epsnoo__eetttttee sequence in taken, sequences before pedding ser added. idtype-int, shape (8])
langchs: indicates the tree length
each
speaker ids; ints indicsting the speaxer ID.
(dcypemint, shave- (8])
nux suuappaker:: scalar int indicating the number of speakers.
iii vaiues in speakor_ should in (9, spe_kerer)
nosee: noise sampled tron scanderd isotropic Gauestan (816, 117,
idcypa-fioat, shaps- (Ri 122])
out_offsec: first tisexcep to output. Randarly xappled for training, 0 For sampling.
idrype-int, chape=(8]) out soquence_iength: scaler int length of the output sequence at 200 82. 400 for iraining (2 secunds), 6000 for ssapling (50 swourde) sigma2: sealar tleat terperature (signe+*2) the cottnax.
Returns:
al Li audio aligoed features to be fed inco the desoder. idsype-float, shaps- [8,, out sequsaoee longth, 256]) al aligeed lengthe: the predicted andio-aligned lengtns. idtyps- float, shapo-(M!) eae
Learn eabeddings
input tokens and apeaker
enbedded tobens tokens
niput sizettokeeeeeeeeeee voeah ze, + (6, 268]
cutpat_din: Serrut_dii--2-00 256) (taken_soquonzoe
unbeddeed spaakkrrd
Baised vonpul_ pizeenun output_dia 128) (spoakor_ids)
4- 18, 126]
+ Hake the "claso-conaitioning inpucs for class-conditions! batch nom (CCBM)
a using the enbedded soeaker IDs and the nolse.
scba_cordition. Concat atancddede_eaaaae__ecc dunay sequence soba_cordition con_sondition Nors. 1]
to pebo_pondition noisel, axis=ll ---> (8, 256] broadoest (M, 2, 236]
y Use "lenoths" to make a mask indieating volid entries of toker__cqueeesss
sequence length Loken_ shapee [1] # 606
mask Range (sequence_ length) (None, il < lengtholi, Sone] +>> 18, 6003
pilared 1D convelution stack. 10 blocks x esbesded__oxen
per block 60 convolotions tocal.
conv mesk mask|i, 5, Nonel 5- (N, 600, 1] dumy axie for broadcast.
for in rangs (10):
for a, b in [[I, 21, 14, BI, (16, 32)]: block_inguts Re IC) axstondi fona catchtorsss, oebn conditionn)
8
KackedConviiDeere aerrriieeeppt_ccaaeeeee 256, kernel_sizara dilation al
conv_nask!
MesSiiCiasttnentiiii Betchoorrn,,,,,, ceen conditiont)
KeskedConel Bour chameel_eselee.5. kernel ternel_sizeel, dilationst] (
x, conv_mack)
1= biock_inguta 4 -> 18, 500, 256]
# Bave dilaced conv stack octputs as maalioned_fotturee. aowligned__fe feeturss
(l, 609, 2567
x predicted token lengths.
ccba_condition)] x Ceavin chh s-256, kernel_xiseel]] [x]
x RedurClassooditee wtn.oonntttcnnilBtttioooooeo
ccon_condition)]
x Ceavi jeutput channels-i, kernel_size-1 (x) * -> [6, 600, 1]
token_iengthe Rellixi: 0]1 ->> (25, 6007
Loker_erda (Lnkun( ienggin,,,, axig-1) # > 18, 600!
token_centres token [token_lengths / 2.1 f -> 18, 600/
conpute predicted lonoth the last walld
of toder_ends.
aligned_
[and[_cngth-1] for end, icngth in zip cipcecnn_ccdds lengthai]
# Conpute output grid c> (8, out sequenee www.eoooeeengggghe5000000
out pos = -Range sequence length) [None, i1 1 out offset li, None]
out_pos - cout Kone, :] pat_pos =>> 18, 6000, 600!
i, Bene], flcat) 1 -> 18, 6000, 1]
daf: Loken_ismliew
logite -dittt / eigna2)
6008, 606)
insalid input lonat
(f1ip
1/6);
dunny nutpue axis.
iogits_inv_nace "1. Cast (masklt, None, ih, :loat]
E00)
Ioskei = logita leB logita_
fH, 6090, 600]
weights - Sottnaximssken logits, axis=2: t -> (8, 6050, 600)
# Do a batcl natmul (wribten as an einam)) to cospute the aligged festores. + testures reatures => (8, 5003, 2567 Liignad_featuzer (Lnsun("nol, nid-ood,,,, veights, [eatuuee]]
_ligned_feetturee uret, lengtes
Figure 3: Pseudocode for our proposed EATS aligner.
16

---

Published as a conference paper at ICLR 2021
import mansorflow.compate as tf
def get_rel__sseeeee waa.ueee__ectttrrarn (waveforms, invert_mu_lax mo=255., jitter-False, (e-_jiittr__stepa--3 "*Comuutes mel specttranss for the given waveforme.
Azgs:
wavetorms: tf. Tensor corresponding to batch of waveforms sampled
(dtype-tf, float322, shape-(M, sequence lengthh)
invert_mu law: whether to apply law inversion to the input waveforms.
In EATS both the real data and generator outputs are mu-law'ed, so this is always set to True. mu: The mu value used invert law-True (ignored otherwise).
jitter: whether to apply random jitter to the input waveforms before computing spectrograms.
True only for eT spectrograme input the
prediction loss.
max_iitter_stepp maximum number of steps by which the input waveforms aze randomly jittered if jitter-Irue (ignored otherwise).
Returns:
tensor with spectrograms the corresponding input waveforms. (dtype=tE.float32,
shaps=(N, num frames=ceei (seppnnce_length/1122))) num_bins=86))
waveforms sereforms.shap...sseee asspe. saaeesassstttha_aaaa rank waveforms
if jitter:
assert mex_jitter_
crop_shape [t. stapeeee
= (t. [[O, 01, [max_jitte__steps, max_fitter_steps]]]
f Jitter independently for each batch ftem.
(1ambda
image..e... ragge..anddo_mrrppptt crop_shape),
if invert_mu_law:
((1 =t) (t) tf.signal.sttt ft (t, frame_lengee 204g Frame_step=1024, pad_end-Trus) tf.abs
-ff.signtt /
mel mel_ hti_welght_maariis tf wom_sppetrrogrmmbinstttttt sta..obeenttttt..--ll-.
nur_mel_bins-te num_
sarp)ee_rte==44000. awer__edge_hertz-000 opper_edge_hhrrt==76000
tf..eesordoott, mel, www.wweweggtttttteeeeeeeet axes-1)
t1. 10000.+t) return
gen_apect rww.ettorrammm_mem rogramm sacaaammm__oo__ree ptterrram__r___ee________ loss
(gen
jitter-false)
real_ ael_aapectoogamm for pred loss
jitter-True)
Figure 4: TensorFlow code for mel-spectrogram computation.
D MEL-SPECTTROGRM COMPUTATION
In Figure 4w we include the TensorFlow (Abadi al. 2015) code used to compute the mel-spectrograass fed into the spectrogram discriminator (Section (..3 and the spectrogram prediction loss (Section (2.4). Note that for in the prediction losses Lpred L call this function with jitter=Trus for real spectrograms and jitter-Fal for generated spectrograms. When used for the spec- trogram discriminator inputs, we do upply jitter either real generated spectrograms, setting j bitter=Fa l se in both cases.
17

---

Published as a conference paper at ICLR 2021
def (values, temperature):
** "Compute the soft minimum with the given temperature. *** -temperature leg (sum (exp leva temperature))
def skex_natrix(x)) (x)
""askew a matrix so that the diagonals become the rows, height, width x.shape y zeros (height width 1, width)
for range (height width 1);
for in (vidth:: & shift each column J down by j stepe.
yii, i] - x(elip(1 j, 0, height - 1), 31
return y
def rogram, dtw__erorrrroeoooeeoddooddeoeeee (spec_a, spec_b, parp_penalty-1.., temperature-0.01) "CCompute DIW error given pair of spectrograms. Compute cost matrix.
diffs abs (spec_a [None, 3, :] spec_b[: None, :1)
coata mean (diffs, axis--1) 8 paizwise cost, square the diffs for 12. size cost. shape (-1]
Initialise path costs.
path_coss - INFINITY ones (size 1) path_cost_prev INFINITY oneatsize + 1) path_ccst__prev10 0.0
Aggregate path costs from cost 0] costi-1, -1],
cost akew_matrix (cost) $ Shape is now (2 size 1, size).
for i in range size - 1):
directions beath_cost_prev[::11
path_cost + warp_penalty,
path_cost[:-11 warp warp_penaltyl
path_cost_next cost [i] soft_mirimun (directions, temperature)
# Replace soft minimum with reguiar minimum for regular DIW.
path_cost_next concaterate (I (IINEENNTY), path_cost__en
path_cost, path_cost_pres return path_cost[-1]
path_cost_next, path_cost
Figure 5: Pseudocode for dynamic time warping.
E DYNAMIC TIME WARPING PSEUDOCODE
In Figure 5 we present pseudocode for the soft dynamic time warping (DTW) procedure we use in the spectrogram prediction loss Spredt
Note that the complexity of this implementation is quadratic. It could be made more efficient using Itakura or Sakoe-Chiba bands (Itakura 1975] Sakoe & Chiba 1978), but we found that enabling or disabling DTW for the prediction loss did not meaningfully affect training time, this optimisation is not necessary in practice.
18

---

Published as a conference paper at ICLR 2021
Output symbol
x
Substitute symbol
k
k
Table 4: The symbols in this table replaced removed when they appear in phonemizer's output.
F TEXT PREPROCESSING
We use phonemi zer (Bernard 2020) (version 2.2) to perform partial normalisation and phone- misation of the input text (for all our results except for the No Phonemes ablation, where character sequences as input directly), We used the espeak backend (with espeak ng version 1.50), which produces phoneme sequences using the International Phonetic Alphabet (IPA). We enabled the following options that phonemizer provides:
wi with_stress, which includes primary and secondary stress marks in the output;
strip, which removes spurious whitespace;
cisserve_punctuse ion, which ensures that punctuation left unchanged. This is important because punctuation meaningfully affect prosody.
The phoneme sequences produced by phonemizer contain some rare symbols (usually in non-English words), which we replace with more frequent symbols. The substitutions we perform are listed in Table This results set of 51 distinct symbols. The character sequence
Modern text-to- text-to-speech synthesis pipelines typically involve multiple processing stages.
becomes
m'addann t'eksttosp'itt 'tt plassssi st'erdziz.
s'mños
p'arplamz
t'iprkli
mnv'all
m,altipol
G TRANSFORMER-BASER ATTENTION ALIGNER BASELINE
In this Appendix we describe transformer-basee attention aligner baseline, used in Section to compare against our monotonic interpolation-base aligner described in Section 2.1 We use transformer attention (Vaswani et al. 2017) with output positional features the queries, and of input positional features and encoder output as the keys. The encoder outputs are from the same dilated convolution stack used in our EATS model, normalised using Layer Normalization (Ba et al. 2016) before input into the transformer. We omit the fully-connected output layer following the attention mechanism. Both sets of positional features ct al. (2017). We use heads with key and value dimensions of 64 per head. Its outputs are taken the audio-aligned feature representations, after which we apply Batch Normalisation and ReLU non- non-linearity before upsampling via the decoder.
the sinusoidal encodings from Vaswani
19

---

Published as a conference paper at ICLR 2021
18
283
252
196
Token Indox
160
112
273
84
271
4-
269
20
2-
267
0
265
10
0.00
0.25
0.50
0.75
1.00
1.25
12.75
13.00
13:25 13.50 Time (6)
13.75
Time (c)
(s)
Figure 6: Positions of the tokens over time for 128 utterances generated from the same text, with different latent vectors z. Close-ups of the start and end of the sequence show the variability of the predicted lengths.
14
12
Count
13.65
13.70
13.75
13.80
Length
Figure 7: Histogram of lengths for 128 utterances generated from the same text, with different latent vectors
H VARIATION IN ALIGNMENT
To demonstrate that the aligner module makes use of the latent vector account for variations in token lengths, we generated 128 different renditions of the second sentence from the abstract: "In this work, we take the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs." Figure 6 shows the positions of the tokens over time, with close-ups of the start and end of the sequence, to make the subtle variations in length more visible. Figure 7 shows histogram of the lengths of the generated utterances. The variation is subtle (less than 2% for this utterance), but noticeable. Given that the training data consists of high-quality recordings of human speech performed by professional voice actors, only modest degree of variation is to be expected.
20

---

Published as a conference paper at ICLR 2021
Model
MOS
FDSD
Natural Speech
4.55 = 0.075
0.682
No Discriminators
1.407 0.040
1.594
No RWDs
2.526 0.060
0.757
No Phonemes
3.423 0.073
0.688
No MelSpecD
3.525 0.057
0.849
No Mon. Int.
3.551 0.073
0.724
No DTW
3.559 + 0.065
0.694
EATS
4.083 + 0.049
0.702
Table 5: Mean Opinion Scores (MOS) and Fréchet DeepSpeech Distances (FDSD) for final EATS model and the ablations deseribed in Section sorted by MOS. FDSD scores presented here computed held-out validation multi-speakkrr set and therefore could not be obtained for the Single Speaker ablation. Due to dataset differences, these are also not comparable with the FDSD values reported for GAN-TTS by Binkkwski et al. (2020).
I EVALUATION WITH FRÉCHET DEEPSPEECH DISTANCE
We found Fréchet DeepSpeech Distances (Binkowski et al. 2020). both conditional and unconditional, unreliable in our setting. Although they provided useful guidance at the early stages of model iteration - i.e. were able to clearly distinguish the models that do and do not train - FDSD scores of the models of reasonable quality were not in line with their Mean Opinion Scores, shown for ablations in T
possible reason for FDSD working less well in our setting is the fact that our models rely on features extracted from spectrograms similar those computed at the DeepSpeech preprocessing stage. As our models combine losses computed on raw audio and mel-spectrograms, it might be the case that the speech generated by some model is of lower quality, yet has convineing spectrograms. Comparison of two of our ablations seems to affirm this hypothesis: the No MelSpecD model achieves much higher MOS (== 3.5) than the No RWDs ablation (#= 2.5) which is optimised only against spectrogram- bbased losses. Their FDSDs, however, suggest the opposite ranking of these models.
Another potential cause for the discrepancy between MOS and FDSD the difference in samples for which these scores were established. While FDSD was computed on samples randomly held out from the training set, the MOS was computed on more challenging, often longer utterances. As we did not have ground truth audio for the latter, we could not compute FDSD for these samples. The sample sizes commonly used for the metrics based on Fréchet distance, e.g. (Heusel et al. 2017 Kurach et al.) 2019 Binkkwski et al. 2020), are also usually larger than the ones used for MOS testing (yan den Oord et al.] 2016 Binkowski et al. 2020); we used 5,120 samples for FDSD and 1,000 for MOS.
We also note that conditional FDSD is not immediately applicable in our setting, as it requires fixed length (two second) samples with aligned conditionings, while our case there is no fixed alignment between the ground truth characters and audio.
We hope that future research will revisit the challenge of automatic quantitative evaluation of text-to- speech models and produce reliable quality metric for models operating in our current regime.
21

---

Published as a conference paper at ICLR 2021
Stages
Stage
Notes
WaveNet van den Oord ct
2016
Ling AR, Au
SampleRNN
Mehri el al. 2017
Au
X
not a TTS model
Deep Voice
Arik et 2017
Ph FF Ling
Au
X
uses segmentation model
WaveRNN
Kalchbrenner al. 2018
Ling AlR Au
LPCNet Valin Skoglund
2019
Cep AR
WaveGlow (Prenger 2019
MciS " Au
FloWaveNer
Kim
2019
MelS IF
Au
WaveFlow PPng al.
20196
MciS AR Au
partially autoregressive
Por. WaveNet (van den
Oord al.
2018
Ling PP" Au
X
distillation
ClariNer Ping et al.
2019a
teacher
Ch/Ph AR, Au
ClariNer Ping ct al.
2019a
student
Ch/Ph FF" Au
distillation
WaveGAN Donahue et 2019
PF Au
not a TTS model
MclGAN Kumar et al.
2019
McIS Au
Par. WaveGAN Yamamoto WaveeooNYYammmotoooo al. 2020
McIS FE Au MeiS FF MagS
AdVoc
(Neekhara
2019
GAN-TTS Binkowski
2020
Ling FF Au
Tacotron
Wang al.
2017
Ch AR, MelS FF MagS
Au
Griffin & Lim 1984
Tacotron
Shen al.
2018
Ch AR MelS AR Au
Deep Voice 2
Gibiansky et al.
2017
Ch Ph
FF Ling AR
Au
X
segmentation model
DV2 Tacotron
Gibiansky al
2017
Ch AR MagS AR
Au
X
Deep Voice. 3
Ping et al.
2018
Ch MelS - Au
X
several alternative vocoders
TransformerT7S
Ci al.
2019
Ch Ph McIS AR Au
AR
X
Flowtron Valle et al. 2020
Ch AR MelS FF Au
X
VoiceLoop
Taigman al.
2017
Ling + Au
X
GAN Exposure
Guo ct al.
McIS Au
X
MelNet
Vasquez
Lewis
2019
MelS >
X
ParaNet
Peng
2019
Ch/Ph
IT, Mels
X
distillation
FastSpeech
Ren et
2019
Ph
FF
McI FF
distillation
Flow-TTS
Miao al
2020
Ch
PF MelS FE Au
X
Glow-TTS
Kim et al.
2020
Ph
FF
MelS FF Au
Char2wav
Sotelo et al.
2017
Ch
AR
Ling Au
AR
X
end-tu-end finetuning
EATS (Ours)
Ch/Ph Au
Table 6: A comparison of TTS methods. The model stages described in each paper are shown by linking together the inputs, outputs and intermediate representations that are used: characters (Ch), phonemes (Ph), mel-spectrograms (MelS), magnitude spectrograms (MagS), cepstral fea- tures (Cep), linguistic features (Ling, such as phoneme durations and fundamental frequencies, or WORLD (Morise et al., 2016) features for Char2wav (Sotelo et al. 2017) and VoiceLoop (Taigman et al. 2017)), and audio (Au). Arrows with various superscripts describe model components: autore- gressive (AR), feed-forwart (FF), feed-forward requiring distillation (FF*). Arrows without superscript indicate components that do not require learning. Stage means the model is trained in single stage to map from unaligned text/phonemes audio (without, e.g., distillation separate vocoder training). EATS is the only feed-forwart model that fulfills this requirement.
22

---

Published as a conference paper at ICLR 2021
J COMPARISON OF TTS METHODS
In Table 6 we compare recent TTS approaches in terms of the inputs and outputs to each stage of the pipeline, and whether they are learnt in a single stage or multiple stages. Differentiating EATS from each prior approach is the fact that it learns audio end-to-end in a single stage, without requiring distillation or separate vocoder training. The ClariNet teacher model (Ping al. (019a) also trained in a single stage, but uses teacher forcing to achieve this, requiring the model to be autoregressive. A separate distillation stage is necessary to obtain a cerwafo model in this case.
feed-forward mapping from text/phonemes to
23


Reviews:
[{"review_id": "u2k3RT7lCkw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a fully end-to-end TTS system with adversarial training. The proposed method has three main advantages: 1) a simple, end-to-end pipeline with only two submodules, with a fully differentiable and efficient architecture; 2) a flexible dynamic time warping to compute the loss between the predicted (generated) and true spectrograms; 3) a good MOS performance compared with the other state-of-the-art systems with more supervision. The evaluation is done very thoroughly with a variety of ablation studies.\n\nOverall, I vote for accepting the paper. First of all, the paper is very well organized and easy to follow. Related works are well summarized, while focusing on the main differences on the proposed method. Method is explained in great detail with easy-to-follow descriptions, proper equations, and very appropriate figures. Solid evaluation is performed, and experimental results and speech samples are convincing.\n\nPros:\n+ The structure of the paper allows an easy read.\n+ The main contributions are clearly stated and supported by the experiments.\n+ Major works on the similar topic are widely covered and referenced.\n+ Evaluation is thorough enough to support the arguments with in-depth ablation studies.\n+ Appendices provide useful, supplementary information.\n\nCons:\n- No comparison over the computational cost nor model size is presented. It is of particular interest because the proposed model is non-autoregressive, and thus may be capable of a causal, real-time inference.\n- No use of widely accepted benchmark datasets. More direct comparison would be of interest.\n\nMinor comments/questions:\n- If I understood correctly, the training sample has no phoneme-level alignment. Instead, only a sentence-speech pair is provided. If so, how do you select the corresponding text snippet from a sentence when you randomly sample 2 seconds of audio from the training examples whose length varies from 1 to 20 seconds?\n- Section 2.5 on DTW is rather lengthy. DTW is a quite well-known algorithm for alignment between the two sequences, the detailed explanation on the algorithm may be omitted without the loss of readability, in my opinion.\n- In Section 2.1, T is used to denote the total number of output times steps of the aligner, while T in Section 2.4 denotes the number of mel-frequency frames. Are these T's identical?\n- The proposed aligner module doesn't seem to be very useful compared with the attention-based aligner as seen in the ablation study (Table 1): very small improvement from 3.551 to 3.559 MOS. Can you provide more explanations?", "extra_scores": {}, "extra_text": {}}, {"review_id": "X8I_4qbVlcI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel TTS system that 1) relies on almost no intermediate representation; and 2) is entirely feedforward instead of autoregressive. There are two major strengths in the paper. First, several modules in the proposed system are novel and smart, including the aligner and the dynamic programming loss, and it is, to my knowledge, the first feedforward text-to-speech system that does not rely on the intermediate representation. Second, the experiment result is convincing. There are, however, some room for improvement and questions for the author to clarify.\n\nFirst, the elegance in the architecture is overshadowed by the complicated training algorithm. The training loss is like the superposition of common loss terms in the speech synthesis community, making the method look a bit heuristic. The complicated training algorithm also makes the proposed method harder to reproduce. It would be helpful if the authors can provide brief guidelines for readers trying to reimplement EATS, such as how to tune the hyperparameters.\n\nSecond, notice that EATS performs slightly worse than GAN-TTS, which does not quite show the benefit of end-to-end training (unlike ClariNet). I understand that there are a lot of challenges in training EATS, and the authors have briefly discussed this in Section 5. However, it is worthwhile to expand the discussion a bit by showing further experiments that demonstrate the potential benefit of end-to-end training.\n\nFinally, although end-to-end comes with the (potential) merits of improved data efficiency and improved quality, it also has its downsides. Without a clearly interpretable hidden representation, it is harder to have direct control over prosody. How would prosody control be possible under the end-to-end framework?\n\nDespite the weaknesses, this paper makes sufficiently novel contributions in TTS, making it above the acceptance threshold. I would look forward to further justifications of the EATS paradigm.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KAiTv-9pk4", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## Summary\n\nThe authors propose EATS, a method for TTS from unaligned audio and text data, directly to the waveform. Previous work either use aligned phonetic features, or output spectrograms that are later converted to a waveform by a deep vocoder model.\n\nIn order to achieve this, the authors had to use several tricks, some already existing, for instance taken from the GAN TTS architecture, and some novel. The three key novelties are:\n- differentiable monotonic attention with gaussian kernel and length prediction.\n- dynamic time warping for the spectrogram loss.\n- using both spectrogram and waveform domain discriminators\n\nThe authors provide a comprehensive ablation study with MOS score, although their model is under the state of the art by a significant margin.\n\n## Review\n\nThis paper builds on GAN TTS, and tries to make it trainable end-to-end without aligned features.\n\nThe two main contributions, namely dynamic time warping and monotonic attention with gaussian kernel are both elegant, and can likely be used for many other applications related to time series with heterogeneous time scales. In particular, the time warping loss allows to accomodate both for the natural irregularities in spoken speech, as well as providing sufficient signal for the monotonic attention to work.\n\nThe rest of the architecture is very similar to GAN TTS except for the spectrogram domain discriminator that was added.\n\nWhile the model is under the state of the art for TTS, the samples are already quite convincing. The authors conduct a thorough ablation study, both with MOS and audio samples.\n\nOverall I think this is a really good paper, that is likely to prove quite useful for the development of end to end speech synthesis solution. As I already mentioned, I also believe that the approach of using dynamic time warping and monotonic attention can be used for other kind of time series.\n\n\n## Remarks and questions to the authors\n\n- Table 1, MOS for Tacotron 2 would be very informative. All the baselines are trained on aligned data while Tacotron is a legitimate contender for EATS as it can be trained on the same data. The point of the authors is that their methods is simpler because the training is in one stage. However, given the large number of losses and components in their model, with their respective hyper-parameters to tune, I'm not entirely sold on the simplicity argument. The tacotron 2 paper reports a MOS of 4.5 but on a private dataset.\n- Section 3, [1] used the same simple L1 + log spectrogram loss as used here.\n- I was surprised by the bad performance of the transformer attention, in particular in the audio samples, the output for this model is garbage towards the end of the signal. Any clue on why this would happen?\n- It would be interesting to have a benchmark, in particular, can the model generate speech in real time on GPU and on CPU?\n\n[1] SING: Symbol-to-Instrument Neural Generator, Defossez et al. Neurips 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8i0Rsbolbje", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach for end-to-end speech synthesis, where every step is learned jointly with the others. Specifically,  the proposed model takes a character sequence as input and outputs an audio signal directly. The model is trained using an combination of losses, including an adversarial loss. In the experimental section, the proposed approach is compared against strong baselines, and ablation studies are presented.\n\nPros:\n- The proposed approach is novel and a significant step toward competitive end-to-end models.\n- The related work is thorough as far as I can tell.\n- The experiments are insightful, showing the impact of each part of the system.\n\nCons:\n- The performance are promising, but still below the baselines.\n- The end-to-end claim is a bit misleading as the character-based model is not performing well, and the phoneme-based model is not really end-to-end, as the g2p part is not trained jointly.\n- The paper is sometime not easy to follow.\n\nDetailed comments:\n- The reason behind using an adversarial loss is not really explained in the paper. A few lines before section 2.1 would help clarify that.\n- The order of the sub-sections (2.1-2.7) in Section 2 is not intuitive and seems a bit random, making the section slightly hard to follow.  \n- Section 2.7: \"inconsistent spelling rules of the English language\" -> It's not about inconsistent rules: every language has inconsistent rules, several dialects and variations in the pronunciation of words. Please rephrase.\n- It's not clear which dataset was used in the experiment. If it is a private dataset, please state it clearly.\n\nOverall, despite the paper's two main weaknesses (not fully end-to-end and lower performance),  I think it is a significant step towards fully end-to-end model and should be accepted. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates a speech synthesis approach that directly generates raw audios from text or phoneme inputs in an end-to-end fashion.  The approach first maps the input texts/phonemes into a representation sequence that is aligned with the output at a lower sampling frequency by a differentiable aligner and then upsamples the representation sequence to the full audio frequency by a decoder.  A number of techniques including adversarial training and soft DTW are applied to improve the training.  The experimental results are good. There are raised concerns from the reviewers which are mostly cleared by the rebuttal of the authors.  After the rebuttal and discussion, all reviewers are supportive on accepting the paper. 
Decision: Accept (Oral)