Non-Attentive Tacotron: Robust and controllable neural TTS synthesis including unsupervised duration modeling
Authors: Jonathan Shen
Ye Jia
Mike Chrzanowski
Yu Zhang
Isaac Elias
Heiga Zen
Yonghui Wu
Venue:  (2021)
Keywords: tts
text-to-speech

Abstract: This paper presents Non-Attentive Tacotron based on the Tacotron 2 text-to-speech model, replacing the attention mechanism with an explicit duration predictor. This improves robustness significantly as measured by unaligned duration ratio and word deletion rate, two metrics introduced in this paper for large-scale robustness evaluation using a pre-trained speech recognition model. With the use of Gaussian upsampling, Non-Attentive Tacotron achieves a 5-scale mean opinion score for naturalness of 4.41, slightly outperforming Tacotron 2. The duration predictor enables both utterance-wide and per-phoneme control of duration at inference time. When accurate target durations are scarce or unavailable in the training data, we propose a method using a fine-grained variational auto-encoder to train the duration predictor in a semi-supervised or unsupervised manner, with results almost as good as supervised training.

Under review as a conference paper at ICLR 2021
NON-ARTENTIVE TACOTRON: ROBUST AND CONTROLLABLE NEURAL TTS SYNTHESIS
INCLUDING UNSUPERVISED DURATION MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper presents Non-Artentive Tacotron based on the Tacotron 2 text-to-speech model, replacing the attention mechanism with an explicit duration predictor. This improves robustness significantly as measured by unaligned duration ratio and word deletion rate, two metrics introduced in this paper for large-scale robustness evaluation using a pre-trained speech recognition model. With the use of Gaussian upsampling, Non-Attentive Tacotron achieves 5-scale mean opinion score for naturalness of 4.41, slightly outperforming Tacotron 2. The duration predictor enables both utterance-wide and per-phoneme control of duration inference time. When accurate target durations are scarce or unavailable in the training data, we propose method using fine-grained variational auto-encoder train the duration predictor in a semi semi-supervised or unsupervised manner, with results almost as good as supervised training.
INTRODUCTION
Autoregressive neural text-to- toeee-h (TTS) models using an attention mechanism are known be able to generate speech with naturalness on par with recorded human speech. However, these types of models are known to less robust than traditional approaches (He et al., 2019) Zheng et al. 2019)] Guo et al.) 2019: Battenberg et al. (2020). These autoregressive networks that predict the output one frame at a time are trained to decide whether to stop at each frame, and thus a misprediction on a single frame result in serious failures such early cut-off. Meanwhile, there are little to hard constraints imposed on the attention mechanism to prevent problems such as repetition, skipping, long pause babbling. To exacerbate the issue, these failures rare and are thus often not represented in small test sets, such as those used in subjective listening tests. However, in customer-facing products, even a one-in-a-milllee -iil-nnmmllonn chance of such problems can severely degrade the user experience.
There have been various works aimed at improving the robustness of autoregressive attention- neural TTS models. Some of them investigated reducing the effect of the exposure bias on the autoregressive decoder, using adversarial training (Guo et al., 2019) or adding regularization to encourage the forward and backward attention to be consistent (Zheng et al. 2019). Others utilized or designed alternative attention mechanisms, such as Gaussian mixture model (GMM) attention (Graves) 2013 Skerry-Ryaa et al., 2018), forward attention (Zhang al. 2018), stepwise monotonic attention (He et al. 2019), or dynamic convolution attention (Battenberg et al. 2020). Nonetheless, these approaches do not fundamentally solve the robustness issue.
Recently, there has been surge in the use of non-autoregressive models for TTS. Rather than predicting whether to stop on each frame, non non-autoregressive models need to determine the output length ahead of time, and input token. A side benefit of such a duration predictor is that it is significantly more resilient to the failures afflicting the attention mechanism.
way to do is with explicit prediction of the duration for each
However, one-to-mann regression problems like TTS can benefit from an autoregressive decoder as the previous mel-spectrogram frames provides context disambiguate between multi-modal outputs.

---

Under review as a conference paper at ICLR 2021
In this paper, propose, Non-Attentive Tacotron neural TTS model that combines the robust duration predictor with the autoregressive decoder of Tacotron (Shen et al. 2018
Our work is similar to DurIAN (Yu al. 2019 Zhang et al. 2020 which incorporates the duration predictor with an autoregressive decoder. But besides the differences in architecture, we also introduce couple of novel features in our model.
The key contributions of this paper are summarized as follows:
1. Replacing the attention mechanism in Tacotron with duration prediction and upsampling modules leading to better robustness with the naturalness matching recorded natural speech;
2. Introduction of Gaussian upsampling significantly improving the naturalness compared to vanilla upsampling through repetition;
3. Global and fine-grained controlling of durations at inference time;
4. Semi-supervised and unsupervised duration modeling of Non-Attentive Tacotron, allowing the model to be trained with few to duration annotations; and
5. More reliable evaluation metrics for measuring robustness of TTS models, well as comparing Non-Attentive Tacotron with Tacotron with respect to those metrics.
RELATED WORKS
In the past decade, model-based TTS synthesis has evolved from hidden Markov model (HMM)-based approaches (Zen et al. 2009) to using deep neural networks. Over this period, the concept of using an explicit representation of token (phoneme) durations has not been foreign. Early neural parametric synthesis models (Zen et al. 2013) require explicit alignments between input and target and include durations as part of the bag of features used to generate vocoder parameters. Explicit durations continue to be used with the advent of the end-to-end neural vocoder WaveNet (Oord et al. 2016) in
works such Deep Voice (Arik et al. 2017 Gibiansky al. 2017) and CHiVE (Kenter
2019)
As general focus turned towards end-to-end approaches, the autoregressive sequence-to-seqquene model with attention mechanism used in neural machine translation (NMT) (Bahdanau al. 2015)
and automatic speech recognition (ASR) (Chan et al. (2016) became an attractive option, removing the need represent durations explicitly. This led to works such as Char2Wav (Sotelo et al. 2017).
Tacotron (Wang et al., 2017, [Shen et al. (018). Deep Voice 3 (Ping et al. 2018). and Transformer TTS (Li et al. 2019). Similar models have been used for more complicated problems, like direct speech-to-speech translation (Jia et al. 2019), speech conversion (Biadsy et al. 2019), and speech enhancement (Ding et al. 2020).
Tacotron 2, on which our work is based, is one such model. It connects character-level encoder and an autoregressive decoder producing mel spectrogram frames with the use of a location-sensilee attention mechanism (Chorowski et al. 2015).
Recently, there has been a surge of non-autoregresssvve models, bringing back the use of explicit duration prediction. This approach initially surfaced in NMT (Gu et al. 2017). then made its way into
TTS with models such as FastSpeech (Ren et al.) 2019 2020). AlignTTss (Zeng et al. 2020). TalkNet (Beliaev al., 2020), and JDI-T (Lim al. 2020 See Appendix C for rough categorization of these models.
To train the duration predictor, FastSpeech uses target durations extracted from pre-trained autore- gressive model in teacher forcing mode, while JDI-T also extracts target durations from a separate autoregressive model but co-trains with the feed-forward model. TalkNet uses a CTC-based ASR model to extract target durations, while CHiVE, FastSpeech 2, and DurIAN use target durations from an external aligner module utilizing forced alignment. Finally, Align forgoes target durations completely and uses specialized alignment loss inspired by the Baum- Welch algorithm to train mixture density network for alignment.
Audio samples available in supplemental materials.
2

---

Under review as a conference paper at ICLR 2021
Nradicted Feotures y
Segment encoder
Duration predictor
Predicted Festures
Cone Poss Not
Encuder
Decoder
Inputs
Outputs
Specerogram Frame
Training only
Inference only
Target Features
Projectionn
Projection
Add
Concat
Positional Eriieddiing
Tocal
Duration
Gausstan Uneempping f
Duration availseie)
Predictied Duration
Prior Latenta
Turget Durationa
Predkcted Duration
Daration Predictor
Pasterior
Latenta A
Projection
Projection
Conditnnaal VAF A
Conteonte
2 BILSTM
2B:LSTM
Speaker ids
Speaker Embedding
Aligned
Speaker ids
Speaker Emoeddinn
Phoneme ids
Encoder
Attention
Phonemo X
Token Embedding
Corv
Target Features y
Coni
2 LISTMM
(a) Full model.
(b) Semi-supervisedd supervised/Ussspprviied duration modelling with FVAE.
Figure 1: Architecture of Non-Attentive Tacotron.
3 MODEL
Modern neural TTS models typically consist of two separate networks: (1) a feature generation network that transforms input tokens (e.g., grapheme phoneme ids) into acoustic features (e.g., mel-spectrogram), and (2) a vocoder network that transforms the acoustic features into a time-domain audio waveform. This paper focuses on the feature generation network, and can be used with any vocoder network, e.g., WaveNet (Oord et al., 20166). WaveRNN (Kalchbrenner et al. 2018). WaveGlow (Prenger al., 2019), MelGAN (Kumar et 2019). or WaveGrad (Chen et al. 2020). The architecture of Non-Attentiv Tacotron is illustrated in Figure la See See|Appendix A for specific
parameter value settings.
The model follows that of Tacotron 2 (Shen et al. 2018 transforming input ids X = (æ11,... xN) of length N into mel-spectrogram predictions as inputs, and include a silence token at word boundaries as well as an end-of-sequence token. The ids are used to index into a learned embedding and is then passed through an encoder consisting of X (dropout, batch normalization. convolution) layers followed by single bi-directional LSTM with ZoneOut to generate a 2-dimensional output of length N. This output is concatenated with a speaker embedding vector to produce the final encoder output H (h1....,h., h.).
yy...... yr) of size K. Phonemes are used
The autoregressive decoder also follows Tacotron 2, and predicts mel-spectrograms one frame at time. At training time, teacher forcing (Williams & Zipser 1989) is employed and the previous groundtruth mel-spectrogram frame is used as input, while at inference time the previous predicted mel-spectrogram frame used. This previous frame is passed through pre-net containing two fully-connected layers of ReLU units with dropout, then concatenated with an upsampled (aligned) encoder output corresponding to the current frame. The upsampled encoder outputs for future frames are not visible to the decoder at the current frame. In Tacotron 2. this upsampling or alignment attention mechanism (Chorowski et al.) 2015), while this
achieved using a location-sensittee work the attention mechanism is not used and separate upsampling mechanism described later is used in its stead. The result is then passed through two uni-directional LSTT layers with ZoneOut. The LSTM output is concatenated with the upsampled encoder output yet again then projected to the mel-spectrogram dimension as frames of a preliminary predicted spectrogram Y'. Once all the mel-spectrogram frames have been predicted, they are passed through a 5-layer batch normalized convolutional post-net with tanh activation on all except the last layer. The post-net predicts a residual to add to the prediction Y' to obtain the final prediction Y.
In place of the attention mechanism used in Tacotron 2, duration- based models upsample the encoder outputs using per-token duration information. This can be done by simply repeating each encoder
3

---

Under review as a conference paper at ICLR 2021
output by its duration in FastSpeech (Ren et al. 2019), but instead we adopt different process call Gaussian upsampling, which is described in subsection 3.1, Note that while durations in seconds used for loss computation, they are converted durations integer frames for upsampling.
For Gaussian upsampling, a duration and a range parameter must be predicted for cach token. The range parameter is called thus because it controls the range of token's influence. The duration predictor passes the encoder output through two bi-directional LSTM layers followed by a projection layer to predict the numeric duration d (di ........ dN) ) for each input token. During training, these predicted durations are only used for loss computation, and the target durations are used instead in the upcoming steps The range parameter predictor passes the encoder output concatenated with durations through two bi-directional LSTM layers followed by projection layer and SoftPlus activation to predict positive range parameter for each input token.
After the encoder outputs are upsampled, Transformer-style sinusoidal positional embedding (Vaswani et al. 2017) is concatenated. The positional embedding tracks the index of each upsampled frame within each token; if the duration values are [2, 1, 3), the indices for the positional embedding would be [1, 2, 1, 1,33
The model is trained using combination of duration prediction loss and mel-spectrogram recon- struction loss. The duration prediction loss is the T.2 loss between predicted and target durations in seconds, and the mel-spectrogran reconstruction loss is L + L loss between the predicted and the groundtruth mel-spectrogram both before and after the post-net (following Jia et al. (2018).
L = Cspce + Adur
Laur
ld d' 112
(2)
T
Lupoo TK
(119 - vill, + 1111 v2212 + 11 97ll + +113 2ī112)
(3)
3.1 GAUSSIAN UPSAMPLING
Given sequence of vectors to be upsampled H (h1 hN), integer duration values d (di,.... dx), and range parameter values o (0 AN), we compute the upsampled vector sequence U (u1:...; uT)
i-1 G: = + Σ dj, 2 j=1
di
N (f)) Whi EN (t:c,,o)))
N Ut = Σ wah,
That is, we place a Gaussian distribution with standard deviation o; the center of the output segment corresponding to the i-th input token as determined by the duration values d, and for each frame we take a weighted sum of the encoder outputs in accordance with the values of Gaussian distributions at that frame. This is similar to the softmax based aligner in Donahue et al. (2020), except learned or rather than a fixed temperature hyperparameter is used here.
Compared with vanilla upsampling by repetition (as in Ren et al. (019
), which can be seen case of learning hard monotonic attention, Gaussian upsampling results an alignment that more akin to single-componett GMM attention. Another benefit of Gaussian upsampling is that it is fully differentiable, which critical to semi-supervised and unsupervised duration modeling (section 4 as it allows the gradients from the spectrogram losses to flow through to the duration predictor.
3.2 TARGET DURATIONS
Neural TTS models using duration need alignments between input tokens and output features. This can be accomplished by implementing an aligner module in the model or by using external aligner.
Note that target durations may not be required with semi-supervised and unsupervised duration modeling.
See section 4
4

---

Under review as a conference paper at ICLR 2021
In our work, target durations extracted by external, flatstart trained, speaker-dependent HMM- based aligner with a lexicon (Takin & Wightman 1994). However, sometimes it is difficult to train reliable aligner model and/or extract accurate alignments due to data sparsity, poor recording conditions, or unclear pronunciations. To address this problem, we introduce semi-supervised and unsupervised duration modeling.
SEMI-SUPERVISED AND UNSUPERVISHD DURATION MODELING
A naïve approach to unsupervised duration modeling would be to simply train the model using the predicted durations (instead of the target durations) for upsampling, and use only mel-spectrogram reconstruction loss for optimization. To match the length between the predicted durations and the target mel-spectrogram frames, the predicted per-token durations can be scaled by T/Σ, d,. In addition to that, utterance-level duration loss Lu N (T -;--;eee d4) could be added to the total loss. However, experiments show that such an approach does not produce satisfying naturalness in the synthesized speech (subsection 6.3)
The proposed unsupervised duration modeling is illustrated in Figure lb. We instead utilize a fine-grained VAE (FVAE) similar to (Snn et al. (2020) to model the alignment between the input tokens and the target mel-spectrogram frames, and extract per-token latent features from the target mel-spectrogram based on this alignment. The token encoder output H is aligned the target spectrogram Y using an attention mechanism following Lee & Kim (2019):
Aht fapea (Y"))
where h; is used as the query in the attention, and fsp is a spectrogram encoder whose output per frame is used the values in the attention. A simple dot-product attention from Luong al. (2015) was used in this work. A latent feature z; is then computed from e, and h; using variational auto-encoder (VAE) (Kingma & Welling. 2014) with Gaussian prior A( I), optimized through the evidence lower bound (ELBO):
log (Y H) ΣDEI DKI. (z; ha,c)) (z:)) + (2 (,, [logp (Y H, Z)]
(4)
where the first term is the KI. divergence between the prior and posterior, and the second term can be approximated by drawing samples from the posterior.
Because these latent features are extracted from the target spectrogram with an alignment, they capable of carrying duration related information. At training time, the per-token duration d is predicted from the concatenation of the token encoder output H and the posterior latent Z; while at inference time, the prior latent is used for Z (either sampled from the distribution or using the distribution mode), and the internal attention mechanism of the FVAE is not used.
not utilized for factorizing latent dimensions. These latent features are only used for duration prediction, and are not used for range parameter prediction mel-spectrogram reconstruction. We also cap the range parameters for each token to twice its predicted duration in this setup for better training stability. As in Section 3 durations predicted seconds are used for loss computation, but are converted durations integer frames for upsampling.
Unlike Sun et al. 2020), scheduled sampling
The overall loss used for semi-supervised and unsupervised training thus
C = Sype Caur + Aa + ÁKL
(5)
where DKL and Lypec correspond to the first and second terms in Equation 4 respectively, and Laur is only counted for examples with target duration labels (i.e. supervised examples). The last three terms all weighted per valid token.
ROBUSTNESS EVALUATION
Previous work typically evaluated the robustness of TTS systems small set of handpicked "hard cases" (He et al. 2019 Zheng et al., 2019 Guo et al. 2019). Although such evaluation is helpful for guiding improvements, is not reflective of the overall robustness of the system. The handpicked
5

---

Under review as a conference paper at ICLR 2021
samples may be biased to the weaknesses of a certain system, and is prone to lead further optimization to overfit to the specific evaluation set.
In this work, we evaluate the robustness of TTS systems large evaluation sets in automated way by leveraging existing ASR systems. We run ASR and forced alignment evaluations on the synthesized speech against the verbalized text, and report two metrics measuring and under-generation:
1. Unaligned duration ratio (UDR): The synthesized speech is forced aligned with the verbalized input text using an ASR system. Each token the input text aligned segment in the synthesized audio. Any long audio segments (> second) not aligned to any input token are typically due to over-generation from the TTS system, such as long pauses, babbling, word repetitions, or failures stop after finishing the utterance. The total duration such long unaligned segments divided by the total output duration the UDR. Note that short unaligned segments are ignored. If the synthesized speech unable to be aligned with the input text, is considered as having a UDR of 100%.
2. ASR word deletion rate (WDR): This is the deletion emoor portion a standard ASR word error rate (WER) evaluation. Uinder-generatinn in the synthesized speech, such early cutoff and word skipping, is reflected by higher WDR.
As the ASR system will make mistakes, the metrics above are just an upper-bound on the actual failures of the TTS system.
6 EXPERIMENTS
All models were trained on proprictary dataset with 66 speakers with 4 different English accents (US, British, Australian, and Nigerian). The amount of data per speaker varied from merely 5 seconds to 47 hours, totaling 354 hours.
preliminary experiment comparing different attention mechanisms (including monotonic, stepwise monotonic, dynamic convolution and GMM attention (GMMA)) showed that GMMA performed the best. We therefore compared our non-attentive Tacotron not only with Tacotron with location- sensitive attention (LSA) which was used in the original Tacotron 2 paper but also with Tacotron 2 with GMMA. The Tacotron models used reduction factor 2 and L I loss.
Following Shen et al. (2018), predicted features were obtained in teacher-forcing mode from Tacotron model and used to train WaveRNN vocoder which was then used for all experiments.
NATURALNESS
The naturalness of the synthesized speech was evaluated through subjective listening tests, including 5-scale Mean Opinion Score (MOS) tests and side-by-side preference tests. The sentences were synthesized using 10 US English speakers (5 male / 5 female) in round-robin fashion. The amount of training data for the evaluated speakers varied from hours to 47 hours.
cabte/ss MOS results. Non-Attentive Tacotron with Gaussian upsampling matched Tacotron (GMMA) in naturalness, and both were close to the groundtruth audio. A preference test between
Table 1: MOS with 95% confidence intervals.
600
Model
MOS
400
Tacotron
200
1:37
158
w/ LSA
4.35 + 0.05 4.37 + 0.04
40
w/ GMMA
Much Woose
Slightly Woese the Sume Better
Slightly
Non-Attentive Tacotron
Better
w/ Gauss. upsampling
4.41 + 0.04
w/ vanilla upsampling
4.13 0.05
Figure 2: Preference test result with Non-Attentive Tacotron with Gaussian upsampling compared against Tacotron (GMMA).
Ground truth
4.42 + 0.04

---

Under review as a conference paper at ICLR 2021
Non-Attentive Tacotron and Tacotron 2 (GMMA) further confirmed this result, as shown in Figure Non-Attentive Tacotron with vanilla (repeating) upsampling was rated as significantly less natural than with Gaussian upsampling.
The effectiveness learned range parameter versus a fixed temperature hyperparameter set at 10.0 per Donahue al. (2020) is compared using preference test in Table 2 While there is only a slight perceived benefit in using a learned range parameter, reduces the need to tune another dataset-dependent hyperparameter. Additionally, in multi-speaker setups it is possible that the optimal may be speaker-dependent.
Table 2: Preference test between a learned a versus a fixed a set at 10.0. Pace is defined as in subsection 6.2 A negative preference value means that the learned a is preferred over the fixed σ.
Pace
0.8x
1.0x
1.25x
Preference
0.017 #: 0.057 -0.055 + 0.054 -0.01 + 0.055
6.2
PACE CONTROL
Table shows WER and MOS results after modifying the utterance-wide pace by dividing the predicted durations by various factors. The WER computed on speech synthesized on transcripts from the LibriTTS test-clean subset with the same 10 speakers in subsection 6.1 and then transcribed by ASR model de- scribed in Park et al. (2020) with a WER of 2.3% the ground truth audio.
With pace between 0.8x - 1.25x, the WERs hardly impacted. The WER signif- icantly worse when the pace was increased 1.5x normal, partially because the ASR model used was not optimized for speech so fast. In contrast, the subjective MOS de- creased rapidly when the pace was sped up slowed down significantly. However, most the comments from raters plaining about the pace, such as "too slow to be natural" (0.8x) "way too fast" (1.25x).
simply com-
Non- Atn-Attentive Tacotron also able to control the pace of the synthesized speech at finer granularity, such as per-word or per-pha per-phoneme, while still maintain the naturalness of the syn- thesized speech. Figure 3 shows examples of controlling the pace for specific words in a sentence.
Single word pace control with sentence "T m saddened about the devastation in Big Basin." The top spectrogram is with regular pace. The rest slow down the words "saddened", "devastation" and "Big Basin" respectively to 0.67 x the regular pace by scaling the predicted duration by 1.5x.
Figure 3:
Table 3: Performance of controlling the utterance-wide pace of the synthesized speech.
Pace
0.67
0.8
0.9x
1.0x
1.11x
1.25
1.5x
WER
3.3%
2.8%
2.6%
2.6%
2.5%
2.7%
6.1%
MOS
3.28 + 0.06
3.87 + 0.05
4.24 + 0.04
4.41 + 0.04
4.28
0.04
3.79 + 0.06
3.18 + 0.06
6.3
SEMI-SUPERVISED AND UNSUPERVISED DURATION MODELING
Ten different US English speakers (5 male/ / female) each with about 4 hours of training data were used for evaluating the performance of the unsupervised and semi-supervised duration modeling.
7

---

Under review as a conference paper at ICLR 2021
Prediced (soomaad
Prediced loanated
PVAS
Target
280
64
682
Figure 4: Alignment on text "What time do I need show up my sky diving lesson?" from the unsupervised model. The predicted alignments are from Gaussian upsampling.
Table 4: Performance of unsupervised and semi-supervised duration modeling. Zero vectors used as FVAE latents for inference. MAE denotes the mean absolute error.
Training
Model
Dur. MAE (ms)
MOS
Unsupervised
w/o FVAE
124.4
2.91 0.09
w/ FVAE
41.3
4.31 0.04
Semi-supervised
w/o FVAE
21.5
4.19 0.05
w/ PVAE
18.3
4.35 0.04
Supervised
Non-Attentive Tacotron
15.4
4.37 0.04
Tacotron GMMA
4.35 0.04
Ground truth
4.52 0.03
The duration labels for these 10 speakers (i.e. about 11% of the training data) were withheld for semi-supervised models, and all duration labels were withheld for the unsupervised models.
Figure s]shows predicted alignment after Gaussian upsampling and the internal alignment from the attention module in the FVAE compared with the alignment computed from the target durations, for the unsupervised model. Despite not having access to any target durations, both the FVAE and duration predictor were able to produce alignment close that computed from the target durations.
As shown in Table 4 with the use of the FVAE, the naturalness both semi-supervised and unsupervised models emoos were higher. The autoregressive decoder trained with teacher forcing may have been powerful enough to correct the duration prediction errors some degree. However, the naturalness degraded significantly without the use of the FVAE. Although the duration eтrоr from the semi-supervised model without FVAE was lower than that from the unsupervised model with FVAE, the former was significantly less natural than the latter. This may be due to a lower consistency between supervised and unsupervised speakers without FVAE.
very close to that of the supervised models, though duration prediction
Although these models were close to the supervised model in MOS, manual investigation found that samples from both semi-supervised models and unsupervised models had small chance of containing slight errors that do not occur in the supervised model, such as unclear pronunciations, phoneme repetitions, or extra pauses. However, they are significantly less severe than similar errors from Tacotron 2, mostly impacting just one few phonemes. These errors are further confirmed in the large scale robustness evaluation (subsection 6.4).
The utterance-wide or fine-grained and unsupervised models as well. However, as the alignments are not as accurate, the synthesized speech with fine-grained pace control not natural from the supervised model. The duration may be extended by simply inserting more silence, and the extended portion may include phoneme repetitions unclear pronunciations.
control (susecction 6.2 can be applied the semi-supervised
8

---

Under review as a conference paper at ICLR 2021
Table 5: Robustness measured by UDR and WDR two large evaluation sets. The evaluation
speakers are unsupervised ones in the semi-supervised and unsupervised models.
LibriTTS
web-long
System
UDR (%)
WDR (%)
UDR (%)
WDR (%)
Tacotron 2
w/ LSA
16.96
0.4
46.04
4,4
w/ GMMA
3.812
0.1
6.157
1.3
Non-Auentive Tacotron
Supervised
0.005
0.1
0.011
1.1
Semi-supervised
0.266
0.9
0.695
3.5
Unsupervised
0.223
0.7
0,527
3.2
6.4 ROBUSTNESS
We evaluated the robustness of the neural TTS models by measuring UDR and WDR on two large evaluation sets: LibriTTS: 354K sentences from all train subsets from the LibriTTS corpus (Zen et al. 2019}; and web-long: 100K long sentences mined from the web, which included a small amount of irregular text such programming code. The median text lengths of the two were 74 and 224 characters, respectively. The input was synthesized using the same 10 speakers in |subsectionn 6.3 in round-robin fashion. All model outputs were capped at 120 seconds.
We used the ASR model trained on the LibriSpeech (Panayotov et al. 2015 and LibriLight (Kahn) al. 2020) corpora from Park al. (2020) for measuring WDR, and a confidence islands-based forced alignment model (Chiu et al.) 2018) for measuring UDR.
Table the robustness metries for Tacotron 2 and Non-Attentive Tacotron. Tacotron (LSA) suffered from severe ver-genneattion as measured by UDR, especially on long inputs. Manual investigation uncovered that they were typically long babbling or long silence, often the end (failure to stop). It also had high level of under- sedee-ggneration as measured by WDR, typically due to early cutoff. Tacotron 2 (GMMA) performed almost as well as the supervised Non-Attentive Tacotron in WDR because of its soft monotonic nature, which made end-of-sentence prediction easier. However, still had significantly higher level of over-generation compared to Non-Attentive Tacotron, even when unsupervised or semi-supervised duration modeling is used for the latter. The robustness of semi- sumi-supervised and unsupervised Non-Attentive Tacotron is significantly worse than the supervised one. Manual investigation uncovered that the typical failure pattern is that part of the spectrogram not correctly synthesized (often as silence, but sometimes as babbling), despite that the duration prediction seems reasonable. Such failure pattern contributes to both UDR and WDR. This indicates further improvements to be made. Even then, the semi- sumi--upereised and unsupervised Non-Attentn Attentive Tacotron still performs significantly better on weergennration compared Tacotron
In practice, we also observed that Tacotron required significantly more care in data preprocessing achieve this level of robustness, including consistent trimming of leading and trailing silences and filtering out utterances with long pauses. On the other hand, Non-Attentive Tacotron is significantly less sensitive the data preprocessing steps.
CONCLUSIONS
This paper presented Non-Attentvee Tacotron, showing significant improvement in robustness compared to Tacotron 2 as measured by unaligned duration ratio and word deletion rate, while also slightly outperforming it in naturalness. This was achieved by replacing the attention mechanism in Tacotron 2 with an explicit duration predictor and Gaussian upsampling. We demonstrated the ability to control the pacing of the entire utterance as well as individual words using the duration predictor. We also described method of modeling duration in semi-supervised or unsupervised manner using Non-Attentive Tacotron when accurate target duration are scarce or unavailable by using fine-grained variational auto-encoder, with results almost as good supervised training.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Sercan O Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky. Yongguo Kang, Xian Li, John Miller, Andrew Ng. Jonathan Raiman, Shubho Sengupta, and Mohammad Shoeybi. Deep Voice: Real-Time Neural Text-to-Speech. In Proc. ICML, pp. 195-204 2017.
Damitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In Proc. ICLR, 2015.
Eric Battenberg, RJ Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby. Docation-Relatie. Attention Mechanisms For Robust Long-Form Speech Synthesis. In Proc. ICASSP, 2020.
Stanislav Beliaev, Yurii Rebryk, and Boris Ginsburg. TalkNet: Fully-Convolutionn Non- Autoregressive Speech Synthesis Model. arXiv preprint arXiv:2005.05514 2020.
Fadi Biadsy, Ron Weiss, Pedro J. Moreno, Dimitri Kanvesky, and Ye. Jia. Parrotron: An End-to-Ein Speechh Conversion Model and its Applications Immpired Speech and Speech Separation. In Proc. Interspeech, pp- 4115 4115-4119, 2019.
W. Chan, N. Jaitly, Q. Le, and O. Vinyals. Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition. In Proc. ICASSP, pp. 4960-4964, 2016.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. WaveG- rad: Estimating Gradients for Waveform Generation. arXiv preprint (tt::200900071133 2020.
Chung-Cheng Chiu, Anshuman Tripathi, Katherine Chou, Chris Co. Navdeep Jaitly, Diana Jaun- zeikare, Anjuli Kannan, Patrick Nguyen, Hasim Sak, Ananth Sankar, Justin Tansuwan, Nathan Wan, Yonghui Wu, and Xuedong Zhang. Speech Recognition for Medical Conversations. In Proc. Interspeech, pp. 2972-2976, 2018.
Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-basee models for speech recognition. In Proc. NeurlPS, 2015.
Shaojin Ding, Ye Jia, Ke Hu, and Quan Wang. Textual Echo Cancellation. arXiv preprint arXiv:2008. 06006, 2020.
Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan. End-to-End Adversarial Text-to-Speech. arXiv preprint arXiv:2006.03575 2020.
Andrew Gibiansky, Serean Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. Deep voice 2: Multi-speaker neural text-to-speech. In Advances in neural information processing systems, pp. 2962-2970, 2017.
Alex Graves. Generating Sequences with Recurrent Neural Networks.
arXiv preprint
1rXiv:1308.0850 2013.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregreesiive neural machine translation. arXiv preprint arXiv: /71 .00221,, 22281,2011 2017.
Haohan Guo, Frank K Soong, Lei He, and Lei Xie. A New GAN-based End-to-End TTS Training Algorithm. In Proc. Interspeech, pp. 1288 1292. 2019.
Mutian He, Yan Deng, and Lei He. Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS. In Proc. Interspeech, pp. 1293-1297, 2019.
Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez-Moreno, and Yonghui Wu. Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis. In Proc. NewrIPS, 2018.
Ye Jia, Ron Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Translation with a Sequence-to te-eueee------ueneeee Model. In Proc. Inter-
Wu. Direct speech, pp. 1 1127, 2019.
10

---

Under review as a conference paper at ICLR 2021
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhoma- nenko, Gabriel Synnaeve, Armand Joulin, Abdelrahman Mohamed, and Emmanuel Dupoux. Libri- light: A Benchmark for ASR with Limited or No Supervision. In Proc. ICASSP, pp. 7669-7673, 2020.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient Neural Audio Synthesis. In Proc. ICMI., pp. 2410-2419, 2018.
Tom Kenter, Vincent Wan, Chun-An Chan, Rob Clark, and Jakub Vit. CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network. In Proc. ICML, pp. 5331-3340, 2019.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes, In Proc. ICLR, 2014.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, and Aaron Courville. MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis. In Proc. NeurIPS, 2019.
Younggun Lee and Taesu Kim. Robust and fine-gran ane-grained prosody control of end-to-end speech synthesis. In Proc. ICASSP, pp. 5911-5915, 2019.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural Speech Synthesis with Transformer Network. In Proc. AAAI, volume 33, pp. 6706-6713, 2019.
Dan Lim, Won Jang, Hyeyeong Park, Bongwan Kim, and Jesam Yoon. JDI-T: Jointly Trained
Duration Informed Transformer for Text-To-Speech without Explicit Alignment. arXiv preprini arXiv:2005.07799 2020.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv: 158.005000000055,, 2015.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A Generative Model for Raw Audio. arXiv preprint 100v:6609.333000 2016.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR Corpus Based on Public Domain Audio Books. In Proc. ICASSP, pp. 5206-5210, 2015.
Daniel Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoe V Le. Improved Noisy Student Training for Automatic Speech Recognition. In Proc. Interspeech, 2020.
to appear.
Wei Ping, Kainan Peng, Andrew Gibiansky, Serean O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep Voice 3: Scaling text-to-speech with convolutional sequence learning. Proc. ICLR, 2018
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. WaveGlow: A Flow-based Generative Network for Speech Synthesis. In Proc. ICASSP, 2019.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. FastSpeech: Fast, Robust and Controllable Text Speech. In Proc. NeurlPS, 2019.
Yi Ren, Chenxu Hu, Tao Qin, Sheng Zhao, Zhou Zhao, and Yan Liu. FastSpeech 2: Fast and High-Quality End-to-End Text-to-Speech. arXiv preprint arXiv:2006. 045528000004588 2020.
Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdcep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerrv-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis. and Yonghui Wu. Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions. In Proc. ICASSP, pp. 4779 2018.
RJ Ekerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron Weiss, Rob Clark, and Rif A Saurous. Towards End-to-Ent to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron. In Proc. ICMI., pp. 4700-4709, 2018.
11

---

Under review as a conference paper at ICLR 2021
Jose Sotelo, Soroush Mehri, Kundan Kumar, Joao Felipe Santos, Kyle Kastner, Aaron C. Courville, and Yoshua Bengio. Char2Wav: End-to-End speech synthesis. In Proc. ICLR workshop, 2017. 2017.
Guangzhi Sun, Yu Zhang, Ron J Weiss, Yuan Cao, Heiga Zen, and Yonghui Wu, Fully-Hierarchical Fine-Grained Prosody Modeling for Interpretable Speech Synthesis. In Proc. ICASSP, pp. 6264- 6268, 2020.
David Talkin and Colin W Wightman. The aligner: Text to speech alignment using markov models and a pronunciation dictionary. In The Second ESCA/IEEE Workshop on Speech Synthesis, 1994.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Proc. NIPS, 2017.
Yuxuan Wang, RJ Skerry-Raan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang. Ying Xiao, Zhifeng Chen, Samy Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. Saurous. Tacotron: Towards End-to-End Speech Synthesis. In Proc. Interspeech, pp. 4006- 4010, 2017.
Ronald J. Williams and David Zipser. A Learning Algorithm for Continually Running Fully Recurrett Neural Networks. Neural Computation, 1(2):270-280, 1989.
Chengzhu Yu, Heng Lu, Na Hu, Meng Yu, Chao Weng, Kun Xu, Peng Liu, Deyi Tuo, Shiyin Kang, Guangzhi Lei, Dan Su, and Dong Yu. DurIAN: Duration informed attention network for multimodal synthesis. arXiv: 1909000709000... 2019.
H. Zen, K. Tokuda, and A. Black. Statistical Parametric Speech Synthesis. Speech Communication, 51(11):1039-1066 1064, 2009.
Heiga Zen, Andrew Senior, and Mike Schuster. Statistical Parametric Speech Synthesis Using Deep Neural Networks. In Proc. ICASSP, pp. 7962-7966, 2013.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech. In Proc. Interspeech, pp.
1526-15360 2019.
Zhen Zeng, Jianzong Wang, Ning Cheng, Tian Xia, and Jing Xiao. AlignTTS: Efficient Feed-Forward Text-to-Speech System without Explicit Alignment. In Proc. ICASSP, pp. 67 171446718, 2020.
Jing-Xuan Zhang, Zhen-Huu Hua Ling, and Li-Rong Dai. Forward attention in sequence-to-sequencc acoustic modeling for speech synthesis. In Proc. ICASSP, pp. 4789-4793, 2018
Zewang Zhang, Qiao Tian, Heng Lu, Ling-Hui Chen, and Shan Liu. Adadurian: Few-shot adaptation for neural text-to-speech with durian. arXiv preprint 2xX.i:0005005642 2020.
Yibin Zheng, Jianhua Tao, Wen. Zhengqi, and Jiangyan Yi. regularizing end-to-end tts. JEEE/ACM Trans. Audio Speech & Lang. Process., 27(12):2067-2077 2019.
decoding sequence for
12

---

Under review as a conference paper at ICLR 2021
A
MODEL PARAMETERS
Table 6: Model parameters.
Common
Training mode
Synchronous
Batch (per replica)
32
Replicas
Parameter init L regularization
Xavier
1 x 10-6
Learning rate
0.001
Learning rate schedule
Linear rampup 4K steps then decay half every 50K steps. Adam(0.9, 0.999, x 10 9)
Optimizer
LSTM zone-out prob
LSTM cell abs value cap
10.0
Inputs and Targets
Sampling rate (Hz)
24,000
Normalize waveform
No
Pre-emphasis
No
Frame size (ms)
50
Frame hop (ms)
12.5
Windowing
Hanning
FFT window size (point)
2048
Mel channels K
128
Mel frequency lower bound (Hz)
20
Mel frequency upper bound (Hz)
12,000
Mel spectrogram dynamic range compression
log (x - 0.001) 512
Token embedding dim
Speaker embedding dim
64
Encoder
Conv kernel
5 x 1
Cony dim
[512, 512,
Conv activation
[None, None, None] 0.999 512
Conv batch norm decay
Bi-LSTM dim
FVAE
Segment encoder conv kernel
3 1 512, 512]
Segment encoder conv dim
Segment encoder Bi-LSTM dim
256 x 2
Layer norm attention inputs
Yes 8 projected to 16
Latent dim
Duration Predictor
Bi-LSTM dim
512 2
Projection activation
None
Adur supervised
2.0
Adur semi-supervised
100.0
A semi-supervised
100.0
ÁKI. semi-supervised
1 10 3
Au unsupervised
10
AXL unsupervised
x 10
Range Parameter Predictor
Bi-I.TTM dim
512 x 2
Projection activation
SoftPlus
Positional Embedding
Embedding dim
32
Timestep denominator
10,000
Decoder
Pre-net dim supervised
[256, 256]
Pre-nct dim scmi/unsupervised
[128, 128]
Pre-net activation
[ReLU, ReLU]
Pre-net dropout prob
[0.5. 0.5]
LSTM dim
1,024
LSTM init
uniform (0.1) uniform (0.1)
Projection init
Post-net kernel
5 1
Post-net conv dim
[512, 512, 512, 512, 128] [tanh, tanh, tanh, tanh, None] uniform(0.
Post-net cony activation
Post-net conv init
13

---

Under review as a conference paper at ICLR 2021
B WER BREAKDOWNS IN THE ROBUSTNESS EVALUATION
Table 7: WER breakdowns in the robustness evaluation. Deletion rate (del) is the WDR in Table 5
LibriTTS
web-long del
System
WER
del
ins
sub
WER
ins
sub
Tacotron 2
w/ LSA
1.8
0.4
0.3
1.1
13.0
4.4
2.0
6.7
w/ GMMA
1.7
0.1
0.1
1.5
10.1
1.3
1.3
7.4
Non-Attentive Tacotron Supervised
1.4
0.1
0.1
1.2
9.3
1.1
1.3
6.9
Semi-supervised
0.9
14.1
3.5
1.6
9.0
Unsupervised
3.5
0.7
0.3
2.6
15.3
3.2
2.0
10.1
C CLASSIFICATION OF SOME TTS MODELS
Table 8: Classification of some TTS models into autoregressive (AR)/teed-forward (FF), RNN/Trans-
former/fully convolutional, and attention- www.bbbduuatioonbaaaaaaeeee based..
Model
Year
AR
FF
RNN
Transformer
Full Conv
Attention
Duration
Deep Voice
2017
Char2Wav
2017
Tacotron
2017
Deep Voice
2017
Tacotron
2018
Deep Voice 3
2018
Transformer TTS
2019
CHiVE
2019
DurIAN
2019
Fastspeech
2019
TalkNet
2020
AlignTTS
2020
JDI-T
2020
Non- Attentive Tacotron
2020
14


Reviews:
[{"review_id": "sJceBIR9_Y3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach based on the Tacotron model for speech synthesis, where the attention mechanism is replaced by a duration predictor. It also presents a short study on semi-supervised and unsupervised training. The paper also introduces two metrics to evaluate the robustness of the model. The experiments shows that the proposed model is on par with the Tacotron baselines in terms on MOS score and better in terms of the new metrics.\n\nPros:\n- The presented approach improves upon the Tacotron model.\n- The semi- and unsupervised learning capability of the system is significant.\n- The new metrics are very welcome, as the evaluation tools for TTS are limited. \n- The paper is clearly written.\n\nCons:\n- The novelty is limited as it is mainly an incremental improvement to an established approach. \n- The references are limited.\n\nDetailed comments:\n-  In terms of clarity, it's hard to say which part of the proposed approach is novel and which parts are from the original Tacotron. The authors should provide a brief description of the attentive Tacotron and clearly explain which parts they modified.\n- A Related Work section is missing. Some related works are presented in the introduction, but I think the paper could benefit to have a separate section for that. It could also be a good spot to present the Tacotron approach.\n\nOverall, the paper's novelty is limited, mainly due to it's incremental nature, but the unsupervised training capabilities and the new metrics are  significant, so I put it just above the acceptance threshold. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ldocLkL41_V", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nIn this paper the authors tackle the problem of alignment between input tokens and output acoustic features. The key contribution of this paper is replacing the attention mechanism of the Tacotron 2 with an explicit representation of token durations. The attention mechanism is vulnerable to issues such as pauses, repetitions, and skips, and hence using durations directly takes care of such issues. The challenge lies in obtaining the durations. The authors propose different methods toward that end. \n\nFirst, they introduce a duration predictor in the Tacotron 2 model architecture which utilizes the encoder features to predict the durations. This duration predictor may be supervised if target durations are available. The authors train HMM-based aligners to obtain target durations for this method. They also introduce a fine-grained variational autoencoder (FVAE) which is a conditional VAE that model the alignment between phonemes and acoustic features to extract token-level features which are in turn used by the duration predictor to predict the durations. The FVAE may be semi-supervised or unsupervised.\n\nThe proposed non-attentive Tacotron model achieves similar naturalness scores to Tacotron 2 which is very close to ground truth naturalness. Additionally, the authors evaluate the proposed model for robustness. They propose two metrics which measure the severity of alignment issues in synthesized speech. The proposed model outperforms the attention-based Tacotron 2 for multiple datasets. The authors also demonstrate the ability to control the pace of utterances by modulating predicted durations. Finally, the authors demonstrate the ability of the semi-supervised and unsupervised models to achieve similar results to the supervised model by utilizing the FVAE duration model.\n\nPros:\n1. The use of speech recognition-based metrics for measuring robustness is very interesting. More widespread adaptation of robustness as a criteria in addition to MOS scores is required. These objective metrics which can be computed for large datasets can be very helpful.\n2. The duration modeller working in the supervised, semi-supervised, and unsupervised setting is another good feature of this model. \n\nCons:\n1. The connection between the duration model and the full model is not very clear in the text. Why does the duration model have a section separate from the Model section, instead of a subsection? A suggestion would be to at least mention early in section 2 that ground truth target durations are not required since there is a semi-supervised duration modeller. There is a delay of 4-5 pages before it is clear that phoneme durations are not a requirement to use the non-attentive tacotron.\n2. In section 5.4, the experiment setup is not very clear. Do you mean that out of the 66 speakers in the full dataset, 10 speakers' durations are removed (for the semi-supervised experiment)?", "extra_scores": {}, "extra_text": {}}, {"review_id": "aXZ6D5Furiw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nIn this paper, the authors introduce a text-to-speech model based on Tacotron 2, called Non-Attentive Tacotron. Instead of an attention mechanism, a duration predictor is utilized to improve robustness, which is evaluated by two metrics, unaligned duration ratio (UDR) and word deletion rate(WDR). The authors propose semi-supervised and unsupervised duration modeling with a fine-grained variational auto-encoder (FVAE).\n\nReasons for score:\n\nThe paper is well-written. The experiment results of improvement in robustness are convincing. And the duration modeling in an unsupervised manner is appealing. However, using similar ideas to improve the robustness of end-to-end TTS has been investigated.\n\n\nPros:\n\n-- With the use of Gaussian upsampling instead of location-sensitive attention (LSA), the results in Table 2 demonstrate the improvement of robustness compared to Tacotron 2.\n\n-- Two metrics, UDR and WDR, focusing on TTS over-generation and under-generation, respectively, are introduced for robustness evaluation, which seems convincing and suitable for large-scale evaluation.\n\n-- A method of unsupervised duration modeling is proposed. As shown in Table 4, with FVAE, the TTS system can achieve high MOS in an unsupervised manner when the target durations are not available.\n\n\nCons:\n\n-- This paper mainly focuses on improving Tacotron 2 with regard to robustness. However, the similar weakness in Tacotron 2 has been investigated in DurIAN and FastSpeech, as mentioned in section 1. There should be more experiments on the comparison between Non-Attentive Tacotron and similar work, such as DurIAN, non-autoregressive FastSpeech, etc.\n\n-- The introduced Gaussian upsampling and FVAE improve the robustness compared to Tacotron2, but similar ideas have been investigated before, as mentioned in the paper. In Donahue et al. (2020), a similar concept to Gaussian upsampling has been introduced. The authors should have the experiments verifying the benefit of a learned range parameter sigma. Besides, a similar duration modeling method that utilizes attention between target spectrogram and phone embedding also appears in Sun et al. (2020).\n\n\nQuestions:\n\n-- The ratio of labeled duration data and unlabeled data in the semi-supervised experiment is unclear. The amount of training data and the ratio of the unlabeled duration should be compared in Table 4.\n\n-- In Sun et al. (2020), besides the duration, pitch and energy are also predicted by the VAE. Can the similar method be applied to Non-Attentive Tacotron with the VAE? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Wt4_drsmMH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new text-to-speech synthesis (TTS) framework that does not require attention mechanism at inference time. The authors suggest to use Duration Predictor to estimate the duration of each phoneme in a phoneme sequence.\nThe estimated duration of phonemes are upsampled by the proposed approach GaussianUpsampling, which shows significantly better listening test results than Vanilla repetition upsampling method. The authors also combine prior work, FVAE, to enable semi-supervised or unsupervised training (no duration labels). The authors show that naively predicting the duration in the unsupervised setting does not work very well, whereas combining FVAE makes it work significantly better.\nFinally, the authors suggest two evaluations metrics, 1. Unaligned Duration Ratio (UDR) and 2. ASR word deletion rate (WDR), to test the robustness of TTS systems. The first metric was used to test the problem of over-generation, and the second metric was used to test a word skipping problem. The authors show that the proposed Non-attentive Tacotron model shows superior performance on both metrics compared to the attention-based Tacotron2. Lastly, the authors also show that both semi-supervised and unsupervised approach perform well as much as the supervised one in terms of MOS.\n\nStrength: \nThe authors improve Tacotron2 to solve the problem of attention-based model such as over-generation and word skipping problem.  The experiments are well defined to show the robustness of Non-attentive model. It shows the robustness of Non-attentive TTS model by showing UDR and WDR. The MOS score is almost close to that of human voice, which is impressive. \n\nWeakness:\n\n---Lack of novelty---\nWhile the results are impressive (e.g., MOS score) I feel like the proposed approaches lack novelty.\nFor example, \n1)Duration prediction has been there for a while since Fastspeech1.\n2)GaussianSampling is similar to EATS (https://arxiv.org/abs/2006.03575).\n3)FVAE is also not proposed by the authors\n\n---Lack of some experiments---\nI assume the biggest advantage of Non-attentive Tacotron is the robustness in generation quality. If so, does the unsupervised Non-attentive Tacotron still has the advantages of less over-generation and word skipping problems? It’d have been nicer if the authors had shown UDR and WDR results in the unsupervised setting too.\n\n---Writings---\nAlthough using FVAE makes the unsupervised experiments successful, the insights on using FVAE to tackle this problem is not clearly stated. The authors must state why is combining FVAE approach expected to predict duration better than the naïve w/o FVAE version.  Also, I ask the authors to keep readers in mind little more when writing the paper. The authors should make the writings clearer when explaining the existing methods. For example, in Section3, the depth of explanations on FVAE is too short. Please elaborate more on how it can be formulated as conditional VAE framework.\n\nRating:\nI consider it “not bad” paper, but I think the impact of this paper is not strong enough to pass the bar of ICLR because of the lack of novelty (as written in Weakness part). Therefore, I recommend rejection. However, I’d be happy to listen to the authors’ opinion regarding this issue.\n\nQuestions:\n\n--- Questions on Vanilla upsampling ---\nIt has been shown by Fastspeech1 that the Vanilla upsampling  gives similar score to Tacotron2 model (3.84 vs 3.86). In this paper, however, it seems like the Vanilla upsampling is not working very well compare to Tacotron2 (4.13 vs 4.37). I’d like to ask the authors what the source of this difference could be.\n\n--- Questions on training WaveRNN ---\nThe authors have written that WaveRNN model was trained on *predicted features*. In this case, I assume the ground truth waveform and the predicted features are *unaligned* because Tacotron2 autoregressively decodes features, which must be different to the ground truth mel-spectrogram. How could WaveRNN be trained well enough in this setting? Were the *predicted features* predicted using Teacher-forcing?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates a non-attentive architecture of Tacotron 2 for TTS where the attention mechanism is replaced by a duration predictor.  The authors show that this change can significantly improve the robustness. In addition, the authors propose two evaluation metrics for TTS robustness, namely, unaligned duration ratio (UDR) and word deletion rate (WDR), which appear to be novel to the TTS community. The proposed non-attentive architecture yields good MOS scores in the experiments.  

Overall, the paper is well written but the reviewers commented on the technical novelty of the work as it is essentially an improvement within the Tacotron 2 framework. There is also a lack of comparative study with other existing frameworks with similar techniques.  Although the authors put together a detailed rebuttal to address the comments, in the end the above two major concerns remain. 
Decision: Reject