WaveGrad: Estimating Gradients for Waveform Generation
Authors: Nanxin Chen
Yu Zhang
Heiga Zen
Ron J Weiss
Mohammad Norouzi
William Chan
Venue: ICLR 2021 Poster (2021)
Keywords: vocoder
diffusion
score matching
text-to-speech
gradient estimation
waveform generation

Abstract: This paper introduces WaveGrad, a conditional model for waveform generation which estimates gradients of the data density. The model is built on prior work on score matching and diffusion probabilistic models. It starts from a Gaussian white noise signal and iteratively refines the signal via a gradient-based sampler conditioned on the mel-spectrogram.
WaveGrad offers a natural way to trade inference speed for sample quality by adjusting the number of refinement steps, and bridges the gap between non-autoregressive and autoregressive models in terms of audio quality.
We find that it can generate high fidelity audio samples using as few as six iterations.
Experiments reveal WaveGrad to generate high fidelity audio, outperforming adversarial non-autoregressive baselines and matching a strong likelihood-based autoregressive baseline using fewer sequential operations.  Audio samples are available at https://wavegrad.github.io/.

Published as a conference paper at ICLR 2021
WAVEGRAD: ESTIMATING GRADIENTS FOR WAVEFORM GENERATION
Nanxin Chen
Johns Hopkins University, Center for Language and Speech Processing bobchennang juu. edu
Yu Zhang! Heiga Zen, Ron J. Weiss, Mohammad Norouzi, William Chan Google Research, Brain Team {ngyuzh, haigazen, ronw, www.,mmorouz, howp:/ooooozz...llllaaaaaageeeep
ABSTRACT
This paper introduces WaveGrad, a conditional model for waveform generation which estimates gradients of the data density. The model built on prior work score matching and diffusion probabilistic models. It starts from Gaussian white noise signal and iteratively refines the signal via a gradient-based sampler conditioned the mel-spectrogram. WaveGrad offers natural way trade in- ference speed for sample quality by adjusting the number of refinement steps, and bridges the gap between non-autoregressive and autoregressive models in terms of audio quality. We find that it can generate high fidelity audio samples using as few as six iterations. Experiments reveal WaveGrad to generate high fidelity au- dio, outperforming adversarial non aon-autoregressvee baselines and matching strong Aikelihood-basee autoregressive baseline using fewer sequential operations. Audio samples are available at https: / //w////aaaaaaaae g thub. io/.
INTRODUCTION
Deep generative models have revolutionized speech synthesis (Oord et al., 2016; Sotelo al., 2017; Wang et al., 2017; Biadsy et al., 2019; Jia et al., 2019; Vasquez & Lewis, 2019). Autoregressive models, in particular, have been popular for raw audio generation thanks to their tractable likeli- hoods, simple inference procedures, and high fidelity samples (Oord et al., 2016; Mehri et al., 2017; Kalchbrenner al., 2018; Song et al., 2019; Valin & Skoglund, 2019). However, autoregressive models require a large number of sequential computations to generate an audio sample. This makes it challenging deploy them in real-world applications where faster than real time generation is essential, such as digital voice assistants on smart speakers, even using specialized hardware.
There has been a plethora of research into non autoregressive models for audio generation, including normalizing flows such as inverse autoregressive flows (Oord et al., 2018; Ping et al., 2019), gener- ative flows (Prenger al., 2019; Kim et al., 2019), and continuous normalizing flows (Kim al., 2020; Wu & Ling, 2020), implicit generative models such as generative adversarial networks (GAN) (Donahue et al., 2018; Engel et al., 2019; Kumar et al., 2019; Yamamoto et al., 2020; Binkowski et al., 2020; Yang et al., 2020a;b; McCarthy & Ahmed, 2020) and energy score (Gritsenko et al., 2020), variational auto-encoder models (Peng et al., 2020), as well as models inspired by digital sig- nal processing (Ai & Ling, 2020; Engel et al., 2020), and the speech production mechanism (Juvela et al., 2019; Wang et al., 2020). Although such models improve inference speed by requiring fewer sequential operations, they often yield lower quality samples than autoregressive models.
This paper introduces WaveGrad, conditional generative model of waveform samples that esti- mates the gradients of the data log-density opposed to the density itself. WaveGrad is simple to train, and implicitly optimizes for the weighted variational lower-bound of the log-likelihood.
Work done during an internship at Google Brain.
1 Equal contribution.

---

Published as a conference paper at ICLR 2021
Figure 1: A visualization of the WaveGrad inference process. Starting from Gaussian noise (n 0), gradient-based sampling applied using as few as 6 iterations to achieve high fidelity audio (n = 6). Left: signal after each step of gradient- based sampler. Right: zoomed view of 50 segment.
WaveGrad is non-autoregresssvve, and requires only constant number of generation steps during inference. Figure visualizes the inference process of WaveGrad.
WaveGrad builds class of generative models that emerges through learning the gradient of the data log-density, also known as the Stein score function (Hyvärinen, 2005; Vincent, 2011). Dur- ing inference, can rely the gradient estimate of the data log-density and gradient-based samplers (e.g., Langevin dynamics) to sample from the model (Song Ermon, 2019). Promising results have been achieved image synthesis (Song Ermon, 2019; 2020) and shape generation (Cai et al., 2020). Closely related are diffusion probabilistic models (Sohl-Dickstein et al., 2015), which capture the output distribution through Markov chain of latent variables. Although these models do not offer tractable likelihoods, one can optimize (weighted) variational lower-bound on the log-likelihood. The training objective can be reparameterized to resemble deonising score matching ( (Vincent, 201 1), and can be interpreted estimating the data log-density gradients. The model is non-autoregressive during inference, requiring only a constant number of generation steps, using Langevin dynamics-like sampler to generate the output beginning from Gaussian noise.
The key contributions of this paper are summarized as follows:
WaveGrad combines recent techniques from score matching (Song et al., 2020; Song & Ermon, 2020) and diffusion probabilistic models (Sohl-Dickstein et al., 2015; et al., 2020) to address conditional speech synthesis.
We build and compare two variants of the WaveGrad model: (1) WaveGrad conditioned on discrete refinement step index following Ho et al. (2020), (2) WaveGrad conditioned on a continuous scalar indicating the noise level. We find this novel continuous variant more effective, especially because once the model is trained, different number of refinement steps can be used for inference. The proposed continuous noise schedule enables our model to use fewer inference iterations while maintaining the same quality (e.g., 6 vs. 50).
We demonstrate that WaveGrad is capable of generating high fidelity audio samples, outper- forming adversarial non-autoregressive models (Yamamoto et al., 2020; Kumar et al., 2019; Yang al., 2020a; Biríkowski et al., 2020) and matching of the best autoregressive mod- els (Kalchbrenner et al., 2018) in terms of subjective naturalness. WaveGrad is capable of generating high fidelity samples using as few as six refinement steps.
2 ESTIMATING GRADIENTS FOR WAVEFORM GENERATION
We begin with a brief review of the Stein score function, Langevin dynamics, and score matching. The Stein score function (Hyvärinen, 2005) is the gradient of the data log-density logp(y) p(y) with respect the datapoint y:
s(y) V. log p(v).
2

---

Published as a conference paper at ICLR 2021
Dennising Ps(th (ineiaa)
Diffision ViN6a Min)
WaveGrad directed graphical model for training, conditioned on iteration index. 43 yn) iteratively adds Gaussian noise to the signal starting from the waveform yo- 4(u++1/00 yo) the noise distribution used for training. The inference denoising process progressively removes noise, starting from Gaussian noise yn. akin to Langevin dynamics. Adapted from Ho et al. (2020).
Figure 2:
Given the Stein score function s(), one can draw samples from the corresponding density, ÿ ~ p(y), via Langevin dynamics, which can be interpreted as stochastic gradient ascent in the data space:
J++1 Us + 2 ((.) + v 2i:
where 7) > is the step size, 2, NO, I). and denotes an identity matrix. variant (Ho al., 2020) is used as our inference procedure.
generative model can be built by training neural network to learn the Stein score function di- rectly, using Langevin dynamics for inference. This approach, known as score matching (Hyvärinen, 2005; Vincent, 2011 has seen success image (Song & Ermon, 2019; 2020) and shape (Cai al., 2020) generation. The denoising score matching objective (Vincent, 2011) takes the form:
(o) - Volog 40 v) |
Bys-its (())(
where p(-) is the data distribution, and q(-) is a noise distribution.
Recently, Song & Ermon (2019) proposed weighted denoising score matching objective, in which data is perturbed with different levels of Gaussian noise, and the score function se s0 o)) is condi- tioned on a, the standard deviation the noise used:
Σ Eyrpin) EgnNiin [bot0
y o²
(4)
where is set of standard deviation values that are used to perturb the data, and A(o) is a weighting WaveGrad is variant of this approach applied to learning conditional generative models of the form p(y x). WaveGrad adopts a similar objective which combines the idea of Vincent (201 I); Ho et al. (2020); Song & Ermon (2019). WaveGrad learns the gradient of the data density, and uses a sampler similar to Langevin dynamics for inference.
function for different
The denoising score matching framework relies ing the gradient of the data log density (i.e., q in Equation 3, and aN, in Equation 4). The choice of the noise distribution is critical for achieving high quality samples (Song & Ermon, 2020). As shown in Figure 2, WaveGrad relies on the diffusion model framework (Sohl-Dickstein et al., 2015; Ho al., 2020) generate the noise distribution used learn the score function.
noise distribution to provide support for learn-
2.1 WAVEGRAD A DIFFUSION PROBABILISTIC MODEL
Ho al. (2020) observed that diffusion probabilistic models (Sohl-Dickstein et al., 2015) and score matching objectives (Song& & Ermon, 2019; Vincent, 2011: I; Song & Ermon, 2020) are closely related. As such, we will first introduce WaveGrad
diffusion probabilistic model.
We adapt the diffusion model setup in Ho et al. (2020), from unconditional image generation to conditional raw audio waveform generation. WaveGrad models the conditional distribution po po(v
3

---

Published as a conference paper at ICLR 2021
Algorithm 1 Training. WaveGrad directly condi- tions the continuous noise level va. is from predefined noise schedule.
Algorithm 2 Sampling. WaveGrad generates
ples following gradient-based sampler similar Langevin dynamics.
1: repeat
UN ~ No 1) 2: for = N...., do
2: Yo ~ ((3)
3:
([.. S}}
3: 2 ~~ I)
4:
Vā~ Uniform((4-1:44)
on 8n Vi on
en (Vn. (ān))
5:
<~ <~~N0,,
4: Vn-1
van
6:
Take gradient descent step
5: if L, 1n-1 Un-1 + on
Valle-egh yo + v1 ā v,,.,,va vā)ll
end for
7:
until converged
7: return yo
2) where is the waveform and z contains the conditioning features corresponding to yo, such as linguistic features derived from the corresponding text, mel-spectrogram features extracted from yo- or acoustic features predicted by Tacotron-style text-to-speech synthesis model (Shen et al., 2018):
Po(00 x) = Po(yo:N x>) dyi:.,
(5)
where ./..... .. VN is series of latent variables, each of which are of the same dimension as the data Yo. and is the number of latent variables (iterations). The posterior (3h:N yo) called the diffusion process (or forward process), and is defined through the Markov chain:
( vo) II (Vn 1n-1));
(6)
where each iteration adds Gaussian noise:
(3m, 9n-1) N (yni v(1 Ba) Un 1 1,,, BBT),
under some (fixed constant) noise schedule .1,....By. BN We emphasize the property observed by Ho al. (2020), the diffusion process can be computed for any step n in closed form:
Un Vān Yo + v(1 a,)e
(8)
where N/0, an == 1- An and dh = Il (ss. The gradient of this noise distribution is
V um log ((nn Uo) v1 O.
Ho et al. (2020) proposed to train on pairs (No. Vn)) and to reparameterize the neural network to model eg. This objective resembles denoising score matching in Equation (Vincent, 2011) 1):
Ey,, Cn ||ea (Vān, 30 + VI 11-an, E, <|(2]
(10)
where Cn is constant related to 3n- In practice Ho et al. (2020) found it beneficial to drop the Cn term, resulting in a weighted variational lower bound the log-likelihood. Additionally Ho et al. (2020), 6g conditions the discrete index n, will discuss further below. We also found that substituting the original L2 distance metric with Lj offers better training stability.
2.2 NOISE SCHEDULE AND CONDITIONING ON NOISE LEVEL
In the score matching setup, Song & Ermon (2019; 2020) noted the importance of the choice of noise distribution used during training, since it provides support for modelling the gradient distribu- tion. The diffusion framework can be viewed as a specific approach to providing support to score matching, where the noise schedule is parameterized by 3I...., .x.. as described in the previous ,.....,3.. section. This is typically determined via some hyperparameter heuristic, e.g., a linear decay sched- ule (Ho et al., 2020). We found the choice of the noise schedule to be eritical towards achieving high fidelity audio in our experiments, especially when trying to minimize the number of inference iterations N make inference efficient. A schedule with superfluous noise may result in model
4

---

Published as a conference paper at ICLR 2021
unable to recover the low amplitude detail of the waveform, while a schedule with too little noise may result in a model that converges poorly during inference. Song & Ermon (2020) provide some insights around tuning the noise schedule under the score matching framework. We will connect some of these insights and apply them to WaveGrad under the diffusion framework.
Another closely related problem is determining N, the number of diffusion/denoising steps. A large N equips the model with more computational capacity, and may improve sample quality. However using a small N results in faster inference and lower computational costs. Song & Ermon (2019) used N = 10 to generate 32 x 32 images, while Ho et al. (2020) used 1,000 iterations to generate high resolution 256 256 images. In our case, WaveGrad generates audio sampled at 24 kHz.
We found that tuning both the noise schedule and N in conjunction was critical to attaining high fidelity audio, especially when N is small. these hyperparameters are poorly tuned, the training sampling procedure may provide deficient support for the distribution. Consequently, during infer- ence, the sampler may converge poorly when the sampling trajectory encounters regions that deviate from the conditions seen during training. However, tuning these hyperparameters can be costly due to the large search space, as large number of models need to be trained and evaluated. We make empirical observations and discuss this in more details in Section 4.4.
We address some of the issues above in our WaveGrad implementation. First, compared to the diffusion probabilistic model from Ho et al. (2020), we reparameterize the model to condition on the continuous noise level ā instead of the discrete iteration index n. The loss becomes
Ea,c [|
(vā + vI vI- vā) -
(11)
A similar approach was also used in the score matching framework (Song & Ermon, 2019; 2020), wherein they conditioned on the noise variance.
There is one minor technical issue we must resolve in this approach. In the diffusion probabilistic model training procedure conditioned on the discrete iteration index (Equation 10), we would sample Uniform ...,...,,,,. ..)),...,,,, and then compute its corresponding On, When directly conditioning on the continuous noise level, we need to define a sampling procedure that can directly sample ā. Recall that ā, == II" (1 3.) [0, 1]. While we could simply sample from the uniform distribution ā ~ Uniform(O, 1), we found this to give poor empirical results. Instead, we use a simple hierarchical sampling method that mimics the discrete sampling strategy. We first define noise schedule with S iterations and compute all of its corresponding vā.:
lo 1, L. VII 3.).
(12)
We first sample segment s ~ U(11....... ((. SH), which provides segment (4)-1,40), and then sample from this segment uniformly to give Vā. The full WaveGrad training algorithm using this sampling procedure is illustrated in Algorithm
One benefit of this variant is that the model needs to be trained only once, yet inference can be run over large space of trajectories with- out the need to be retrained. To be specific, once we train model, we can use differ- ent number of iterations N during inference, making it possible to explicitly trade off be- tween inference computation and output qual- ity in one model. This also makes fast hyper- parameter search possible, as we will illustrate in Section 4.4. The full inference algorithm is explained in Algorithm 2. The full WaveGrad architecture is visualized in Figure 3. Details are included in Appendix A.
3 x Cony (1)
a FILM
UBlock (128, x2)
5.x Cony (32)
UBlock (128, x2)
FiLM
DBlock (128, /2)
UBlock (256, x3)
FiLM
DBlock ( (128, /2)
UBlock (512, x5)
FiLMM
DBlock (256, /3)
UBlock (512, x5)
FiLM
DBlock (512, /5)
3 Conv (768)
3 RELATED WORK
This work is inspired in part by Sohl-Dickstein et al. (2015), which applies diffusion proba-
Figure 3: WaveGrad network architecture. The in- puts consists of the mel-spectrogram conditioning signal æ, the noisy waveform generated from the previous iteration yn. and the noise level Vā. The model produces e, each iteration, which can be interpreted as the direction to update yn-

---

Published as a conference paper at ICLR 2021
bilistic models to unconditional image synthe- sis, whereas we apply diffusion probabilistic models to conditional generation of waveform. The objective we use also resembles the Noise Conditional Score Networks (NCSN) objective of Song & Ermon (2019). Similar to Song & Ermon (2019; 2020), our models condition on a continuous scalar indicating the noise level. Denoising score matching (Vincent, 2011 1) and sliced score matching Song et al. (2020) also use similar objective functions, however they do not condi- tion on the noise level. Work of Saremi et al. 22018) on score matching is also related, in that their objective accounts for a noise hyperparameter. Finally, Cai et al. (2020) applied NCSN to model conditional distributions for shape generation, while our focus is waveform generation.
WaveGrad also closely relates to masked-basee generative models (Devlin et al., 2019; Lee et al., 2018; Ghazvininejad et al., 2019; Chan et al., 2020; Saharia al., 2020), insertion-base genera- tive models (Stern et al., 2019; Chan et al., 201 9b;;;;; Li & Chan, 2019) and edit-based generative models (Sabour et al., 2019; Gu et al., 2019; Ruis al., 2019) found in the semi- semi-autooeerresive quence generation literature. These approaches model discrete tokens and use edit operations (e.g., insertion, substitution, deletion), whereas in our work, we model the (continuous) gradients in tinuous output space. Edit-based models can also iteratively refine the outputs during inference (Lee al., 2018; Ghazvininejad et al., 2019: Chan et al., 2020), while they do not rely on (continuous) gradient-based sampler, they rely on (discrete) edit-based sampler. The noise distribution play key role, token masking based on Bernoulli (Devlin al., 2019), uniform (Saharia al., 2020), or hand-crafted (Chan et al., 2020) distributions have been used to enable learning the edit distribution. We rely on a Markov chain from the diffusion framework (Ho et al., 2020) to sample perturbations.
We note that the concurrent work of Kong et al. (2020) also applies the diffusion framework of Ho al. (2020) to waveform generation. Their model conditions on a discrete iteration index whereas we find that conditioning on continuous noise level offers improved flexibility and enables gen- crating high fidelity audio as few as six refinement steps. By contrast, Kong et al. (2020) report performance using 20 refinement steps and evaluate their models when conditioned on ground truth mel-specttogram. We evaluate WaveGrad when conditioned on Tacotron 2 mel-spectrogram predic- tions, which corresponds more realistic TTS setting.
The neural network architecture of WaveGrad is heavily inspired by GAN-TTS (Binkowski et al., 2020). The upsampling block (UBlock) of WaveGrad follows the GAN-TTS generator, with minor difference that no BatchNorm is used.
4 EXPERIMENTS
We compare WaveGrad with other neural vocoders and carry out ablations using different noise schedules. We find that WaveGrad achieves the same sample quality as the fully autoregressive state-of-the-art model of Kalchbrenner et al. (2018) (WaveRNN) on both internal datasets (Table 1) and LJ Speech (Ito & Johnson, 2017) (Table C.1) with less sequential operations.
4.1 MODEL AND TRAINING SETUP
We trained models using a proprietary speech dataset consisted of 385 hours of high quality English speech from 84 professional voice talents. For evaluation, we chose a female speaker in the train- ing dataset. Speech signals were downsampled to 24 kHz then 128- --8dimmnniioal mel-spectrogram features (50 ms Hanning window, 12.5 ms frame shift, 2048-point FFT, 20 Hz. & 12 kHz lower & upper frequency cutoffs) were extracted. During training, mel-spectrograms computed from ground truth audio were used as the conditioning features x. However, during inference, we used pre- dicted mel-spectrograms generated by a Tacotron 2 model (Shen et al., 2018) as the conditioning signal. Although there was a mismatch in the conditioning signals between training and inference, unlike Shen et al. (2018), preliminary experiments demonstrated that training using ground truth mee-spectrssrams as conditioning had no regression compared to training using predicted features. This property is highly beneficial it significantly simplifies the training process of text-to-speech

---

Published as a conference paper at ICLR 2021
models: the WaveGrad vocoder model can be trained separately large corpus without relying on a pretrained sppett-sppet model.
compared: Base and Large. The WaveGrad Base model took 24 frames corresponding to 0.3 seconds audio (7,200 samples) as input during train- ing. We set the batch size to 256. Models were trained using 32 Tensor Processing Unit (TPU) v2 cores. The WaveGrad Base model contained 15M parameters. For the WaveGrad Large model, and another without.
Model Size: Two network size variations
we repeated each UBlock/DBlock twice, one with ppsampping/downsampiinn Each training sample included 60 frames corresponding to a 0.75 second of audio (18,000 samples). We used the same batch size and trained the model using 128 TPU v3 cores. The WaveGrad Large model contained 23M parameters. Both Base and Large models were trained for about 1M steps. The network architecture is fully convolutional and con-autoreeressvve thus it highly parallelizable at both training and inference.
Noise Schedule: All noise schedules we used can be found in Appendix B.
4.2 EVALUATION
The following models were used as baselines in this experiment: (1) WaveRNN (Kalchbrenner et al., 2018) conditioned on mel-spectrograms predicted by a Tacotron 2 model in teacher-forcing mode following Shen et al. (2018); The model used a single long short-term memory (LSTM) layer with 1,024 hidden units, convolutional layers with 512 channels as the conditioning stack to process the specstrcroo features, and 10-component mixture of logistic distributions (Salimans al., 2017) as its output layer, generating 16-bit samples at 24 kHz. It had 18M parameters and was trained for I M steps. Preliminary experiments indicated that further reducing the number of units in the LSTM layer hurts performance. (2) Parallel WaveGAN (Yamamoto et al., 2020) with 1.57M parameters, trained for IM steps. (3) MeIGAN (Kumar et al., 2019) with 3.22M parameters, trained for 4M steps. (4) Multi-band MelGAN (Yang et al., 2020a) with 2.27M parameters, trained for IM steps. (5) GAN-TTS (Binkoowki et al., 2020) with 21 .M parameters, trained for 1M steps.
All models were trained using the same training set as the WaveGrad models. Following the orig- inal papers, Parallel WaveGAN, MelGAN, and Multi-band MelGAN were conditioned on the mel- spectrograms computed from ground truth audio during training. They were trained using publicly available implementation at https: //tg///tttttt github........ com/ kan- baya sh / Paaral lel Wa veGAN. Note that hyper-parameters of these baseline models were not fully optimized for this dataset.
To compare these models, we report subjective listening test results rating speech naturalness on a 5-point Mean Opinion Score (MOS) scale, following the protocol described in Appendix D. Con- ditioning mel-spectrograms for the test set were predicted using a Tacotron 2 model, which were passed to these models to synthesize audio signals. Note that the Tacotron 2 model was identical to the one used to predict mel-spectrograms for training WaveRNN and GAN-TTS models.
4.3 RESULTS
Subjective evaluation results are summarized in Table 1. Models conditioned on discrete indices fol- lowed the formulation from Section 2. 1, and models conditioned on continuous noise level followed the formulation from Section 2.2. WaveGrad models matched the performance of the autoregressive WaveRNN baseline and outperformed the non-autoregressive baselines. Although increasing the model size slightly improved naturalness, the difference was not statistically significant. The Wave- Grad Base model using six iterations achieved a real time factor (RTF) of 0.2 on an NVIDIA V100 GPU, while still achieving an MOS above 4.4. As a comparison, the WaveRNN model achieved a RTF of 20.1 on the same GPU, 100 times slower. More detailed discussion is in Section 4.4. Ap- pendix C contains results on a public dataset using the same model architecture and noise schedule.
4.4 DISCUSSION
To understand the impact of different noise schedules and reduce the number of iterations in the noise schedule from 1,000, we explored different noise schedules using fewer iterations. We found that well-hehaved inference schedule should satisfy two conditions:
7

---

Published as a conference paper at ICLR 2021
Table 1: Mean opinion scores (MOS) of various models and their confidence intervals. All models except WaveRNN are non-autoregressiiee. WaveGrad, Parallel WaveGAN, MelGAN, and Multi- band MelGAN were conditioned on the mel-spectrograms computed from ground truth audio during training. WaveRNN and GAN-TTS used predicted features for training.
Model
MOS (1)
WaveRNN
4.49 + 0.04
Parallel WaveGAN
3.92 + 0.05
MeIGAN
3,95 + 0.06
Multi-band MeIGAN
4.10 + 0.05
GAN-TTS
4.34 + 0.04
WaveGrad
Base (6 iterations, continuous noise levels)
4.41 + 0.03
Base (1,000 iterations, discrete indices)
4.47 + 0.04
Large (1,000 iterations, discrete indices)
4.51 = 0.04
Ground Truth
4.58 +: 0.05
1. The KL--divergence DRL. (4(2N yoo || (0, 7)) between yn and standard normal distribution (0, 1) needs be small. Large Kl--divergence introduces mismatches between training and inference. To make the KL-divergence small, some 3s need to be large enough.
2. 3 should start with small values. This provides the model training with fine granularity details, which found crucial for reducing background static noise.
In this section, all the experiments were conducted with the WaveGrad Base model. Both objective and subjective evaluation results are reported. The objective evaluation metrics include
1. Log-met log-mel spectrogram mean squared error metrics (LS-MSE), computed using 50 ms window length and 6.25 frame shift;
2. Mel cepstral distance (MCD) (Kubichek, 1993), a similar MSE metric computed using 13- dimensional mel frequency cepstral coefficient features;
3. Po Frame Error (FFE) (Chu & Alwan, 2009), combining Gross Pitch Error and Voicing Deci- sion metries measure the signal proportion whose estimated pitch differs from ground truth.
Since the ground truth waveform is required to compute objective evaluation metrics, we report results using ground truth -pesspectss as conditioning features. We used validation set of 50 utterances for objective evaluation, including audio samples from multiple speakers. Note that for MOS evaluation, we used the same subjective evaluation protocol described in Appendix D. We experimented with different noise schedules and number of iterations. These models were trained with conditioning on the discrete index. Subjective and quantitative evaluation results are in Table 2.
We also performed a detailed study on the the WaveGrad model conditioned on the continuous noise level in the bottom part of Table Compared to the model conditioned on the discrete index with fixed training schedule (top of Table 2), conditioning better, especially if the number of iterations was small. It can be seen from Table 2 that degradation with the model with six iterations was not significant. The model with six iterations achieved real time factor (RTF)= = 0.2 on an NVIDIA V100 GPU and RTF 1.5 on an Intel Xeon CPU (16 cores, 2.3GHz). As we did not optimize the inference code, further speed ups likely possible.
the continuous noise level generalized
5 CONCLUSION
In this paper, we presented WaveGrad, a novel conditional model for waveform generation which estimates the gradients of the data density, following the diffusion probabilistic model (Ho et al., 2020) and score matching framework (Song et al., 2020; Song & Ermon, 2020). WaveGrad starts from Gaussian white noise and iteratively updates the signal via a maddient-based sampler condi- tioned on the specttrca.. WaveGrad is non-autoregressive, and requires only constant num- ber of generation steps during inference. We find that the model can generate high fidelity audio samples using few six iterations. WaveGrad is simple train, and implicitly optimizes for the
8

---

Published as a conference paper at ICLR 2021
Table Objective and subjective metries of the WaveGrad Base models. When conditioning on the discrete index, a separate model needs to be trained for each noise schedule. In contrast, a single model can be used with different noise schedules when conditioning on the noise level directly. This variant yields high fidelity samples using few six iterations.
Iterations (schedule)
LS-MSE (1)
MCD (L)
FFE (L)
MOS (1)
WaveGrad conditioned on discrete index
25 (Fibonacci)
283
3.93
3.3%
3,86 = 0.05
50 (Linear (1 10 0.05))
181
3.13
3.1%
4.42 0.04
1,000 (Linear (1 10-1, ,00055)
116
2.85
3.2%
4.47 + 0.04
WaveGrad conditioned continuous noise level (Manual)
217
3.38
2.8%
4.41 = 0.04
25 (Fibonacci)
185
3.33
2.8%
4.44 = 0.04
50 (Linear (1 x 10 10-4 0.05))
177
3.23
2.7%
4.43 = 0.04
1,000 (Linear (1 10 10-1,00005) 0.005))
2.85
3.0%
4.46 + 0.03
weighted variational lower-bound of the log-likelihood. The empirical experiments demonstrated WaveGrad to generate high fidelity audio samples matching a strong autoregressive baseline.
AUTHOR CONTRIBUTIONS
Nanxin Chen wrote code, proposed the idea, all experiments and wrote the paper. Yu Zhang cruited collaborators, co-managed/advised the project, conducted evaluation, debugging model and editing paper. Heiga Zen helped conducting text-to-speech experiments and advised the project. Ron Weiss implemented the objective evaluation metrics and advised the project. Mohammad Norouzi suggested the use of denoising diffusion models for audio generation and helped with writ- ing and advising the project. William Chan conceived the project, wrote code, wrote paper, and co-managed/advised the project.
ACKNOWLEDGMENTS
The authors would like to thank Durk Kingma, Yang Song, Kevin Swersky and Yonghui Wu for providing insightful research discussions and feedback. We also would like to thank Norman Casagrande for helping us to include the GAN-TTS baseline.
REFERENCES
Yang Ai and Zhen-Hua Ling. Knowledge-and--Dat--Diii Amplitude Spectrum Prediction for Hierarchical Neural Vocoders. arXiv preprint arXiv:20044078832 2020.
Eric Battenberg, RJ Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby. Location-relative Attention Mechanisms for Robust Long-form Speech Synthe- sis. In ICASSP. 2020.
Fadi Biadsy, Ron J. Weiss, Pedro J. Moreno, Dimitri Kanevsky, and Ye Jia. Parrotron: An End-to- End Speech-to-Speech Conversion Model and its Applications to Bearing-Impainng Speech and Speech Separation. In INTERSPEECH, 2019.
Mikolaj Binkkoski,, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan. High Fidelity Speech Synthesis with Adversar- ial Networks. In ICLR, 2020.
Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning Gradient Fields for Shape Generation. In ECCV, 2020.
Harris Chan, Jamie Kiros, and William Chan. Multilingual KERMIT: It's Not Easy Being Genera- tive. In NeurlPS: Workshop on Perception Generative Reasoning, 2019a.

---

Published as a conference paper at ICLR 2021
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. KERMIT: Genera- tive Insertion-Basee Modeling for Sequences. arXiv preprint arXiv:1906.01604,222 2019b.
William Chan, Mitchell Stern, Jamie Kiros, and Jakob Uszkoreit. An Empirical Study of Generation Order for Machine Translation. arXiv preprint arXiv: 13437, 2019c.
William Chan, Chitwan Saharia, Geoffrey Hinton, Mohammad Norouzi, and Navdeep Jaitly. Im- puter: Sequence Modelling via Imputation and Dynamic Programming. In ICML, 2020.
Wei Chu and Abeer Alwan. Reducing FO Frame Error of FO Tracking Algorithms under Noisy Conditions with an Unvoiced/Voicek Classification Frontend. In ICASSP, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training Deep
Bidirectional Transformers for Language Understanding. In NAACI, 2019.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial Audio Synthesis. arXiv preprint arXiv:1802.04200 2018
Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise Transformations. Distill, 2018. doi: 1/23391/distill.000 https://disill..ubb200 http://ddsllllllooo2//////aerrre
Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. GANSynth: Adversarial Neural Audio Synthesis. In ICLR, 2019.
Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts. Signal Processing. In 2020.
DDSP: Differentiable Digital
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-Predict: Parallel Decoding of Conditional Masked Language Models. In EMNLP, 2019.
Alexey A. Gritsenko, Tim Salimans, Rianne van den Berg, Jasper Snoek, and Nal Kalchbrenner. A Spectral Energy Distance for Parallel Speech Synthesis. arXiv preprint arXiv:2008.01 160, 2020.
Jiatao Gu, Changhan Wang, and Jake Zhao. Levenshtein Transformer. In NeurlPS, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. arXiv preprint arXiv:2006. 10.000661/2333 /239, 2020.
Aapo Hyvärinen. Estimation of Non-Noomal Statistical Models by Score Matching. Journal of Machine Learning Research, 6, April 2005.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML, 2015.
Keith Ito and Linda Johnson. LJ-Speed Speecheeeehhee SneechhDataset...
The LJ Speech Dataset, 2017. https://keithito..om
Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. Direct Speech-to-Speeee Translation with Sequence-to- to-eequncee Model. In INTER. SPEECH, 2019.
Lauri Juvela, Bajibabu Bollepalli, Vassilis Tsiaras, and Paavo Alku. GlotNet-A Raw Waveform Model for the Glottal Excitation in Statistical Parametric Speech Synthesis. /EEE/ACM Transac- tions on Audio, Speech, and Language Processing, 1716 2019.
Nal Kalchbrenner. Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aäron van den Oord, Sander Dieleman, and Koray Kayukcuoglu, Efficient Neural Audio Synthesis. In ICML., 2018
10

---

Published as a conference paper at ICLR 2021
Hyeongju Kim, Hyeonseung Lee, Woo Hyun Kang. Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. WaveNODE: A Continuous Normalizing Flow for Speech Synthesis. In ICML: Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2020.
Sungwon Kim, Sang-Gil Lee, Jongyoon Song, Jaehyeon Kim, and Sungroh Yoon. FloWaveNet: A Generative Flow for Raw Audio. In ICML, 2019.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A Versatile Diffusion Model for Audio Synthesis. arXiv preprint arXiv:2009.09761, 2020.
Robert Kubichek. Mel-Cepstral Distance Measure for Objective Speech Quality Assessment. In IEEE PACRIM, 1993.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, and Aaron Courville, MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis. In NeurlPS, 2019.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic Non-Autoregressive Neural Se- quence Modeling by Iterative Refinement. EMNLP, 2018.
Lala Li and William Chan. Big Bidirectional Insertion Representations for Documents. In EMNLP: Workshop of Neural Generation and Translation, 2019.
Ollie McCarthy and Zohaib Ahmed. HooliGAN: Robust, High Quality Neural Vocoding. arXiv preprint arXiv:2008.0244 02493, 2020.
Soroush Mehri, Kundan Kumar, Ishaan Guirajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. SampleRNN: An Unconditional End-to-End Neural Audio Generation Model. In ICLR, 2017.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A Generative Model for 2016.
Raw Audio. arXiv preprint 100xv:6609.038000
Aäron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg. Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalch- brenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. Parallel WaveNet: Fast High-Fidelity Speech Synthesis. In ICML, 2018.
Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic Image Synthesis with Opatially-Adaptivy Normalization. CVPR, 2019.
Kainan Peng, Wei Ping, Zhao Song, and Kexin Zhao. Non-Autoregressive Neural Text-to-Speech. In ICML, 2020.
Wei Ping, Kainan Peng, and Jitong Chen. ClariNet: Parallel Wave Generation in End-to-End Text- to-Speech. In ICLR. 2019.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. WaveGlow: A Flow-based Generative Network for Speech Synthesis. In ICASSP, 2019.
Laura Ruis, Mitchell Stern, Julia Proskumia, and William Chan. Insertion-Deletion Transformer. In EMNLP: Workshop of Neural Generation and Translation, 2019.
Sara Sabour, William Chan, and Mohammad Norouzi. Optimal Completion Distillation for Se-
quence Learning. In ICLR, 2019.
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. Non-Autoregresssve Machine Translation with Latent Aligmments. arXiv preprint arXiv:2004.07437 2020.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik Kingma. PixeICNN++: Improving the PixeICNN with Discretized Logistic Mixture Likelihood and Other Modifications. In ICLR, 2017.

---

Published as a conference paper at ICLR 2021
Saeed Saremi, Arash Mehrjou, Bernhard Schölkopf, and Aapo Hyvärinen. Deep Energy Estimator Networks. arXiv preprint arXiv:1805.0830000 2018.
Andrew M. Saxe, James I.. McClelland, and Surya Ganguli. Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks. In ICLR, 2014.
Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerrv-Ryan, Rif A. Saurous, Yannis Agiomyrgian- nakis, and Yonghui Wu. Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions. In ICASSP. 2018.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsuper- vised Learning using Nonequilibrium Thermodynamics. In ICML, 2015.
Eunwoo Song, Kyungguen Byun, and Hong-Goo Goo Kang. ExcitNet Vocoder: A Neural Excitation Model for Parametric Speech Synthesis Systems. In EUSIPCO, 2019.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribu- tion. In NeurIPS, 2019.
Yang Song and Stefano Ermon. Improved Techniques for Training Score-Based Generative Models. arXiv preprint arXiv:2006.0901,, 2020.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced Score Matching: A Scalable Ap- proach to Density and Score Estimation. In Uncertainty in Artificial Intelligence, pp. 574-584, 2020.
Jose Sotelo. Soroush Mehri, Kundan Kumar, Joao Felipe Santos, Kyle Kastner, Aaron C. Courville, and Yoshua Bengio. Char2Wav: End-to-End Speech Synthesis. In ICLR, 2017.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion Transformer: Flexible Sequence Generation via Insertion Operations. In ICML, 2019.
Jean-Mare Valin and Jan Skoglund. LPCNet: Improving Neural Speech Synthesis through Linear
Prediction. In ICASSP, 2019.
Sean Vasquez and Mike Lewis. MelNet: A Generative Model for Audio in the Frequency Domain. arXiv preprint arXiv: www..0006.....3 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NIPS, 2017.
Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 2011.
Xin Wang, Shinji Takaki, and Junichi Yamagishi. Neural Cource-Filter Waveform Models for Sta- tistical Parametric Speech Synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:402-415, 2020.
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoe Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous. Tacotron: Towards End-to-Ein -nd Speech Synthesis. In INTER- SPEECH, 2017.
Ning-Qie Qian Wu and Zhen-Hua Ling. WaveFFJORD: FFJORD-Based Vocoder for Statistical Para- metric Speech Synthesis. In ICASSP, 2020.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel WaveGAN: Fast Waveform Gen- eration Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram. In ICASSP. 2020.
Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, and Lei Xie. Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech. arXiv preprint arXiv:2005.055 05000, 2020a.
Jinhyeok Yang, Junmo Lee, Youngik Kim, Hoonyoung Cho, and Injung Kim. VocGAN; A High- Fidelity Real-time Vocoder with a Hierarchically-nese nested Adversarial Network. arXiv preprint arXiv:2007. 15256, 2020b.
12

---

Published as a conference paper at ICLR 2021
Corx
LReLU
Tostare-wke Affirr
[RdA
Fature wine Affice
3x
Coo
x Core
lipaapplc
Festure
x Cors
LRRLU
Upoample
Diamaanple
x
1x Car
Cope
Figure A block diagram of feature-wise linear modula- tion (FiLM) module. We con- dition the noise level v of diffusion//denisiing pro- cess, and pass to posi- tional encoding function.
Figure 4: A block diagram of the Upsampling Block (UBlock). We upsample the signal modulated with infor- mation from the FiLM mod- ule.
Dewecamplc
Figure 5: A block diagram of the downsampling block (DBlock).
A NEURAL NETWORK ARCHITECTURE
To convert the mel-spectrogram signal (80 Hz) into raw audio (24 kHz), five upsampling blocks (UBlock) are applied to gradually upsample the temporal dimension by factors of 5, 2, with the number of channels of 512, 512, 256, 128, 128 respectively. Additionally, one convolutional layer is added before and after these blocks.
The UBlock is illustrated in Figure 4. Each UBlock includes two residual blocks (He et al., 2016). Neural audio generation models often use large receptive field (Oord al., 2016; Binkkoski al., 2020; Yamamoto et al., 2020). The dilation factors of four convolutional layers are 1, 2, 4, 8 for the first three UBlocks and 1, 2, 1, 2 for the rest. Upsampling is carried out by repeating the nearest input. For the large model, we use 1, 2, 4, 8 for all UBlocks.
an iterative approach, the network prediction is also conditioned on noisy waveform Van Mo + vI- c. Downsampling blocks (DBlock), illustrated in Figure 5, are introduced to downsample the temporal dimension the noisy waveform. The DBlock similar to UBlock except that only one residual block is included. The dilation factors 2, in the main branch. Downsampling is carried out by convolution with strides. Orthogonal initialization (Saxe et al., 2014) used for all UBlocks and DBlocks.
The feature-wise linear modulation (FiLM) (Dumoulin et al., 2018) module combines information from both noisy waveform and input mel-spectrogram. We also represent the iteration index n, which indicates the noise level of the input waveform, using Transformer-style sinusoidal positional embeddings (Vaswani al., 2017). To condition on the noise level directly, we also utilize the sinusoidal embeddings where 5000 āa instead of n is used. The FiLM module produces both scale and bias vectors given inputs, which are used UBlock for feature-wise affine transformation as
TD, V O U + E(D, Vā),
(13)
where 7 and & correspond the scaling and shift vectors from the FiLM module. D is the out- put from corresponding DBlock, U is an intermediate output in the UBlock, and c denotes the Hadamard product.
An overview the FiLM module is illustrated in Figure 6. The structure is inspired by spatially- adaptive denormalization (Park al., 2019). However batch normalization (loffe & Szegedy, 2015) not applied in our work since each minibatch contains samples with different levels noise. Batch statistics not accurate since they heavily dependent sampled noise level. Experiment
13

---

Published as a conference paper at ICLR 2021
v 0n with various noise schedules.
0.8
1000 (Linear)
0.6
50 (Linear)
0.4
25 (Fibonacci)
0.2
6 (Manual)
10"
10a
10a
10a
Iteration Index n
Figure Plot of different noise schedules.
results also verified our assumption that models trained with batch normalization generate low-
quality audio.
B NOISE SCHEDULE
For the WaveGrad Base model, tested different noise schedules during training. For 1000 and 50 iterations, we set the forward process variances to constants increasing linearly from 31 to SN. defined Linear( Sy. N). We used Linear(1 x 10 0.005, 1000) for 1000 iterations and Linear(1 x 10-4, 0.05, 50) for 50 iterations. For 25 iteration, a different Fibonacci- schedule was adopted (referred as Fibonacci(N)):
3 = 1 10-6
= 10
(14)
= Bn-2 Vn 2 2.
When fixed schedule was used during training, the same schedule was used during inference. We found that a mismatch in the noise schedule degraded performance.
To sample the noise level Vā, we the maximal iteration S 1000 and precompute 1 lg from Linear(1 10 -6, 0.01, 1000). Unlike the base fixed schedule. WaveGrad support using a different schedule during inference thus "Manual" schedule was also explored to demonstrate the possibilities with WaveGrad. For example, the 6-iteration inference schedule was explored by sweeping the 3s over following possibilities:
x 1 10-5 1100 10-,,00" 1003, 1,,0,,,100 101001
(15)
Again, we did not need to train individual models for such hyper-parameter tuning. Here we used LS-MSE as a metric for tuning.
All noise schedules and corresponding Vā are plotted in Figure 7.
C RESULTS FOR LJ SPEECH
We ran experiments using the IJ Speech dataset (Ito & Johnson, 2017), a publicly available dataset consisting of audiobook recordings that segmented into utterances of up to 10 seconds. We trained on a 12.764-utterance subset (23 hours) and evaluated on held-out 130-utterance subset, following Battenberg et al. (2020). During training, mel-spectrograms computed from ground truth audio was used as the conditioning features. We used the held-out subset for evaluating synthesized speech with ground truth features. Results are presented in Table C. For this dataset, larger net- work size is beneficial and WaveGrad also matches the performance of the autoregressive baseline.
D SUBJECTIVE LISTENING TEST PROTOCOL
The test set included 1,000 sentences. Subjects were asked to rate the naturalness each stimulus after listening it. Following previous studies, five-point Likert scale score (1: Bad, 2: Poor, 3:
14

---

Published as a conference paper at ICLR 2021
Table C.I: Mean opinion scores (MOS) for LJ speech datasets.
Model
MOS (1)
WaveRNN
4.49 0.05
WaveGrad
Large (6 iterations, continuous noise levels)
4.47 = 0.04
Large (1000 iterations, continuous noise levels)
4.35 0.05
Base (6 iterations, continuous noise levels)
4.35 = 0.05
Base (1000 iterations, continuous noise levels)
4,40 = 0.05
Ground Truth
4.55 + 0.04
Reported opinion scores (MOS) of various models and their confidence inter- vals. "Linguistic" and "Mel" in the "Features" " column indicate that linguistic features and mel- spectrogram were used conditioning, respectively.
Table E.2:
Model
Features
Sample Rate
MOS
Autoregressive
WaveNet (Oord et al., 2016)
Linguistic
16 kHz
4.21 + 0.08
WaveNet (Oord al., 2018)
Linguistic
24 kHz
4.41 = 0.07
WaveNet (Shen et al., 2018)
Mel
24 kHz
4.53 = 0.07
WaveRNN (Kalchbrenner al., 2018)
Linguistic
kHz
4.46 0.07
Non-autoregressive
Parallel WaveNet (Oord et al., 2018)
Linguistic
kHz
4.41 0.08
GAN-TTS (Bińkkowsii al., 2020)
Linguistic
24 kHz
4.21 - 0.05
GED (Gritsenko et al., 2020)
Linguistic
kHz
4.25 = 0.06
Fair, 4: Good, 5: Excellent) adopted with rating increments of 0.5. Each subject allowed evaluate up to six stimuli. Test stimuli were randomly chosen and presented for each subject. Each stimulus presented to subject in isolation and was evaluated by subject. The subjects were paid and native speakers of English living in United States. They were requested to use headphones in quiet room.
E SUBJECTIVE SCORES REPORTED IN THE PRIOR WORK
Table E.2 shows the reported
opinion scores of the prior work which used the same speaker. Although different papers listed here used the same female speaker, their results are not directly comparable due differences in the training dataset, sampling rates, conditioning features, and sentences used for evaluation.
15


Reviews:
[{"review_id": "jo6d7RFDm-g", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work uses diffusion probabilistic models for conditional speech synthesis tasks, specifically to convert mel-spectrogram to the raw audio waveform. Results from the proposed approach match the state-of-the-art WaveRNN model. The paper is very well-written and it is quite easy to follow. The study of the total number of diffusion steps and two different ways (continuous and discrete) ways to feed it in the network is very interesting. It is quite relevant and important for speech synthesis tasks. Using this, authors are able to find a 6-step inference procedure that yields very competitive performance to WaveRNN while still being computationally feasible.\n\nPros:\n1. Great results for the neural vocoding task\n2. Exhaustive study of the diffusion steps and how to feed them to the network is valuable and original.\n\nCons:\n1. Study of the width/depth of the network can be more exhaustive.\n\n*Further comments*: Authors mention in the conclusion ````Wavegrad is simple to train`. What makes the authors say so? It would be great to substantiate it with evidence/comparison?", "extra_scores": {}, "extra_text": {}}, {"review_id": "FzxTmyUQtPd", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Motivated by recent works on score matching and diffusion probabilistic models, the authors presented a conditional, non-autoregressive model for waveform generation. The paper is well motivated and easy to follow. \nThe model starts from Gaussian white noise signal and iteratively refines the signal via Langevin dynamics-like sampler. The model is capable to generate high fidelity audio, outperforming a few adversarial non-autoregressive baselines even using only 6 iterations. Though the fundamental approaches of this paper have been applied in other domains, making the approach work well for waveform generation is relatively new and not easy.\nThe authors also had a deep dive on noise schedule, and proposed a variant of WaveGrad which conditioned on continuous noise level. The authors showed clear benefits of this variant over conditioning on discrete index, including using few iterations to generate high fidelity audio, being able to tradeoff between inference computation and output quality, fast hyper-parameter searching. However, one minor concern is that, if inference computation is not a bottleneck, when using linear schedule, for both 50/1000 iteration scenarios, WaveGrad conditioned on continuous noise level does not show clear benefits over conditioning on discrete index.\n\nThe other minor concern is regarding the model size. When comparing with adversarial counterparts, most models are with fewer parameters than the proposed WaveGrad. Though the authors claimed that model size has little effect on the performance of WaveGrad (according to Table 1), but WaveGrad does seem to clearly benefit from larger model according to Table C.1, the MOS for LJ speech datasets. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FA8lIxoFMvj", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors propose a conditional waveform synthesis model: WaveGrad, which combines recent methods from score matching and diffusion probabilistic models. \n\nPros:\n\n(1) The proposed diffusion probabilistic model-based approach achieves comparable results as the autoregressive WaveRNN and outperformed the non-autoregressive baselines.\n\n(2) The WaveGrad Base model using six iterations is much faster than the WaveRNN.\n\nCons:\n\n(1) The authors borrow ideas from Ho et al. (2020) for image generation to address the conditional waveform synthesis problem.  There are really limited innovations on the model side since the proposed method just combines some techniques from recent score matching and diffusion probabilistic models.  \n\n(2) Why the proposed WaveGrad Vocoder can synthesize high-quality speech comparing to LSTMs in WaveRNN and WaveNet in the original Tacotron 2? The authors should give more insights and explanations to better illustrate the proposed method. \n\n(3) The original Tacotron 2 with WaveNet can achieve 4.526 ± 0.066, which is better than the WaveGrad. The authors should also include the results of the WaveNet Vocoder. \n\n(4) The authors should also provide results from the compared methods in https://wavegrad-iclr2021.github.io/ to validate the superiority of the proposed method. \n\n*** Post-Rebuttal ***\n\nThank the authors for responding to my concerns in the rebuttal. \n\nFor the contribution part,  the major technical contribution is the continuous noise schedule. However, it is very obscure in the original paper. As also suggested by R1, the authors should carefully revise the paper to make it more clear. In addition, I found the paper largely borrowed content from Ho et al. (2020). From the framework figure and Algorithms to texts and equations, only limited modifications are made. After reading the paper, my first impression is that the paper just uses a new model on image synthesis to address speech synthesis. The work is not well motivated and largely copying another paper in the method section is not a professional way.  \n\nHope the authors can modify the paper by adding the new clarifications and explanations in the rebuttal to improve the paper.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7bhtsbU6-ae", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a neural vocoder based on denoising diffusion probabilistic model. It matches autoregressive neural vocoder in terms audio fidelity, but only requires constant number of sequential steps at synthesis.\n\nDetailed comments:\n\n1, The first key contribution is not convincing. The proposed method is a direct application of Ho et al. (2020) for waveform synthesis. The combination/connection between score matching and diffusion probabilistic model is a contribution in Ho et al. (2020).\n\n2, In general, flow-based models (e.g., WaveGlow, WaveFlow) can also match autoregressive model in terms of audio fidelity, while being much faster at synthesis (e.g., >20 times faster than real-time). What is the advantage of using WaveGrad as a neural vocoder? It provides comparable audio quality as flow-based models, but it is slower at synthesis (e.g., up to 5x faster than real-time). BTW, the authors may discuss WaveFlow (a SOTA flow-based model), which also introduces a trade-off between autoregressive and non-autoregressive models. It also needs constant number of sequential steps at synthesis. \n\n3, Section 2 is not well written and self-contained. For example, the authors simply list Equation (2-4) and references without further explanation. Indeed, they are not closely related to the technical contribution of this paper. One may either introduce in more detail, or skip some content as this work focuses on a particular parameterization of diffusion probabilistic model introduced by Ho et al. (2020).\n\nPros:\n- Solid technical contribution.\n- Good empirical results.\n\nCons:\n- It's unclear to me whether WaveGrad is a promising neural vocoder (see my comment 2).\n- Paper writing can be improved.\n\nI would like to raise my rating if my concerns are properly addressed.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers and myself agree that the contribution is clear, significant, and has enough originality. Hence, my recommendation is to ACCEPT the paper. As a brief summary, I highlight below some pros and cons that arose during the review and meta-review processes.

Pros:
- Solid technical contribution, specially the use of continuous noise levels.
- Clever application of diffusion/score-matching models to a new domain and task, with conditioning.
- Good empirical results, both objective and subjective.
- Listening samples provided.

Cons:
- Lack of formal comparison with flow-based vocoders.
- Potentially limited novelty.
- No official code available.

Note: Readers may also be interested in concurrent work https://openreview.net/forum?id=a-xFK8Ymz5J ("DiffWave: A Versatile Diffusion Model for Audio Synthesis").
Decision: Accept (Poster)