DiffWave: A Versatile Diffusion Model for Audio Synthesis
Authors: Zhifeng Kong
Wei Ping
Jiaji Huang
Kexin Zhao
Bryan Catanzaro
Venue: ICLR 2021 Oral (2021)
Keywords: diffusion probabilistic models
audio synthesis
speech synthesis
generative models

Abstract: In this work, we propose DiffWave, a versatile diffusion probabilistic model for conditional and unconditional waveform generation. The model is non-autoregressive, and converts the white noise signal into structured waveform through a Markov chain with a constant number of steps at synthesis. It is efficiently trained by optimizing a variant of variational bound on the data likelihood. DiffWave produces high-fidelity audios in different waveform generation tasks, including neural vocoding conditioned on mel spectrogram, class-conditional generation, and unconditional generation. We demonstrate that DiffWave matches a strong WaveNet vocoder in terms of speech quality (MOS: 4.44 versus 4.43), while synthesizing orders of magnitude faster. In particular, it significantly outperforms autoregressive and GAN-based waveform models in the challenging unconditional generation task in terms of audio quality and sample diversity from various automatic and human evaluations.

Published as a conference paper at ICLR 2021
DIFF A VERSATILE DIFFUSION MODEL FOR AUDIO S SYNTHESIS
Zhifeng Kong
Wei Ping
Computer Science and Engineering, UCSD 24kong@eng. ucsd. edu
NVIDIA wp wping@nviidia.com
Jiaji Huang, Kexin Zhao
Bryan Catanzaro NVIDIA
Baidu Research kexinzhao kexinniiai (huangjiaji,
@baidu. com
boatanzaro@tvidiaa.com
ABSTRACT
In this work, we propose DiffWave, versatile diffusion probabilistic model for con- ditional and unconditional waveform generation. The model is non-autoregressive, and converts the white noise signal into structured waveform through a Markov chain with a constant number of steps at synthesis. It is efficiently trained by optimizing a variant of variational bound on the data likelihood. DiffWave pro- duces high-fidelity audio in different waveform generation tasks, including neural vocoding conditioned on mel spectrogram, class-conditional generation, and un- conditional generation. We demonstrate that DiffWave matches strong WaveNet vocoder in terms of speech quality (MOS: 4.44 versus 4.43), while synthesizing orders of magnitude faster. In particular, it significantly outperforms autoregressive and GAN-based waveform models the challenging unconditional generation task in terms of audio quality and sample diversity from various automatic and human evaluations.
INTRODUCTION
Deep generative models have produced high-fidelity raw audio speech synthesis and music generation. In previous work, likelihood-based models, including autoregressive models (van den Oord et al., 2016; Kalchbrenner et al., 2018; Mehri et al., 2017) and flow-based models (Prenger et al., 2019; Ping et al., 2020; Kim et al., 2019), have predominated in audio synthesis because of the simple training objective and superior ability of modeling the fine details of waveform in real data. There other waveform models, which often require auxiliary losses for training, such as flow- models trained by distillation (van den Oord et al., 2018; Ping et al., 2019), variational auto-encoder (VAE) based model (Peng al., 2020), and generative adversarial network (GAN) based models (Kumar et al., 2019; Binkowski et al., 2020; Yamamoto et al., 2020).
Most of previous waveform models focus audio synthesis with informative local conditioner (e.g.. mel spectrogram or aligned linguistic features), with only a few exceptions for unconditional genera- tion (Mehri al., 2017: Donahue et al., 2019). It has been noticed that autoregressive models (e.g.. WaveNet) tend to generate made-up word-like sounds (van den Oord et al., 2016), or inferior sam- ples (Donahue et al., 2019) under unconditional settings. This is because very long sequences need to be generated (e.g.. 16,000 time-steps for one second speech) without any conditional information.
Diffusion probabilistic models (diffusion models for brevity) are class of promising generative models, which use a Markov chain to gradually convert simple distribution (e.g.. isotropic Gaussian) into complicated data distribution (Sohl-Dicksted et al., 2015; Goyal et al., 2017; Ho et al., 2020). Although the data likelihood is intractable, diffusion models can be efficiently trained by optimizing the variational lower bound (ELBO), Most recently, a certain parameterization has been shown suc- cessful in image synthesis (Ho et al., 2020), which is connected with denoising score matching (Song
"Contributed to the work during an internship at Baidu Research, USA. Audio samples in: https: /i ffwave-dmo.. g thub. io/

---

Published as a conference paper at ICLR 2021
Myy g (TT
(data(Fo))
g( (mn)
g(<2k<1)
diffusion process
zo
21
22
#T-1
DT
reverse process
po (0ol(2)))
po(21|022
polor- x Platent (DT)
Figure 1: The diffusion and reverse process in diffusion probabilistic models. The reverse process gradually converts the white noise signal into speech waveform through a Markov chain po
& Ermon, 2019). Diffusion models can use a diffusion (noise-adding) process without learnable parameters to obtain the "whitened" latents from training data. Therefore, no additional neural networks are required for training in contrast other models (e.g., the encoder in VAE (Kingma & Welling, 2014) or the discriminator in GAN (Goodfellow et al., 2014)). This avoids the challenging "posterior collapse" "mode collapse" issues stemming from the joint training of two networks, and hence is valuable for high-fidelity audio synthesis.
In this work, propose DiffWave, versatile diffusion probabilistic model for audio synthesis. DiffWave has several advantages over previous work: i) It is ron-autoregressive thus can synthesize high-dimensional waveform in parallel. ii) It flexible
it does not impose any architectural constraints in contrast to flow-based models, which need to keep the bijection between latents and data (e.g., see more analysis in Ping et al. (2020)). This leads mootp-f neural vocoders that still generate high-fidelity speech. iii) It uses single ELBO-based training objective without any auxiliary losses (e.g., spectrogram- that produces high-quality audio signals for both conditional and unconditional waveform generation.
losses) for high-fidelity synthesis. iv) It versatile model
Specifically, we make the following contributions:
1. DiffWave uses a by WaveNet (van den Oord et al., 2016). matches the strong WaveNet vocoder in terms of speech quality (MOS: 4.44 vs. 4.43), while synthesizing orders of magnitude faster as it only requires a few sequential steps (e.g.. 6) for generating very long waveforms.
and bidirectional dilated convolution architecture motivated
2. Our small DifeWave has 2.64M parameters and synthesizes 22.05 kHz high-fidelin speech (MOS: 4.37) more than 5x faster than real-time on a V100 GPU without engineered kernels. Although it is still slower than the sate-of-the-ae flow-bat bow-dased models (Ping et al., 2020; Prenger et al., 2019), it has much smaller footprint. We expect further speed-up by optimizing its inference mechanism in the future.
3. DiffWave significantly outperforms WaveGAN (Donahue et al., 2019) and WaveNer in the challenging unconditional and class-conditional waveform generation tasks in terms of audio quality and sample diversity measured by several automatic and human evaluations.
We organize the rest of the paper follows. We present the diffusion models in Section 2, and intro- duce DiffWave architecture in Section 3. Section 4 discusses related work. We report experimental results in Section 5 and conclude the paper Section
DIFFUSION PROBABILISTIE MODELS
We define qdata (Xo) as the data distribution on R.,, where L is the data dimension. Let my E R for t 0, 1.... I be sequence of variables with the same dimension, where t is the index for diffusion steps. Then, diffusion model T steps is composed two processes: the diffusion process, and the reverse process (Sohl-Dickstein al., 2015). Both of them are illustrated in Figure 1.
2

---

Published as a conference paper at ICLR 2021
Algorithm Training
Algorithm 2 Sampling
f 1ori=1, 2, .. Niter do Sample 20 Adata (O, 1), and f &~ Uniform([1.--. ,T}
Sample 2T ~ Platent = (0(
for t T,T 1,... do
Compute po( t) and (0 t) using Eq. (5)
Take gradient step on
Sample - Po (o .N (a (), ( 1)31) end for return 20
Velle eø To + vI Ojc, t)ll according to Eq. (7)
end for
The diffusion process is defined by a fixed Markov chain from data 2o to the latent variable znp
T II E=
g(01, ... "
where each of g(ze/xx=11 is fixed (Z, -I- B, 3,1) for small positive constant 3,. The function of g( (zz-1) is to add small Gaussian noise to the distribution of xt-1. The whole process gradually converts data z6 whitened latents øy according to variance schedule 31, ,8r.
The reverse process is defined by a Markov chain from IT to zo parameterized by 0:
T Platent (xT) = (O I), and po(xo, IT -1-1 ( = II
Po Dt 1(xe),
(2)
where Platent (IT) is isotropic Gaussian, and the transition probability po (rr) (0(2 as 11 (1 t), (), with shared parameter 8. Note that both (4y and dy take two inputs: the diffusion-step / E N, and variable 2; E TR2, ws outputs an I-dimensional vector as the mean, and og outputs real number the standard deviation. The goal of po(2)-1 (2x) is eliminate the Gaussian noise (i.e. denoise) added in the diffusion process.
is parameterized
Sampling: Given the reverse process, the generative procedure is first sample an FT (O I), and then sample _t-1 (o (xx) for = T, 1,... 1. The output 2o is the sampled data.
Training: The likelihood (T f po JT (xT) Platent (FT) drer is intractable to calculate in general. The model is thus trained by maximizing its variational lower bound (ELBO):
(Po. ... FT -T 1 Platent (XT)
E E (ma) log poppo (zo) = Eggaa (mr) log E
Blgar r "nr..roooo (mo)
(E11, ... .T (mo)
2 Eg(xa.... .er) log po poo. ... 2T (TT) Platent (IT)
ELBO.
((01 ... Fr (20)
Most recently, Ho et al. (2020) showed that under certain parameterizzaton, the ELBO of the diffusion model can be calculated in closed-form. This accelerates the computation and avoids Monte Carlo estimates, which have high variance. This parameterization is motivated by its connection to denoising score matching with Langevin dynamics (Song & Ermon, 2019; 2020). To introduce this parameterization, we first define some constants based on the variance schedule {B:] in the diffusion process as in Ho et al. (2020):
Br, āi = II ass 3, -=-1 Br for t> 1 and 8 = B Cr.
a = 1 1 111-,,
(4)
a=1
Then, the parameterizations μs and og are defined by
3r VI - - and og(24,)) =3²
po(2e, t) aan xz
(5)
where R8 x N R_ is neural network also taking x; and the diffusion-step i as inputs. Note
that on (Ze, t) is fixed to constant 3/ for every step under this parameterization. In the following proposition, explicitly provide the closed-form expression of the ELBO.
One find that g(=r( approaches isotropic Gaussian with large in Eq. (1 the Appendix A.
3

---

Published as a conference paper at ICLR 2021
Diflusion-step embedding
FC
Fully connected
Condl x1
Comv1x1
Com11x1
FC
Bi DilCom-22"
Bi-directional Dilated Conv (dilation 2"|
FC
Broadcast over length
Flement-wise addition
Bi DilCov-2 2
Clement-wise multiplication
ComdX1
Connect to next layer
Conditioner
residual layer
Canol Ne
Cunoixt
Skip connections
Convixt
Convixt
Henedaa Layeri
Residual layer
Output
Figure 2: The network architecture of DiffWave in modeling co RR -R.
Proposition (Ho et al., 2020) Suppose a series of fixed schedule BB. are given. Let e N0, I) and xo qdata Then, under the purameterization in Eq. (5), we huve
ELBO c Σ
KyLe ||e colvanzo -1- ay, t)]
(6)
8x for some constants and Kt, where Kit 20.(1-G,-) -1 for t >>> 1, and R1
Note that e irrelevant for optimization purpose. The key idea in the proof is to expand the ELBO into a sum of KL divergences between tractable Gaussian distributions, which have a closed-form expression. We refer the readers to look at Section A the Appendix for the full proof.
In addition, Ho et al. (2020) reported that minimizing the following unweighted variant of the ELBO leads to higher generation quality:
min Lunweighted (0) Exa,,r, ||e - -o (Vari + /1 Cie, 1)
where is uniformly taken from 1,...., Therefore, we also use this training objective in this paper. We summarize the training and sampling procedures in Algorithm 1 and 2, respectively.
Fast sampling: Given a trained model from Algorithm 1, we noticed that the most effective denoising steps at sampling occur near = 0 (see Section IV on demo website). This encourages to design a fast sampling algorithm with much fewer denoising steps Tinter (e.g., 6) than T at training (e.g.. 200). The key idea is to "collapse" the T-step reverse process into Stee process with carefully designed variance schedule. We provide the details in Appendix B.
DIFFWAVE ARCHITECTURE
In this section, we present the architecture of DiffWave (see Figure 2 for an illustration). We build the network éa RR N + I& in Eq. (5) based on a bidirectional dilated convolution architecture that is different from WaveNet (van den Oord et al., 2016), because there is no autoregressive generation constraint." The similar architecture has been applied for source separation (Rethage al., 2018 18; Lluís et al., 2018). The network is non-autoregressive, so generating an audio with length L from latents Øт requires 7 rounds of forward propagation, where T (e.g., 50) is much smaller than the waveform length L. The network is composed of stack of N residual layers with residual channels
Indeed, found the causal dilated convolution architecture leads to audio quality DiffWave.
4

---

Published as a conference paper at ICLR 2021
C. These layers grouped into blocks and each block has N layers. We use bidirectional dilated convolution (Bi-DilCony) with kernel size in each layer. The dilation is doubled at each layer within each block, i.e., [1, 2, 4. 4.... ___ We sum the skip connections from all residual layers as in WaveNet. More details including the tensor shapes are included in Section C in the Appendix.
3.1 DIFFUSION- EMBEDDING
is important to include the diffusion-step as part of the input, as the model needs to output different ca t) for different t. We use an 128-dimensional encoding vector for each (Vaswani et al., 2017);
fembedding sin [10 0s t).... sin (10 0ax4
t). cos (10 -0 t)
cos (10
(8)
We then apply three fully connected (FC) layers on the encoding, where the first two FCs share parameters among all residual layers. The last residual-layer-speer second FC into a C-dimensional embedding vector. We next broadcast this embedding vector over length and add it to the input of every residual layer.
FC maps the output of the
3.2 CONDITIONAL GENER ATION
Local conditioner: In speech synthesis, a neural vocoder can synthesize the waveform conditioned on the aligned linguistic features (van den Oord et al., 2016; Anık al., 2017b), the mel spectrogram Shen et al., 201 2018), or the hidden states within the text-to-wave architecture (Ping et al., 2019; Donahue et al., 2020). In this work, we test DiffWave as a neural vocoder conditioned on mel spectrogram. We first upsample the mel spectrogram to the same length as waveform through transposed 2-D convolutions. After a layer-specific Conv mapping its mel-band into 2C' channels, the conditioner is added as bias term for the dilated convolution in each residual layer. The hyperparameters can be found in Section 5.1.
from a text-to-spectioe -pptttt-ppect model (Ping et al., 2018;
Global conditioner: In many generative tasks, the conditional information is given by global discrete labels (e.g.. speaker IDs or word IDs). We use shared embeddings with dimension diabel = 128 in all experiments. In each residual layer, we apply a layer-specific Convl x 1 map diabel to 2C channels, and add the embedding as a bias term after the dilated convolution in each residual layer.
3.3 UNCONDITIONAL GENERATION
In unconditional generation task, the model needs to generate consistent utterances without conditional information. It is important for the output units of the network to have receptive field size (denoted as r) larger than the length L of the utterance. Indeed, we need r 2 2L, thus the left and right-most output units have receptive fields covering the whole L-dimensional inputs illustrated in Figure 4 in Appendix. This posts a challenge for architecture design even with the dilated convolutions.
For a stack of dilated convolution layers, the receptive field size of the output is up to: (k: - 1) E; dj + 1, where & is the kernel size and d, is the dilation at i-th residual layer. For example, 30-layer dilated convolution has a receptive field size r 6139, with k 3 and dilation cycle [1, .... .51]]. This only amounts to 0.38s of 16kHz audio. We can further increase the number of layers and the size of dilation cycles; however, we found degraded quality with deeper layers and larger dilation cycles. This is particularly true for WaveNet. In fact, previous study (Shen et al., 2018) suggests that even a moderate large receptive field size (e.g., 6139) not effectively used in WaveNet and it tends to focus on much shorter context (e.g., 500). DiffWave has an advantage in enlarging the receptive fields of output zo: by iterating from Fr to Tip in the reverse process, the receptive field size can be increased up to T x r, which makes Wave suitable for unconditional generation.
RELATED WORK
In the past years, many neural text-to-speech (TTS) systems have been introduced. An incomplete list includes WaveNet (van den Oord et al., 2016). Deep Voice 1 & 2 & 3 (Arik et al., 20177;bb Ping et al., 2018). 1). Tacotron & 2 (Wang et al., 2017; Shen et al., 2018), Char2War (Sotelo et al., 2017), VoiceLoop (Taigman et al., 2018), Parallel WaveNet (van den Oord et al., 2018), WaveRNN (Kalch- brenner et al., 2018), ClariNet (Ping et al., 2019). ParaNet (Peng et al., 2020), FastSpeech (Ren et al., 2019), GAN-TTS (Biákowski et al., 2020), and Flowtron (Valle al., 2020). These systems first
5

---

Published as a conference paper at ICLR 2021
generate intermediate representations (e.g.. aligned linguistic features, mel spectrogram, hidden representations) conditioned on text, then use a neural vocoder to synthesize the raw waveform.
Neural vocoder plays the most important role in the recent success of speech synthesis. Autoregressive models like WaveNet and WaveRNN can generate high- fideliiy speech, but in a sequential way of generation. Parallel WaveNet and ClariNet distill parallel flow-based models from WaveNet, thus can synthesize waveform in parallel. In contrast, WaveFlow (Ping et al., 2020), WaveGGlww (Prenger et al., 2019) and FloWaveNet (Kim et al., 2019) are trained by maximizing likelihood. There are other waveform models, such as VA VAE-based models (Peng et al., 2020), GAN-based models (Kumar et al., 2019; Yamamoto et al., 2020; Binkowski et al.. 2020), and neural signal processing models (Wang et al., 2019; Engel et al., 2020: Ai & Ling. 2020). In contrast to likelihood-bart
models, they often require auxiliary training losses to improve the audio fidelity. The proposed DiffWave another promising neural vocoder synthesizing the best quality of speech with a single objective function.
Unconditional generation of audio in the time domain a challenging task in general. Likelihood- based models are forced to learn all possible variations within the dataset without any conditional information, which can be quite difficult with limited model capacity. produce made-up word-like sounds or inferior samples (van den Oord et al., 2016; Donahue et al., 2019). VQ- VAE (van den Oord et al., 2017) circumvents this issue by compressing the waveform into compact latent code, and training an autoregressive model in latent domain. GAN-based models are believed to be suitable for unconditional generation (e.g., Donahue et al., 2019) due to the "mode seeking" behaviour and success in image domain (Brock et al., 2018). Note that unconditional generation of audio in the frequency domain is considered easier, as the spectrogram is much shorter 200 than waveform (Vasquez& & Lewis, 2019; Engel et al., 2019; Palkama et al., 2020).
practice, these models
In this work, we demonstrate the superior performance of DiffWave in unconditional generation of waveform. In contrast to the exact-likelihood models, DiffWave maximizes a variational lower bound of the likelihood, which can focus on the major variations within the data and alleviate the requirements for model capacity. In contrast to GAN or VAE-based models (Donahue al., 2019; Peng et al., 2020), it is much easier to train without mode collapse, posterior collapse, or training instability stemming from the joint training of two networks. There a concurrent work (Chen et al., 2020) that uses diffusion probabilistic models for waveform generation. In contrast to DifeWave, it uses a neural architecture similar to GAN-TTS and focuses on the neural vocoding task only. Our DiffWave vocoder has much fewer parameters than WaveGrad 2.64MM vs. 15M for Base models and 6.91M vs. 23M for Large models. The small memory footprint is preferred in production TTS systems, especially for on-device deployment. In addition, Wave requires smaller batch size (16 vs. 256) and fewer computational resources for training.
5 EXPERIMENTS
We evaluate DiffWave neural vocoding, unconditional and class-conditional generation tasks.
5.1
NEURAL VOCODING
Data:
We use the LJ speech dataset (Ito, 2017) that contains ~22 hours of audio recorded in home environment with sampling rate of 22.05 kHz, contains 13,100 utterances from a female speaker.
Models: We compare DiffWave with several state-of-the-ar neural vocoders, including WaveNet, ClariNet, WaveGlow and WaveFlow. Details of baseline models can be found in the original papers. Their hyperparameters can be found in Table 1. Our DiffWave models have 30 residual layers, kernel size 3. and dilation cycle [1. 2. 512). We compare DiffWave models with different number of
diffusion steps T E [20, 40, 50, 200} and residual channels C' E (64, 12 128). We use linear spaced schedule for 3. E [1 10 0.02] for DiffWave with T = 200, and Br E [1 x 10-1, 0.005 for DiffWave with T < 50. The reason to increase 3g for smaller I' is to make g (zo) close to Platent- In addition, we compare the fast sampling algorithm with smaller Tinter (see Appendix B), denoted as DiffWave (Fast), with the regular sampling (Algorithm 2). Both of them the same trained models.
We use the 80-band mel spectrogram of the original audio as the conditioner to test these neural vocoders in previous work (Ping al., 2019; Prenger et al., 2019; Kim et al., 2019). We set FFT size to 1024, hop size to 256, and window size to 1024. We upsample the mel spectrogram 256 times by applying two layers of transposed 2-D convolution (in time and frequency) interleaved
Conditioner:

---

Published as a conference paper at ICLR 2021
Table I: The model hyperparameters, model footprint, and 5-scale Mean Opinion Score (MOS) with 95% confidence intervals for WaveNet, ClariNet, WaveFlow, WaveGlow and the proposed Wave on the neural vocoding task. means the number is the higher the better, and + means the number is the lower the better.
Model
T
Tinfer
layers
channels
#param L)
MOS(C)
WaveNet
30
128
4.57M
4 + 0.10
ClariNet
60
64
2.17M
4.09
WaveGlow
96
256
87.88M
4.32
WaveFlow
64
64
5.91M
0.11
WaveFlow
64
128
22.25M
4.00
Wave BASE
20
20
30
64
2.64M
4.31
Dave BASE
40
40
30
64
2.64M
#.10
Dave BASE
50
50
30
64
2.64M
4.38 0.08
DaveWave LARGE
200
200
30
128
6.91M
4.44 0.07
DiveWave BASE (Fast)
50
6
30
64
2.64M
4.37 0.07
DiffWave LARGE (Fast)
200
30
128
6.91M
4.42 - 0.09
Ground-truth
4.52 0.06
with leaky ReLU (a = 0.4). For each layer, the upsamling stride time is and 2-D filter sizes are (32,3]. After upsampling, we a layer-specific Convl x map the 80 mel bands into 2 residual channels, then add the conditioner as bias term for the dilated convolution before the gated-tanh nonlincarities in each residual layer.
Training: We train DiffWave on Nvidia 2080Ti GPUs using random short audio clips of 16,000 samples from each utterance. We use Adam optimizer (Kingma & Ba, 2015) with batch size of 16 and learning rate x 10-4. We train all DiffWave models for 1M steps. For other models, we follow the training setups in the original papers.
We use the crowdMOS tookit (Ribeiro et al., 2011) for speech quality evaluation, where the test utterances from all models
Results:
presented to Mechanical Turk workers. We report the 5-scale Mean Opinion Scores (MOS), and model footprints in Table
Our DiffWave larGe model with residual channels 128 matches the strong WaveNet vocoder in terms of speech quality (MOS: 4.44 vs. 4,43). The WiffWave BASE with residual channels 64 also generates high quality speech (e.g., MOS: 4.35) even with small number of diffusion steps (e.g., T 40 or 20). For synthesis speed, DiffWave BASE (T 20) in FP32 generates audio 2.1 x faster than real sime and DiffWave BASE (T 40) in FP32 1.1 X faster than real-time on Nvidia V100 GPU without engineering optimization. Meanwhile, DiffWave BASE (Fast) and Diff LARGE (Fast) can be 5.6x and 3.5x faster than real- time respectively and still obtain good audio fidelity. contrast, a WaveNet implementation can be 500x slower than real-time synthesis without engineered kernels. DiffWave is still slower than the state-of-the-ee flow-based models (c.g., a 5.91M WaveFlow is > 40x faster than real-time in FP1 but has smaller footprint and slightly better quality. Because DiffWave does not impose any architectural constraints as in flow- based models, we expect further speed-up up by optimizing the architecture and inference mechanism in the future.
5.2 UNCONDITIONAL GENERATION
In this section, we apply DiffWave to an unconditional generation task based on raw waveform only.
Data: We use the Speech Commands dataset (Warden, 2018), which contains many spoken words by thousands of speakers under various recording conditions including some very noisy environment. We select the subset that contains spoken digits (0~-)), which we call the SC09 dataset. The SC09 dataset contains 358 training utterances (~8..7 hours in total) by 2,032 speakers, where each audio has length equal to one second under sampling rate 16kHz. Therefore, the data dimension L is 16,000. Note that the SC09 dataset exhibits various variations (e.g., contents, speakers, speech rate, recording conditions); the generative models need to model them without any conditional information.
Models: We compare DiffWave with WaveNet and WaveGAN, We also tried remove the mel conditioner in a sfate-of-the-an aht GAN-based neural vocoder (Yamamoto et al., 2020), but found it could
'The MOS evaluation for DifaWave(Fast) with Timer 6 was done after paper submission and may not be directly comparable previous scores.
7

---

Published as a conference paper at ICLR 2021
Table 2: The automatic evaluation metries (FID, IS, mls, AM, and NDB/K), K), and 5-scale MOS with 95% confidence intervals for WaveNet, WaveGAN, and WiffWave on the unconditional generation
task. means the number is the higher the better, and + means the number is the lower the better.
Model
FID(C)
IS(1)
ml)(t)
AM())
NDB/ (()
MOS(1)
WaveNet- 128
3.279
2.54
7.6
1.368
0.86
1.34 0.29
WaveNet-256
2,947
2.84
10.0
1,260
0.86
1.43 - 0.30
WaveGAN
1.349
4.53
36.6
0.796
0.78
2.03 0.33
DiffWave
1.287
5.30
59,4
0.636
0.74
3.39 = 0.32
Trainset
0.000
8.48
281.4
0.164
0.00
Testset
0.011
8.47
275.2
0.166
0.10
3.72 = 0.28
Table 3: The automatic evaluation metrics (Accuracy, FID-class, IS, mlS), and 5-scale MOS with 95% confidence intervals for WaveNet and DiftWave on the class-conditional generation task.
Model
Accuracy(t)
FID-class(L)
IS(1)
mlS(1)
MOS(T)
WaveNet-
56.20%
7.876.12.469
3.29
15.8
1.46 + 0.30
WaveNet-256
60.70%
6.95412.114
3.46
18.9
1.58 + 0.36
DiftWave
91.20%
1.113:00569
6.63
117.4
3.50 0.31
DiffWave (deep & thin)
94.00 %
0.932-+0:450
6.92
133.8
3.44 + 0.36
Trainset
99.06%
0.000110.000
8,48
281.4
Testset
98.76%
0.04410.016
8.47
275.2
3.72 + 0.28
not generate intelligible speech this unconditional task. We use 30 layer-Wavelle models with residual channels 128 (denoted as WaveNet-1288 and 256 (denoted as WaveNet-256), respectively. We tried increase the size of the dilation cycle and the number of layers, but these modifications lead to worse quality. In particular, large dilation cycle (e.g., up to 2048) leads to unstable training. For WaveGAN, we use their pretrained model on Google Colab. We use 36-layer DiffWave model with kernel size 3 and dilation cycle 21, 2048). We set the number of diffusion steps T 200 and residual channels C' 256. We use linear spaced schedule for 3, E [1 x 10 0.02].
Training: We train WaveNet and DiffWave on 8 Nvidia 2080Ti GPUs using full utterances. We use Adam optimizer with a batch size of 16. For WaveNet, we set the initial learning rate as 1 x 10-3 and halve the learning rate every 200K iterations. For DiffWave, fix the learning rate to 2 10-4. We train WaveNet and DiffWave for IM steps.
Evaluation: For human evaluation, report the 5-scale MOS for speech quality similar to Section 5.1. To automatically evaluate the quality of generated audio samples, we train a ResNeXT classifier (Xie et al., 2017) on the SC09 dataset according an open repository (Xu & Tuguldur, 2017). The classifier achieves 99.06% accuracy on the trainset and 06%76% accuracy on the testset. We use the following evaluation metries based on the 1024-dimensional feature vector and the 10-dimensional logits from the ResNeXT classifier (see Section D in the Appendix for the detailed definitions):
Fréchet Inception Distance (FID) (Heusel et al., 2017) measures both quality and diversity of generated samples, and favors generators that match moments in the feature space.
Inception Score (IS) (Salimans et al., 2016) measures both quality and diversity gener- ated samples, and favors generated samples that be clearly determined by the classifier.
Modified Inception Score (mIS) (Gurumurthy al., 2017) measures the within-class diversity of samples in addition to IS.
AM Score (Zhou et al., 2017) takes into consideration the marginal label distribution of training data compared to IS.
Number of Statistically-Diffeeen Bins (NDB) (Richardson & Weiss, 2018) measures diversity of generated samples.
Results: We randomly generate 1,000 audio samples from each model for evaluation. We report results in Table 2. Our DiffWave model outperforms baseline models under all metrics, including both automatic and human evaluation. Notably, the quality of audio samples generated by DiffWave is much higher than WaveNet and WaveGAN baselines (MOS: 3.39 vs. 1.43 and 2.03). Note that the quality of ground-truth audio exhibits large variations. The automatic evaluation metrics also indicate that DiffWave is better at quality, diversity, and matching marginal label distribution of training data.
8

---

Published as a conference paper at ICLR 2021
5.3 CLASSICONDITIONAL GENERATION
In this section, we provide the digit labels as the conditioner in DiffWave and compare our model to WaveNet. We omit the comparison with conditional WaveGAN due to its noisy output audio (Lee et al., 2018). For both DiffWave and WaveNet, the label conditioner is added to the model according to Section 3.2. We use the same dataset, model hyperparameters, and training settings as in Section 5.2.
We use slightly different automatic evaluation methods in this section because audio samples are generated according to pre-specified discrete labels. The AM score and NDB are removed because they are less meaningful when the prior label distribution of generated data is specified. We keep IS and mIS because favors sharp. clear samples and mIS measures within-class diversity. We modify FID to FIDD-laas: for each digif from 0 to 9, we compute FID) between the generated audio samples that are pre-specified as this digit and training utterances with the same digit labels, and report the mean and standard deviation of these ten FID scores. We also report classification accuracy based on the ResNeXT classifier used in Section 5.2.
Evaluation:
Results: We randomly generate 100 audio samples for each digit (0 to 9) from all models for evaluation. We report results in Table 3. Our DiffWave model significantly outperforms WaveNet on all evaluation metrics. It produces superior quality than WaveNet (MOS: 3.50 vs. 1.58), and greatly decreases the gap to ground between WaveNet and ground-truth). The automatic evaluation metries indicate that DiffWave is much better at speech clarity (> 91% accuracy) and within-class diversity (its mIS is 6x higher than WaveNet). We additionally found a deep and thin version of DiffWave with residual channels C' 128 and 48 residual layers can achieve slightly better accuracy but lower audio quality. One may also compare quality of generated audio samples between conditional and unconditional generation based on IS, mlS, and MOS. For both WaveNet and DiffWave, IS increases by >> -0%, mIS almost doubles, and MOS increases by 2 0.11. These results indicate that the digit labels reduces the difficulty of the generative task and helps improving the generation quality of WaveNet and DiffWave.
(the gap between DiffWave and ground-truth is 100% of the gap
5.4 ADDITIONAL RESULTS
The unconditional DiffWave model can readily perform speech denois- ing. The SC09 dataset provides six types of noises for data augmentation in recognition tasks: white noise, pink noise, rumning tap, exercise bike, dude miaowing, and doing the dishes. These noises are not used during the training phase of our unconditional WiffWave in Section 5.2. We add 10% of each type of noise test data, feed these noisy utterances into the reverse process at 25, and then obtain the outputs Zo 's. The audio samples are in Section V on the demo website. Note that our model is not trained on denoising task and has zero knowledge about any noise type other than the white noise added in diffusion process. It indicates DiffWave learns a good prior of raw audio.
Zero-shot speech denoising:
Interpolation in latent space: We can do interpolation with the digit conditioned DiffWave model in Section 5.3 on the SC09 dataset. The interpolation of voices xr. 2. between two speakers a, d is done in the latent space at = 50. We first sample x ~ ((xz|=z)) and x ~ q(2; (a)) for the two speakers. We then do linear interpolation between x9 and z 27 = (1 Ajæt + Axy for 0 < X < 1. Finally, we sample z ~ Po ( The audio samples are in Section VI on the demo website.
6 CONCLUSION
In this paper, we present DiffWave, a versatile generative model for raw waveform. In the neural vocoding task, it readily models the fine details of waveform conditioned on mel spectrogram and matches the strong autoregressive neural vocoder in terms of speech quality. In unconditional and class-conditional generation tasks, properly captures the large variations within the data and produces realistic voices and consistent word-level pronunciations. To the best of our knowledge, DiffWave the first waveform model that exhibits such versatility. DiffWave raises a number of open problems and provides broad opportunities for future research. For example, it would be meaningful to push the model to generate longer utterances, as DiffWave potentially has very large receptive fields. Second, optimizing the inference speed would be beneficial for applying the model in production TTS, because DiffWave is still slower than flow-based models. We found the most effective denoising steps in the reverse process occur near Zu, which suggests an smaller T is possible in DiffWave. In addition, the model parameters 9 are shared across the reverse process. so the persistent kernels that stash the parameters on-chip would largely speed-up inference on GPUs (Diamos et al., 2016).

---

Published as a conference paper at ICLR 2021
REFERENCES
Yang Ai and Zha Ling. A neural vocoder with hierarchical generation of amplitude and phase spectra for statistical parametric speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:839-851, 2020.
Sercan Ô. Arık, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky. Yongguo Kang, Xian Li, John Miller, Jonathan Raiman. Shubho Sengupta, and Mohammad Shoeybi. Deep Voice: Real-time neural text-to-speech. In ICMI., 201 7a.
Sercan Ô. Arik, Gregory Diamos, Andrew Gibiansky, John Miller, Kainan Peng, Wei Ping. Jonathan Raiman, and Yanqi Zhou. Deep Voice 2: Multi-speaker neural text-to-speech. NIPS, 2017b.
Mikotaj Birikowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks, In ICLR, 2020.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arxiv:/809.1 2018.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. WaveG- rad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00/1 2020.
Greg Diamos, Shubho Sengupta, Bryan Catanzaro, Mike Chrzanowski, Adam Coates, Erich Elsen, Jesse Engel, Awni Hannun, and Sanjeev Satheesh. Persistent mns: Stashing recurrent weights on-chip. In International Conference on Machine Learning, pp. 2024-2033, 2016.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In ICLR, 2019.
Jeff Donahue, Sander Dieleman, Mikotaj Binkowski, Erich Elsen, and Karen Simonyan. End-to-end adversarial text-to-speech. arXiv preprint (XXi::2006003575 2020.
Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. Gansynth: Adversarial neural audio synthesis. In ICLR, 2019.
Jesse Engel, Lamtharn Hantrakul, Chenjie Gu and Adam Roberts. Ddsp: Differentiable digital signal processing. arXiv preprint arXiv:2001.04644 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp. 2672-2680, 2014.
Anirudh Goyal Alias Parth Goyal, Nan Rosemary Ke, Surya Ganguli, and Yoshua Bengio. Variational walkback: Learning a transition operator stochastic recurrent net. In Advances in Neural Information Processing Systems, pp. 4392-4402, 2017.
Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 166-174, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 1 1239, 2020.
Keith Ito. The LJ speech dataset. 2017.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In JCML, 2018.
Sungwon Kim, Sang-gil Lee, Jongyoon Song, and Sungroh Yoon. FloWaveNet: A generative flow for raw audio. In /CML,,011
10

---

Published as a conference paper at ICLR 2021
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR, 2014.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. In Advances in Neural Information Processing Systems, pp. 14881-14899 2019.
Chae Young Lee, Anoop Toffy, Gue Jun Jung, and Woo-Jin Han. Conditional wavegan. arXiv preprint 1809.
2018.
Francesc Lluís, Jordi Pons, and Xavier Serra. End-to-end music source separation: is it possible in the waveform domain? arXiv preprint arXiv: /211://70../187,, 2018.
Soroush Mehri, Kundan Kumar, Ishaan Guirajani, Rithesh Kumar, Shubbam Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. SampleRNN: An unconditional end-to-end neural audio generation model. In ICLR, 2017.
Kasperi Palkama, Lauri Juvela, and Alexander Ilin. Conditional spoken digit generation with stylegan. In Interspeech, 2020.
Kainan Peng, Wei Ping, Zhao Song, and Kexin Zhao. Kon-autoreereesive neural text-to-speech. In ICML,, 2020.
Wei Ping, Kainan Peng. Andrew Gibiansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep Voice Scaling text-to-speech with convolutional sequence learning. In /CLR, 2018.
Wei Ping, Kainan Peng, and Jitong Chen. ClariNet: Parallel wave generation in end-to-end text-to- speech. ICLR, 2019.
Wei Ping, Kainan Peng. Kexin Zhao, and Zhao Song. WaveFlow: A compact flow-based model for raw audio. In ICML., 2020.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. WaveGlow: A flow-based generative network for speech synthesis. In IEEE International Conference on Acoustics, Speech and Signal Processing (/CASSP),201)
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. arXiv preprint (eXii:/905.092600 2019.
Dario Rethage Jordi Pons, and Xavier Serra. A wavenet for speech denoising. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5069-5073. IEEE, 2018.
Flávio Ribeiro, Dinei Florêncio, Cha Zhang, and Michael Seltzer. CrowdMOS: An approach for crowdsourcing mean opinion score studies. In ICASSP, 2011.
Eitan Richardson and Yair Weiss. On gans and gmms. In Advances in Neural Information Processing Systems, pp. 5847-5858, 2018.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, pp. 2234-2242, 2016.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan et al. Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. In ICASSP, 2018.
Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv preprint arXiv: / 503.03358, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11918-1192 2019.
11

---

Published as a conference paper at ICLR 2021
Yang Song and Stefano Ermon. Improved techniques for training score- basee generative models. arXiv preprint (rXiv:2006.09011, 2020.
Jose Sotelo, Soroush Mehri, Kundan Kumar, Joao Felipe Santos, Kyle Kastner, Aaron Courville, and Yoshua Bengio. Char2wav: End-to-end speech synthesis. ICLR workshop, 2017.
Yaniv Taigman, Lior Wolf, Adam Polyak, and Eliya Nachmani. VoiceLoop: Voice fitting and synthesis via phonological loop. In ICLR, 2018.
Rafael Valle, Kevin Shih, Ryan Prenger, and Bryan Catanzaro. Flowtron: autoregressive flow-based generative network for text-to-speech synthesis. arXiv preprint arXiv:2005.05957, 2020.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint wnXiv:1609.034999 2016.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances Neural Information Processing Systems, pp. 6306-6315, 2017.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George den Driessche, Edward Lockhart, Luis C Cobo, Florian Stimberg, et al. Parallel WaveNet: Fast high-fidelity speech synthesis. In JCML, 2018.
Sean Vasquez and Mike Lewis. Melnet: A generative model for audio in the frequency domain. arXiv preprint arXiv: 1906.0108,, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Xin Wang, Shinji Takaki, and Junichi Yamagishi. Neural source- fiuee--ilte--beeed waveform model for statistical parametric speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5916-5920. IEEE, 2019.
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly. Zongheng Yang. Ying Xiao, Zhifeng Chen, Samy Bengio, Quoe Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous. Tacotron: Towards end-to-end speech synthesis. In Interspeech, 2017.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv: 1804.0000403209, 2018
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 1492-1500, 2017
Yuan Xu and Erdene-Ochir Tuguldur. Convolutional neural networks for Google speech commands data set with PyTorch, 2017.
wwwoorcc-sppeccccoomeee
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. ICASSP 2020-208 2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6199 6203. IEEE, 2020.
Zhiming Zhou, Han Cai, Shu Rong, Yuxuan Song, Kan Ren, Weinan Zhang, Yong Yu, and Jun Wang. Activation maximization generative adversarial nets. arXiv preprint arXiv: 2017.
12

---

Published as a conference paper at ICLR 2021
A PROOF OF PROPOSITION 1
Proof.
We expand the ELBO Eq. (3) into the sum of sequence tractable KI. divergences below.
(o .T ( x Platent (XT)
ELBO Eg log
((01,-.. IT (20)
Po(
Bg
log Platent (TT)
Σ log a( (111((1-1)
log (o g(01 (m:)
log po (2 (111-112))) + log
(( (mo) )
Eg
log Platent (2T) (xT)
g(0; [Tc (0)
t=2
Platent (IT) popoo(tol ((T
po (o xx,e)
Eg
log
log
g [2t, zin)
t=2
T
-E
KL (1go
KL (g(z 20) (p log po (2rol (zzo (z))
Before we calculate these terms individually, we first derive q( (mo) and ( 12,, mu). Let c, 's be independent standard Gaussian random variables. Then, by definition of g and using the notations of constants introduced in Eq. (4), we have
Ft
VotIi-1 + VBcc
12 + V /aB VBee 14 + ya 11 V Var + vBee
Varzo + Vapa-1.... 02
Vanir-iE--1 BBee
Note that (()))20) is still Gaussian, and the mean of xı is Varzo, and the variance matrix is (0:ak-1-..0 31 ..... a,3,-1 + 8.) = (1 - L.).. Therefore, ( = -Nxx; Vāzzo, - ())
(10)
is worth mentioning that,
c x0) NeTr; Vārzo (1-āT)
(11)
where är = -1 B.) approaches zero with large T.
Next, by Bayes rule and Markov chain property, (1F,-1/2,,,00)
g( g(1x-1
(( (00)
N(rii / V0,51-1 1, B.T) N ((1 04 i.-1))
N(xri Vā,to.
(1 11
(2
$ {(2T(1 ))-1)))
(2iT(1 āt))
|
/apTi-11
| /ār
[2 - v a,
exp
23
2(1 a1-1)
2(1 (cz)
V
0 B
Var(l - 12)
(2 exp
Ft-1
20
23
Cz
Oz
Therefore,
Vare1 -
g mo)
N(
To
+
Biy B.T).
(12)
āe
1- 1-ar
Now, we calculate each term of the ELBO expansion in Eq. (9). The first constant term
Ey KL (g(2r (a) (xT))
KI, N(VāTzo -ar)
1 1 äT
Eza BzallVor22 0ll dl log vI- αT
2
GT
Exn | 2
d
(ār + log(1 1 -ār))
2
2
13

---

Published as a conference paper at ICLR 2021
Next, we compute KL 200 Gaussian with the same covariance matrix Be T, the KL divergence between them is times
Because both q zu) and po po (21-111 (22)
2/h
the squared & distance between their means. By the expression of g(0)(1o)) we have di von vI - âge. Therefore, we have
E KL (g(x) - 1 It; 20, (11
1
Exo
1
To +
var(1 04--1)
It vai
It
Be
co(
2
1 αe
1-
vI
ar
1
V 13
21 VI- aGe
vat Ō4-1)
1
Br
l|
Emoi
E
It.
23,
at
Vān
at
Vot
vI
āc
1
37
Exoie 110 I't -
ca (x)e 1)]?
23.
0+(1
att
87
Bzvie ||e - c0( t)1?
2 11 1 3,00(1
(12)
a-
1:
ExO.. ||c co(
20 - Ō(-1)
Finally, as 21 = Van V1 -ac= Vaizo + V1 anc, we have
31
Eg log (20o(21)
= Eg log.
T0:
31
co(21
(.11)
Vai
1
d
1
B1
12
Eg
log
2T
31
40
211
co (2) 1)
2,
v
/n
vI- -
1
1
B1
c 1) )|
log 2TB
E
Ezoie
20
(ai220 + vI -I-afe
2
2
vai
vI a1
vB
log
2T
E Ezo,c
(e - 50(
2
2
van
d
log 2T,31
E
Exzo.ele
lle
60(x1, 1)1l
2
201
The computation the ELBO now finished.
14

---

Published as a conference paper at ICLR 2021
B DETAILS OF THE FAST SAMPLING ALGORITHM
Let Tinter & T be the number of steps the reverse process (sampling) and {T) Tinter be the user- defined variance schedule, which can be independent with the training variance schedule { Then, compute the corresponding constants in the same way Eq. (4):
24 = 1 nt: 9 II 7s 7t 1 1- 1/s for t t> and i/2 = 1/- - 90
(13)
As step during sampling, we need to select an & and use ca(: () to eliminate noise. This is realized by aligning the noise levels from the user-defined and the training variance schedules. Ideally, want va, = VT.- However, since this is not always possible, we interpolate VT between two consecutive training noise levels v0 and vā,, if v between them. We therefore obtain the desired aligned diffusion step t, which denote is talign, via the following equation:
Vor Ts
aliga t
if is V0++1, v
(14)
Vā
Va++1
Note that, galign is floating-point number, which is different from the integer diffusion-step at training. Finally, the parameterizations of pe and og are defined in similar way Eq. (5):
fast (I Mg
I2 VI is
T
es (I, ( t ralien)) and oB fast (I 8) nT.
(15)
To
The fast sampling algorithm is summarized in Algorithm 3.
Algorithm 3 Fast Sampling
Sample TTinto Platent = (N, T)
for = Tinfer; Tinter - -1..... do Compute fast (I); and ffrtttt (Ia, 8) using Eq, (15) Sample fa-1 ~ NNx 15 ( end for return 20
(N 8), o fast (I.: 8)21
In neural vocoding task, we use user-defined variance schedules {0.0001, 0.001, 0.01, 0.05, 0.2, for DiffWave LARGE and {0.0001, 0.001, 0.01, 0.05, 0.2, 0.53 for DiffWave BASE in Section 5.1.
The fast sampling algorithm is similar to the sampling algorithm in Chen et al. (2020) in the sense of considering the noise levels controllable variable during sampling. However, the fast sampling algorithm for DiffWave does not need to modify the training procedure (Algorithm 1), and can just reuse the trained model checkpoint with large T.
15

---

Published as a conference paper at ICLR 2021
C
DETAILS OF THE MODEL ARCHITECTURE
Diffusion-step embedding
Input
(8.120)
(9.1.4)
Conv1x1
(9,512)
(.C.4)
(0,512)
FC
(0,C)
(0,C,4)
(0,C,4)
Bi-DilConv-22 mod
(0.20.
Conv1x1
(9.C.4)
Conditioner
(8,C,4)
(R.C.1)
(8,C,))
Skip connections
Com11x1
(8,0,1)
Canvl
Comvix1
Residual layer BI
(8.1.L)
Resicual layer
Output
Residual layer
Figure 3: The network architecture of WiveWave in modeling co (0,, t). including tensor shapes at each stage and activation functions. B is the batch size, C is the number of residual/skip channels of the network, and L is data dimension.
N-th Bi- Di-iilConv
Bi-DilConv input
receptive fields
L-dimensional data
Figure 4: The Receptive fields of the output units within DiffWave network.
16

---

Published as a conference paper at ICLR 2021
D DETAILS OF AUTOMATIC EVALUATION METRICS IN SECTION 5.2 AND 5.3
The automatic evaluation metrics used in Section 5.2 and 5.3 are described as follows. Given an input audio x, an 1024-dimensional feature vector (denoted Freature (ar) computed by the ResNeXT F, and is then transformed to the 10-dimensional multinomial distribution (denoted as p=(x)) with fully connected layer and a softmax layer. Let Xtrain be the trainset, Pgen be the distribution of generated data, and X Xgen Pgen Pgen(i.i.d. (i.i.d.) be the set of generated audio samples. Then, we compute the following automatic evaluation metrics:
Fréchet Inception Distance (FID) (Heusel et al., 2017) computes the Wasserstein-22 dis- tance between Gaussians fitted to Freature (Xtrain) and Feetuue (Xgen). That is,
FID
10
pell + Tr (2 + En 2(EEEE,)
where pt, E, are the mean vector and covariance matrix of Fleature (Xrrai)) and where Mg. Eg are the mean vector and covariance matrix of Fleature (Xgen))
Inception Score (IS) (Salimans et al., 2016) computes the following:
TS exp (Bz-ogen KI. (Pa
Pr (F(
where Ey' P5 (z') is the marginal label distribution.
Modified Inception Score (mIS) (Gurumurthy et al., 2017) computes the following:
mIS exp (Ez,z'~Poon KL (p=(()))))))
AM Score (Zhou et al., 2017) computes the following:
AM KL (En'~ Pr
Pxx Prt") Eu-ponn H(p>(z)),
where H(-) computes the entropy. Compared IS, AM score takes into consideration the the prior distribution of (X
Number of Ktatisticall--Difeeees
Bins (NDB) (Richardson & Weiss, 2018): First, Xtrain is clustered into K bins by K-Means in the feature space (where K in our evaluation). Next, each sample in X Xgen is assigned to its nearest bin. Then, NDB is the number of bins that contain statistically different proportion of samples between training samples and generated samples.
17


Reviews:
[{"review_id": "u_hE0-YuxC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The Diffusion Probabilistic model is gaining popularity as a generative model. Diffwave explores the same for speech synthesis tasks. They show very good results i.e. matching autoregressive Wavenet on the conditional and outperforming baselines on the unconditional audio waveform synthesis tasks.\n\nThe paper is clearly written, and easy to follow. The work does not provide any novel machine learning or generative modeling insights. However, the work is significant for speech synthesis applications since it shows great results, with a small foot-print network with a very new method. This work can be expected to spark a plethora of follow-up works for speech synthesis and other real-valued time-series modeling tasks.\n\nPros:\n1. Very good results on conditional neural vocoding task.\n2. State-of-the-art results on unconditional speech synthesis task\n\nCons:\n1. Lack of novelty in terms of insights/approach\n2. Motivation and ablation-study for various design choices are missing\n\n*Further Comments*: An ablation study that establishes the impact of various hyper-parameters/components (e.g. choice of diffusion step embedding function, more detailed analysis of width/depth of the network, etc.) would help the readers get a lot more value out of the paper. A qualitative study of samples, specifically, pointing out any bias that underlies generative modeling via the diffusion process would be great.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UaanPP5rnWY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper develops a speech synthesis model using denoising diffusion processes, a generative model framework recently demonstrated in image generation (Ho et al. 2020).    The application is straightforward and there is little if any theoretical difference from the Ho et al. paper.    I didn't check the proofs included in the appendix, but they along with the learning and sampling procedure seem to be already developed in (Ho et al. 2020).  The authors should take care to be very clear about the mathematical developments that are directly taken from the prior literature, and what developments are introduced in this paper.   \nNevertheless the experiments on this application is valuable, and makes significant progress on a problem that has proven surprisingly difficult to solve in an efficient way.  \nThe experiments and demos are convincing, and the results could be considered highly competitive in conditional generation and  state of the art for class-conditional and non-conditional generation.  \n\nThe writing at times could use improvement.   In the abstract, line 1, \"we propose DiffWave, a versatile Diffusion probabilistic model for conditional and unconditional Waveform generation\".   I don't like this style of capitalizing things in a sentence that are not proper nouns.  If you want to introduce an abbreviation derived from a term or phrase, a widely accepted conventional method is to italicize the phrase and define the acronym the first time it is used, as in \"\\emph{diffusion waveform} (DiffWave) model\".   Later you have \"DiffWave produces high-fidelity audios in Different Waveform generation tasks\":  Why is \"Different Waveform\" capitalized?   DiffWave has already been defined relative to \"diffusion waveform\".   If this is supposed to be cute, it's not.   \nDefining the acronym in the abstract is OK, but not necessary.  In any case, you still have to define it again in the body of the paper, since the abstract is considered a standalone summary of your document. \nAlso \"audios\" is not a word.  Please use \"audio signals\". This is repeated throughout the paper. \nOther examples:  \n\"This avoids the ... issues *stemmed from the joint training\"   stemmed -->  stemming\n\" for generating very long waveform\" :  waveform --> waveforms\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LTDmB3vxjc0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a neural vocoder based on a diffusion probabilistic model. The model utilizes a fixed-length markov chain to convert between a latent uncorrelated Gaussian vector and a full-length observation. The conversion from observation to latent is fixed and amounts to adding noise at each step. The conversion from latent to observation reveals slightly more of the observation from the latent at each step via a sort of cancellation. This process is derived theoretically based on maximizing the variational lower bound (ELBO) of the model and follows Ho et al. (2020) who derived it for image generation. Thorough experiments show that the model produces high quality speech syntheses on the LJ dataset (MOS comparable to WaveNet and real speech) when conditionally synthesizing from the true mel spectrogram, while generating much more quickly than WaveNet. Perhaps more interesting and surprising, however, is that it generates very high quality and intelligible short utterances with no conditioning, and also admits to global conditioning, e.g., with a digit label.\n\nThe paper is very clearly written, with the description of the model going into sufficient detail in the main body of the paper for the reader to understand it without getting bogged down in the less immediately relevant details. The experiments are thorough and well executed, comparing with listening tests to many state of the art neural vocoders for the conditional task. It describes a thorough evaluation of the unconditional generation task, which is in general difficult, but in this case was constrained in such a way as to make it feasible and informative, using reasonable metrics that clearly show the advantages of the proposed approach. The literature review is thorough and comes at a point in the paper where the reader understands the proposed approach and can appreciate the nuances of the differences between the approaches. Audio samples are provided on a companion website and demonstrate the effectiveness of the approach along with some additional interesting properties of the model not even mentioned in the paper (denoising most impressively, interpolation between speakers is less convincing). \n\nThe paper has two minor weaknesses. First is that it does not make a clearer distinction between the concurrent work from Chen et al (2020) along similar lines, although presumably this paper was not released prior to submission of the current paper. An extended comparison would be welcome in a camera ready version of this paper. Second, that it doesn't explicitly state the real-time factor of WaveNet generation in the results discussion on page 6, which is presumably much smaller than 1. This section compares to WaveNet in terms of quality and WaveFlow in terms of speed, slightly being slightly worse in both comparisons, but better in the opposite, partially missing, comparisons.\n\nOverall, this paper makes a strong contribution to the field of neural vocoding and to the field of representation learning more generally for long-duration intricately structured signals (i.e., speech).", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gta7978HCx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Well-written paper with strong, well-presented results. The MOS results attain or surpass the best WaveNet results. The presentation is on the whole clear. \n\nI was a bit confused by the core model description at first. In particular, the index t on the samples x_t is not a time-index, correct? Rather, it's just a step in the diffusion process? At first I thought it was a time-index, and so the model seemed very much like an AR model, leaving me very confused.\n\nSome additional things I liked about the work:\n\nCompelling contrast with existing methods (WaveNet, VAE, GANs).\nUse of multiple metrics in the evaluation, 5 objective metrics in addition to MOS, e.g. Tables 2 & 3, with details on the metrics provided in an Appendix.\nUse of multiple models as reference models, WaveNet, WaveGlow, WaveFlow, Clarinet, WaveGAN, in addition to the proposed model.\nFocus on unconditional generation, which AIU has not received that much attention in the community\nWhere I am unsure is the originality of the work. I personally am not aware of the diffusion approach having been applied to TTS, but this is not my primary area of expertise. Obviously, if there is related work in TTS with diffusion models, this should be cited.\n\nAlso, what I don't see in the Conclusion is any discussion of the weaknesses and challenges for the model going forward. The paper would be strengthened by having a more balanced conclusion.\n\nSome caveats regarding my review:\n\nI am not familiar with the specific datasets used, so cannot fully appreciate the significance of the results reported.\nI did not check the math in detail; the notation overall seemed clear and consistent to me (though see my first set of comments).\n\nI have a few more specific comments.\n\nThroughout the paper, \"... audios ...\" : \"audio\" is not usually used as a plural noun. \n\nE.g. \"We randomly generate 1,000 audios\" --> \"We randomly generate 1,000 audio waveforms\"?\n\n\"Notably, the quality of audios ... \" --> \"Notably, the quality of audio ... \"\n\n\"Note that, the quality of ground-truth ...\": nit, no comma after \"that\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "8-3DEPdJmj", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors adapt the recent trend of work on denoising diffusion probablistic models to the task of conditional or unconditional waveform generation.\nUsing the same principles as in (Ho et al 2020), as well as a Wavenet-like non causal model, the authors provide state of the art results for both tasks, as evaluated on spoken digits dataset (for unconditional and conditional generation) and on the LJ speech dataset (for deep vocoding). \nThe model proposed is faster to evaluate than WaveNet, and has less parameters than WaveGlow. It achieves a slightly higher MOS than WaveFlow for a comparable model size. The generation speed is comparable to previous methods.\n\n## Review\n\nThe paper is clear, well structured and the authors provide many experiments to validate their approach.\nWhile serious, the paper does lack novelty, as the method is completely taken from (Ho et al. 2020). The architecture is similar to Wavenet, but non causal. Note that this exact same non causal wavenet architecture has already been used for source separation [Rethage et al. 2018, Lluis et al. 2019].\n\nOne limitation is that unconditional, or weakly conditioned generation (i.e. not conditioned on a mel-spectrogram) is only evaluated on single digit generation, which is relatively limite. While the samples shows an improvement over WaveNet, it seems the proposed architecture would still struggle to generate longer sequences, like an entire sentence for instance.\nIt would be interesting to add WaveGlow or WaveFlow to the SC09 comparison.\n\nOverall I recommend acceptance as the paper show that denoising diffusion process can be used for waveform generation, even though the paper does not bring further novelty.\n\nReferences:\nRethage et al. 2018: A Wavenet for Speech Denoising\nLluis et al. 2019: End-to-end music source separation: is it possible in the waveform domain?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I join all five reviewers in recommending acceptance.

There was some discussion about a comparison with WaveGrad (Chen et al., 2020), a contemporaneous work that explores a similar modelling approach for speech generation. While I agree that such a comparison is a useful addition to the manuscript, I do not think it is reasonable to request anything beyond an acknowledgement and citation of the work from the authors as a condition for acceptance. Further discussion and comparison experiments could be valuable, but I believe that should not factor into the final decision. My position is most similar to Reviewer 4's in this sense. The current version of the manuscript briefly discusses the differences between WaveGrad and DiffWave, which I think is more than sufficient. (As an aside, another difference potentially worth discussing is that the "noise schedule" for WaveGrad can be adapted at inference time, enabling a trade-off between inference speed and sample quality, which I believe is not possible for DiffWave in its current form.)

There was some debate about the weakly conditioned generation results; I believe they are a nice addition to the paper, although it would have been suitable for publication without them. They certainly do not detract from it, and might inspire further work in weakly conditioned audio generation (e.g. music). There were also concerns about the clarity of writing, which I believe the authors have addressed in the current version of the manuscript.

This work stands out because it applies a relatively fresh idea in generative modelling to a domain of great practical importance, which has long been dominated by traditional likelihood-based models, with compelling results. While this implies a limited degree of technical novelty, I do not think that is grounds for rejection, and in fact I would argue that making new ideas work well for practical problems is just as important.
Decision: Accept (Oral)