PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior
Authors: Sang-gil Lee
Heeseung Kim
Chaehun Shin
Xu Tan
Chang Liu
Qi Meng
Tao Qin
Wei Chen
Sungroh Yoon
Tie-Yan Liu
Venue: ICLR 2022 Poster (2022)
Keywords: diffusion-based model
generative model
speech synthesis

Abstract: Denoising diffusion probabilistic models have been recently proposed to generate high-quality samples by estimating the gradient of the data density. The framework assumes the prior noise as a standard Gaussian distribution, whereas the corresponding data distribution may be more complicated than the standard Gaussian distribution, which potentially introduces inefficiency in denoising the prior noise into the data sample because of the discrepancy between the data and the prior. In this paper, we propose PriorGrad to improve the efficiency of the conditional diffusion model (for example, a vocoder using a mel-spectrogram as the condition) by applying an adaptive prior derived from the data statistics based on the conditional information. We formulate the training and sampling procedures of PriorGrad and demonstrate the advantages of an adaptive prior through a theoretical analysis. Focusing on the audio domain, we consider the recently proposed diffusion-based audio generative models based on both the spectral and time domains and show that PriorGrad achieves faster convergence and superior performance, leading to an improved perceptual quality and tolerance to a smaller network capacity, and thereby demonstrating the efficiency of a data-dependent adaptive prior.

Published as a conference paper at ICLR 2022
PRIORGRAD: IMPROVING CONDITIONAL DENOISING DIFFUSION MODELS WITH DATA-DEPENDENT ADAP- TIVE PRIOR
Sang-gil Lee
Heeseung Kim!
Chachun Shin
Xu Tan
Chang Liu
Qi Menga
Tao Qin
Wei Chen
Sungroh 1.on
Tan Liu?
1 Data Science & Al Lab., Scoul National University
Microsoft Research Asia
3 AIIS, ASRI, INMC, ISRC, NSI, and Interdisciplinary Program in Artificial Intelligence, Scoul National University
[tkdr1ff202,, gmltmd789, chaehuny, sryoon] .rr (xxuta, changliu, meq, taoqin, wehe, tylin)emicrosoft.com
ABSTRACT
Denoising diffusion probabilistic models have been recently proposed to generate high-quality samples by estimating the gradient of the data density. The frame- work defines the prior noise as a standard Gaussian distribution, whereas the cor- responding data distribution may more complicated than the standard Gaussian distribution, which potentially introduces inefficiency in denoising the prior noise into the data sample because of the discrepancy between the data and the prior. this paper, we propose PriorGrad to improve the efficiency of the conditional diffu- sion model for speech synthesis (for example, a vocoder using mel--ppectrogram the condition) by applying adaptive prior derived from the data statistics based on the conditional information. We formulate the training and sampling pro- cedures of PriorGrad and demonstrate the advantages of an adaptive prior through a theoretical analysis. Focusing on the speech synthesis domain, we consider the recently proposed diffusion-based speech generative models based on both the spectral and time domains and show that PriorGrad achieves faster convergence and inference with superior performance, leading to an improved perceptual qual- ity and robustness to a smaller network capacity, and thereby demonstrating the efficiency of data-dependent adaptive prior.
INTRODUCTION
Deep generative models have been achieving rapid progress, by which deep neural networks ap- proximate the data distribution and synthesize realistic samples from the model. There wide range of this type of approach, ranging from autoregressive models (Oord et al., 2016a] gener- ative adversarial networks (Goodfellow et al.] 2014] Brock et al., 2019), variational autoencoders (Kingma & Welling 2013) Vahdat & Kautz 2020), and normalizing flows (Rezende & Mohamed, Kingma Dhariwal, 2018). Denoising diffusion probabilistic models (DDPMs) (Ho et al, 2020 and score matching (SM) (Song & Ermon 2019) are recently proposed categories that can be used to synthesize high-fide high-fideltyy samples with competitive or sometimes better quality than previous state-of-the-ar approaches. Consequently, there have been variety of applications based on DDPM or SM (Saharia et al.) 20011: Kawar et al. (02T). Speech synthesis is one of the most successful ap- plications, where the diffusion model can synthesize spectral time-domain audio conditioned text or spectral information, respectively, achieving a competitive quality but faster sampling (Chen)
2015
al. 2021 Kong al. 2021 Jeong et al. 2021 Lee & Han 202I) than autoregressive models (Oord et al. 2016b Kalchbrenner et al. 2018).
"Work done during an internship at Microsoft Research Asia 1 "Corrsponning Authors

---

Published as a conference paper at ICLR 2022
However, although the diffusion-based speech synthesis models have achieved high-quality speech audio generation, they exhibit potential inefficiency, which may necessitate advanced strategies. For example, the model suffers from a significantly slow convergence during training, and prohibitively large training computation time is required to learn the approximate reverse diffusion process. We investigate the diffuion-basee based models and observe the discrepancy between the real data distribution and the choice of the prior. Existing diffusion-based models define a standard Gaussian as the prior distribution and design non-parametric diffusion process that procedurally destroys the signal into the prior noise. The deep neural network is trained to approximate the reverse diffusion process by estimating the gradient of the data density. Although applying the standard Gaussian as the prior is simple without any assumptions on the target data, it also introduces inefficiency. For example, in time-domain waveform data, the signal has extremely high variability between different segments such as voiced and unvoiced parts. Jointly modeling the voiced and unvoiced segments with the same standard Gaussian prior may be difficult for the model to cover all modes of the data, leading to training inefficiencies and potentially spurious diffusion trajectories.
Given the previous reasoning, we assessed the following question: For a conditional diffusion-basee model, can we formulate more informative prior without incorporating additional computational or parameter complexity? To investigate this, we propose a simple yet effective method, called PriorGrad, that uses adaptive noise by directly computing the mean and variance for the forward dif- fusion process prior, based on the conditional information. Specifically, using a conditional speech synthesis model, we propose structuring the prior distribution based on the conditional data, such as a mel-spectrogram for the vocoder Chen et al. 2021; Kong et al., 2021) and a phoneme for the acoustic model (Jeong et al., 202I). By computing the statisties from the conditional data at the frame level (vocoder) or phoneme- level (acoustic model) granularity and mapping them as the mean and variance of the Gaussian prior, we can structure the noise that is similar to the target data distribution at an instance level, easing the burden of learning the reverse diffusion process.
We implemented PriorGrad based on the recently proposed diffusion- speech generative mod- els (Kong et al., [021]] Chen et al. 2021 Jeong et al.) (021), and conducted experiments on the LJSpeech (Ito Johnson| (017)) dataset. The experimental results demonstrate the benefits of Prior- Grad, such significantly faster model convergence during training, improved perceptual quality, and an improved tolerance to a reduction in network capacity. Our contributions are as follows:
To the best of our knowledge, our study is one of the first to systematically investigate the effect using a non-standard Gaussian distribution as the forward diffusion process prior to the conditional generative model.
Compared to previous non-parametric forward diffusion without any assumption, we show that the model performance is significantly improved with faster convergence by leveraging the conditional information as the adaptive prior.
We provide comprehensive empirical study and analysis of the diffusion model behavior in speech generative models, in both the spectral and waveform domains, and demonstrate the effectiveness of the method, such significantly accelerated inference and improved quality.
2 BACKGROUND
In this section, we describe the basic formulation of the diffusion- model and provide related studies, along with description of our contribution with PriorGrad.
Basic formulation Denoising diffusion probabilistic models (DDPM) (Ho et al.) 2020 are re- cently proposed deep generative models defined by two Markov chains: forward and reverse pro- cesses. The forward process procedurally destroys the data Xo into a standard Gaussian Xy, as follows:
T
((11T) Xn) = II [(xx/xx--1) q(x//=-1)== - / 3xx-1,,
where q(x, (x))) represents the transition probability at the f-th step using a user-defined noise schedule BB E {B1,..., Br). Thus, the noisy distribution of x, is the closed form of q(xx|xo)
2

---

Published as a conference paper at ICLR 2022
N(x; Vaxxu (1-āz) where a := 1- S1 =II Og. ((xx(xx) converges in distribution to the standard Gaussian (xy: 0,I) if ör is small enough based on carefully designed noise schedule. The reverse process that procedurally transforms the prior noise into data is defined as follows:
T
po(Xo:T) = p(xx) II po (x (x(x)-1(xe), po(xx-1(x)) = N (x 1), Ex(x,,))),
(2)
where p(xr) = (xT: O, 1) and po (x x+) corresponds to the reverse of the forward transition probability, parameterized using deep neural network. We define the evidence lower bound (ELBO) loss as the training objective of the reverse process:
L(0) B,
KL (g(x7 (pxx))(())(((()))))
Σ
log pa (xo|x1)
(3)
As shown in
Ho et al. 2020), ((xx Xx, xo) be represented by Bayes rule as follows:
9(x4-- X1, Xb) = (X f((x, Xo), B,I),
(4)
v Var- 1/3
Var(l ā
Xt; Br 1 (1t-1
(i( Xo)
Xo +
Be. ag
(5)
1 αe
By fixing p(xr) as a standard Gaussian, KL (x))ll rameterized. The original framework in Ho al. (0200 fixed Ex( () as a constant B& and set the standard Gaussian noise e the optimization target instead of jů, by reparameterizing Xo va, (x: - /I ā,E) from g(x, to minimize the second and third terms equation
becomes constant and is not pa-
Based on this setup, in Ho et al. (2020), the authors further demonstrated that we can drop the weighting factor of each term and use simplified training objective that provides higher sample quality:
B
- ELBO = C
E Ex,.
||e 60l c0lVarxo + v1 -1,, 0)12
(6)
20
simple (0) = Ee,x, [ (xx, t)]?
Related work Since the introduction of the DDPM, there have been variety of further studies
(Nichol Dhariwal 2021 Song et al. 2020). applications (Chen et al., 2021 Jeong et al.) 202T] Kong al. 202 Saharia al. 2021), and symbiosis of diffusion (Ho al. 2020 Sohl--iccsteei ct al. 20155 and score-bare based models (Song & Ermon 2019 2020) as a unified view with stochastic differential equations (SDEs) (Song al. (021). From application perspective, several condi-
tional generative models have been proposed. Waveform synthesis models (Chen et al., 2021 Kong al. (022I) are one of the major applications in which the diffusion model is trained to generate time-domain speech audio from the prior noise, conditioned on a mel-spectrogram. A diffusion- based decoder has also been applied to text-to-spectroggram generation models (Jeong et al., 202T Popov et al. 202T). PriorGrad focuses on improving the efficiency of training such methods from the perspective of conditional generative model. We investigate the potential inefficiency of the cur- rent methods which require unfeasibly large computing resources to train and generate high-quality samples.
Studies on formulating an informative prior distribution for deep generative model are not new, and there has been a variety of studies investigating a better prior, ranging from hand-crafted (Nalisnick) & Smyth| 2017] Tomezak & Welling 20188), autoregressive (Chen et al. 2017), vector quantization (Razavi et al. (219). prior encoder (Rezende & Mohamed 2013). and data-dependent approaches similar to (Li al. 2019). We tackle the problem of training inefficiency of diffusion- based models by crafting better priors in a data-dependent manner, where our method can provide better trajectory and reduce spurious modes, enabling more efficient training. Nachmani et al. (2011)
3

---

Published as a conference paper at ICLR 2022
condition
data
forward process (xx,(x,-1)
data-dependent prior
X1
X
XT
N (U Ec)
reverse process Po 1x
Figure High-level overview of the proposed method with the directed graphical model.
used Gamma distribution as the diffusion prior. Note that there has also been a concurrent study con- ducted on leveraging the prior distribution on the acoustic model, Grad-TTS (Popov et al. 202T), in which the effectiveness of using the mean-shifted Gaussian investigated. Unlike the method in Popov et al. (202I), which enforces the encoder output match the target mel-spectrogram by using an additional encoder loss, our approach augments the forward diffusion prior directly through data and the encoder has no restriction on latent feature representa- tions. In Popov et al. (2021), the forward diffusion prior is jointly trained and may induce additional overhead on convergence as the prior changes throughout the training, whereas our method provides guaranteed convergence through the fixed informative prior.
prior with the identity variance was
METHOD
We investigate the following intuitive argument: When we structure the informative prior noise closer the data distribution, can we improve the efficiency of the diffusion model? In this section, present general formulation of PriorGrad, describe training and sampling algorithms,, and pro- vide benefits of PriorGrad through a theoretical analysis. PriorGrad offers a generalized approach for the diffusion- based model with the non- non-standard Gaussian the prior and can be applied to variety applications.
3.1 GENERAL FORMULATION
In this section, we provide general formulation of the method regarding using the non-standard Gaussian ( Σ) as the forward diffusion prior. PriorGrad leverages the conditional data to directly compute instance- level approximate priors in an adaptive manner, and provides the ap- proximate prior as the forward diffusion target for both training and in- ference. Figure [ and 2 presents visual high-level overview. We take the same parameterization of μe and о as the original DDPM (Ho al. 2020) from Section 2 as follows:
P_
M10.1
Vi
Sampling
(x( Vac
X;
V1 āi t)) ox(xi, t) = 3
Bi
(8)
Figure 2: Illustrative de- scription of the diffusion trajectory with the non- standard Gaussian.
Assuming that have access optimal Gaussian (M,, Σ) the forward diffusion prior distribution, we have the following modified
ELBO:
Proposition 1 Let NO, E) and Xy Adata- Then, under the parameterization in Eq. equa- tion g the ELBO loss will be
ELBO C(E)
Σ - n) vI C,,,t))] 1,
t=1
8 for some constant C, where ||x|| = 20 for t > 1, and 7 201
The Proposition D is an extension the ELBO in Equation 6 with the non-standard Gaussian dis- tribution. Contrary to original DDPM, which used (O the prior without any assumption data, through Proposition [ we train the model with A(, E). whose mean and variance are ex- tracted from the data, as the prior for the forward process. See appendix for full derivation. We
4

---

Published as a conference paper at ICLR 2022
also drop 7 the simplified loss L ||c cx(x,, -) for training, following the previous work. Algorithms [ and describe the training and sampling procedures augmented by the data- dependent prior (jt, 2). Because computing the data-dependent prior is application-dependeer
in Section and 5 we describe how to compute such prior based on the conditional data on the given task.
Algorithm Training of PriorGrad
Algorithm Sampling of PriorGrad
repeat
(ju, Σ) data-dependent prior
(1,. Σ) data data-dependent prior
Sample XT (O, 2)) for | T,T - 11....... do
Sample X ~ Qdata E (0, Σ)
Sample / ~ (11
T1)
Sample (0, Σ) if 1; else z
x x: V
11) + arE
X;-1 vaz (x+ Vi-a: (x (21) +o,
[ C= || 100c-e(xx, (x, 1) --
Update the model parameter ! with VoL until converged
end for return Xo 4
3.2 THEORETICAL ANALYSIS
In this section, describe the theoretical benefits of PriorGrad. First, discuss about the simpli- fied modeling with the following proposition:
Proposition 2 Let Lip, E,x;; 0) denote the -ELBO loss in Proposition 1. Suppose that is linear function. Under the constraint that det(E) det((I, we have ming Li,, E, Xu; 0) S ming L(O, I, Xy;
The proposition shows that setting the prior whose covariance Σ aligns with the covariance of data Xy leads to a smaller loss if we use a linear function approximation for Es. This indicates that we can simple model to represent the mean of ( under the data-dependent prior, whereas need to use a complex model to achieve the same precision under prior with isotropic covariance. The condition det Σ) det(I) means that the two Gaussian priors have equal entropy, which is condition for a fair comparison.
Second, we discuss the convergence rate. The convergence of optimization depends the condition number of the Hessian matrix of the loss function (denoted as H) (Nesterov) 2003), that (H)) where Amax and Amin are the maximal and minimal eigenvalues of H, respectively. smaller condition number leads to a faster convergence rate. For L(p, E, xo; 0), the Hessian is calculated as
is, Amax(H]
0L des deh
Ocg a0
OL
8c 062
H =
a0
Ocg
Again, if we assume Eg linear function, we have H x I for L(u, E, 0) and H Σ + I for (O0, No; 0). It is clear that if we set the prior to be N(,, E), the condition number of H equals 1, achieving the smallest value of the condition number. Therefore, it can accelerate the convergence. Readers may refer to the appendix A.2 for more details.
4 APPLICATION TO VOCODER
In this section, we apply PriorGrad vocoder model, as visually described in Figure
4.1 PRIORGRAD FOR VOCODER
Our formulation of PriorGrad applied to vocoder is based on DiffWave (Kng et al. 202 where the model synthesizes time-domain waveform conditioned on a mel-spectrogram that contains a compact frequency feature representation of the data. Unlike the previous method that used N I), PriorGrad network is trained to estimate the noise e ~~ (O Σ) given the destroyed signal Va,X0 vT d4e. Same Kong et al. (2021), the network is also conditioned on the discretized
5

---

Published as a conference paper at ICLR 2022
forward process
condition
data-dependent prior
N ( Σe)
reverse process
Figure 3: Visual description of PriorGrad for vocoder.
index of the noise level vāz with the diffusion-stepp embedding layers, and the mel-spectrogram c with the conditional projection layers.
Based on the mel-spectroggrm condition, we propose leveraging a normalized frame-level energy of the mel-spectrogram for acquiring data-dependent e exploiting the fact that the spectral energy contains an exact correlation to the waveform variancd First, we compute the frame-level energy by applying roots of sum of exponential c, where is the mel-spectnogram from the training dataset. We then normalize the frame-level energy to a range of (0, 1] to acquire the data-dependent the frame-level energy
diagonal variance Ee. In this way, deviation for the waveform data we want to model. This can be considered as non-linear mapping between the mel-scale spectral energy and the standard deviation of the diagonal Gaussian.
proxy of the standard
We set (0, Σ) as the forward diffusion prior for each training step by upsampling Eg in the frame- level to Σ in the waveform-level using hop length for the given vocoder. We chose a zero-mean prior in this setup reflecting the fact that the waveform distribution is zero-mean. In practice, we imposed the minimum standard deviation of the prior to 0.1 through clipping to ensure numerical stability during training" We have tried several alternative sources of conditional information to compute the prior, such as voiced/unvoiced labels and phoneme-level statistics, but they resulted in worse performance. We justify our choice of using frame-level energy in the appendix A.3
4.2 EXPERIMENTAL SETUP
We used LISpeech (Ito & Johnson 2017) dataset for all experiments, which is a commonly used open-souree 24h speech dataset with 13,100 audio clips from single female speaker. We used an 80-band mel-spectrogram feature at the log scale from the 22,050Hz normme-normalized speech, with the 1024-point FFT, 80Hz and 7, 7,600Hz low- and high-frequency cutoff, and a hop length of 256. We used 13,000 clips as the training set, 5 clips as the validation set, and the remaining 95 clips the test set used for an objective and subjective audio quality evaluation. We followed the publicly available implementatiorf where it uses a 2.62M parameter model with an Adam optimizer Kingma & Ba 2014) and learning rate of 2 10-4 for total of 1M iterations. Training for 1M iterations took approximately 7 days with a single NVIDIA A40 GPU. We used the default diffusion steps with T 50 and the linear beta schedule ranging from 1 10-4 4 5 x 10-2 for training and inference, which is the default setting of DiffWave model used in Kong et al. (2021). We also used the fast Tinfer = 6 inference noise schedule from DiffWave B. BASE without modification.
4.3 EXPERIMENTAL RESULTS
We conducted experiments to verify whether our method can learn the reverse diffusion process faster, by comparing to the baseline DiffWave model with ~ NO, I). First, we present the model convergence result by using a spectral domain loss on the test set to show the fast training of Pri- orGrad. Second, provide both objective and subjective audio quality results, where PriorGrad offered the improved quality. Finally, we measure a tolerance to the reduction of the network ca- pacity, where PriorGrad exhibits improved parameter efficiency. We leave the description of the objective metrics we collected to the appendix A.4
'hii property about the spectral density is dictated by Parseval's 's theorem (toica et al. 2005
II our preliminary study, we applied grid search the minimum standard deviation from 0.1 0.0001, and it showed negligible difference as long as the minimum is clipped to the reasonably low value. Not clipping it entirely can result in numerical instability.
https: / thub. com/ 1mnt com /di ffwave

---

Published as a conference paper at ICLR 2022
Table 1: 5-scale subjective mean opinion (MOS) sults of PriorGrad vocoder with 95% confidence intervals.
0.60
DiffWave PriorGrad
Method
Tinter
Training Steps
500K
IM
0.58
GT
4.42 = 0.07
0.56
LS-MAA
6
3.98
0.08
4.01 + 0.08
DiffWave
0.54
50
4.12
0.08
4.12 + 0.08
0.52
PriorGrad
6
4.02
0.08
4.14 1 0.08
50
4.21
0.08
4.25 1 0.08
0.50
200K
800K Iteration
600K
Table 2: MOS results of PriorGrad vocoder under reduced model capacity, evaluated at 1M training step.
Figure 4: Model convergence result of vocoder models measured by log-mel spectrogram mean absolute error (LS- MAE).
Parameters
Method
Base (2.62M) Small (1.23M)
GT
4.38 + 0.08
DiffWave
4.06 0.08
3.90 + 0.09
PriorGrad
4.12 0.08
4.02 + 0.08
Table 3: Objective metric results of PriorGrad vocoder at 1M training steps. Lower is better.
Method
LS-MAF (4)
MR-STFT (L)
MCD (L)
Fg RMSE (L)
S(Zr, (4)
S(ão, zo) (4)
DiffWave
0.5264
1.0920 0.9976
9.7822
16.4035
72698.62
1650.22
PriorGrad
0.5048
9.2820
15.5542
42236.93
1608.89
Model convergence We used widely adopted spectral metric with log-mel spectrogram mean absolute error (LS-MAE) as the proxy of the convergence for the waveform synthesis model. We can see from Figure 4 that PriorGrad exhibited a significantly faster spectral convergence compared to the baseline. In the auditory test, we observed that PriorGrad readily removed the background white noise early in training, whereas the baseline needed to learn the entire reverse diffusion process starting from e ~ (O, I) which contains little information.
Table T shows the 5-scale subjective mean opinion score (MOS) test of PriorGrad from Amazon Me- chanical Turk. We observed that PriorGrad outperformed the baseline DiffWave model with 2 times fewer training iterations. PriorGrad also outperformed the baseline on objective speech and distance metrics as shown in Table 3 such as Multi-resolltion STFT error (MR-STFT) (Yamamoto et al.) 2020), Mel cepstral distortion (MCD) (Kubichek] 1993) and Fo root mean square error (F, RMSE) and a debiased Sinkhorn divergence (Feydy et al. (2019) between the prior and the ground-truth ())), or the generated samples and the ground-truth S (@0. zg), consistent with the subjective quality results. Refer to A.7 for comparison to other state-of-the-art approaches with varying number of inference steps.
Parameter efficiency We measured tolerance to reduced diffusion network capacity by set- ting the width of the dilated convolutional layers by half, leading to a smaller model with 1.23M parameters. We trained the small Wiff and PriorGrad with the same training configurations for 1M steps. With Tonferr = 50, Table 2 confirmed that the performance degradation of PriorGrad reduced compared the baseline DiffWave model. The small PriorGrad was able to maintain the quality close to the larger baseline model, suggesting that having access to the informative prior distribution can improve parameter efficiency of the diffusion-based generative model.
5 APPLICATION TO ACOUSTIC MODEL
In this section, we present the application of PriorGrad to the acoustic models, as visually described in Figure 5
7

---

Published as a conference paper at ICLR 2022
condition
forward process
In being comparatively modern
data-dependent prior
N (He:
reverse process
Figure 5: Visual description of PriorGrad for acoustic model.
5.1 PRIORGRAD FOR ACOUSTIC MODEL.
The acoustic models generate a mel-spectrogram given a sequence phonemes with the encoder- decoder architecture. In other words, the acoustic model analogous to the text-conditional im- age generation task (Reed et al., 2016 Ramesh et al. (021). where the target image is 2D mel- spectrogram. We implement the PriorGrad acoustic model by using the approach in Ren et al. (2200) as feed-forward phoneme encoder, and using diffusion based decoder with dilated con- volutional layers based on Kong al. (202I). Note that similar decoder architecture was used in Jeong et al. (202I).
To build the adaptive prior, we compute the phoneme-level statistics of the 80-band mel-spectrogram frames by aggregating the frames that correspond to the same phoneme from the training data, where phoneme-to-frame alignment provided by the Montreal forced alignment (MFA) toolkit (McAuliffe et al. (017). Specifically, for each phoneme, we acquire the 80-dimensional diagonal mean and variance from the aggregation of all occurrences in the training dataset. Then, we construct the dictionary of N(u, Σ) per phoneme. To use these statistics for the forward diffusion prior, need to upsample the phoneme-level prior sequence to the frame-level with the matching duration. This be done by jointly upsampling this phoneme-level prior as same the phoneme encoder output based on the duration predictor (Ren et al. 2019) module.
Following Algorithm D train the diffusion decoder network with the mean-shifted noisy mel- spectrogram X (x (4) -I-V O22E as the input. The network estimates the injected noise NO, Σ) as the target. The network additionally conditioned the aligned phoneme encoder output. The encoder output is added as bias term of each dilated convolution layers of the gated residual block with the layer-wise
convolution.
5.2 EXPERIMENTAL SETUP
For the acoustic model experiments, we implemented the Transformer phoneme encoder architec- ture identical to FastSpeech 2 (Ren et al. (2020) and the convolutional diffusion decoder architecture based on Jeong et al. (202T). We adopted the open-source implementation of the DiffWave architec- ture (Kong et al. 2021) with 12 convolutional layers, same Jeonn et al. (2021). We used the beta schedule with with grid search method (Chen et al. 2021). We followed the same training and inference proto- cols Ren et al. (2020) and used a pre-trained Parallel WaveGAN (PWG) (Yamamoto et al.) (020)) vocoder for our controlled experiments. For state-of-the-art comparison, we used HiFi-GAN (Kong et al. 2020) vocoder to match the previous work. We conducted a comparative study of PriorGrad acoustic model with different diffusion decoder network capacity, i.c., a small model with 3.5M parameters (128 residual channels), and large model with 10M parameters (256 residual channels). Training for 300K iterations took approximately 2 days on single NVIDIA P100 GPU. We leave the additional details in the appendix A.5
400 for training and used the fast reverse sampling schedule with Tinfer =
5.3 EXPERIMENTAL RESULTS
Model convergence We can see from Table that applyyin M(, Σ) with PriorGrad significantly accelerated the training convergence and exhibited higher-quality speech for different decoder ca- pacities and training iterations. A high-capacity (10M) PriorGrad was able to generate high-quality mel-spectrogram with as few as 60K training iteration, whereas the baseline model was not able to match the performance even after 300K iterations. This is because the training iterations were insufficient for the baseline to learn the entire diffusion trajectory with the standard Gaussian as the prior, leading to lower performance. The small (3.5M) baseline model exhibited improved
8

---

Published as a conference paper at ICLR 2022
Table 4: MOS results of PriorGrad acoustic model with 95% confidence interval. We used pre-
trained Parallel WaveGAN Yamamoto et al. 2020 for the vocoder.
Parameters (Decoder)
Training Steps
Method
60K
300K
GT
4.41 0.08
GT (Vocoder)
4.22 + 0.08
Baseline
10M
3.84 = 0.10
3.91 + 0.09
PriorGrad
10M
4.04 + 0.07
4.09 + 0.08
Baseline
3.5M
3.87 = 0.09
4.00 = 0.08
PriorGrad
3.5M
3.96 0.07
4.08 = 0.07
quality compared to the large baseline for the same training iterations, which further suggests that the convergence of the large baseline model is slow. By contrast, by easing the burden of learn- ing the diffusion process by having access to the phoneme-level informative forward prior, Prior- Grad achieved high-quality samples significantly earlier in the training. Refer Appendix [A.6 for additional experimental results including an alternative model with the jointly trainable diffusion prior estimation, and Appendix (A.7 for an expanded comparison to previous work, where PriorGrad achieves a new state-of-the-ee acoustic model.
Parameter efficiency Similar to the vocoder experiment, we assessed whether we can retain the high performance with a reduced diffusion decoder capacity from 10M to 3.5M parameters. The small PriorGrad also consistently outperformed the baseline model and achieved an almost identical perceptual quality the high-capacity PriorGrad. The small PriorGrad outperformed both the small and large baseline models, which confirms that PriorGrad offers an improved parameter efficiency and tolerance to the reduced network capacity. This suggests that PriorGrad provides way to build practical and efficient diffusion-batt based generative models. This holds a special importance for speech synthesis models, where the efficiency plays key role in model deployment in realistic environments.
DISCUSSION AND CONCLUSION
In this study, we investigated the potential inefficiency recently proposed diffusion-based model conditional generative model within the speech synthesis domain. Our method, PriorGrad, di- rectly leverages the rich conditional information and provides an instance-level non-standard adap- tive Gaussian the prior of the forward diffusion process. Through extensive experiments with the recently proposed diffusion-based speech generative models, we showed that PriorGrad achieves faster model convergence, better denoising of the white background noise, an improved percep- tual quality, and parameter efficiency. This enables the diffusion-based generative models to be significantly more practical, and real-world applications favor lightweight and efficient models for deployments.
Whereas our focus is on improving the diffusion-bassee speech generative models, PriorGrad has potential to expand the application beyond the speech synthesis domain. In the image domain, for example, PriorGrad can potentially be applied to image super-resolution tasks (Saharia et al.) 2021) by using the patch-wise mean and variance of the low-resolution image to dynamically control the forward/reverse process. The method could potentially be used for depth map conditional image synthesis (Esser et al.) 2020), where the depth map can be mapped as the prior variance using the fact that the monocular depth corresponds to the camera focus and visual fidelity between the foreground and background. We leave these potential avenue of PriorGrad for future work.
PriorGrad also has limitations that require further investigation. Although PriorGrad offers a vari- ety of practical benefits as presented, may also require well-thought out task- design to compute the data-dependent statistics (or its proxy), which may be unsuitable depending on the gran- ularity of the conditional information. Building advanced approach to enable generalized realization of PriorGrad will be an interesting area of future research.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work was supported by the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University 2021, Institute of Information & communi- cations Technology Planning & Evaluation (IITP) grant funded by the Korea govermeen((MSIT) AN0.2021-0-01344 Artificial Intelligence Graduate School Program (Seoul National University)], and the MSIT (Ministry of Science, ICT), Korea, under the High- Pigh-Potettial Individuals Global Train- ing Program (2020-0-01649) supervised by the ITTP (Institute for Information & Communications Technology Planning & Evaluation), and Microsoft Research Asia.
REFERENCES
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL
htttps://openrevier nnt/ htt.e//ewe..netttfoorln.ooe--... Forummild--
Arturo Camacho and John G Harris. A sawtooth waveform inspired pitch estimator for speech and music. The Journal of the Acoustical Society of America, 124(3): 1638 1652, 2008.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In International Conference on Learning Representations, 2017.
Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming transformers for high-resolution image synthesis. arXiv preprint SiIi:20022.08444 2020.
Jean Feydy, Thibault Séjourné, François-Xavies Vialard, Shun-ichi Amari, Alain Trouvé, and Gabriel Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2681-2690.
PMLR, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-266620 2014.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Asso-
ciates, Inc., 2020. URL httpps:/ neurips...... wttp:pps..cpppppr/2///1... 4c5bcfec858 waw.eea5584aa009665112
wbw.eb-aaaaapppeee pdf
Keith Ito and Linda Johnson. The lj speech dataset.
https://eeihitt..cm
SS-Speechh-Dataseee 2017.
Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and nam Soo Kim. Diff- TTS: Denoising Diffusion Model for Text-to-Speech. In Proc. Interspeech 2021, pp. 3605- 3609, 2021.
doi: 09224437/mmerspcch22221--69
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pp. 2410-2419. PMLR, 2018.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Stochastic image denoising by sampling from the posterior distribution. arXiv preprint arXiv:2101.09552,
2021. 2021.
Valentin Khrulkov and Ivan Oseledets. Understanding ddpm latent codes through optimal transport. arXiv preprint arXiv:2202.07477, 2022.
10

---

Published as a conference paper at ICLR 2022
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. arXiv preprint arXiv:2005.11129, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 44116698,, 2014.
Diederik P Kingma and Max Welling. 1321v:1312.611, 2013.
Auto-encoding variational bayes.
arXiv preprint
Durk Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolu- tions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL https: ptttps://proceeedingg neuripss....r....oooo.......r..m http:///psss.ccppppp//////////// 61136666222200222c11122222333333
Pap000-ppeepppp pdf
Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. arXiv preprint arXiv:2010.05646, 2020.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint 2tt1:22006000132. 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
Robert Kubichek. Mel-cepstral distance measure for objective speech quality assessment. In Pro- ceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing, volume 1, pp. 125-128. IEEE, 1993.
Junhyeok Lee and Seungu Han. Nu-wave: diffusion probabilistic model for neural audio upsam- pling, 2021.
Zuchao Li. Rui Wang, Kehai Chen, Masso Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao. Data-dependent gaussian prior objective for language generation. In International Confer- ence on Learning Representations, 2019.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pp. 498-502, 2017.
Eliya Nachmani, Robin San Roman, and Lior Wolf. Denoising diffusion gamma models. arXiv preprint 1rX.v:2110005944, 2021.
Eric Nalisnick and Padhraic Smyth. Stick-breaking variational autoencoders, In International Con- ference on Learning Representations, 2017.
Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2003.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, 2021.
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks.
In Maria Florina Balcan and Kilian Q. Weinberger (eds.). Proceedings of The 33rd Interna- tional Conference on Machine Learning. volume 48 of Proceedings of Machine Learning Re- search, pp. 1747-1756, New York, New York, USA, 20-22 Jun 2016a. PMLR. URL http: 7/proceed ngs. ml press v48/ oordl 6. html
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint
2016b.
Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song. Waveflow: A compact flow-based model for raw audio. In International Conference Machine Learning, pp. 7706-7716. 16. PMLR, 2020.

---

Published as a conference paper at ICLR 2022
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech, 2021.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3617-3621, IEEE, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2 /2092, 2021.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. arXiv preprint 1906: 2019.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International Conference on Machine Learning, pp. 1060-1069. PMLR, 2016.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech: Fast, robust and controllable text to speech. arXiv preprint
2019.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006. 04558, 2020.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows In Interna- tional Conference on Machine Learning, pp. 1530-1538. PMLR, 2015.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement, 2021.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Learning Representations, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- lional Conference on Learning Representations, October 2020. URL httpt:tt//aaiii...///// / abs/ abs/201 220200100 0. 02502
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Processing Systems, pp. 11895-11907, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Tien Lin (eds.), Advances in Neural Information Processing Systems, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021. URL https::/ net/
Forum??d-PXTIG122RR
Petre Stoica, Randolph L Moses, et al. Spectral analysis of signals. 2005.
Jakub Tomezak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelligence and Statistics, pp. 1214-1223. PMLR, 2018
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Infor- mation P Processing Systems (NeurIPS), 2020.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acousties, Speech and Signal Processing IEEE, 2020.
(ICASSP), pp.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 DERIVATION OF THE ELBO LOSS
Proposition 1 Let NO 2) and Xo Adata- Then, under the parameterization Eq. equa- tion 8 the ELBO loss will be
-EBBO C(2) + Σ Trlxxo.elee -xoVVrxXo 11)
vI di,et)ll
8 for t 1, and 7/1 2a for some constant C, where [ix|| = -x 71 20f01(1-00
Proof:
According to Algorithm 1, the input xy is normalized by subtracting its mean μ.. In the following, we denote Xn = XD and VoxXa + vI a.c.
According to Equation (3), the ELBO loss is
ELBO -E, (KL(g(>>\<o))ll<x>))) Eiz KL(g(Xx- xo) (x:)) log po (x (80lx1)
Let G'S tid. (0, E). Similarly the Equation (1), have (xxx(100) N(xi; vario (1 E)).
By Bayes rule and Markov chain property,
q(xi-1 (x,, xo) g(x,) x, ()((x)x11)))))))))))))) (x))
((x)
N(xxi Vaix 1,,,)) B,E
(i (x1- 100--iiaa-- //a-1800 (1 az- Σ))
N(xi v /xxo (1 2x)
-(2m3.)- exp
Xt-1 Vāl-
X0
2/3
Ctt
E
where ||x| --= xx Therefore,
V Vai(1 ãx-1)
a(x:-1 Xc, xo) N
Xt-li
Xo
Xe. 3
at
ae
Then, we calculate each term of the ELBO expansion. The first term
(KK = äT
d (ār + log(1 -ār)).
Ex,
(10)
2
2
The second term is
81
E Ex (0(
(11)
20,(1
-(-1)
Finally, we have
1
Eg (Xgoo()) log( 1) det(())
Ex..clle - Ex 1)
(12)
201
Combining Equation 10). and 12) together, we can get the result the Proposition.
A.2 THEORETICAL BENEFITS OF PRIORGRAD
Proposition 2 Let L(p, Xu: 8) denote the - -ELBO loss in Proposition 1. Suppose that eø is a linear function. Under the constraint that det Σ) = det(I). we have ming L(p, E, xxi 0) s ming T(O.L I, 0).
13

---

Published as a conference paper at ICLR 2022
Proof: We use L(μ, Σ, Xu: 8) to denote -ELBO. According Equation (0). 11 and 12)
we have
L(,, E,;
Exo |
log det(E) Σ
mBxx. (aan:lleeeelvaie) + v1 ā.e, t)ll-)
1m1
d lox (0((2))1)- - d g( + log(1 (rr)), 2
+
where Xy Xo n. We assume that EO(+) is scalar function with parameter freedom
d, i.e., ax + - vl- = 0 + v1 - ( where 8 E Rddd with constraint
A = Q0 = ding (0, ... ..) and Q the eigenmatrix for D- i.e., E = Q" Q'E-QQ with
Σ diag( o . For the term Σt MeBxo.ell - (v + vI ā |
we have
7 R. el. (VVox + v1 c
Σ
Yr (alaall((((e@aa V(Va + vI- -,))
T
0j + (1 --00)080 + 0g88 20, -.88)
t=l
Σ
Σ Pt
0 0 20, v1 a,8,,)
Σ
Σ
2t [1+ 03 221 - -r00))
(13)
where the second equation is established by taking expectation of X and e. The above equation will achieve its minimum at 8, 2(-17vI-0 Vj, and the minimum is Ej=1 EEEI (E moT Ei
Eiz 7
For L(O,I, Xo:0), 0), we have
L(0, x;: 0) Beo |xol 2 +
1
log det(l) + Σ Mlxxo.alle 0(Voi8o + vI aie, t)l?
2
2
log (og((
2 (ār + log(1 aT)),
Similarly, we can get the minimum of Σ Y1 Ex, ele- -(vāx āie, t)ll with a diagonal
0 is E Et= M
EE mv
Under the condition that det det(E) det((I), we have
Els
the minimum of Ej=1 Etm 7t
TVV (over all possibilities of (a1, .a.....,.a))) .a)))
2i= 71 (1
will be Ej=1 EE
"t- I TvTT
(by solving the constrained minimization problem).
Ei 7 Thus, we get the result in the Proposition.
Remark: From Equation (12), we have the second order derivative of L(u, Σ, No: 0) with respect to 8 equals Σ "e for all j 1.... d. Thus the condition number of the Hessian matrix equals 1. Similarly, the second order derivative of LO, 1, Xo; 0) with respect to 0, equals ΣT= 7c - āi +
ā,oz). Thus the Hessian matrix equals e I eg Σ with = Ei (1 - -ūx) and = Σ 7 whose condition number is no less than 1.
A.3 EXPLORATION ON THE CONDITIONAL INFORMATION SOURCE OF PRIORGRAD VOCODER
In this section, we provide further discussion regarding the source of the conditional information for constructing PriorGrad vocoder and empirical justification of selecting the frame-level spectral
14

---

Published as a conference paper at ICLR 2022
0.75
Magnitud
0.25
0.00
4.25
4.75
-1.00
-1.00
6.0
0.2
Data-dapaiddent
0.8
Data- Sata-dependent prior
Data-dependent prior
Figure 6: Scatter plots of waveform audio signals from the test set under different choices of the conditional information for PriorGrad vocoder. Left: V/UV label-based prior. Middle: Phoneme label-based prior. Right: Energy-basse prior.
energy as the prior. Considering the waveform synthesis model as a vocoder, the main application of such model is a spectrogram inversion module of the text-to-speech pipeline. That is, the vocoder model is usually combined with the acoustic model as a front-end, where the acoustic model gen- erates the mel-spectrogram based on text input. FastSpeech 2 (Ren al.] 2020) demonstrated significant improvement in the quality of the acoustic model leveraged by carefully designed ad- ditional speech-related features for fine-grained supervision, such as voiced/unvoiced (V/UV) label obtained from FO contours, or phoneme-to-frame alignment labels. We explored V/UV or phoneme labels as alternative sources of information for constructing the prior.
Figure 6 scatter plots of the waveform audio clip from the test set corresponding to the stan- dard deviation labels assigned by E. acquired from training set. Because the V/UV label binary,
only assign two different prior variances to the waveform distribution, which is an overly coarse assumption the data distribution. Phoneme-level prior can offer more fine- ane-grained approximate prior. However, we found that the variance statisties acquired from the training is misaligned with the unobserved waveform, where the actual variance can be significantly different the target prior, leading an inconsistent result. The frame-level spectral energy for PriorGrad exhibited the best alignment of the label and the unobserved waveform data and demonstrated con- sistent results in quality.
A.4 DESCRIPTION OBJECTIVE METRICS FOR PRIORGRAD VOCODER
In this section, we describe the objective metrics we collected to evaluate the performance of Prior- Grad vocoder.
Log-mel spectrogram mean absolute error (LS-MAE) This is spectral regression error be- tween the log-me mel spectrogram computed between the synthesized waveform and the ground ground-truth. We used the same STFT function for computing the spesttecc as the conditional input of DiffWave and PriorGrad.
Multi-resolution STFT error (MR-STFT) This measures the spectral distance across multiple resolutions of STFT. MR-STFT widely adopted training objective of recent neural vocoders because using multiple resolution windows can capture the time- frequeeyy distributions of the re- alistic speech signal. We used the resolution hyperparameter proposed in Parallel WaveGAN (Ya-) mamoto et al. 2020). which is implemented in an open-soune library
Mel-cepstral distortion (MCD) MCD a widely adopted speech metric (Kubichek) 1993) which measures the distance between the mel cepstra. We used an open-source implementation with default parameter
Fo root mean square error (F, RMSE) This measures the accuracy (measured by RMSE) the fundamental frequency (F)) which is an approximate frequency of the (quasi-)periodic structure of
15

---

Published as a conference paper at ICLR 2022
Table Sampling noise schedule used for PriorGrad acoustic model experiments obtained by grid search method (Chen et al. )021).
Method
Tifer
Sampling Noise Schedule
2
[0.8, 0.9]
Baseline
[0.0006, 0.003, 0.01, 0.07, 0.8, 0.9]
12
[0.0002, 0.0005, 0.002, 0.008, 0.03, 0.04, 0.06, 0.07, 0.5, 0.7, 0.8, 0.9]
[0.3, 0.9]
PriorGrad
[0.0001, 0.008, 0.01 0.05, 0.7, 0.9]
12
[0.0002, 0.0007, 0.004, 0.009, 0.01, 0.02, 0.06, 0.08, 0.1, 0.3, 0.5, 0.9]
voiced speech signals. We used the Saw-tooth Waveform Inspired Pitch Estimation (SWIPE) CCa macho & Harris 2008) with a hop size of 128 to obtain Fo using an open open-source implementatior
Debiased Sinkhorn divergence This a positive definite approximation of optimal transport (Wasserstein) distance between two data distributions (Feydy al. 2019). This metric be used as a mathematically grounded assessment of the informative prior, which is evidenced by a recent study that the diffusion process coincides with the optimal transport map (Khrulkov & Oseledets) 2022). We used the open-source library with default parameters. Using DiffWave and PriorGrad with fer 6, We calculated the test set average of the Sinkhorn divergence using Monte Carlo estimate with 100 samples per test data point. We denote S(æT, zo) as the distance between the prior and the real data, and S(Fo, no) as the distance between the generated samples and the real data.
A.5 ADDITIONAL DETAILS OF PRIORGRAD ACOUSTIC MODEL
In this section, we describe additional details the experimental design of the PriorGrad acoustic model. The feed-forward Transformer-basee phoneme encoder has 11.5M parameters trained with the Adam optimizer with the learning rate schedule identically described in Ren et al. (2020). The diffusion decoder is simultaneously trained with the same training configurations.
For a fair comparative study between different diffusion models, we searched for the best performing baseline model with (), first, then applied PriorGrad to this baseline. We applied T 400 for training the diffusion decoder with a linearly spaced beta schedule ranging from 1 x 10 to 10-2 We found that 10-4 to 2 10-2 used for waveform domain in Kong al. (2021) performed poorly for the baseline, where the model failed to capture the high-frequency details of the mel-spectrogram. This indicates that the optimal training noise schedule can be different, depending on the data domain (Kong & Ping 2021). On the contrary. PriorGrad's performance was similar under different choices training noise schedules. suggesting that PriorGrad also features robustness regarding designing the noise schedules for model training.
We found that the fast reverse noise schedule with Tinfer performed poorly for the baseline model with (NO, Thus, we applied grid search over the Tinfer = 6 reverse schedule for each model, which is a similar approach to that in Chen et al. 2021 This enabled a fair comparative study of each model configuration when using the fast noise schedule for sampling. We applied a fine- fine-graine see-ggrined grid search over every possible combination of the mono- tonically increasing betas based the Ll loss between the model prediction and the target from the validation set. We applied the following range for the grid search: 11,.....8 8,99 x {10-1, 1,10-
described in Kong et al. (2021)
for Tenfer 2, 21, 2, ..., 8.9) {10 -4 10 10-02 2,1022 10-11 1101110-11 11101) for T Tinfer 6, and 21 ...,. 8,9} x 10- Tinfer = 12.
4 10 10-3. 10 10 10-2 10 20-2.10- 101011 101011 10-1 1001] for
We found that setting relatively high values of beta at the last steps important for the quality of the baseline model, whereas PriorGrad was significantly more robust to the choice of the noise schedule. Table 5 shows the optimal beta schedules for each model configuration from the grid search method.
https:// http://wwwwwwter.eeeeeeeee.eeeet kernel opperaattloppraiiiiaommo ons wtw.//////ooooooooooooooo./oo oss/
16

---

Published as a conference paper at ICLR 2022
Table 6: Additional MOS results of acoustic models including an alternative method with jointly trainable estimation of the diffusion prior. We used pre-trained Parallel WaveGAN Yamamoto al. 2020) for the vocoder.
Parameters (Decoder)
Training Steps
Method
60K
300K
Baseline PriorGrad
10M
3.84 = 0.10
3.91 = 0.09
10M
4.04 + 0.07
4.09 - 0.08
Trainable ((1,, Ee)
10M
FAIL
3.32 + 0.12
Σ
Sample
GT
Baseline N
PriorGrad
Trainable
Figure 7: Visualized example of PriorGrad acoustic model. Top: Bascline model generates data from the standard Gaussian, leading to higher level of noise and slow training. Middle: PriorGrad model generates data from the data-dependent non-standard Gaussian that improves the quality and accelerates model training. Bottom: Alternative model with jointly trainable diffusion prior, where the estimation noisy and quality is worse. Right: ground-truth mel-spectrogram.
A.6 COMPARISON TO TRAINABLE DIFFUSION PRIOR
In this section, we present additional results regarding the challenges of formulating a jointly train- able estimation of the diffusion prior. Based on the improvements achieved by PriorGrad, it is natural to suppose that the jointly trainable diffusion prior distribution with additional parameteri- zation might perform similar or better than the data-dependent prior. In this section, we explored feasibility of the diffusion prior with trainable parameters. For acoustic model, explored two setups: 1. replacing the fixed data-dependent prior with the estimated N N(ps, Ee) from the projec- tion layer of the phoneme encoder, 2. parameterizing N N(us Eg) with independently defined convolutional layers with the data-dependent (j., Σ) we applied to PriorGrad as an input to acquire refined prior. For the vocoder model, instead of the spectral energy the proxy of the waveform variance, we explored an independently defined diffusion prior estimator by using the smaller-scale convolutional network with the mel-spectrogram as the input.
We found that all approaches failed to jointly estimate or refine the informative prior distribution. For the acoustic model, the estimated mean was significantly noisier than our data-dependent one, and the estimated variance is collapsed to a single point which is uninformative. These models performed significantly worse than the baseline method with the fixed (O, I), as measured by Table 6 We visualize the data-dependent prior compared to the estimated version in Figure 7 For the vocoder model. the diffusion prior estimator resulted in a divergence the estimated mean and collapse of the estimated variance to zero, leading to the training failure. We do not draw the conclusive claim that the jointly trainable diffusion prior is not possible. However, the results suggest that the fixed dependentt diffusion prior with PriorGrad is effective and easy-to-use method to improve the efficiency the diffusion-based generative model without introducing additional complexities to the network.
17

---

Published as a conference paper at ICLR 2022
Table Expanded vocoder model results compared previous work with 95% confidence intervals. t: pretrained weights obtained from the open-source repository with different train/test split.
Method
Tinfer
MOS
RTF
Parameters
GT
4.60 + 0.05
6
4.10 + 0.08 / 4.20 = 0.08
0.1388
DiffWave / PriorGrad
12
4.15 + 0.08 / 4.29 + 0.08
0.2780
2.62M
50
4.19 + 0.07 / /433 = 0.07
1.1520
WaveGlow
4.09 = 0.08
0.0780
87.9 87.9M
WaveFlow
4.01 = 0.09
0.1759
22.3M
HiFi-GAN (V1) f
4.44 + 0.05
0.0068
14.0M
Table 8: Expanded acoustic model results compared previous work with 95% confidence intervals. We used pretrained HiFi-GAN (V1) (Kong et al., 2020) for the vocoder. t: pretrained weights obtained from the open-source repository with different train/test split.
Parameters
Method
T Tinfer
MOS
RTF
Encoder Decoder
GT GT (Vocoder)
4.65 = 0.05
4.50 + 0.06
2
2.80 + 0.17 4.25 + 0.08
0.0069
Baseline / PriorGrad
6
3.67 = 0.12 4.29
0.07
0,0113
11.5M
3.5M
12
4.14 = 0.08 / 4.39
+ 0.08
0.0176
2
3.43 + 0.15
0.0090
Grad-TTSi
7.2M
7.6M
10
4.38 = 0.05
0.0308
FastSpeech Glow-TTS;
4.19 + 0.08
0.0040
11.5M
11,5M
4.23 + 0.08
0.0081
7.2M
21.4M
A.7 COMPARISON TO STATE-OF-THEE-RT
In this section, we present additional results of PriorGrad by comparing to recent state-of-the-art speech synthesis models and show that PriorGrad is competitive with or sometimes outperforms the previous models. We provide detailed analysis with a varying number of inference denoising steps (Tinfer), along with their speed measured by real-time factor (RTF) the NVIDIA A40 GPU, and the model capacity measured by the number of parameters.
Vocoder comparison Table 7 provides an expanded MOS result of the fully converged PriorGrad vocoder with the 1M training steps. For the fast inference noise schedule, we used firfer defined as [0.0001, 0,001, 0.01, 0.05, 0.2, 0.5] from DiftWave without modification, and used Tinfer 12 defined as [0.0001, 0.0005, 0.0008, 0.001, 0.005, 0.008, 0.01, 0.05, 0.08, 0.1, 0.2, 0.5]. For the full fer = 50, PriorGrad significantly outperformed the baseline DiffWave. Furthermore, PriorGrad achieved an even larger performance gap with a significantly reduced network capac- ity, compared to the flow-based vocoders such as WaveGlow (Prenger et al. 2019) and WaveFlow (Ping al. 2020). This suggests that PriorGrad competitive likelihood-based neural vocoder, and enables step closer to the current state-of-the-an GAN-based model (HiFi-GAN, Kong et al. 22020)) The speed is close to real-time (RTF being close to 1), but is noticeably slower than other approaches.
For the fast inference with fer = 6 denoising steps, PriorGrad was able to match the performance to the baseline DiffWave model with fer 50 while still outperforming the flow-based vocoders with comparable RTF This further showease the efficiency of PriorGrad, where having access to the informative non-standard Gaussian prior accelerates the inference of the diffusion-based model.
18

---

Published as a conference paper at ICLR 2022
Acoustic model comparison We additionally trained our baseline cuffusion-based acoustic model and the enhanced model with PriorGrad which are compatible with the pretrained HiFi-GAN (Kong et al. 2020) vocoder. This enabled fair assessment of PriorGrad compared to recent state-of-the- art acoustic models. We trained our acoustic model for the full convergence with 1M training steps. We found that the small PriorGrad with 3.5M decoder parameters performed almost identical the high-capacity (10M) model. Therefore, we chose the small decoder model as our final choice for comparison.
Table provides an expanded MOS result of PriorGrad acoustic model with varying Tinfer and comparison to the recent representative acoustic model from different categories: Feed-forward (FastSpeech 2. Renn et al. (2020),, Flow-based (Glow-TT) TTS, KKim et al.] (2020)) and the concurrent diffusion-based model (Grad-TTS, Poppv et al. (202T)). The results show that PriorGrad acous- tic model is competitive to the state-of-the-art models. PriorGrad with finerr = 12 provided an identical quality to the state-of-the-ard Grad-TTS with Tinfer 10. This is achieved with an ap- proximately 1.75 times faster RTF and sets PriorGrad a new state-of-the- art acoustic model.
Armed with the informative non-standard Gaussian prior, PriorGrad robust to the extremely small number of inference steps with feneer = 2. The fast PriorGrad matches the performance with the state-of-thee -frt--rr flow-based acoustic model, Glow-TTS, with fewer parameters. The RTF becomes close to the feed-forward FastSpeech 2 well scoring higher MOS. By contrast, the bascline acoustic model with.V ((0, I) suffered significant degradation for smaller Tinfer- This result further highlights the efficiency of PriorGrad, where it is robust to the reduced number of denoising steps and enables a significantly accelerated inference.
A.8 MOS EVALUATION DETAILS
We conducted standard and widely adopted protocol for the MOS evaluation using Amazon Me- chanical Turk. We constructed 20 randomly selected clips from the test set, and applied total of 450 evaluations for each model with the 5-scale scoring on audio naturalness. To encompass the diverse set of listeners, single listener can only evaluate the three randomly exposed samples. Therefore, there are 150 unique listeners for each model. For reliability, only skilled listeners with the eval- uation approval rate higher than 98% and the number of previously approved evaluations higher than 100 are qualified to conduct the test. The listeners were rewarded $0.2 for each evaluation. Approximately $130 were spent as compensation for each set of the MOS test.
A.9 AUDIO SAMPLES DEMO PAGE
We provide demo page of the audio samples used in this study;
github. i hwtp:iirrpprrrrrall//oooo
19


Reviews:
[{"review_id": "i0Rtry-6j0Y", "reviewer": "Reviewer_Wdw2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well written and the key idea is described in a simple way. The main motivation stems from the question whether it is possible to induce diffusion probabilistic models (DDPM) to use a more informative prior, rather than the typical isotropic unit Gaussian, *without* additional computational or parameter complexity.\n\nAs a side comment, I would like to clarify that while it is true that the DDPM as formulated in this work bears no overheads, the task of estimating mean and covariances of the data (especially for \"complex\" data such as the one used in the experiments) comes with its own cost and design decisions, which should not underestimated.\n\nThe background section is typical for a DDPM paper (using the discrete-time Markov chain formulation). Although this is nitpicking, I think some clarifications are in order: the ELBO from eq (3), which is derived as in Ho et. al. 2020, is studied as three separate components, and the focus is on the second one, which is labeled $L_{t-1}$ in Ho et al. 2020. Then, eq (6) is not the ELBO, but only the $L_{t-1}$ term.\nAlso, since this work argues for taking into account an informative prior where $\\mu$ and $\\Sigma$ are instrumental, I would argue that the constant $C$ that appears in eq (6) should be developed, as there are terms that depend on it. This could help the reader gain a better understanding of the proposed method (and the limitations of proposition 1 and 2).\n\nThe presented method is discussed in an intuitive manner at first, using Fig 2. Nevertheless, after some thoughts, I think this figure can be misleading (even if it is just an illustration). In a two-dimensional toy example, $p(\\mathbf{x}_T) = \\mathcal{N}(\\mu, \\Sigma)$ could be \"closer\" to the true data distribution, actually, on top of it. This is not necessarily true for high-dimensional spaces though.\nMore to the point, since you \"simulate\" the transition from $\\mathbf{x}_0$ to $\\mathbf{x}_T$ in the forward process for a finite $T$, then the process does not end up in $ \\mathcal{N}(0,\\mathbf{I})$ (for the un-informative baseline) nor in $ \\mathcal{N}(\\mu, \\Sigma)$ (for the data-dependent prior).\nThe backward process starts from the prior, and again, since it is \"simulated\" for a finite amount of time $T$, it will not reach $\\mathbf{x}_0$.\nThis kind of illustration could be helpful to better understand the claim of proposition 2.\n\nAlso, note that in proposition 1, the quantity under study is not the ELBO, but $L_{t-1}$. This is also where the constant $C$ is \"hiding\" a dependence on $\\mu$ and $\\Sigma$.\n\nProposition 2 states that the use of a more informative prior is better in terms of a tighter bound. This seems intuitive at first, as there are more degrees of freedom than using zero mean and unit covariance. However, the proposition holds when the function that learns the denoising effect of the backward process is linear, which is not true in practice. Do you have any intuition if the proposition holds also in this more general case? Are there any other arguments that can be used to show that indeed the ELBO is smaller?\n\nThe observation that the Hessian of the $L_{t-1}$ term is better conditioned when using an informative prior w.r.t. the case of a unit Gaussian prior is also based on the assumption of linearity of the denoising function. However, results in Fig 4 do not show that the rate of convergence is improved in a practical endeavor. There, we see that the considered error metric is better for the informative prior case, but the rates of the two curves appear to be similar, just shifted. Do you have any comments on this effect?\n\nAlthough the application domain used to evaluate the proposed method falls outside my competence, I appreciate the efforts and the timeliness of the architectures and design choices made in the execution of the experiments, which I find somehow compelling to corroborate that the paper presents a valid idea. That being said, the evaluation only compares the non-informative prior baseline against the informative variant. Wouldn't it be interesting to compare additional methods from the state of the art, which the authors appear to know very well? I particularly liked the idea of studying \"parameter efficiency\", and I wonder whether the efficiency of an informative prior carries on when comparing to different approaches.\n\nTo conclude my comments, I think this paper introduces a relevant idea which shows potential in applications. I am less excited about the methodological sections of this paper, which are not strong and precise enough to be convincing.\n", "summary_of_the_paper": "This work builds on denoising diffusion probabilistic models (DDPM), and argues for to modify the forward and backward diffusion processes such that instead of using an uninformative prior $p(\\mathbf{x}_T) = \\mathcal{N}(0,\\mathbf{I})$, they use a data-dependent prior $p(\\mathbf{x}_T) = \\mathcal{N}(\\mu, \\Sigma)$, where the mean and covariance are derived from training data, in a \"pre-processing\" step.\n\nThe authors show that, under restrictive conditions, the ELBO obtained with the proposed prior is smaller than that obtained with the uninformative prior. Under similar restrictive conditions, the authors argue that convergence rate of the parameter optimization is better conditioned, and as a consequence faster.\n\nThis work illustrates the benefits of the proposed method using two applications, one to a vocoder, and one to an acoustic model. Experiments rely on several very recent related work, especially concerning technical details of the architectures, and data pre-processing choices. Overall, two methods are compared, a baseline DDPM and the proposed DDPM that uses training data to build a better prior.", "summary_of_the_review": "Interesting and relevant idea, to improve the behavior of diffusion models. Experimental results indicate the idea has a positive effect on metrics used in specific application domains, including training performance and parameter efficiency.\nHowever, the theoretical justification of the proposed method is somehow weak, and does not provide sufficiently compelling arguments on the observed behavior of the proposed method in practice.\n\n\n*** Post-rebuttal / discussions remarks ***\n\nThanks for the useful discussions and for the additional work to improve the paper. I have increased my ranking for the paper from 5 to 6.\nThe main reason why I didn't rank the paper higher is that the theoretical contribution of this work, despite being very promising, is still not mature and sufficiently mathematically grounded.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cpKYDZcr506", "reviewer": "Reviewer_D8iM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nMost of the theory explanations focus on known \\mu and \\Sigma, but I see very little about how these parameters are estimated and pros and cons about  different estimation techniques. \n\nRelating to point above, can you clarify where is the \\mu coming from in case? Noting that original paper had prior as standar normal, you have \\mu also. I understand that noise processis zero-mean, but \\mu appears in your ELBO. Can you clarify this point. \n\nIn 4.1: Why non-clipping variance causes instability in training? Is there a theoretical explanation to it or is it only empirical observation? Relating to  the accompanied footnote, did you try to see if different corpora would have different clipping threshold? \n\nIn the Table 1, is GT a the clean, i.e. non-vocoded sample? At least it is supposed to be. And in the same vein, why is 500k training steps better for the proposed than 1M training steps? Can you check statistical significance of this, as if result is significant, then there should be a reason for it. \n\nIntro: 2nd para: \"procedurally destroys signal into\" ??\nIn Algorithm 2, line after If-statement seems to have wrong index x_{t-1} to x_t. \nNotation seems to be a bit mismatched in style. I would like to see consistent style, for example all vectors to be bolded non-caps and all matrices to be bolded and capitalized. Then it is easier for the reader to follow the story. \n\nReview has been updated based on author rebuttal and discussion. ", "summary_of_the_paper": "In this paper authors propose to use informative prior for conditional diffusion model when applied to neural  vocoding task.  ", "summary_of_the_review": "Paper provides quite nice theoretical justifications for their decision to use informative prior instead of classic non-informative one. However, I feel that modification itself is too simple to warrant an ICLR paper. I hope authors can provide a nice rebuttal to this point in assessing the level of contribution of this paper. \n\nThe other point that lowers my score is that the final selection of noise variance is based on ad-hoc technique and not justified theoretically, whereas other parts of the paper are.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XZNYQygE5Wz", "reviewer": "Reviewer_spyq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Novel idea to change the initial sampling distribution to improve inference speed and training convergence.\n- Tested for both vocoder and acoustic model.\n\nWeaknesses:\n\n- Mathematical justification of such prior is lacking.\n- Same related work is missing.\n- Experimental evaluation lacks comparison to alternative models, both for vocoder and acoustic model.\n- Baseline acoustic model is non-standard, which makes it even more relevant to compare to other alternative models.\n- Acoustic model comparison uses non-optimal vocoder (PWG).\n- No discussion and numbers on the inference speed and real-time-factor (RTF), neither for vocoder or acoustic model.\n- Code not published.\n\nDetails:\n\nIs prior grad mathematically motivated?\n\"The (DDPM) framework assumes the prior noise as a standard Gaussian distribution\"\nNo, it is not an assumption. It is a mathematical consequence of the definitions of the framework.\n\nMost of the math theory in the paper just shows that convergence rate and training behavior might be better for the proposed method. But it does not give a justification why the data-dependent prior is mathematical sound within the framework.\n\nRelated work section seems good, at least related diffusion (or grad-based) models, related prior modeling.It misses many other state-of-the-art vocoder models (MelGAN, UnivNet).\n\nGAN-based vocoders are currently among the best vocoders, such as:\n\n- MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis (https://arxiv.org/abs/1910.06711)- Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech (https://arxiv.org/abs/2005.05106): MelGAN 3.87 MOS, proposed MB-MelGAN 4.22 MOS, Recording 4.58 MOS.\n- HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis (https://arxiv.org/abs/2010.05646): proposed HiFi-GAN V1 4.36 MOS, Recording 4.45 MOS, MelGAN 3.79 MOS, WaveGlow 3.81 MOS, WaveNet (MoL) 4.02 MOS.\n- UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation (https://arxiv.org/abs/2106.07889): MelGAN: 3.56 MOS, Parallel WaveGAN 3.07 MOS, HiFi-GAN 3.89 MOS, proposed UnivNet-c32 3.93 MOS, Recordings 4.16 MOS.\n\nThose are not compared here.\n\nThen there is WaveGlow, WaveFlow and similar vocoder models which are also not compared.\n\nSo experimental validation on vocoder experiments is weak.\n\nExperimental validation on acoustic model is weak as well. Custom-implemented baseline with non-optimal vocoder (PWG). Why not test against other acoustic models such as Diff-TTS or Grad-TTS or FastSpeech 2? Why not use a better vocoder?\n\nVocoder experiments: LJSpeech. PriorGrad based on DiffWave. T=50 steps for training and inference, for both DiffWave and PriorGrad.\nIt would be interesting to study the influence of T on both models. Maybe DiffWave with T=100 compensates the effect. Or PriorGrad with T=25. Just some variation on T for both models. You can argue that using the conditional data-dependent prior is an unfair comparison when using the same T.\n\nAcoustic model (TTS) experiment with fixed vocoder:\nTable 4 should make more clear what \"baseline\" is. From the text, it seems to be DiffWave, just as before.\nThis baseline is also some custom implementation.\nWhy not use existing implementation, such as Grad-TTS?\n\nSpeed (RTF) not evaluated? Not even discussed?\n\nCode?\n", "summary_of_the_paper": "This paper presents an extension to conditional denoising diffusion models for text-to-speech (TTS). The initial noise in the inference procedure is not sampled from a standard Gaussian but from a data-dependent conditional non-standard Gaussian, which is called PriorGrad. The motivation is to increase inference speed and training convergence. This method is applied to DiffWave used as a vocoder, and also applied to DiffWave used as an acoustic model (phonemes to mel spect).\n", "summary_of_the_review": "The basic idea of using a better initial prior distribution to improve inference speed and training convergence is nice, although it lacks some mathematical justification. The experimental section is lacking too much.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3gu552yj45T", "reviewer": "Reviewer_518v", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Time-wise and instance-based means and variances provide additional information that models are able to exploit, yielding improved results in terms of convergence speed and generation quality.\n- Results are reported for both objective and subjective assessments. Baselines are clear and appropriate.\n- As far as I can judge, I think the paper is technically correct.\n- Well-written paper, except from some minor typos (see below).\n\nWeaknesses:\n- Obvious adaptation from standard to non-standard Gaussian prior.\n- Potential lack of generality to other conditional audio tasks or other domains such as image.\n- Non-trivial and case-specific methodology to compute means and variances for the non-standard Gaussian priors.\n- I don't think one of the three main claims is well supported by the experiments (see below).\n\nExtended review:\n1. I have the impression that the title, parts of the text, and even the abstract, try to encompass a general application of the methodology. However, results are only provided for two specific speech processing tasks. The title, for instance, does not even mention speech, and the abstract mentions audio when only speech is considered. While in principle the proposed methodology could be applied to other tasks and domains, the benefit of it is not shown for those.\n1. The procedure to compute instance- and time-specific means and variances for the prior seems non-straightforward and highly dependent on the type of conditioning signal, further complicating the generalization of the approach. The authors even state that with other conditioning signals for the considered task (V/U segments and phoneme-level statistics) they did not obtain any improvements. This also raises some doubts beyond generalization. In particular, regarding how hard is to adapt the approach to other tasks/domains or even conditioning signals, or how limited is the proposed idea. Even in the acoustic modeling setting the approach needs to rely on mel band energies (leaving the reader suspicious as if this is the only information that works).\n1. There is some unclear formulation of the approach. Expanding the last point to focus more on the considered case: How is frame-level energy normalized? Which is the exact formula? How do different procedures affect final performance? Why there should be a range of $(0,1]$ (later the paper mentions it is not really $(0,1]$ but clipped to some empirically-chosen value...)?\n1. There is at least concurrent work (https://arxiv.org/abs/2110.05948) investigating the effect of using other priors beyond the standard Gaussian. Perhaps contribution 1 could be softened in this regard.\n1. No details on how MOS subjective tests were performed. This can be a serious issue.\n1. In my opinion, the \"tolerance to a reduction in network capacity\" claim is not supported by the results. First of all, it is debatable whether reducing the width of convolutional layers is sound or meaningful in this setting (Sec 4.3, \"parameter efficiency\"). But, more importantly, the relative change in MOS in Table 2 does not support the claim (3.9/4.06=0.961 which is extremely close to 4.02/4.12=0.975; that is, both DiffWave and PriorGrad present almost the same relative reduction in performance (4% vs 3%). This complexity-leverage claim again does not hold with results in Table 4 when we think of relative terms.\n\nMinor considerations:\n- Talking about \"inefficiency\" in conditional speech synthesis when for instance WaveGrad shows that with only 6 steps one can achieve almost perfect accuracy is, at least, misleading.\n- \"Given the above investigation\" --> Given the previous reasoning?\n- \"our method can provide a better trajectory...\" --> Claim not shown either theoretically or empirically.\n- In Algorithm 2, it is perhaps unclear that $\\sigma_t$ corresponds to a noise source or that it includes it (the authors use $\\epsilon$ for Algorithm 1).\n- I think Figs 3 and 5 are unnecessary.\n- \"by comparing the baseline\" --> by comparing to the baseline\n- References Jeong et al and Kawar et al have no venue or arxiv or paper link.\n\n", "summary_of_the_paper": "The paper proposes to use a data-dependent, adaptive prior for the noise used in conditional DDPMs. In particular, it proposes to move from the standard to the non-standard Gaussian prior. For that, the ELBO loss is reformulated by taking into account adaptive means and variances and the sampling procedure is also adapted. Statistics (directly or indirectly derived) from conditioning signals are used to compute time-wise and instance-based means/standard deviations. An empirical evaluation is conducted on two conditional audio generation tasks: vocoding and acoustic modeling, showing some favorable results.", "summary_of_the_review": "The paper is technically correct and shows some improvements in the empirical evaluation. However, I think that moving from standard to non-standard Gaussian priors is rather obvious. In addition, I am concerned about the generality of the approach, as it only focuses on two specific speech generation tasks and deriving appropriate means and variances from certain conditioning signals can be problematic or may not yield to an improvement. One of the three main claims is not well supported.\n\n[Update: After authors' rebuttal I'm changing my score from 5 to 6]", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper suggests using a conditional prior in conditional diffusion-based generative models. Typically, only the score function estimator is provided with the conditioning signal, and the prior is an unconditional standard Gaussian distribution. It is shown that making the prior conditional improves results on speech generation tasks.

Several reviewers initially recommended rejection, but after extensive discussion and interaction with the authors, all reviewers have given this work a "borderline accept" rating. 

Criticisms included that the idea is too simple or obvious to warrant an ICLR paper. I am inclined to disagree: simple ideas that work are often the ones that persist and see rapid adoption (dropout regularisation is my favourite example). Like the authors, I believe simplicity is an advantage in this respect, rather than a disadvantage. Of course, simple ideas do require extensive and convincing empirical validation to be worth publishing at ICLR. After the authors' updates, I believe the work meets this bar.

Another issue raised by several reviewers is the limited theoretical justification for the approach. However, combined with the simplicity of the method, I believe the empirical results of the revised version sufficiently justify the approach on their own. Nevertheless, I would recommend that the authors consider further how they could address this issue in the final version of their manuscript, as they have already begun to do during the discussion phase.

Another way to strengthen the paper further would be to demonstrate how the generic approach can be applied in a different domain (e.g. conditional image generation), but I do not consider this addition necessary for the work to warrant publication.

In light of this, I am recommending acceptance.
Decision: Accept (Poster)