PRIORGRAD: IMPROVING CONDITIONAL DENOISING DIFFUSION MODELS WITH DATA-DEPENDENT ADAP-
Year: 2022

Abstract:
Denoising diffusion probabilistic models have been recently proposed to generate high-quality samples by estimating the gradient of the data density. The framework defines the prior noise as a standard Gaussian distribution, whereas the corresponding data distribution may be more complicated than the standard Gaussian distribution, which potentially introduces inefficiency in denoising the prior noise into the data sample because of the discrepancy between the data and the prior. In this paper, we propose PriorGrad to improve the efficiency of the conditional diffusion model for speech synthesis (for example, a vocoder using a mel-spectrogram as the condition) by applying an adaptive prior derived from the data statistics based on the conditional information. We formulate the training and sampling procedures of PriorGrad and demonstrate the advantages of an adaptive prior through a theoretical analysis. Focusing on the speech synthesis domain, we consider the recently proposed diffusion-based speech generative models based on both the spectral and time domains and show that PriorGrad achieves faster convergence and inference with superior performance, leading to an improved perceptual quality and robustness to a smaller network capacity, and thereby demonstrating the efficiency of a data-dependent adaptive prior.

text: Denoising diffusion probabilistic models have been recently proposed to generate high-quality samples by estimating the gradient of the data density. The framework defines the prior noise as a standard Gaussian distribution, whereas the corresponding data distribution may be more complicated than the standard Gaussian distribution, which potentially introduces inefficiency in denoising the prior noise into the data sample because of the discrepancy between the data and the prior. In this paper, we propose PriorGrad to improve the efficiency of the conditional diffusion model for speech synthesis (for example, a vocoder using a mel-spectrogram as the condition) by applying an adaptive prior derived from the data statistics based on the conditional information. We formulate the training and sampling procedures of PriorGrad and demonstrate the advantages of an adaptive prior through a theoretical analysis. Focusing on the speech synthesis domain, we consider the recently proposed diffusion-based speech generative models based on both the spectral and time domains and show that PriorGrad achieves faster convergence and inference with superior performance, leading to an improved perceptual quality and robustness to a smaller network capacity, and thereby demonstrating the efficiency of a data-dependent adaptive prior.

heading: 1 INTRODUCTION
text: Deep generative models have been achieving rapid progress, by which deep neural networks approximate the data distribution and synthesize realistic samples from the model. There is a wide range of this type of approach, ranging from autoregressive models (Oord et al., 2016a;b), generative adversarial networks (Goodfellow et al., 2014; Brock et al., 2019), variational autoencoders (Kingma & Welling, 2013; Vahdat & Kautz, 2020), and normalizing flows (Rezende & Mohamed, 2015; Kingma & Dhariwal, 2018). Denoising diffusion probabilistic models (DDPMs) (Ho et al., 2020) and score matching (SM) (Song & Ermon, 2019) are recently proposed categories that can be used to synthesize high-fidelity samples with competitive or sometimes better quality than previous state-of-the-art approaches. Consequently, there have been a variety of applications based on DDPM or SM (Saharia et al., 2021; Kawar et al., 2021). Speech synthesis is one of the most successful applications, where the diffusion model can synthesize spectral or time-domain audio conditioned on text or spectral information, respectively, achieving a competitive quality but faster sampling (Chen et al., 2021; Kong et al., 2021; Jeong et al., 2021; Lee & Han, 2021) than autoregressive models (Oord et al., 2016b; Kalchbrenner et al., 2018).
∗Work done during an internship at Microsoft Research Asia †Corresponding Authors
However, although the diffusion-based speech synthesis models have achieved high-quality speech audio generation, they exhibit potential inefficiency, which may necessitate advanced strategies. For example, the model suffers from a significantly slow convergence during training, and a prohibitively large training computation time is required to learn the approximate reverse diffusion process. We investigate the diffuion-based models and observe the discrepancy between the real data distribution and the choice of the prior. Existing diffusion-based models define a standard Gaussian as the prior distribution and design a non-parametric diffusion process that procedurally destroys the signal into the prior noise. The deep neural network is trained to approximate the reverse diffusion process by estimating the gradient of the data density. Although applying the standard Gaussian as the prior is simple without any assumptions on the target data, it also introduces inefficiency. For example, in time-domain waveform data, the signal has extremely high variability between different segments such as voiced and unvoiced parts. Jointly modeling the voiced and unvoiced segments with the same standard Gaussian prior may be difficult for the model to cover all modes of the data, leading to training inefficiencies and potentially spurious diffusion trajectories.
Given the previous reasoning, we assessed the following question: For a conditional diffusion-based model, can we formulate a more informative prior without incorporating additional computational or parameter complexity? To investigate this, we propose a simple yet effective method, called PriorGrad, that uses adaptive noise by directly computing the mean and variance for the forward diffusion process prior, based on the conditional information. Specifically, using a conditional speech synthesis model, we propose structuring the prior distribution based on the conditional data, such as a mel-spectrogram for the vocoder (Chen et al., 2021; Kong et al., 2021) and a phoneme for the acoustic model (Jeong et al., 2021). By computing the statistics from the conditional data at the frame level (vocoder) or phoneme-level (acoustic model) granularity and mapping them as the mean and variance of the Gaussian prior, we can structure the noise that is similar to the target data distribution at an instance level, easing the burden of learning the reverse diffusion process.
We implemented PriorGrad based on the recently proposed diffusion-based speech generative models (Kong et al., 2021; Chen et al., 2021; Jeong et al., 2021), and conducted experiments on the LJSpeech (Ito & Johnson, 2017) dataset. The experimental results demonstrate the benefits of PriorGrad, such as a significantly faster model convergence during training, improved perceptual quality, and an improved tolerance to a reduction in network capacity. Our contributions are as follows:
• To the best of our knowledge, our study is one of the first to systematically investigate the effect of using a non-standard Gaussian distribution as the forward diffusion process prior to the conditional generative model.
• Compared to previous non-parametric forward diffusion without any assumption, we show that the model performance is significantly improved with faster convergence by leveraging the conditional information as the adaptive prior.
• We provide a comprehensive empirical study and analysis of the diffusion model behavior in speech generative models, in both the spectral and waveform domains, and demonstrate the effectiveness of the method, such as a significantly accelerated inference and improved quality.

heading: 2 BACKGROUND
text: In this section, we describe the basic formulation of the diffusion-based model and provide related studies, along with a description of our contribution with PriorGrad.
Basic formulation Denoising diffusion probabilistic models (DDPM) (Ho et al., 2020) are recently proposed deep generative models defined by two Markov chains: forward and reverse processes. The forward process procedurally destroys the data x0 into a standard Gaussian xT , as follows:
q(x1:T |x0) = T∏ t=1 q(xt|xt−1), q(xt|xt−1) := N (xt; √ 1− βtxt−1, βtI), (1)
where q(xt|xt−1) represents the transition probability at the t-th step using a user-defined noise schedule βt ∈ {β1, ..., βT }. Thus, the noisy distribution of xt is the closed form of q(xt|x0) =
N (xt; √ ᾱtx0, (1− ᾱt)I), where αt := 1−βt, ᾱt := ∏t s=1 αs. q(xT |x0) converges in distribution to the standard Gaussian N (xT ; 0, I) if ᾱT is small enough based on a carefully designed noise schedule. The reverse process that procedurally transforms the prior noise into data is defined as follows:
pθ(x0:T ) = p(xT ) T∏ t=1 pθ(xt−1|xt), pθ(xt−1|xt) = N (xt−1;µθ(xt, t),Σθ(xt, t)) , (2)
where p(xT ) = N (xT ; 0, I) and pθ(xt−1|xt) corresponds to the reverse of the forward transition probability, parameterized using a deep neural network. We can define the evidence lower bound (ELBO) loss as the training objective of the reverse process:
L(θ) = Eq [ KL (q(xT |x0)||p(xT )) +
T∑ t=2 KL(q(xt−1|xt,x0)||pθ(xt−1|xt))− log pθ(x0|x1)
] .
(3)
As shown in Ho et al. (2020), q(xt−1|xt,x0) can be represented by Bayes rule as follows:
q(xt−1|xt,x0) = N(xt−1; µ̃(xt,x0), β̃tI), (4)
µ̃t(xt,x0) :=
√ ᾱt−1βt
1− ᾱt x0 +
√ αt(1− ᾱt−1)
1− ᾱt xt, β̃t := 1− ᾱt−1 1− ᾱt βt. (5)
By fixing p(xT ) as a standard Gaussian, KL(q(xT |x0)||p(xT )) becomes constant and is not parameterized. The original framework in Ho et al. (2020) fixed Σθ(xt, t) as a constant β̃tI and set the standard Gaussian noise as the optimization target instead of µ̃t by reparameterizing x0 =
1√ ᾱt
(xt − √
1− ᾱt ) from q(xt|x0) to minimize the second and third terms in equation 3. Based on this setup, in Ho et al. (2020), the authors further demonstrated that we can drop the weighting factor of each term and use a simplified training objective that provides a higher sample quality:
− ELBO = C + T∑ t=1 Ex0, [ β2t 2σ2tαt(1− ᾱt) ‖ − θ( √ ᾱtx0 + √ 1− ᾱt , t)‖2 ] , (6)
Lsimple(θ) := Et,x0, [ ‖ − θ (xt, t)‖2 ] . (7)
Related work Since the introduction of the DDPM, there have been a variety of further studies (Nichol & Dhariwal, 2021; Song et al., 2020), applications (Chen et al., 2021; Jeong et al., 2021; Kong et al., 2021; Saharia et al., 2021), and a symbiosis of diffusion (Ho et al., 2020; Sohl-Dickstein et al., 2015) and score-based models (Song & Ermon, 2019; 2020) as a unified view with stochastic differential equations (SDEs) (Song et al., 2021). From an application perspective, several conditional generative models have been proposed. Waveform synthesis models (Chen et al., 2021; Kong et al., 2021) are one of the major applications in which the diffusion model is trained to generate time-domain speech audio from the prior noise, conditioned on a mel-spectrogram. A diffusionbased decoder has also been applied to text-to-spectrogram generation models (Jeong et al., 2021; Popov et al., 2021). PriorGrad focuses on improving the efficiency of training such methods from the perspective of a conditional generative model. We investigate the potential inefficiency of the current methods which require unfeasibly large computing resources to train and generate high-quality samples.
Studies on formulating an informative prior distribution for deep generative model are not new, and there has been a variety of studies investigating a better prior, ranging from hand-crafted (Nalisnick & Smyth, 2017; Tomczak & Welling, 2018), autoregressive (Chen et al., 2017), vector quantization (Razavi et al., 2019), prior encoder (Rezende & Mohamed, 2015), and data-dependent approaches similar to ours (Li et al., 2019). We tackle the problem of training inefficiency of diffusion-based models by crafting better priors in a data-dependent manner, where our method can provide a better trajectory and can reduce spurious modes, enabling more efficient training. Nachmani et al. (2021)
used Gamma distribution as the diffusion prior. Note that there has also been a concurrent study conducted on leveraging the prior distribution on the acoustic model, Grad-TTS (Popov et al., 2021), in which the effectiveness of using the mean-shifted Gaussian as a prior with the identity variance was investigated. Unlike the method in Popov et al. (2021), which enforces the encoder output to match the target mel-spectrogram by using an additional encoder loss, our approach augments the forward diffusion prior directly through data and the encoder has no restriction on latent feature representations. In Popov et al. (2021), the forward diffusion prior is jointly trained and may induce additional overhead on convergence as the prior changes throughout the training, whereas our method provides guaranteed convergence through the fixed informative prior.

heading: 3 METHOD
text: We investigate the following intuitive argument: When we structure the informative prior noise closer to the data distribution, can we improve the efficiency of the diffusion model? In this section, we present a general formulation of PriorGrad, describe training and sampling algorithms, and provide benefits of PriorGrad through a theoretical analysis. PriorGrad offers a generalized approach for the diffusion-based model with the non-standard Gaussian as the prior and can be applied to a variety of applications.
3.1 GENERAL FORMULATION
In this section, we provide a general formulation of the method regarding using the non-standard GaussianN (µ,Σ) as the forward diffusion prior. PriorGrad leverages the conditional data to directly compute instancelevel approximate priors in an adaptive manner, and provides the approximate prior as the forward diffusion target for both training and inference. Figure 1 and 2 presents a visual high-level overview. We take the same parameterization of µθ and σθ as in the original DDPM (Ho et al., 2020) from Section 2, as follows:
µθ(xt, t) = 1 √ αt
( xt −
βt√ 1− ᾱt θ(xt, t)
) , σθ(xt, t) = β̃ 1 2 t (8)
Assuming that we have access to an optimal Gaussian N (µ,Σ) as the forward diffusion prior distribution, we have the following modified ELBO:
Proposition 1 Let ∼ N (0,Σ) and x0 ∼ qdata. Then, under the parameterization in Eq. equation 8, the ELBO loss will be
− ELBO = C(Σ) + T∑ t=1 γtEx0, ‖ − θ( √ ᾱt(x0 − µ) + √ 1− ᾱt , t)‖2Σ−1 ,
for some constant C, where ‖x‖2Σ−1 = x TΣ−1x, γt = β2t 2σ2tαt(1−ᾱt) for t > 1, and γ1 = 12α1 .
The Proposition 1 is an extension of the ELBO in Equation 6 with the non-standard Gaussian distribution. Contrary to original DDPM, which used N (0, I) as the prior without any assumption on data, through Proposition 1, we train the model with N (µ,Σ), whose mean and variance are extracted from the data, as the prior for the forward process. See appendix A.1 for full derivation. We
also drop γt as the simplified loss L = ‖ − θ(xt, c, t)‖2Σ−1 for training, following the previous work. Algorithms 1 and 2 describe the training and sampling procedures augmented by the datadependent prior (µ,Σ). Because computing the data-dependent prior is application-dependent, in Section 4 and 5, we describe how to compute such prior based on the conditional data on the given task.
Algorithm 1 Training of PriorGrad repeat
(µ,Σ) = data-dependent prior Sample x0 ∼ qdata, ∼ N (0,Σ) Sample t ∼ U({1, · · · , T}) xt = √ ᾱt(x0 − µ) + √ 1− ᾱt L = ‖ − θ(xt, c, t)‖2Σ−1 Update the model parameter θ with∇θL
until converged
Algorithm 2 Sampling of PriorGrad (µ,Σ) = data-dependent prior Sample xT ∼ N (0,Σ) for t = T, T − 1, · · · , 1 do
Sample z ∼ N (0,Σ) if t > 1; else z = 0 xt−1 =
1√ αt (xt− 1−αt√1−ᾱt θ(xt, c, t)) +σtz end for return x0 + µ

heading: 3.2 THEORETICAL ANALYSIS
text: In this section, we describe the theoretical benefits of PriorGrad. First, we discuss about the simplified modeling with the following proposition:
Proposition 2 Let L(µ,Σ,x0; θ) denote the −ELBO loss in Proposition 1. Suppose that θ is a linear function. Under the constraint that det(Σ) = det(I), we have minθ L(µ,Σ,x0; θ) ≤ minθ L(0, I,x0; θ).
The proposition shows that setting the prior whose covariance Σ aligns with the covariance of data x0 leads to a smaller loss if we use a linear function approximation for θ. This indicates that we can use a simple model to represent the mean of q(xt−1|xt) under the data-dependent prior, whereas we need to use a complex model to achieve the same precision under a prior with isotropic covariance. The condition det(Σ) = det(I) means that the two Gaussian priors have equal entropy, which is a condition for a fair comparison.
Second, we discuss the convergence rate. The convergence rate of optimization depends on the condition number of the Hessian matrix of the loss function (denoted as H) (Nesterov, 2003), that is, λmax(H)λmin(H) , where λmax and λmin are the maximal and minimal eigenvalues of H, respectively. A smaller condition number leads to a faster convergence rate. For L(µ,Σ,x0; θ), the Hessian is calculated as
H = ∂2L ∂ 2θ · ∂ θ ∂θ · ( ∂ θ ∂θ )T + ∂L ∂ θ · ∂ 2 θ ∂θ2
(9)
Again, if we assume θ is a linear function, we have H ∝ I for L(µ,Σ,x0; θ) and H ∝ Σ + I for L(0, I,x0; θ). It is clear that if we set the prior to beN (µ,Σ), the condition number of H equals 1, achieving the smallest value of the condition number. Therefore, it can accelerate the convergence. Readers may refer to the appendix A.2 for more details.

heading: 4 APPLICATION TO VOCODER
text: In this section, we apply PriorGrad to a vocoder model, as visually described in Figure 3.

heading: 4.1 PRIORGRAD FOR VOCODER
text: Our formulation of PriorGrad applied to a vocoder is based on DiffWave (Kong et al., 2021), where the model synthesizes time-domain waveform conditioned on a mel-spectrogram that contains a compact frequency feature representation of the data. Unlike the previous method that used ∼ N (0, I), PriorGrad network is trained to estimate the noise ∼ N (0,Σ) given the destroyed signal√ ᾱtx0 + √ 1− ᾱt . Same as Kong et al. (2021), the network is also conditioned on the discretized
index of the noise level √ ᾱt with the diffusion-step embedding layers, and the mel-spectrogram c with the conditional projection layers.
Based on the mel-spectrogram condition, we propose leveraging a normalized frame-level energy of the mel-spectrogram for acquiring data-dependent prior, exploiting the fact that the spectral energy contains an exact correlation to the waveform variance1. First, we compute the frame-level energy by applying roots of sum of exponential to c, where c is the mel-spectrogram from the training dataset. We then normalize the frame-level energy to a range of (0, 1] to acquire the data-dependent diagonal variance Σc. In this way, we can use the frame-level energy as a proxy of the standard deviation for the waveform data we want to model. This can be considered as a non-linear mapping between the mel-scale spectral energy and the standard deviation of the diagonal Gaussian.
We setN (0,Σ) as the forward diffusion prior for each training step by upsampling Σc in the framelevel to Σ in the waveform-level using a hop length for the given vocoder. We chose a zero-mean prior in this setup reflecting the fact that the waveform distribution is zero-mean. In practice, we imposed the minimum standard deviation of the prior to 0.1 through clipping to ensure numerical stability during training2. We have tried several alternative sources of conditional information to compute the prior, such as voiced/unvoiced labels and phoneme-level statistics, but they resulted in a worse performance. We justify our choice of using frame-level energy in the appendix A.3.

heading: 4.2 EXPERIMENTAL SETUP
text: We used LJSpeech (Ito & Johnson, 2017) dataset for all experiments, which is a commonly used open-source 24h speech dataset with 13,100 audio clips from a single female speaker. We used an 80-band mel-spectrogram feature at the log scale from the 22,050Hz volume-normalized speech, with the 1024-point FFT, 80Hz and 7,600Hz low- and high-frequency cutoff, and a hop length of 256. We used 13,000 clips as the training set, 5 clips as the validation set, and the remaining 95 clips as the test set used for an objective and subjective audio quality evaluation. We followed the publicly available implementation3, where it uses a 2.62M parameter model with an Adam optimizer (Kingma & Ba, 2014) and a learning rate of 2 × 10−4 for a total of 1M iterations. Training for 1M iterations took approximately 7 days with a single NVIDIA A40 GPU. We used the default diffusion steps with T = 50 and the linear beta schedule ranging from 1× 10−4 to 5× 10−2 for training and inference, which is the default setting of DiffWaveBASE model used in Kong et al. (2021). We also used the fast Tinfer = 6 inference noise schedule from DiffWaveBASE without modification.

heading: 4.3 EXPERIMENTAL RESULTS
text: We conducted experiments to verify whether our method can learn the reverse diffusion process faster, by comparing to the baseline DiffWave model with ∼ N (0, I). First, we present the model convergence result by using a spectral domain loss on the test set to show the fast training of PriorGrad. Second, we provide both objective and subjective audio quality results, where PriorGrad offered the improved quality. Finally, we measure a tolerance to the reduction of the network capacity, where PriorGrad exhibits an improved parameter efficiency. We leave the description of the objective metrics we collected to the appendix A.4.
1This property about the spectral density is dictated by Parseval’s theorem (Stoica et al., 2005). 2In our preliminary study, we applied grid search over the minimum standard deviation from 0.1 to 0.0001, and it showed negligible difference as long as the minimum is clipped to the reasonably low value. Not clipping it entirely can result in numerical instability.
3https://github.com/lmnt-com/diffwave
Table 1: 5-scale subjective mean opinion score (MOS) results of PriorGrad vocoder with 95% confidence intervals.
Method Tinfer Training Steps
500K 1M
GT - 4.42 ± 0.07
DiffWave 6 3.98 ± 0.08 4.01 ± 0.0850 4.12 ± 0.08 4.12 ± 0.08
PriorGrad 6 4.02 ± 0.08 4.14 ± 0.0850 4.21 ± 0.08 4.25 ± 0.08
Model convergence We used a widely adopted spectral metric with log-mel spectrogram mean absolute error (LS-MAE) as the proxy of the convergence for the waveform synthesis model. We can see from Figure 4 that PriorGrad exhibited a significantly faster spectral convergence compared to the baseline. In the auditory test, we observed that PriorGrad readily removed the background white noise early in training, whereas the baseline needed to learn the entire reverse diffusion process starting from ∼ N (0, I) which contains little information. Table 1 shows the 5-scale subjective mean opinion score (MOS) test of PriorGrad from Amazon Mechanical Turk. We observed that PriorGrad outperformed the baseline DiffWave model with 2 times fewer training iterations. PriorGrad also outperformed the baseline on objective speech and distance metrics as shown in Table 3, such as Multi-resolution STFT error (MR-STFT) (Yamamoto et al., 2020), Mel cepstral distortion (MCD) (Kubichek, 1993) and F0 root mean square error (F0 RMSE) and a debiased Sinkhorn divergence (Feydy et al., 2019) between the prior and the ground-truth S(xT , x0), or the generated samples and the ground-truth S(x̃0, x0), consistent with the subjective quality results. Refer to A.7 for comparison to other state-of-the-art approaches with varying number of inference steps.
Parameter efficiency We measured a tolerance to a reduced diffusion network capacity by setting the width of the dilated convolutional layers by half, leading to a smaller model with 1.23M parameters. We trained the small DiffWave and PriorGrad with the same training configurations for 1M steps. With Tinfer = 50, Table 2 confirmed that the performance degradation of PriorGrad is reduced compared to the baseline DiffWave model. The small PriorGrad was able to maintain the quality close to the larger baseline model, suggesting that having access to the informative prior distribution can improve parameter efficiency of the diffusion-based generative model.

heading: 5 APPLICATION TO ACOUSTIC MODEL
text: In this section, we present the application of PriorGrad to the acoustic models, as visually described in Figure 5.

heading: 5.1 PRIORGRAD FOR ACOUSTIC MODEL
text: The acoustic models generate a mel-spectrogram given a sequence of phonemes with the encoderdecoder architecture. In other words, the acoustic model is analogous to the text-conditional image generation task (Reed et al., 2016; Ramesh et al., 2021), where the target image is 2D melspectrogram. We implement the PriorGrad acoustic model by using the approach in Ren et al. (2020) as a feed-forward phoneme encoder, and using a diffusion-based decoder with dilated convolutional layers based on Kong et al. (2021). Note that a similar decoder architecture was used in Jeong et al. (2021).
To build the adaptive prior, we compute the phoneme-level statistics of the 80-band mel-spectrogram frames by aggregating the frames that correspond to the same phoneme from the training data, where phoneme-to-frame alignment is provided by the Montreal forced alignment (MFA) toolkit (McAuliffe et al., 2017). Specifically, for each phoneme, we acquire the 80-dimensional diagonal mean and variance from the aggregation of all occurrences in the training dataset. Then, we construct the dictionary of N (µ,Σ) per phoneme. To use these statistics for the forward diffusion prior, we need to upsample the phoneme-level prior sequence to the frame-level with the matching duration. This can be done by jointly upsampling this phoneme-level prior as same as the phoneme encoder output based on the duration predictor (Ren et al., 2019) module.
Following Algorithm 1, we train the diffusion decoder network with the mean-shifted noisy melspectrogram xt = √ ᾱt(x0−µ) + √ 1− ᾱt as the input. The network estimates the injected noise
∼ N (0,Σ) as the target. The network is additionally conditioned on the aligned phoneme encoder output. The encoder output is added as a bias term of each dilated convolution layers of the gated residual block with the layer-wise 1× 1 convolution.

heading: 5.2 EXPERIMENTAL SETUP
text: For the acoustic model experiments, we implemented the Transformer phoneme encoder architecture identical to FastSpeech 2 (Ren et al., 2020) and the convolutional diffusion decoder architecture based on Jeong et al. (2021). We adopted the open-source implementation of the DiffWave architecture (Kong et al., 2021) with 12 convolutional layers, same as Jeong et al. (2021). We used the beta schedule with T = 400 for training and used the fast reverse sampling schedule with Tinfer = 6 with a grid search method (Chen et al., 2021). We followed the same training and inference protocols of Ren et al. (2020) and used a pre-trained Parallel WaveGAN (PWG) (Yamamoto et al., 2020) vocoder for our controlled experiments. For state-of-the-art comparison, we used HiFi-GAN (Kong et al., 2020) vocoder to match the previous work. We conducted a comparative study of PriorGrad acoustic model with a different diffusion decoder network capacity, i.e., a small model with 3.5M parameters (128 residual channels), and a large model with 10M parameters (256 residual channels). Training for 300K iterations took approximately 2 days on a single NVIDIA P100 GPU. We leave the additional details in the appendix A.5.

heading: 5.3 EXPERIMENTAL RESULTS
text: Model convergence We can see from Table 4 that applyingN (µ,Σ) with PriorGrad significantly accelerated the training convergence and exhibited higher-quality speech for different decoder capacities and training iterations. A high-capacity (10M) PriorGrad was able to generate high-quality mel-spectrogram with as few as 60K training iteration, whereas the baseline model was not able to match the performance even after 300K iterations. This is because the training iterations were insufficient for the baseline to learn the entire diffusion trajectory with the standard Gaussian as the prior, leading to lower performance. The small (3.5M) baseline model exhibited an improved
quality compared to the large baseline for the same training iterations, which further suggests that the convergence of the large baseline model is slow. By contrast, by easing the burden of learning the diffusion process by having access to the phoneme-level informative forward prior, PriorGrad achieved high-quality samples significantly earlier in the training. Refer to Appendix A.6 for additional experimental results including an alternative model with the jointly trainable diffusion prior estimation, and Appendix A.7 for an expanded comparison to previous work, where PriorGrad achieves a new state-of-the-art acoustic model.
Parameter efficiency Similar to the vocoder experiment, we assessed whether we can retain the high performance with a reduced diffusion decoder capacity from 10M to 3.5M parameters. The small PriorGrad also consistently outperformed the baseline model and achieved an almost identical perceptual quality to the high-capacity PriorGrad. The small PriorGrad outperformed both the small and large baseline models, which confirms that PriorGrad offers an improved parameter efficiency and tolerance to the reduced network capacity. This suggests that PriorGrad provides a way to build practical and efficient diffusion-based generative models. This holds a special importance for speech synthesis models, where the efficiency plays a key role in model deployment in realistic environments.

heading: 6 DISCUSSION AND CONCLUSION
text: In this study, we investigated the potential inefficiency of recently proposed diffusion-based model as a conditional generative model within the speech synthesis domain. Our method, PriorGrad, directly leverages the rich conditional information and provides an instance-level non-standard adaptive Gaussian as the prior of the forward diffusion process. Through extensive experiments with the recently proposed diffusion-based speech generative models, we showed that PriorGrad achieves a faster model convergence, better denoising of the white background noise, an improved perceptual quality, and parameter efficiency. This enables the diffusion-based generative models to be significantly more practical, and real-world applications favor lightweight and efficient models for deployments.
Whereas our focus is on improving the diffusion-based speech generative models, PriorGrad has a potential to expand the application beyond the speech synthesis domain. In the image domain, for example, PriorGrad can potentially be applied to image super-resolution tasks (Saharia et al., 2021) by using the patch-wise mean and variance of the low-resolution image to dynamically control the forward/reverse process. The method could potentially be used for depth map conditional image synthesis (Esser et al., 2020), where the depth map can be mapped as the prior variance using the fact that the monocular depth corresponds to the camera focus and visual fidelity between the foreground and background. We leave these potential avenue of PriorGrad for future work.
PriorGrad also has limitations that require further investigation. Although PriorGrad offers a variety of practical benefits as presented, it may also require a well-thought out task-specific design to compute the data-dependent statistics (or its proxy), which may be unsuitable depending on the granularity of the conditional information. Building an advanced approach to enable a more generalized realization of PriorGrad will be an interesting area of future research.

heading: ACKNOWLEDGEMENT
text: This work was supported by the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2021, Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)], and the MSIT (Ministry of Science, ICT), Korea, under the High-Potential Individuals Global Training Program (2020-0-01649) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation), and Microsoft Research Asia.

heading: A APPENDIX

heading: A.1 DERIVATION OF THE ELBO LOSS
text: Proposition 1 Let ∼ N (0,Σ) and x0 ∼ qdata. Then, under the parameterization in Eq. equation 8, the ELBO loss will be
− ELBO = C(Σ) + T∑ t=1 γtEx0, ‖ − θ( √ ᾱt(x0 − µ) + √ 1− ᾱt , t)‖2Σ−1 ,
for some constant C, where ‖x‖2Σ−1 = x TΣ−1x, γt = β2t 2σ2tαt(1−ᾱt) for t > 1, and γ1 = 12α1 .

heading: Proof:
text: According to Algorithm 1, the input x0 is normalized by subtracting its mean µ. In the following, we denote x̃0 = x0 − µ and xt = √ ᾱtx̃0 + √ 1− ᾱt .
According to Equation (3), the ELBO loss is ELBO = −Eq ( KL(q(xT |x̃0)||p(xT )) + ∑T t=2 KL(q(xt−1|xt, x̃0)||pθ(xt−1|xt))− log pθ(x̃0|x1) ) .
Let i’s i.i.d.∼ N (0,Σ). Similarly to the Equation (1), we have q(xt|x̃0) = N (xt; √ ᾱtx̃0, (1 − ᾱt)Σ).
By Bayes rule and Markov chain property,
q(xt−1|xt, x̃0) = q(xt|xt−1)q(xt−1|x̃0)
q(xt|x̃0)
= N (xt;
√ αtxt−1, βtΣ)N (xt−1; √ ᾱt−1x̃0, (1− ᾱt−1Σ))
N (xt; √ ᾱtx̃0, (1− ᾱt)Σ)
= (2πβ̃t) − d2 exp ( − 1
2β̃t
∥∥∥∥xt−1 − √ᾱt−1βt1− ᾱt x̃0 ∥∥∥∥2
Σ−1
) ,
where ‖x‖2Σ−1 = x TΣ−1x. Therefore,
q(xt−1|xt, x̃0) = N ( xt−1; ᾱt−1β
1− ᾱt x̃0 +
√ αt(1− ᾱt−1)
1− ᾱt xt, β̃tΣ
)
Then, we calculate each term of the ELBO expansion. The first term is
EqKL(q(xT |x̃0)‖p(xT )) = ᾱT 2 Ex̃0‖x̃0‖2Σ−1 − d 2 (ᾱT + log(1− ᾱT )). (10)
The second term is
EqKL(q(xt−1|xt, x̃0)‖pθ(xt−1|xt)) = βt
2αt(1− ᾱt−1) Ex̃0, ‖ − θ(xt, t)‖2Σ−1 . (11)
Finally, we have
Eq log pθ(x̃0|x1) = − 1
2 log(2πβ1) d det(Σ)− 1 2α1 Ex̃0, ‖ − θ(x1, 1)‖2Σ−1 . (12)
Combining Equation (10), (11) and (12) together, we can get the result in the Proposition.

heading: A.2 THEORETICAL BENEFITS OF PRIORGRAD
text: Proposition 2 Let L(µ,Σ,x0; θ) denote the −ELBO loss in Proposition 1. Suppose that θ is a linear function. Under the constraint that det(Σ) = det(I), we have minθ L(µ,Σ,x0; θ) ≤ minθ L(0, I,x0; θ).
Proof : We use L(µ,Σ,x0; θ) to denote -ELBO. According to Equation (10), (11) and (12), we have
L(µ,Σ,x0; θ) = ᾱT 2 Ex0‖x̃0‖ 2 Σ−1 + 1 2 log det(Σ) + T∑ t=1 γtEx0, ‖ − θ( √ ᾱtx̃0 + √ 1− ᾱt , t)‖2Σ−1
+ d
2 log(2πβ1)−
d 2 (ᾱT + log(1− ᾱT )),
where x̃0 = x0 − µ. We assume that θ(·) is a scalar function with parameter freedom d, i.e., θ( √ ᾱtx̃0 + √ 1− ᾱt , t) = θ( √ ᾱtx̃0 + √ 1− ᾱt ), where θ ∈ Rd×d with constraint θ̃ = Qθ = diag(θ̃1, · · · , θ̃d) and Q is the eigenmatrix for Σ−1, i.e., Σ−1 = QT Σ̃−1Q with Σ̃−1 = diag( 1σ1 , · · · , 1 σd ). For the term ∑T t=1 γtEx0, ‖ − θ( √ ᾱtx̃0 + √ 1− ᾱt )‖2Σ−1 , we have
T∑ t=1 γtEx0, ‖ − θ( √ ᾱtx̃0 + √ 1− ᾱt )‖2Σ−1
= T∑ t=1 γtEx0, ‖Q( − θ( √ ᾱtx̃0 + √ 1− ᾱt )‖2Σ̃−1
= d∑ j=1
( 1
σj T∑ t=1 γt ( σj + (1− ᾱt)θ̃2jσj + ᾱtθ̃2jσj − 2σj √ 1− ᾱtθ̃j
))
= d∑ j=1
( 1
σj T∑ t=1 γt ( σj + θ̃ 2 jσj − 2σj √ 1− ᾱtθ̃j
))
= d∑ j=1 ( T∑ t=1 γt ( 1 + θ̃2j − 2 √ 1− ᾱtθ̃j )) (13)
where the second equation is established by taking expectation of x0 and . The above equation will achieve its minimum at θ̃j = ∑T t=1 γt √ 1−ᾱt∑T
t=1 γt ,∀j, and the minimum is∑d
j=1 (∑T t=1 γt − ( ∑T t=1 γt √ 1−ᾱt)2∑T t=1 γt ) .
For L(0, I,x0; θ), we have
L(0, I,x0; θ) = ᾱT 2 Ex0‖x0‖ 2 + 1 2 log det(I) + T∑ t=1 γtEx0, ‖ − θ( √ ᾱtx0 + √ 1− ᾱt , t)‖2
+ d
2 log(2πβ1)−
d 2 (ᾱT + log(1− ᾱT )),
Similarly, we can get the minimum of ∑T t=1 γtEx0, ‖ −θ( √ ᾱtx0+ √ 1− ᾱt , t)‖2 with a diagonal
θ is ∑d j=1 (∑T t=1 γt − ( ∑ t=1 γt √ 1−ᾱt)2∑T t=1 γt(1−ᾱt+ᾱtσj) ) . Under the condition that det(Σ) = det(I), we have
the minimum of ∑d j=1 (∑T t=1 γt − ( ∑ t=1 γt √ 1−ᾱt)2∑T t=1 γt(1−ᾱt+ᾱtσj) ) (over all possibilities of (σ1, · · · , σd))
will be ∑d j=1 (∑T t=1 γt − ( ∑ t=1 γt √ 1−ᾱt)2∑T t=1 γt ) (by solving the constrained minimization problem). Thus, we get the result in the Proposition.
Remark: From Equation (12), we have the second order derivative of L(µ,Σ,x0; θ) with respect to θj equals ∑T t=1 γt for all j = 1, · · · , d. Thus the condition number of the Hessian matrix equals
1. Similarly, the second order derivative of L(0, I,x0; θ) with respect to θj equals ∑T t=1 γt(1−ᾱt+
ᾱtσj). Thus the Hessian matrix equals c1I+ c2Σ with c1 = ∑T t=1 γt(1− ᾱt) and c2 = ∑T t=1 γtᾱt, whose condition number is no less than 1.

heading: A.3 EXPLORATION ON THE CONDITIONAL INFORMATION SOURCE OF PRIORGRAD VOCODER
text: In this section, we provide further discussion regarding the source of the conditional information for constructing PriorGrad vocoder and empirical justification of selecting the frame-level spectral
energy as the prior. Considering the waveform synthesis model as a vocoder, the main application of such model is a spectrogram inversion module of the text-to-speech pipeline. That is, the vocoder model is usually combined with the acoustic model as a front-end, where the acoustic model generates the mel-spectrogram based on text input. FastSpeech 2 (Ren et al., 2020) demonstrated a significant improvement in the quality of the acoustic model leveraged by carefully designed additional speech-related features for fine-grained supervision, such as voiced/unvoiced (V/UV) label obtained from F0 contours, or phoneme-to-frame alignment labels. We explored V/UV or phoneme labels as alternative sources of information for constructing the prior.
Figure 6 shows scatter plots of the waveform audio clip from the test set corresponding to the standard deviation labels assigned by Σc acquired from the training set. Because the V/UV label is binary, we can only assign two different prior variances to the waveform distribution, which is an overly coarse assumption of the data distribution. Phoneme-level prior can offer more fine-grained approximate prior. However, we found that the variance statistics acquired from the training set is misaligned with the unobserved waveform, where the actual variance can be significantly different to the target prior, leading to an inconsistent result. The frame-level spectral energy for PriorGrad exhibited the best alignment of the label and the unobserved waveform data and demonstrated consistent results in quality.

heading: A.4 DESCRIPTION OF OBJECTIVE METRICS FOR PRIORGRAD VOCODER
text: In this section, we describe the objective metrics we collected to evaluate the performance of PriorGrad vocoder.
Log-mel spectrogram mean absolute error (LS-MAE) This is a spectral regression error between the log-mel spectrogram computed between the synthesized waveform and the ground-truth. We used the same STFT function for computing the mel-spectrogram as the conditional input of DiffWave and PriorGrad.
Multi-resolution STFT error (MR-STFT) This measures the spectral distance across multiple resolutions of STFT. MR-STFT is widely adopted as a training objective of recent neural vocoders because using multiple resolution windows can capture the time-frequency distributions of the realistic speech signal. We used the resolution hyperparameter proposed in Parallel WaveGAN (Yamamoto et al., 2020), which is implemented in an open-source library4.
Mel-cepstral distortion (MCD) MCD is a widely adopted speech metric (Kubichek, 1993) which measures the distance between the mel cepstra. We used an open-source implementation with default parameters5.
F0 root mean square error (F0 RMSE) This measures the accuracy (measured by RMSE) of the fundamental frequency (F0) which is an approximate frequency of the (quasi-)periodic structure of
4https://github.com/csteinmetz1/auraloss 5https://github.com/MattShannon/mcd
voiced speech signals. We used the Saw-tooth Waveform Inspired Pitch Estimation (SWIPE) (Camacho & Harris, 2008) with a hop size of 128 to obtain F0 using an open-source implementation6.
Debiased Sinkhorn divergence This is a positive definite approximation of optimal transport (Wasserstein) distance between two data distributions (Feydy et al., 2019). This metric can be used as a mathematically grounded assessment of the informative prior, which is evidenced by a recent study that the diffusion process coincides with the optimal transport map (Khrulkov & Oseledets, 2022). We used the open-source library 7 with default parameters. Using DiffWave and PriorGrad with Tinfer = 6, We calculated the test set average of the Sinkhorn divergence using Monte Carlo estimate with 100 samples per test data point. We denote S(xT , x0) as the distance between the prior and the real data, and S(x̃0, x0) as the distance between the generated samples and the real data.

heading: A.5 ADDITIONAL DETAILS OF PRIORGRAD ACOUSTIC MODEL
text: In this section, we describe additional details of the experimental design of the PriorGrad acoustic model. The feed-forward Transformer-based phoneme encoder has 11.5M parameters trained with the Adam optimizer with the learning rate schedule identically described in Ren et al. (2020). The diffusion decoder is simultaneously trained with the same training configurations.
For a fair comparative study between different diffusion models, we searched for the best performing baseline model with N (0, I) first, then applied PriorGrad to this baseline. We applied T = 400 for training the diffusion decoder with a linearly spaced beta schedule ranging from 1 × 10−4 to 5 × 10−2. We found that 1 × 10−4 to 2 × 10−2 used for waveform domain in Kong et al. (2021) performed poorly for the baseline, where the model failed to capture the high-frequency details of the mel-spectrogram. This indicates that the optimal training noise schedule can be different, depending on the data domain (Kong & Ping, 2021). On the contrary, PriorGrad’s performance was similar under different choices of training noise schedules, suggesting that PriorGrad also features robustness regarding designing the noise schedules for model training.
We found that the fast reverse noise schedule with Tinfer = 6 described in Kong et al. (2021) performed poorly for the baseline model with N (0, I). Thus, we applied grid search over the Tinfer = 6 reverse schedule for each model, which is a similar approach to that in Chen et al. (2021). This enabled a fair comparative study of each model configuration when using the fast noise schedule for sampling. We applied a fine-grained grid search over every possible combination of the monotonically increasing betas based on the L1 loss between the model prediction and the target from the validation set. We applied the following range for the grid search: {1, 2, ..., 8, 9} × {10−1, 10−1} for Tinfer = 2, {1, 2, ..., 8, 9} × {10−4, 10−3, 10−2, 10−2, 10−1, 10−1} for Tinfer = 6, and {1, 2, ..., 8, 9} × {10−4, 10−4, 10−3, 10−3, 10−2, 10−2, 10−2, 10−2, 10−1, 10−1, 10−1, 10−1} for Tinfer = 12.
We found that setting relatively high values of beta at the last steps was important for the quality of the baseline model, whereas PriorGrad was significantly more robust to the choice of the noise schedule. Table 5 shows the optimal beta schedules for each model configuration from the grid search method.
6https://github.com/r9y9/pysptk 7https://www.kernel-operations.io/geomloss/

heading: A.6 COMPARISON TO TRAINABLE DIFFUSION PRIOR
text: In this section, we present additional results regarding the challenges of formulating a jointly trainable estimation of the diffusion prior. Based on the improvements achieved by PriorGrad, it is natural to suppose that the jointly trainable diffusion prior distribution with additional parameterization might perform similar or better than the data-dependent prior. In this section, we explored a feasibility of the diffusion prior with trainable parameters. For acoustic model, we explored two setups: 1. replacing the fixed data-dependent prior with the estimated N (µθ,Σθ) from the projection layer of the phoneme encoder, 2. parameterizing N (µθ,Σθ) with an independently defined convolutional layers with the data-dependent (µ,Σ) we applied to PriorGrad as an input to acquire a refined prior. For the vocoder model, instead of the spectral energy as the proxy of the waveform variance, we explored an independently defined diffusion prior estimator by using the smaller-scale convolutional network with the mel-spectrogram as the input.
We found that all approaches failed to jointly estimate or refine the informative prior distribution. For the acoustic model, the estimated mean was significantly noisier than our data-dependent one, and the estimated variance is collapsed to a single point which is uninformative. These models performed significantly worse than the baseline method with the fixed N (0, I), as measured by Table 6. We visualize the data-dependent prior compared to the estimated version in Figure 7. For the vocoder model, the diffusion prior estimator resulted in a divergence of the estimated mean and collapse of the estimated variance to zero, leading to the training failure. We do not draw the conclusive claim that the jointly trainable diffusion prior is not possible. However, the results suggest that the fixed data-dependent diffusion prior with PriorGrad is an effective and easy-to-use method to improve the efficiency of the diffusion-based generative model without introducing additional complexities to the network.

heading: A.7 COMPARISON TO STATE-OF-THE-ART
text: In this section, we present additional results of PriorGrad by comparing to recent state-of-the-art speech synthesis models and show that PriorGrad is competitive with or sometimes outperforms the previous models. We provide a detailed analysis with a varying number of inference denoising steps (Tinfer), along with their speed measured by a real-time factor (RTF) on the NVIDIA A40 GPU, and the model capacity measured by the number of parameters.
Vocoder comparison Table 7 provides an expanded MOS result of the fully converged PriorGrad vocoder with the 1M training steps. For the fast inference noise schedule, we used Tinfer = 6 defined as [0.0001, 0.001, 0.01, 0.05, 0.2, 0.5] from DiffWaveBASE without modification, and used Tinfer = 12 defined as [0.0001, 0.0005, 0.0008, 0.001, 0.005, 0.008, 0.01, 0.05, 0.08, 0.1, 0.2, 0.5]. For the full Tinfer = 50, PriorGrad significantly outperformed the baseline DiffWave. Furthermore, PriorGrad achieved an even larger performance gap with a significantly reduced network capacity, compared to the flow-based vocoders such as WaveGlow (Prenger et al., 2019) and WaveFlow (Ping et al., 2020). This suggests that PriorGrad is a competitive likelihood-based neural vocoder, and enables a step closer to the current state-of-the-art GAN-based model (HiFi-GAN, Kong et al. (2020)). The speed is close to real-time (RTF being close to 1), but it is noticeably slower than other approaches.
For the fast inference with Tinfer = 6 denoising steps, PriorGrad was able to match the performance to the baseline DiffWave model with Tinfer = 50 while still outperforming the flow-based vocoders with comparable RTF. This further showcase the efficiency of PriorGrad, where having access to the informative non-standard Gaussian prior accelerates the inference of the diffusion-based model.
Acoustic model comparison We additionally trained our baseline diffusion-based acoustic model and the enhanced model with PriorGrad which are compatible with the pretrained HiFi-GAN (Kong et al., 2020) vocoder. This enabled a fair assessment of PriorGrad compared to recent state-of-theart acoustic models. We trained our acoustic model for the full convergence with 1M training steps. We found that the small PriorGrad with 3.5M decoder parameters performed almost identical to the high-capacity (10M) model. Therefore, we chose the small decoder model as our final choice for comparison.
Table 8 provides an expanded MOS result of PriorGrad acoustic model with varying Tinfer and comparison to the recent representative acoustic model from different categories: Feed-forward (FastSpeech 2, Ren et al. (2020)), Flow-based (Glow-TTS, Kim et al. (2020)), and the concurrent diffusion-based model (Grad-TTS, Popov et al. (2021)). The results show that PriorGrad acoustic model is competitive to the state-of-the-art models. PriorGrad with Tinfer = 12 provided an identical quality to the state-of-the-art Grad-TTS with Tinfer = 10. This is achieved with an approximately 1.75 times faster RTF and sets PriorGrad a new state-of-the-art acoustic model.
Armed with the informative non-standard Gaussian prior, PriorGrad is robust to the extremely small number of inference steps with Tinfer = 2. The fast PriorGrad matches the performance with the state-of-the-art flow-based acoustic model, Glow-TTS, with fewer parameters. The RTF becomes close to the feed-forward FastSpeech 2 as well as scoring higher MOS. By contrast, the baseline acoustic model withN (0, I) suffered a significant degradation for smaller Tinfer. This result further highlights the efficiency of PriorGrad, where it is robust to the reduced number of denoising steps and enables a significantly accelerated inference.

heading: A.8 MOS EVALUATION DETAILS
text: We conducted a standard and widely adopted protocol for the MOS evaluation using Amazon Mechanical Turk. We constructed 20 randomly selected clips from the test set, and applied a total of 450 evaluations for each model with the 5-scale scoring on audio naturalness. To encompass the diverse set of listeners, a single listener can only evaluate the three randomly exposed samples. Therefore, there are 150 unique listeners for each model. For reliability, only skilled listeners with the evaluation approval rate higher than 98% and the number of previously approved evaluations higher than 100 are qualified to conduct the test. The listeners were rewarded $0.2 for each evaluation. Approximately $130 were spent as compensation for each set of the MOS test.

heading: A.9 AUDIO SAMPLES DEMO PAGE
text: We provide a demo page of the audio samples used in this study: https://speechresearch. github.io/priorgrad/

Review summaries:
This work builds on denoising diffusion probabilistic models (DDPM), and argues for to modify the forward and backward diffusion processes such that instead of using an uninformative prior $p(\mathbf{x}_T) = \mathcal{N}(0,\mathbf{I})$, they use a data-dependent prior $p(\mathbf{x}_T) = \mathcal{N}(\mu, \Sigma)$, where the mean and covariance are derived from training data, in a "pre-processing" step.  The authors show that, under restrictive conditions, the ELBO obtained with the proposed prior is smaller than that obtained with the uninformative prior. Under similar restrictive conditions, the authors argue that convergence rate of the parameter optimization is better conditioned, and as a consequence faster.  This work illustrates the benefits of the proposed method using two applications, one to a vocoder, and one to an acoustic model. Experiments rely on several very recent related work, especially concerning technical details of the architectures, and data pre-processing choices. Overall, two methods are compared, a baseline DDPM and the proposed DDPM that uses training data to build a better prior.

This paper presents an extension to conditional denoising diffusion models for text-to-speech (TTS). The initial noise in the inference procedure is not sampled from a standard Gaussian but from a data-dependent conditional non-standard Gaussian, which is called PriorGrad. The motivation is to increase inference speed and training convergence. This method is applied to DiffWave used as a vocoder, and also applied to DiffWave used as an acoustic model (phonemes to mel spect). 

The paper proposes to use a data-dependent, adaptive prior for the noise used in conditional DDPMs. In particular, it proposes to move from the standard to the non-standard Gaussian prior. For that, the ELBO loss is reformulated by taking into account adaptive means and variances and the sampling procedure is also adapted. Statistics (directly or indirectly derived) from conditioning signals are used to compute time-wise and instance-based means/standard deviations. An empirical evaluation is conducted on two conditional audio generation tasks: vocoding and acoustic modeling, showing some favorable results.