BDDM: BILATERAL DENOISING DIFFUSION MODELS
Year: 2022

Abstract:
Diffusion probabilistic models (DPMs) and their extensions have emerged as competitive generative models yet confront challenges of efficient sampling. We propose a new bilateral denoising diffusion model (BDDM) that parameterizes both the forward and reverse processes with a schedule network and a score network, which can train with a novel bilateral modeling objective. We show that the new surrogate objective can achieve a lower bound of the log marginal likelihood tighter than a conventional surrogate. We also find that BDDM allows inheriting pre-trained score network parameters from any DPMs and consequently enables speedy and stable learning of the schedule network and optimization of a noise schedule for sampling. Our experiments demonstrate that BDDMs can generate high-fidelity audio samples with as few as three sampling steps. Moreover, compared to other state-of-the-art diffusion-based neural vocoders, BDDMs produce comparable or higher quality samples indistinguishable from human speech, notably with only seven sampling steps (143x faster than WaveGrad and 28.6x faster than DiffWave).

heading: 1 INTRODUCTION
text: Deep generative models have shown a tremendous advancement in speech synthesis (van den Oord et al., 2016; Kalchbrenner et al., 2018; Prenger et al., 2019; Kumar et al., 2019a; Kong et al., 2020b; Chen et al., 2020; Kong et al., 2021). Successful generative models can mainly divide into the GAN (Goodfellow et al., 2014) based approach and the likelihood-based approach. The former uses an adversarial training procedure, but the training is sometimes unstable, and the relevant training objectives are not suitable to compare different GAN models; the latter uses log-likelihood or surrogate objectives for training, but they also have intrinsic limitations. For example, the autoregressive models (van den Oord et al., 2016; Kalchbrenner et al., 2018) have exceedingly slow sampling speed and poor scaling properties on high-dimensional data, as its sampling is inherently a sequential process. Likewise, the flow-based models (Dinh et al., 2016; Kingma & Dhariwal, 2018; Chen et al., 2018; Papamakarios et al., 2021) rely on specialized architectures to build a normalized probability model. The model training is less parameter-efficient and typically more complex than other likelihoodbased generative models (Bond-Taylor et al., 2021). Some other prior works use the surrogate objectives, such as the evidence lower bound in variational auto-encoders (Kingma & Welling, 2013; Rezende et al., 2014; Maaløe et al., 2019) and the contrastive divergence in energy-based models (Hinton, 2002; Carreira-Perpinan & Hinton, 2005). They improve inference speed but often yield lower-quality samples than autoregressive and GAN-based models and typically only work well for low-dimensional data (Bond-Taylor et al., 2021).
An up-and-coming class of likelihood-based models is the diffusion probabilistic models (DPMs) (Sohl-Dickstein et al., 2015), which shed light on the generation of high-quality samples comparable or even superior (Dhariwal & Nichol, 2021) to the state-of-the-art (SOTA) GAN-based models. Sohl-Dickstein et al. (2015) introduced the idea of using an iterative forward process for destroying the structure of a given distribution while learning the reverse process for restoring the structure in the data. With a different modeling objective, the score-based generative models (Song & Ermon, 2019) used a neural network trained with the score matching (Hyvarinen & Dayan, 2005) to produce
samples via Langevin dynamics. As a seminal work of DPMs, Ho et al. (2020) proposed the denoising diffusion probabilistic models (DDPMs), which exploited a connection (Vincent, 2011) between the above two ideas that the diffusion framework can be viewed as a specific approach to provide support to score matching. For the first time, DDPMs present high-quality image synthesis results on par with GAN models, especially on several higher-resolution image benchmarks. Subsequently, score matching and denoising diffusion models were applied to speech syntheses, such as Wavegrad (Chen et al., 2020) and a parallel contribution Diffwave (Kong et al., 2021), and demonstrated the capability of generating high fidelity audio samples outperforming non-autoregressive GAN-based models (Yamamoto et al., 2020; Kumar et al., 2019b; Yang et al., 2020; Bińkowski et al., 2020) meanwhile matching the quality of SOTA autoregressive methods (Chen et al., 2020).
Despite the compelling results, the diffusion generative models are two to three orders of magnitude slower than other generative models such as GANs and VAEs. Their primary limitation is that they require up to thousands of diffusion steps during training to learn the target distribution. Therefore a large number of reverse steps are often required at sampling time. Recently, extensive investigations have been conducted to reduce the sampling steps for efficiently generating high-quality samples, which we will discuss in the related work in Section 2. Distinctively, we conceived that we might train a neural network to efficiently and adaptively estimate a much shorter noise schedule for sampling while achieving generation performances comparable or superior to the conventional DPMs. With such an incentive, after introducing the conventional DPMs as our background in Section 3, we propose in Section 4 bilateral denoising diffusion models (BDDMs), named after a bilateral modeling perspective – parameterizing the forward and reverse processes with a schedule network and a score network, respectively. We theoretically derive that the schedule network should be trained after the score network is optimized. For training the schedule network, we propose a novel objective to minimize the gap between a newly derived lower bound and the log marginal likelihood. We describe the training algorithm as well as the fast and high-quality sampling algorithm in Section 5. The training of the schedule network converges very fast using our newly derived objective, and its training only adds negligible overhead to DDPM’s. In Section 6, our neural vocoding experiments demonstrated that BDDMs could generate high-fidelity samples with as few as three sampling steps. Moreover, our method can produce speech samples indistinguishable from human speech with only seven sampling steps (143x faster than WaveGrad and 28.6x faster than DiffWave).

heading: 2 RELATED WORK
text: Prior works showed that noise scheduling is crucial for efficient and high-fidelity sample generation in diffusion probabilistic models. DDPMs (Ho et al., 2020) shared a linear noise schedule for both training and sampling, which, however, requires thousands of sampling iterations to obtain competitive results. To speed up the sampling process, one class of related work, including (Chen et al., 2020; Kong et al., 2021; Nichol & Dhariwal, 2021), attempts to use a different, shorter noise schedule for sampling. For clarity, we thereafter denote the training noise schedule as β ∈ RT and the sampling noise schedule as β̂ ∈ RN with N < T . In particular, Chen et al. (2020) applied a grid search (GS) algorithm to select β̂. Unfortunately, GS becomes prohibitively slow when N grows large, e.g., N = 6 took more than a day on a single NVIDIA Tesla P40 GPU. This is because the time costs of GS algorithm grow exponentially withN , i.e.,O(9N ) with 9 bins as the default setting in Chen et al. (2020). Instead of searching, Kong et al. (2021) devised a fast sampling (FS) algorithm based on an expert-defined 6-step noise schedule for their score network. However, this specifically tuned noise schedule is hard to generalize to other score networks, tasks, or datasets.
Another class of noise scheduling methods searches for a subsequence of time indices of the training noise schedule, which we call the time schedule. DDIMs (Song et al., 2021a) introduced an accelerated reverse process that relies on a pre-specified time schedule. A linear and a quadratic time schedule were used in DDIMs and showed superior generation quality over DDPMs within 10 to 100 sampling steps. Nichol & Dhariwal (2021) proposed a re-scaled noise schedule for fast sampling, but this also requires pre-specifying the time schedule and the training noise schedule. Nichol & Dhariwal (2021) also proposed learning variances for the reverse processes, whereas the variances of the forward processes, i.e., the noise schedule, which affected both the means and variances of the reverse processes, were not learnable. According to the results of (Song et al., 2021a; Nichol & Dhariwal, 2021), using a linear or quadratic time schedule resulted in quite different performances in different datasets, implying that the optimal choice of schedule varies with the datasets. So, there
remains a challenge in finding a short and effective schedule for fast sampling on different datasets. Notably, Kong & Ping (2021) proposed a method to map a noise schedule to a time schedule for fast sampling. In this sense, searching for a time schedule becomes a sub-set of the noise scheduling problem, which resembles the above category of methods.
Although DPMs (Sohl-Dickstein et al., 2015) and DDPMs (Ho et al., 2019) mentioned that the noise schedule could be learned by re-parameterization, the approach was not investigated in their works. Closely related works that learn a noise schedule emerged until very recently. San-Roman et al. (2021) proposed a noise estimation (NE) method, which trained a neural net with a regression loss to estimate the noise scale from the noisy sample at each time point, and then predicted the next noise scale. However, NE requires a prior assumption of the noise schedule following a linear or Fibonacci rule. Most recently, a concurrent work to ours by Kingma et al. (2021) jointly trained a neural net to predict the signal-to-noise ratio (SNR) by maximizing the variational lower bound. The SNR was then used for noise scheduling. Different from ours, this scheduling neural net only took t as input and is independent of the noisy sample generated during the loop of sampling process. Intrinsically, with limited information about the sampled data, the predicted SNR could deviate from the actual SNR of the noisy data during sampling.

heading: 3 BACKGROUND

heading: 3.1 DIFFUSION PROBABILISTIC MODELS (DPMS)
text: Given i.i.d. samples {x0 ∈ RD} from an unknown data distribution pdata(x0), diffusion probabilistic models (DPMs) (Sohl-Dickstein et al., 2015) define a forward process q(x1:T |x0) =∏T t=1 q(xt|xt−1) that converts any complex data distribution into a simple, tractable distribution after T steps of diffusion. A reverse process pθ(xt−1|xt) parameterized by θ is used to model the data distribution: pθ(x0) = ∫ π(xT ) ∏T t=1 pθ(xt−1|xt)dx1:T , where π(xT ) is the prior distribution for starting the reverse process. Then, the variational parameters θ can be learned by maximizing the standard log evidence lower bound (ELBO):
Felbo := Eq [ log pθ(x0|x1)−
T∑ t=2 DKL (q(xt−1|xt,x0)||pθ(xt−1|xt))−DKL (q(xT |x0)||π(xT ))
] .
(1)

heading: 3.2 DENOISING DIFFUSION PROBABILISTIC MODELS (DDPMS)
text: As an extension to DPMs, denoising diffusion probabilistic models (DDPMs) (Ho et al., 2020) applied the score matching technique (Hyvarinen & Dayan, 2005; Song & Ermon, 2019) to define the reverse process. In particular, DDPMs considered a Gaussian diffusion process parameterized by a noise schedule β ∈ RT with 0 < β1, . . . , βT < 1:
qβ(x1:T |x0) := T∏ t=1 qβt(xt|xt−1), where qβt(xt|xt−1) := N ( √ 1− βtxt−1, βtI). (2)
Based on the nice property of isotropic Gaussians, one can express xt directly conditioned on x0:
qβ(xt|x0) = N (αtx0, (1− α2t )I), where αt = t∏ i=1 √ 1− βi. (3)
To revert this forward process, DDPMs employ a score network1 θ(xt, αt) to define
pθ(xt−1|xt) := N
( 1√
1− βt
( xt −
βt√ 1− α2t θ (xt, αt)
) ,Σt ) , (4)
1Here, θ(xt, αt) is conditioned on the continuous noise scale αt, as in (Song et al., 2021b; Chen et al., 2020). Alternatively, the score network can also be conditioned on a discrete time index θ(xt, t), as in (Song et al., 2021a; Ho et al., 2020). An approximate mapping of a noise schedule to a time schedule (Kong & Ping, 2021) exists, therefore we consider conditioning on noise scales as the general case.
where Σt is the co-variance matrix defined for the reverse process. Ho et al. (2020) showed that setting Σt = β̃tI = 1−α2t−1 1−α2t
βtI is optimal for a deterministic x0, while setting Σt = βtI is optimal for a white noise x0 ∼ N (0, I). Alternatively, Nichol & Dhariwal (2021) proposed learnable variances by interpolating the two optimals with a jointly trained neural network, i.e., Σt,θ(x) := diag(exp(vθ(x) log βt + (1− vθ(x)) log β̃t)), where vθ(x) ∈ RD is a trainable network. Note that the calculation of the complete ELBO in Eq. (1) requires T forward passes of the score network, which would make the training computationally prohibitive for a large T . To feasibly train the score network, instead of computing the complete ELBO, Ho et al. (2020) proposed an efficient training mechanism by sampling from a discrete uniform distribution: t ∼ U{1, ..., T}, x0 ∼ pdata(x0), t ∼ N (0, I) at each training iteration to compute the training loss:
L(t)ddpm(θ) := ∥∥∥∥ t − θ (αtx0 +√1− α2t t, αt)∥∥∥∥2
2
, (5)
which is a re-weighted form ofDKL (qβ(xt−1|xt,x0)||pθ(xt−1|xt)). Ho et al. (2020) reported that the re-weighting worked effectively for learning θ. Yet, we demonstrate it is deficient for learning the noise schedule β in our ablation experiment in Section 6.2.

heading: 4 BILATERAL DENOISING DIFFUSION MODELS (BDDMS)

heading: 4.1 PROBLEM FORMULATION
text: For fast sampling with DPMs, we strive for a noise schedule β̂ for sampling that is much shorter than the noise schedule β for training. As shown in Fig. 1, we define two separate diffusion processes corresponding to the noise schedules, β and β̂, respectively. The upper diffusion process parameterized by β is the same as in Eq. (2), whereas the lower process is defined as qβ̂(x̂1:N |x̂0) = ∏N n=1 qβ̂n(x̂n|x̂n−1) with much fewer diffusion steps (N T ). In our problem formulation, β is given, but β̂ is unknown. The goal is to find a β̂ for the reverse process pθ(x̂n−1|x̂n; β̂n) such that x̂0 can be effectively recovered from x̂N with N reverse steps.

heading: 4.2 MODEL DESCRIPTION
text: Although many prior arts (Ho et al., 2020; Chen et al., 2020; Song et al., 2021a; San-Roman et al., 2021) directly applied a shortened linear or Fibonacci noise schedule to the reverse process, we argue that these are sub-optimal solutions. Theoretically, the diffusion process specified by a new shortened noise schedule is essentially different from the one used to train the score network θ. Therefore, θ is not guaranteed suitable for reverting the shortened diffusion process. This issue motivated a novel modeling perspective to establish a link between the shortened schedule β̂ and the score network θ, i.e., to have β̂ optimized according to θ.
As a starting point, we consider an N = bT/τc, where 1 ≤ τ < T is a hyperparameter controlling the step size such that each diffusion step between two consecutive variables in the shorter diffusion process corresponds to τ diffusion steps in the longer one. Based on Eq. (2), we define the following:
qβ̂n+1(x̂n+1|x̂n = xt) := qβ(xt+τ |xt) = N √α2t+τ α2t xt, ( 1− α2t+τ α2t ) I  , (6) where xt is an intermediate diffused variable we introduced to link the two differently indexed diffusion sequences. We call it a junctional variable, which can be easily generated given x0 and β during training: xt = αtx0 + √ 1− α2t n.
Unfortunately, for the reverse process when x0 is not given, the junctional variable is intractable. However, our key observation is that while using the score by a score network θ∗ trained for the long β-parameterized diffusion process, a short noise schedule β̂(φ) can be optimized accordingly by introducing a schedule network φ. We provide its mathematical derivations in Appendix A.3. Next, we present a formal definition of BDDM and derive its training objectives, L(n)score(θ) and L(n)step(φ; θ∗), for the score network and the schedule network, respectively, in more detail.

heading: 4.3 SCORE NETWORK
text: Recall that a DDPM starts the reverse process with a white noise xT ∼ N (0, I) and takes T steps to recover the data distribution:
pθ(x0) DDPM := EN (0,I) [ Epθ(x1:T−1|xT ) [pθ(x0|x1:T )] ] . (7)
A BDDM, in contrast, starts from the junctional variable xt, and reverts a shorter sequence of diffusion random variables with only n steps:
pθ(x̂0) BDDM := Eqβ̂(x̂n−1;xt, n) [ Epθ(x̂1:n−2|x̂n−1) [pθ(x̂0|x̂1:n−1)] ] , 2 ≤ n ≤ N, (8)
where qβ̂(x̂n−1;xt, n) is defined as a re-parameterization on the posterior:
qβ̂(x̂n−1;xt, n) :=qβ̂
( x̂n−1 ∣∣∣∣∣x̂n = xt, x̂0 = xt − √ 1− α̂2n n α̂n ) (9)
=N  1√ 1− β̂n xt − β̂n√ (1− β̂n)(1− α̂2n) n, 1− α̂2n−1 1− α̂2n β̂nI  , (10) where α̂n = ∏n i=1 √ 1− β̂i, xt = αtx0 + √ 1− α2t n is the junctional variable that maps xt to x̂n given an approximate index t ∼ U{(n − 1)τ, ..., nτ − 1, nτ} and a sampled white noise n ∼ N (0, I). Detailed derivation from Eq. (9) to (10) is provided in Appendix A.2.

heading: 4.3.1 TRAINING OBJECTIVE FOR SCORE NETWORK
text: With the above definition, a new form of lower bound to the log marginal likelihood can be derived such that log pθ(x̂0) ≥ F (n)score(θ) := −L(n)score(θ)−Rθ(x̂0,xt), where
L(n)score(θ) :=DKL ( pθ(x̂n−1|x̂n = xt)||qβ̂(x̂n−1;xt, n) ) , (11)
Rθ(x̂0,xt) :=− Epθ(x̂1|x̂n=xt) [log pθ(x̂0|x̂1)] . (12) See detailed derivation in Proposition 1 in Appendix A.2. In the following Proposition 2, we prove that via the junctional variable xt, the solution θ∗ for optimizing the objective L(t)ddpm(θ),∀t ∈ {1, ..., T} is also the solution for optimizing L(n)score(θ),∀n ∈ {2, ..., N}. Thereby, we show that the score network θ can be trained with L(t)ddpm(θ) and re-used for reverting the short diffusion process over x̂N :0. Although the newly derived lower bound result in the same objective as the conventional score network, it for the first time establishes a link between the score network θ and x̂N :0. The connection is essential for learning β̂, which we will describe next.

heading: 4.4 SCHEDULE NETWORK
text: In BDDMs, a schedule network is introduced to the forward process by re-parameterizing β̂n as β̂n(φ) = fφ ( xt; β̂n+1 ) , and recall that during training, we can use xt = αtx0 + √ 1− α2t n and
β̂n+1 = 1 − α2t+τ α2t . Through the re-parameterization, the task of noise scheduling, i.e., searching for β̂, can now be reformulated as training a schedule network fφ that ancestrally estimates datadependent variances. The schedule network learns to predict β̂n based on the current noisy sample xt – this makes our method fundamentally different from existing and concurrent work, including Kingma et al. (2021) – as we reveal that, aside from β̂n+1, t, or n that reflects diffusion step information, xt is also essential for noise scheduling from a reverse direction at inference time.
Specifically, we adopt the ancestral step information (β̂n+1) to derive an upper bound for the current step while leaving the schedule network only to take the current noisy sample xt as input to predict a relative change of noise scales against the ancestral step. First, we derive an upper bound of β̂n by proving 0 < β̂n < min { 1− α̂ 2 n+1
1−β̂n+1 , β̂n+1
} in Appendix A.1. Then, by multiplying the upper
bound by a ratio estimated by a neural network σφ : RD 7→ (0, 1), we define
fφ(xt; β̂n+1) := min
{ 1− α̂2n+1
1− β̂n+1 , β̂n+1
} σφ(xt), (13)
where the network parameter set φ is learned to estimate the ratio between two consecutive noise scales (β̂n and β̂n+1) from the current noisy input xt.
Finally, at inference time for noise scheduling, starting from a maximum reverse steps (N ) and two hyperparameters (α̂N , β̂N ), we ancestrally predict the noise scale β̂n(φ) = fφ ( x̂n; β̂n+1 ) , for n
from N to 1, and cumulatively update the product α̂n = α̂n+1√ 1−β̂n+1 .

heading: 4.4.1 TRAINING OBJECTIVE FOR SCHEDULE NETWORK
text: Here we describe how to learn the network parameters φ effectively. First, we demonstrated that φ should be trained after θ is well-optimized, referring to Proposition 3 in Appendix A.3. The Proposition also shows that we are minimizing the gap between the lower bound F (n)score(θ∗) and log pθ∗(x̂0), i.e., log pθ∗(x̂0)−F (n)score(θ∗), by minimizing the following objective
L(n)step(φ; θ∗) :=DKL ( pθ∗(x̂n−1|x̂n = xt)||qβ̂n(φ)(x̂n−1;x0, αt) ) , (14)
which is defined as a KL divergence to directly compare pθ∗(x̂n−1|x̂n = xt) against the reparameterized forward process posteriors, which are tractable when conditioned on the junctional noise scale αt and x0.
The detailed derivation of Eq. (14) is also provided in the proof of Proposition 3 to get its concrete formulas as shown in Step (8-10) in Alg. 2.

heading: 5 ALGORITHMS: TRAINING, NOISE SCHEDULING, AND SAMPLING

heading: 5.1 TRAINING SCORE AND SCHEDULE NETWORKS
text: Following the theoretical result in Appendix A.3, θ should be optimized before learning φ. Thereby first, to train the score network θ, we refer to the settings in (Ho et al., 2020; Chen et al., 2020; Song et al., 2021a) to define β as a linear noise schedule: βt = βstart + tT (βend−βstart), for 1 ≤ t ≤ T, where βstart and βend are two hyperparameter that specifies the start value and the end value. This results in Algorithm 1, which resembles the training algorithm in (Ho et al., 2020).
Next, based on the converged score network θ∗, we train the schedule network φ. We draw an n ∼ U{2, . . . , N} at each training step, and then draw a t ∼ U{(n − 1)τ, ..., nτ}. These together can be re-formulated as directly drawing t ∼ U{τ, ..., T − τ} for a finer-scale time step. Then, we
Algorithm 1 Training Score Network (θ)
1: Given T, {βt}Tt=1 2: {αt}Tt=1 = { ∏t i=1 √ 1− βt}Tt=1 3: repeat 4: x0 ∼ pdata(x0) 5: t ∼ U{1, . . . , T} 6: t ∼ N (0, I) 7: xt = αtx0 + √ 1− α2t t 8: L(t)ddpm = ‖ t − θ(xt, αt)‖ 2 2
9: Take a gradient descent step on∇θL(t)ddpm 10: until converged
Algorithm 3 Noise Scheduling
1: Given θ∗, α̂N , β̂N ,xN ∼ N (0, I) 2: for n = N to 2 do 3: x̂n−1 ∼ pθ∗(x̂n−1|x̂n; α̂n, β̂n) 4: α̂n−1 = α̂n√
1−β̂n
5: β̂n−1 = min{1− α̂2n−1, β̂n}σφ(x̂n−1) 6: if β̂n−1 < β1 then 7: return β̂n, . . . , β̂N 8: end if 9: end for
10: return β̂1, . . . , β̂N
Algorithm 2 Training Schedule Network (φ)
1: Given θ∗, τ, T, {αt, βt}Tt=1 2: repeat 3: x0 ∼ pdata(x0) 4: t ∼ U{τ, . . . , T − τ} 5: δt = 1− α2t 6: n ∼ N (0, I) 7: xt = αtx0 + √ δt n
8: β̂n = min { δt, 1−
α2t+τ α2t
} σφ(xt)
9: C = 4−1 log(δt/β̂n) + 2−1D ( β̂n/δt − 1 ) 10: L(n)step = δt2(δt−β̂n)
∥∥∥ n − β̂nδt θ∗(xt, αt)∥∥∥22 + C 11: Take a gradient descent step on∇φL(n)step 12: until converged
Algorithm 4 Sampling
1: Given θ∗, {β̂n}Nsn=1, x̂Ns ∼ N (0, I) 2: {α̂n}Nsn=1 = {∏n i=1 √ 1− β̂n }Ns n=1 3: for n = Ns to 1 do 4: x̂n−1 ∼ pθ∗(x̂n−1|x̂n; α̂n, β̂n) 5: end for 6: return x̂0
sequentially compute the variables needed for calculating L(n)step(φ; θ∗), as presented in Algorithm 2. We observed that, although a linear schedule is used to define β, the noise schedule of β̂ predicted by fφ is not limited to but rather different from a linear one.

heading: 5.2 NOISE SCHEDULING FOR FAST AND HIGH-QUALITY SAMPLING
text: After the score network and the schedule network are trained, the inference procedure can divide into two phases: (1) the noise scheduling phase and (2) the sampling phase.
First, we run the noise scheduling process similarly to a sampling process with N iterations maximum. Different from training, where αt is forward-computed, α̂n is instead a backward-computed variable (from N to 1) that may deviate from the forward one because {β̂i}n−1i are unknown in the noise scheduling phase during inference. To start noise scheduling, we first set two hyperparameters: α̂N and β̂N . We use β1, the smallest noise scale seen in training, as a threshold to early stop the noise scheduling process so that we can ignore small noise scales (< β1) that were never seen by the score network. Overall, the noise scheduling process presents in Algorithm 3.
In practice, we apply a grid search algorithm ofM bins to Algorithm 3, which takesO(M2) time, to find proper values for (α̂N , β̂N ). We used M = 9 as in (Chen et al., 2020). The grid search for our noise scheduling algorithm can be evaluated on a small subset of the training samples. Empirically, even as few as 1 sample for evaluation works well in our algorithm. Finally, given the predicted noise schedule β̂ ∈ RNs , we generate samples with Ns sampling steps, as shown in Algorithm 4.

heading: 6 EXPERIMENTS
text: We conducted a series of experiments on neural vocoding tasks to evaluate the proposed BDDMs. First, we compared BDDMs against several strongest models that have been published: the mixture of logistics (MoL) WaveNet (Oord et al., 2018) implemented in (Yamamoto, 2020), the WaveGlow (Prenger et al., 2019) implemented in (Valle, 2020), the MelGAN (Kumar et al., 2019a) implemented in (Kumar, 2019), the HiFi-GAN (Kong et al., 2020b) implemented in (Kong et al., 2020a) and the two most recently proposed diffusion-based vocoders, i.e., WaveGrad (Chen et al., 2020) and DiffWave (Kong et al., 2021), both re-implemented in our code. The hyperparameter settings of BDDMs and all these models are detailed in Appendix B.
In addition, we also compared BDDMs to a variety of scheduling and acceleration techniques applicable to DDPMs, including the grid search (GS) approach in WaveGrad, the fast sampling (FS)
approach based on a user-defined 6-step schedule in DiffWave, the DDIMs (Song et al., 2021a) and a noise estimation (NE) approach (San-Roman et al., 2021). For fair and reproducible comparison with other models and approaches, we used the LJSpeech dataset (Ito & Johnson, 2017), which consists of 13,100 22kHz audio clips of a female speaker. All diffusion models were trained on the same training split as in (Chen et al., 2020). We also replicated the comparative experiment of neural vocoding using a multi-speaker VCTK dataset (Yamagishi et al., 2019) as presented in Appendix C and obtained a result consistent with that obtained from the LJSpeech dataset.

heading: 6.1 SAMPLING QUALITY IN OBJECTIVE AND SUBJECTIVE METRICS
text: To assess the quality of each generated audio sample, we used both objective and subjective measures for comparing different neural vocoders given the same ground-truth spectrogram s as the condition, i.e., θ(x, s, αt). Specifically, we used two scale-invariant metrics: the perceptual evaluation of speech quality (PESQ) (Rix et al., 2001) and the short-time objective intelligibility (STOI) (Taal et al., 2010) to measure the noisiness and the distortion of the generated speech relative to the reference speech. Mean opinion score (MOS) was also used as a subjective metric for evaluating the naturalness of the generated speech. The assessment scheme of MOS is included in Appendix B.
In Table 1, we compared BDDMs against the state-of-the-art (SOTA) vocoders. To predict noise schedules with different sampling steps (3, 7, and 12), we set three pairs of {α̂N , β̂N} for BDDMs by running on Algorithm 3 a quick hyperparameter grid search, which is detailed in Appendix B. Among the 9 evaluated vocoders, only our proposed BDDMs with 7 and 12 steps and DiffWave with 200 steps showed no statistic-significant difference from the ground-truth in terms of MOS. Moreover, BDDMs significantly outspeeded DiffWave in terms of RTFs. Notably, previous diffusion-
based vocoders achieved high MOS scores at the cost of an unacceptable RTF for industrial deployment. In contrast, BDDMs managed to achieve a high standard of generation quality with only 7 sampling steps (143x faster than WaveGrad and 28.6x faster than DiffWave).
In Table 2, we evaluated BDDMs and alternative accelerated sampling methods, which used the same score network for a pair-to-pair comparison. The GS method performed stably when the step number was small (i.e., N ≤ 6) but not scalable to more step numbers, which were therefore bypassed in the comparisons of 7 and 12 steps. The FS method by Song et al. (2021a) was linearly interpolated to 3 and 7 steps for a fair comparison. Comparing its 7-step and 3-step results, we observed that the FS performance degraded drastically. Both the DDIM and the NE methods were stable across all the steps but were not performing competitively enough. In comparison, BDDMs consistently attained the leading scores across all the steps. This evaluation confirmed that BDDM was superior to other acceleration methods for DPMs in terms of both stability and quality.

heading: 6.2 ABLATION STUDY AND ANALYSIS
text: We attribute the primary advantage of BDDMs to the newly derived objective L(n)step for learning φ. To better reason about this, we performed an ablation study, where we substituted the proposed loss with the standard negative ELBO for learning φ as mentioned by Sohl-Dickstein et al. (2015). We plotted the network outputs with different training losses in Fig. 2. It turned out that, when using L(n)elbo to learn φ, the network output rapidly collapsed to zero within several training steps; whereas, the network trained with L(n)step produced fluctuating outputs. The fluctuation is a desirable property showing the network properly predicts t-dependent noise scales, as t is a random time step drawn from a uniform distribution in training.
By setting β̂ = β, we empirically validated thatF (t)bddm := F (t) score +L(t)step ≥ F (t) elbo with their respective values at t ∈ [20, 180] using the same optimized θ∗. Each value is provided with 95% confidence intervals, as shown in Fig. 3. In this experiment, we used the LJ speech dataset and set T = 200 and τ = 20. Notably, we dropped their common entropy term Rθ(x̂0,xt) < 0 to mainly compare their KL divergences. This explains those positive lower bound values in the plot. The graph shows that our proposed bound F (t)bddm is always a tighter lower bound than the standard one across all examined t. Moreover, we found that F (t)bddm attained low values with a relatively much lower variance for t ≤ 50, where F (t)elbo was highly volatile. This implies that F (t) bddm better tackles the difficult training part, i.e., when the score becomes more challenging to estimate as t→ 0.

heading: 7 CONCLUSIONS
text: BDDMs parameterize the forward and reverse processes with a schedule network and a score network, of which the former’s optimization is tied with the latter by introducing a junctional variable. We derived a new lower bound that leads to the same training loss for the score network as in DDPMs (Ho et al., 2020), which thus enables inheriting any pre-trained score networks in DDPMs. We also showed that training the schedule network after a well-optimized score network can be viewed as tightening the lower bound. Followed from the theoretical results, an efficient training algorithm and a noise scheduling algorithm were respectively designed for BDDMs. Finally, in our experiments, BDDMs showed a clear edge over the previous diffusion-based vocoders.

heading: A THEORETICAL DERIVATIONS FOR BDDMS
text: In this section, we provide the theoretical supports for the following:
• The derivation for upper bounding β̂n (see Appendix A.1).
• The score network θ trained with L(t)ddpm(θ) for the reverse process pθ(xt−1|xt) can be re-used for the reverse process pθ(x̂n−1|x̂n) (see Appendix A.2).
• The schedule network φ can be trained with L(n)step(φ; θ∗) after the score network θ is optimized. (see Appendix A.3).

heading: A.1 DERIVING AN UPPER BOUND FOR NOISE SCALE
text: Since monotonic noise schedules have been successfully applied to in many prior arts including DPMs (Ho et al., 2020; Kingma et al., 2021) and score-based methods (Song et al., 2020; Song & Ermon, 2020), we also follow the monotonic assumption and derive an upper bound for β̂n as below:
Remark 1. Suppose the noise schedule for sampling is monotonic, i.e., 0 < β̂1 < . . . < β̂N < 1, then, for 1 ≤ n < N , β̂n satisfies the following inequality:
0 < β̂n < min
{ 1− α̂2n+1
1− β̂n+1 , β̂n+1
} . (15)
Proof. By the general definition of noise schedule, we know that 0 < β̂1, . . . , β̂N < 1 (Note: no inequality sign in between). Given that α̂n = ∏n i=1 √ 1− β̂i, we also have 0 < α̂1, . . . , α̂t < 1. First, we show that β̂n < 1− α̂2n+1
1−β̂n+1 :
α̂n−1 = α̂n√
1− β̂n < 1⇐⇒ β̂n < 1− α̂2n = 1−
α̂2n+1
1− β̂n+1 . (16)
Next, we show that β̂n < 1− α̂n+1:
α̂n√ 1− β̂n = α̂n
√ 1− β̂n 1− β̂n = α̂n+1 1− β̂n < 1⇐⇒ β̂n < 1− α̂n+1. (17)
Now, we have β̂n < min { 1− α̂ 2 n+1
1−β̂n+1 , 1− α̂n+1
} . When 1− α̂n+1 < 1−
α̂2n+1 1−β̂n+1 , we can show
that β̂n+1 < 1− α̂n+1:
1− α̂n+1 < 1− α̂2n+1
1− β̂n+1 = 1− α̂2n ⇐⇒ α̂n+1 > α̂2n ⇐⇒ α̂2n+1 α̂2n > α̂n+1 (18)
⇐⇒ 1− α̂2n+1 α̂2n < 1− α̂n+1 ⇐⇒ β̂n+1 < 1− α̂n+1. (19)
By the assumption of monotonic sequence, we also have β̂n < β̂n+1. Knowing that β̂n+1 < 1−α̂n+1 is always true, we obtain a tighter bound for β̂n: 0 < β̂n < min { 1− α̂ 2 n+1
1−β̂n+1 , β̂n+1
} .

heading: A.2 DERIVING THE TRAINING OBJECTIVE FOR SCORE NETWORK
text: First, followed from the data distribution modeling of BDDMs as proposed in Eq. (8): pθ(x̂0) := Ex̂n−1∼qβ̂(x̂n−1;xt, n) [ Ex̂1:n−2∼pθ(x̂1:n−2|x̂n−1) [pθ(x̂0|x̂1:n−1)] ] , (20)
we can derive a new lower bound to the log marginal likelihood as follows:
Proposition 1. Given xt ∼ qβ(xt|x0), the following lower bound holds for n ∈ {2, . . . , N}:
log pθ(x̂0) ≥ F (n)score(θ) := −L(n)score(θ)−Rθ(x̂0,xt), (21)
where
L(n)score(θ) := DKL ( pθ(x̂n−1|x̂n = xt)||qβ̂(x̂n−1;xt, n) ) , (22)
Rθ(x̂0,xt) := −Epθ(x̂1|x̂n=xt) [log pθ(x̂0|x̂1)] . (23)

heading: Proof.
text: log pθ(x̂0) = log ∫ pθ(x̂0:n−2|x̂n−1)qβ̂(x̂n−1;xt, n)dx̂1:n−1 (24)
= log ∫ pθ(x̂0:n−2|x̂n−1)qβ̂(x̂n−1;xt, n)
pθ(x̂1:n−1|x̂n = xt) pθ(x̂1:n−1|x̂n = xt) dx̂1:n−1
(25)
= logEpθ(x̂1,n−1|x̂n=xt)
[ pθ(x̂0|x̂1)qβ̂(x̂n−1;xt, n)
pθ(x̂n−1|x̂n = xt)
] (26)
[Jensen’s Inequality] ≥Epθ(x̂1,x̂n−1|x̂n=xt)
[ log
pθ(x̂0|x̂1)qβ̂(x̂n−1;xt, n) pθ(x̂n−1|x̂n = xt)
] (27)
=Epθ(x̂1|x̂n=xt) [log pθ(x̂0|x̂1)]−DKL ( pθ(x̂n−1|x̂n = xt)||qβ̂(x̂n−1;xt, n) ) (28)
=− L(n)score(θ)−Rθ(x̂0,xt) (29)
Next, we show that the score network θ trained with L(t)ddpm(θ) can be re-used in BDDMs. We first provide the derivation for Eq. (9- 10). We have
qβ̂(x̂n−1;xt, n) := qβ̂
( x̂n−1|x̂n = xt, x̂0 = xt − √
1− α̂2n n α̂n
) (30)
= N  α̂n−1β̂n 1− α̂2n xt − √ 1− α̂2n n α̂n + √ 1− β̂n(1− α̂2n−1) 1− α̂2n xt, 1− α̂2n−1 1− α̂2n β̂nI  (31) = N  α̂n−1β̂n α̂n(1− α̂2n) + √ 1− β̂n(1− α̂2n−1) 1− α̂2n xt − α̂n−1β̂n α̂n √ 1− α̂2n n, 1− α̂2n−1 1− α̂2n β̂nI
 (32) = N
 1√ 1− β̂n xt − β̂n√ (1− β̂n)(1− α̂2n) n, 1− α̂2n−1 1− α̂2n β̂nI  . (33) Proposition 2. Suppose xt ∼ qβ(xt|x0), then any solution satisfying θ∗ = argminθL (t) ddpm(θ),∀t ∈ {1, ..., T}, also satisfies θ∗ = argminθL (n) score(θ),∀n ∈ {2, ..., N}.
Proof. By the definition in Eq. (4), we have
pθ(x̂n−1|x̂n = xt) =N  1√ 1− β̂n ( xt − β̂n√ 1− α̂2n θ (xt, α̂n) ) , 1− α̂2n−1 1− α̂2n β̂nI  . (34) Here, from the training objective in Eq. (5), sincext = αtx0+ √ 1− α2t n, the noise scale argument for the score network is known to be αt. Therefore, we can use θ (xt, αt) instead of θ (xt, α̂n) for
expanding L(n)score(θ). Since pθ(x̂n−1|x̂n = xt) and qβ̂(x̂n−1;xt, n) are two isotropic Gaussians with the same variance, the KL divergence is a scaled `2-norm of their means’ difference:
L(n)score(θ) :=DKL ( pθ(x̂n−1|x̂n = xt)||qβ̂(x̂n−1;xt, n) ) (35)
= 1− α̂2n
2(1− α̂2n−1)β̂n ∥∥∥∥∥∥ 1√1− β̂n ( xt − β̂n√ 1− α̂2n θ (xt, αt) ) (36)
−  1√ 1− β̂n xt − β̂n√ (1− β̂n)(1− α̂2n) n ∥∥∥∥∥∥ 2
2
(37)
= (1− β̂n)(1− α̂2n) 2(1− β̂n − α̂2n)β̂n ∥∥∥∥∥∥ β̂n√(1− β̂n)(1− α̂2n) ( n − θ (xt, αt)) ∥∥∥∥∥∥ 2
2
(38)
= (1− β̂n)(1− α̂2n) 2(1− β̂n − α̂2n)β̂n
β̂2n
(1− α̂2n)(1− β̂n) ‖ n − θ (xt, αt)‖22 (39)
= β̂n
2(1− β̂n − α̂2n) ∥∥∥∥ n − θ (αtx0 +√1− α2t n, αt)∥∥∥∥2 2 , (40)
which is proportional to L(t)ddpm := ∥∥∥ n − θ (αtx0 +√1− α2t n, αt)∥∥∥2
2 as defined in Eq. (5).
Thus,
argminθL (t) ddpm(θ) ≡ argminθL (n) score(θ). (41)
Next, we can simplifyRθ(x̂0,xt) to a reconstruction loss for x̂0: Rθ(x̂0,xt) := −Epθ(x̂1|x̂n=xt) [log pθ(x̂0|x̂1)] (42)
=Epθ(x̂1|x̂n=xt)
logN  1√
1− β̂1
( x̂1 −
β̂1√ 1− α̂21 θ(x̂1, α̂1), β̂1I ) (43) =Epθ(x̂1|x̂n=xt) D 2 log 2πβ̂1 + 1
2β̂1 ∥∥∥∥∥∥x̂0 − 1√1− β̂1 x̂1 − β̂1√ β̂1 θ(x̂1, α̂1) ∥∥∥∥∥∥ 2
2  (44)
= D
2 log 2πβ̂1 +
1
2β̂1 Epθ(x̂1|x̂n=xt)  ∥∥∥∥∥∥x̂0 − 1√1− β̂1 ( x̂1 − √ β̂1 θ(x̂1, α̂1) )∥∥∥∥∥∥ 2
2  , (45)
where pθ(x̂1|x̂n = xt) can be efficiently sampled using the reverse process in (Song et al., 2021a). Yet, in practice, similar to the training in (Song et al., 2021a; Chen et al., 2020; Kong et al., 2021), we dropped Rθ(x̂0,xt) when training θ. In theory, we know that Rθ(x̂0,xt) achieves its optimal value at θ∗ = argminθ‖ θ(x̂1, α̂1)− 1‖22, which shares a similar objective as L (t) ddpm. By minimizing
L(t)ddpm, we train a score network θ∗ that best minimizes ∆ t := ‖ t− θ∗(αtx0 + √
1− α2t t, αt)‖22 for all 1 ≤ t ≤ T . Since the first diffusion step has the smallest effect on corrupting x̂0 (i.e., β1 ≈ 0), it suffices to consider a α̂1 = √ 1− β1 = α1, in which case we can jointly minimize Rθ(x̂0,xt) by minimizing L(1)ddpm.
In this sense, during training, given xt ∼ qβ(xt|x0), we can train the score network with the same training objective as in DDPMs and DDIMs. Practically, it is beneficial for BDDMs as we can re-use the score network θ of any well-trained DDPM or DDIM.

heading: A.3 DERIVING THE TRAINING OBJECTIVE FOR SCHEDULE NETWORK
text: Given that θ can be trained to maximize the log evidence with the pre-specified noise schedule β for training, the consequent question of interest in BDDMs is how to find a fast and good enough noise schedule β̂ ∈ RN for sampling given an optimized θ∗. In BDDMs, this problem is reduced to how to effectively learn the network parameters φ .
Proposition 3. Suppose θ has been optimized and hypothetically converged to the optimal θ∗, where by optimal it means that with θ∗ we have pθ∗(x̂n−1|x̂n = xt) = qβ̂(x̂n−1;xt, n) given xt ∼ qβ(xt|x0). When β̂ is unknown but we have x0 = x̂0 and α̂n = αt, we can minimize the gap between the optimal lower bound F (n)score(θ∗) and log pθ∗(x̂0), i.e, log pθ∗(x̂0) − F (n)score(θ∗), by minimizing the following objective with respect to β̂n:
L(n)step(β̂n; θ∗) :=DKL ( pθ∗(x̂n−1|x̂n = xt)||qβ̂n(x̂n−1|x0;αt) ) (46)
= δt
2(δt − β̂n) ∥∥∥∥∥ n − β̂nδt θ∗ ( αtx0 + √ δt n, αt )∥∥∥∥∥ 2
2
+ C, (47)
where
δt = 1− α2t , C = 1
4 log
δt β̂n + D 2 ( β̂n δt − 1 ) . (48)
Proof. Note that x̂0 = x0, α̂n = αt, xt = αtx0 + √
1− α2t n and pθ∗(x̂n−1|x̂n = xt) = qβ̂(x̂n−1;xt, n). When x0 is given to pθ∗ , we can express the probability as follows:
pθ∗(x̂n−1|x̂n = xt(x0), x̂0 = x0) (49)
= ∫ N (z;0, I)pθ∗(x̂n−1|x̂n = αtx0 + √ 1− α2tz)dz (50)
= ∫ N (z;0, I)qβ̂(x̂n−1;xt = αtx0 + √ 1− α2tz, n = z)dz (51)
= ∫ N (z;0, I)N x̂n−1; αtx0 +√1− α2tz√ 1− β̂n − β̂n√ (1− β̂n)(1− α̂2n) z, 1− α̂2n−1 1− α̂2n β̂nI  dz (52)
[See Eq. (2) in (Frey, 1999)]
=N x̂n−1; αtx0√ 1− β̂n ,  √1− α2t√ 1− β̂n − β̂n√ (1− β̂n)(1− α2t ) 2 + 1− α2t /(1− β̂n) 1− α2t β̂n  I 
(53)
=N x̂n−1; αtx0√ 1− β̂n , 1− α2t − β̂n 1− β̂n I  =: qβ̂n(x̂n−1;x0, αt), (54) where, different from pθ∗(x̂n−1|x̂n = xt), from Eq. (49) to Eq. (50), instead of conditioning on a specific xt, when x0 is given xt can be generated using any z ∼ N (0, I).
From this, we can express the gap between log pθ∗(x̂0) and F (n)score(θ∗) in the following form:
log pθ∗(x̂0 = x0)−F (n)score(θ∗) (55)
= log pθ∗(x̂0 = x0)− Epθ∗ (x̂1,n−1|x̂n=xt)
[ log
pθ(x̂0|x̂1)qβ̂(x̂n−1;xt, n) pθ(x̂n−1|x̂n = xt)
] (56)
= log pθ∗(x̂0 = x0)− Epθ∗ (x̂1:n−1|x̂n=xt) [ log
pθ∗(x̂0:n−1|x̂n = xt) pθ∗(x̂1:n−1|x̂n = xt)
] (57)
=Epθ∗ (x̂1:n−1|x̂n=xt) [ log
pθ∗(x̂1:n−1|x̂n = xt) pθ∗(x̂1:n−1|x̂n = xt, x̂0 = x0)
] (58)
=Epθ∗ (x̂n−1|x̂n=xt)
[ log
pθ∗(x̂n−1|x̂n = xt) qβ̂n(x̂n−1;x0, αt)
] (59)
=DKL ( pθ∗(x̂n−1|x̂n = xt)||qβ̂n(x̂n−1;x0, αt) ) (60)
Next, we evaluate the above KL divergence term. By definition, we have
pθ∗(x̂n−1|x̂n = xt) = N  1√ 1− β̂n ( xt − β̂n√ 1− α̂2n θ∗ (xt, α̂n) ) , 1− α̂2n−1 1− α̂2n β̂nI  (61) Together with Eq. (54), we have
L(n)step(β̂n; θ∗) := DKL ( pθ∗(x̂n−1|x̂n = xt)||qβ̂n(x̂n−1;x0, αt) ) (62)
= 1− β̂n
2(1− β̂n − α2t ) ∥∥∥∥∥∥ αt√1− β̂nx0 − 1√ 1− β̂n ( xt − β̂n√ 1− α2t θ∗ (xt, αt) )∥∥∥∥∥∥ 2
2
+ C (63)
= 1− β̂n
2(1− β̂n − α2t ) ∥∥∥∥∥∥ αt√1− β̂nx0 − 1√ 1− β̂n ( αtx0 + √ 1− α2t n − β̂n√ 1− α2t θ∗ (xt, αt) )∥∥∥∥∥∥ 2
2
+ C
(64)
= 1− β̂n
2(1− β̂n − α2t ) ∥∥∥∥∥∥ √ 1− α2t 1− βn n − β̂n√ (1− βn)(1− α2t ) θ∗(xt, αt) ∥∥∥∥∥∥ 2
2
+ C (65)
= 1− α2t
2(1− β̂n − α2t ) ∥∥∥∥∥ n − β̂n1− α2t θ∗(xt, αt) ∥∥∥∥∥ 2
2
+ C (66)
= δt
2(δt − β̂n) ∥∥∥∥∥ n − β̂nδt θ∗ ( αtx0 + √ δt n, αt )∥∥∥∥∥ 2
2
+ C, (67)
where
δt = 1− α2t , C = 1
4 log
δt β̂n + D 2 ( β̂n δt − 1 ) . (68)
As we use a schedule network φ to estimate β̂n from (α̂n+1, β̂n+1) as defined in Eq. (13), we obtain the final step loss for learning φ:
L(n)step(φ; θ∗) = δt
2(δt − β̂n(φ)) ∥∥∥∥∥ n − β̂n(φ)δt θ∗(xt, αt) ∥∥∥∥∥ 2
2
+ 1
4 log
δt β̂n(φ) + D 2
( β̂n(φ)
δt − 1
) .
(69)
This proposed objective for training the schedule network can be interpreted as to better model the data distribution (i.e., maximizing log pθ(x̂0)) by correcting the gradient scale β̂n for the next reverse step (from x̂n to x̂n−1) given the gradient vector θ∗ estimated by the score network θ∗.

heading: B EXPERIMENTAL DETAILS

heading: B.1 CONVENTIONAL GRID SEARCH ALGORITHM FOR DDPMS
text: We reproduced the grid search algorithm in (Chen et al., 2020), in which a 6-step noise schedule was searched. In our paper, we generalized the grid search algorithm by similarly sweeping the N -step noise schedule over the following possibilities with a bin width M = 9:
{1, 2, 3, 4, 5, 6, 7, 8, 9} ⊗ {10−6·N/N , 10−6·(N−1)/N , ..., 10−6·1/N}, (70)
where ⊗ denotes the cartesian product applied on two sets. LS-MSE was used as a metric to select the solution during the search. When N = 6, we resemble the GS algorithm in (Chen et al., 2020). Note that above searching method normally does not scale up to N > 6 steps for its exponential computational cost O(9N ).

heading: B.2 HYPERPARAMETER SETTING IN BDDMS
text: Algorithm 2 took a skip factor τ to control the stride for training the schedule network. The value of τ would affect the coverage of step sizes when training the schedule network, hence affecting the predicted number of steps N for inference – the higher τ is, the shorter the predicted inference schedule tends to be. We set τ = 66 for training the BDDM vocoders in this paper.
For initializing Algorithm 3 for noise scheduling, we could take as few as 1 training sample for validation, perform a grid search on the hyperparameters {(α̂N = 0.1αT i, β̂N = 0.1j)} for i, j = 1, ..., 9, i.e., 81 possibilities in total, and use the PESQ measure as the selection metric. Then, the predicted noise schedule corresponding to the maximum PESQ was stored and applied to the online inference afterward, as shown in Algorithm 4. Note that this searching has a complexity of only O(M2) (e.g., M = 9 in this case), which is much more efficient than O(MN ) in the conventional grid search algorithm in (Chen et al., 2020), as discussed in Section B.1.
B.3 IMPLEMENTATION DETAILS
Our proposed BDDMs and the baseline methods were all implemented with the Pytorch library. The score networks for the LJ and VCTK speech datasets were trained from scratch on a single NVIDIA Tesla P40 GPU with batch size 32 for about 1M steps, which took about 3 days.
For the model architecture, we used the same architecture as in DiffWave (Kong et al., 2021) for the score network with 128 residual channels; we adopted a lightweight GALR network (Lam et al., 2021) for the schedule network. GALR was originally proposed for speech enhancement, so we considered it well suited for predicting the noise scales. For the configuration of the GALR network, we used a window length of 8 samples for encoding, a segment size of 64 for segmentation and only two GALR blocks of 128 hidden dimensions, and other settings were inherited from (Lam et al., 2021). To make the schedule network output with a proper range and dimension, we applied a sigmoid function to the last block’s output of the GALR network. Then the result was averaged over the segments and the feature dimensions to obtain the predicted ratio: σφ(x) = AvgPool2D(σ(GALR(x))), where GALR(·) denotes the GALR network, AvgPool2D(·) denotes the average pooling operation applied to the segments and the feature dimensions, and σ(x) := 1/(1 + e−x). The same network architecture was used for the NE approach for estimating α2t and was shown better than the ConvTASNet used in the original paper (San-Roman et al., 2021). It is also notable that the computational cost of a schedule network is indeed fractional compared to the cost of a score network, as predicting a noise scalar variable is intrinsically a relatively much easier task. Our GALR-based schedule network, while being able to produce stable and reliable results, was about 3.6 times faster than the score network. The training of schedule networks for BDDMs took only 10k steps to converge, which consumed no more than an hour on a single GPU.
Regarding the image generation task, to demonstrate the generalizability of our method, we directly adopted a score network pre-trained on the CIFAR-10 dataset implemented by a third-party opensource repository. Regarding the schedule network, to demonstrate that it does not have to use specialized architecture, we replaced GALR by the VGG11 (Simonyan & Zisserman, 2014), which was also used by as a noise estimator in (San-Roman et al., 2021). The output dimension (number of classes) of VGG11 was set to 1. Similar to the setting for GALR in speech synthesis, we added a sigmoid activation to the last layer to ensure a [0, 1] output. Similar to the training in speech domain, we trained the VGG11-based schedule networks while freezing the score networks for 10k steps, which normally can be finished in about two hours.
Our code for the speech vocoding and the image generation experiments will be uploaded to Github after the final decision of ICLR is released.

heading: B.4 CROWD-SOURCED SUBJECTIVE EVALUATION
text: All our Mean Opinion Score (MOS) tests were crowd-sourced. We refer to the MOS scores in (Protasio Ribeiro et al., 2011), and the scoring criteria have been included in Table 3 for completeness. The samples were presented and rated one at a time by the testers.

heading: C ADDITIONAL EXPERIMENTS
text: A demonstration page at https://bilateral-denoising-diffusion-model. github.io shows some samples generated by BDDMs trained on LJ speech and VCTK datasets.

heading: C.1 MULTI-SPEAKER SPEECH SYNTHESIS
text: In addition to the single-speaker speech synthesis, we evaluated BDDMs on the multi-speaker speech synthesis benchmark VCTK (Yamagishi et al., 2019). VCTK consists of utterances sampled at 48 KHz by 108 native English speakers with various accents. We split the VCTK dataset for training and testing: 100 speakers were used for training the multi-speaker model and 8 speakers for testing. We trained on a 44257-utterance subset (40 hours) and evaluated on a held-out 100- utterance subset. For the score network, we used the Wavegrad architecture (Chen et al., 2020) so as to examine whether the superiority of BDDMs remains in a different dataset and with a different score network architecture.
Results are presented in Table 4. For this multi-speaker VCTK dataset, we obtained consistent observations with that for the single-speaker LJ dataset presented in the main paper. Again, the proposed BDDM with only 16 or 21 steps outperformed the DDPM with 1,000 steps. To the best of our knowledge, ours was the first work that reported this degree of superior. When reducing to 8 steps, BDDM obtained performance on par with (except for a worse PESQ) the costly gridsearched 8 steps (which were unscalable to more steps) in DDPM. For NE, we could again observe a degradation from its 16 steps to 21 steps, indicating the instability of NE for the VCTK dataset likewise. In contrast, BDDM gave continuously improved performance while increasing the step number.

heading: C.2 COMPARING DIFFERENT REVERSE PROCESSES FOR BDDMS
text: This section demonstrates that BDDMs do not restrict the sampling procedure to a specialized reverse process in Algorithm 4. In particular, we evaluated different reverse processes, including that of DDPMs as shown in Eq. (4) and DDIMs (Song et al., 2021a), for BDDMs and compared the objective scores on the generated samples. DDIMs (Song et al., 2021a) formulate a non-Markovian generative process that accelerates the inference while keeping the same training procedure as DDPMs. The original generative process in Eq. (4) in DDPMs is modified into
p (τ) θ (x0:T ) := π(xT ) S∏ i=1 p (γi) θ (xγi−1 |xγi)× ∏ t∈γ̄ p (t) θ (x0|xt), (71)
where γ is a sub-sequence of length N of [1, ..., T ] with γN = T , and γ̄ := {1, ..., T} \γ is defined as its complement; Therefore, only part of the models are used in the sampling process.
To achieve the above, DDIMs defined a prediction function f (t)θ (xt) that depends on θ to predict the observation x0 given xt directly:
f (t) θ (xt) :=
1
αt
( xt − √ 1− α2t θ(xt, αt) ) . (72)
By leveraging this prediction function, the conditionals in Eq. (71) are formulated as
p (γi) θ (xγi−1 |xγi) = N ( αγi−1 αγi (xγi − ς θ(xγi , αγi)) , σ2γiI ) ifi ∈ [N ], i > 1 (73)
p (t) θ (x0|xt) = N (f (t) θ (xt), σ 2 t I) otherwise, (74)
where the detailed derivation of σt and ς can be referred to (Song et al., 2021a). In the original DDIMs, the accelerated reverse process produces samples over the subsequence of β indexed by γ: β̂ = {βn|n ∈ γ}. In BDDMs, to apply the DDIM reverse process, we use the β̂ predicted by the schedule network in place of a subsequence of the training schedule β.
Finally. the objective scores are given in Table 5. Note that the subjective evaluation (MOS) is omitted here since the other assessments above have shown that the MOS scores are highly correlated with the objective measures, including STOI and PESQ. They indicate that applying BDDMs to either DDPM or DDIM reverse process leads to comparable and competitive results. Meanwhile, the results show some subtle differences: BDDMs over a DDPM reverse process gave slightly better samples in terms of signal error and consistency metrics (i.e., LS-MSE and MCD), while BDDM over a DDIM reverse process tended to generate better samples in terms of intelligibility and perceptual metrics (i.e., STOI and PESQ).

heading: C.3 UNCONDITIONAL IMAGE GENERATION
text: For the unconditional image generation task, we evaluated the proposed BDDMs on the benchmark CIFAR-10 (32 × 32) dataset. The score functions, including those initially proposed in DDPMs (Ho et al., 2020) or DDIMs (Song et al., 2021a) and those pre-trained in the above third-party implementations, are all conditioned on a discrete step-index. We estimated the noise schedule β̂ in continuous space using the VGG11 schedule network and then mapped it to discrete time schedule using the approximation method in (Kong & Ping, 2021).
Table 6 shows the performances of different sampling methods for DDPMs in CIFAR-10. By setting the maximum number of sampling steps (N ) for noise scheduling, we can fairly compare the improvements achieved by BDDMs against related methods in the literature in terms of FID. Remarkably, BDDMs with 100 sampling steps not only surpassed the 1000-step DDPM baseline, but also produced the SOTA FID performance amongst all generative models using less than or equal to 100 sampling steps.
2Our implementation was based on https://github.com/openai/improved-diffusion 3The authors of VDM claimed that they tuned the hyperparameters only for minimizing the likelihood and
did not pursue further tuning of the model to improve FID.

Review summaries:
The paper proposes to train a scheduling network in addition to the score network for fast sampling of diffusion models. To train such network a new loss is proposed. Results are presented for the task of neural vocoding (single-speaker and multi-speaker).

The paper aims at improving generative denoising diffusion models for audio synthesis. In particular, it proposes a method to learn a noise prediction or scheduling network that optimally chooses the noise scales (this is, the scales of the noise that is injected during the iterative synthesis process) at inference time. This allows for faster synthesis without much reduction in quality. The method, called Bilateral Denoising Diffusion Model (BDDM), relies on a slightly modified training objective compared to previous works, in which case learning the noise schedule corresponds to optimizing the objective towards a tighter bound on the data log-likelihood. The main score model is learned in a similar fashion like previous diffusion models, such that the objectives for the score model and the noise scheduling network decompose into two separate objectives that can be optimized essentially one after another. Experimentally, BDDM achieves audio generation with a quality almost as good as the current state-of-the-art, while requiring much less synthesis steps. It also outperforms other methods for accelerated sampling from diffusion models for audio synthesis.

This speech synthesis paper proposes to learn the forward diffusion process and reach better ELBO values by doing that, as well as facilitate inference with fewer steps while keeping good generation quality. The experiments were conducted on LJSpeech and VCTK datasets. The paper compared the proposed method to some strong baselines, including WaveNet, GANs and two recent diffusion methods: WaveGrad and DiffWave. The metrics were STOI, PESQ and human evaluation MOS. The MOS results for the proposed method show no significant difference from the ground truth (same as DiffWave) starting from 7 steps while being and an order of magnitude faster than the best baseline DiffWave.