BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis
Authors: Max W. Y. Lam
Jun Wang
Dan Su
Dong Yu
Venue: ICLR 2022 Poster (2022)
Keywords: Speech Synthesis
Vocoder
Generative Model
Diffusion Model

Abstract: Diffusion probabilistic models (DPMs) and their extensions have emerged as competitive generative models yet confront challenges of efficient sampling. We propose a new bilateral denoising diffusion model (BDDM) that parameterizes both the forward and reverse processes with a schedule network and a score network, which can train with a novel bilateral modeling objective. We show that the new surrogate objective can achieve a lower bound of the log marginal likelihood tighter than a conventional surrogate. We also find that BDDM allows inheriting pre-trained score network parameters from any DPMs and consequently enables speedy and stable learning of the schedule network and optimization of a noise schedule for sampling.
Our experiments demonstrate that BDDMs can generate high-fidelity audio samples with as few as three sampling steps. Moreover, compared to other state-of-the-art diffusion-based neural vocoders, BDDMs produce comparable or higher quality samples indistinguishable from human speech, notably with only seven sampling steps (143x faster than WaveGrad and 28.6x faster than DiffWave). We release our code at https://github.com/tencent-ailab/bddm.

Published as a conference paper at ICLR 2022
BDDM: BILATERAL DENOISING DIFFUSION MODELS FOR FAST AND HIGH-QUALITY SPEECH SYNTHESIS
Max W. Y. Lam, Jun Wang, Dan Su
Dong Yu Tencent Al Lab
Tencent AI Lab
Shenzhen, China (maxwylam, joinerwang, dansu)@tencent..co com
Bellevue WA, USA dyuêtencent. com
ABSTRACT
Diffusion probabilistic models (DPMs) and their extensions have emerged as com- petitive generative models yet confront challenges efficient sampling. We propose new bilateral denoising diffusion model (BDDM) that parameterizes both the forward and reverse processes with a schedule network and a score network, which can train with novel bilateral modeling objective. We show that the new surrogate objective can achieve a lower bound of the log marginal likelihood tighter than conventional surrogate. We also find that BDDM al- lows inheriting pre-trained score network parameters from any DPMs and con- sequently enables speedy and stable learning of the schedule network and op- timization of noise schedule for sampling. Our experiments demonstrate that BDDMs can generate high-fidelity audio samples with as few as three sam- pling steps. Moreover, compared to other state-of-the-art diffusion-based neu- ral vocoders, BDDMs produce comparable or higher quality samples indistin- guishable from human speech, notably with only seven sampling steps (143x faster than WaveGrad and 28.6x faster than DiffWave). We release our code at httpps://gittuu..omm
INTRODUCTION
Deep generative models have shown a tremendous advancement in speech synthesis (van den Oord al.
2016 Kalchbrenner et 2018 Prenger et al. 2019 Kumar et al., 2019 Kong et al. [020bb]
2020 Kong et al., 2021) Successful generative models can be mainly divided into two categories: generative adversarial network (GAN) (Goodfellow et al.) (014) based and likelihood- based. The former is based on adversarial learning, where the objective is generate data indistin- guishable from the training data. Yet, the training GANs can be very unstable, and the relevant train- ing objectives are not suitable to compare against different GANs. The latter uses log- likeehholh or surrogate objectives for training, but they also have intrinsic limitations regarding generation speed or quality. For example, the autoregressive models (van den Oord et al. 2016 Kalchbrenner al., 2018). while being capable of generating high-fidelity data, are limited by their inherently slow sampling process and the poor scaling properties on high-dimensional data. Likewise, the flow- based models (Dinh et al. 20166] Kingma & Dhariwal] (2018) Chen et al. 2018 Papamakarios et al.) 202T rely on specialized architectures
Chen et al.
build normalized probability model, whose training is less parameter-effiieet. Other prior works use surrogate objectives, such as the evidence lower bound in variational auto-encoders (Kingma & Welling 2014, Rezende al. (0014) Maaløe al., 2019) and the contrastive divergence in energy-based models (Hinton) 2002 Carreira-Perpinat & Hinton 2005). These models, despite showing improved speed, typically only work well for low- dimensional data, and, in general, the sample qualities are not competitive to the GAN-based and the autoregressive models (Bond- et al. (2021).
An up-and-coming class of tikelihood-based models is the diffusion probabilistic models (DPMs) SShl-Diccstee et al. 2015), which introduces the idea of using a forward diffusion process to sequentially corrupt given distribution and learning the reversal of such diffusion process to restore the data distribution for sampling. From a similar perspective, [Song & Ermon (2019) proposed the score-based generative models by applying the score matching technique (Hyvarinen & Dayan

---

Published as a conference paper at ICLR 2022
2005 to train neural network such that samples can be generated via Langevin dynamics. Along these two lines of research, Ho et al. (2020) proposed the denoising diffusion probabilistic models (DDPMs) for high-quality image syntheses. Dhariwal & Nichol (202I) demonstrated that improved DDPMs Nichol & Dhariwal (2021) are capable of generating high-quality images of comparable or even superior quality to the state-of-the-art (SOTA) GAN-based models. For speech syntheses,
DDPMs were also applied in Wavegrad (Chen et al. 2020) and DiffWave (Kong et al.) 2021) to produce higher-fidelity audio samples than the conventional non-autoregressive models (Yamamoto et al. 2020; Kumar et al. 2019; Yang et al., 2021 Binkowski et al., 2020) and matched the quality
of the SOTA autoregressive methods (Chen et al. 2020).
Despite the compelling results, the diffusion generative models are two to three orders of magnitude slower than other generative models such as GANs and VAEs. Their primary limitation is that they require up to thousands of diffusion steps during training to learn the target distribution. Therefore a large number of reverse steps are often required at sampling time. Recently, extensive investigations have been conducted to reduce the sampling steps for efficiently generating high-quality samples, which we will discuss in the related work in Section 2 Distinctively, we conceived that we might train a neural network to efficiently and adaptively estimate much shorter noise schedule for sam- pling while achieving generation performances comparable or superior to the conventional DPMs. With such incentive, after introducing the conventional DPMs as our background in Section we propose in Section bilateral denoising diffusion models (BDDMs), named after a bilateral mod- cling perspective- parameterizing the forward and reverse processes with schedule network and a score network, respectively. We theoretically derive that the schedule network should be trained after the score network is optimized. For training the schedule network, we propose a novel objec- tive to minimize the gap between newly derived lower bound and the log marginal likelihood. We describe the training algorithm as well as the fast and high-quality sampling algorithm in Section [5] The training of the schedule network converges very fast using our newly derived objective, and its training only adds negligible overhead to DDPM's. In Section [ our neural vocoding experiments demonstrated that BDDMs could generate high- fideliiyl samples with few three sampling steps. Moreover, our method can produce speech samples indistinguishable from human speech with only seven sampling steps (1 (143x faster than WaveGrad and 28.6x faster than DifeWave).
2 RELATED WORK
Prior works showed that noise scheduling is crucial for efficient and high-fidelity data generation in DPMs. DDPMs (Ho et al.] (0220 used a shared linear noise schedule for both training and sampling, which, however, requires thousands of sampling iterations obtain competitive results. To speed up the sampling process, one class of related work, including (Chen et al., 2020 Kong et al. 202T Nichol & Dhariwal 202T), attempts to use a different, shorter noise schedule for sampling. For clarity, we thereafter denote the training noise schedule as B E I2. and the sampling noise schedule as 9 E R. with N < T. In particular, Chen et al. (2020) applied grid search (GS) algorithm to select (3. Unfortunately. GS becomes prohibitively slow when N grows large, e.g., N = 6 took more than a day on a single NVIDIA Tesla P40 GPU. This is because the time costs of GS algorithm grow exponentially with N, i.e., (ggN) with 9 bins as the default setting in Chen et al. (2020). Instead of searching, Kong et al.] (2021) devised a fast sampling (FS) algorithm based on an expert-defined 6-step noise schedule for their score network. However, this specifically tuned noise schedule is hard to generalize to other score networks, tasks, or datasets.
Another class of noise scheduling methods searches for subsequence of time indices of the training noise schedule, which we call the time schedule. DDIMs (Song et al., 2021) introduced an accel- erated reverse process that relies on a pre-specified time schedule. A linear and a quadratic time schedule were used in DDIMs and showed superior generation quality over DDPMs within 10 to 100 sampling steps. Nichol & Dhariwal (202T) proposed a re-scaled noise schedule for fast sam- pling, but this also requires pre-specifying the time schedule and the training noise schedule. Nichol & Dhariwal (202T) also proposed learning variances for the reverse processes, whereas the variances of the forward processes, i.e., the noise schedule, which affected both the means and variances of the reverse processes, were not learnable. According to the results of (Song et al. 2021 Nichol & Dhariwal 2021), ), using a linear or quadratic time schedule resulted in quite different performances in different datasets, implying that the optimal choice of schedule varies with the datasets. So, there remains challenge in finding short and effective schedule for fast sampling on different datasets.
2

---

Published as a conference paper at ICLR 2022
Notably, Kong & Ping (2021) proposed method to map noise schedule to time schedule for fast sampling. In this sense, searching for a time schedule becomes a sub-set of the noise scheduling problem, which resembles the above category of methods.
Although DPMs (Sohl-Dickstein et al. 2013) and DDPMs (Ho al. 2019) mentioned that the noise schedule could be learned by re-parameterization, the approach
not investigated in their works.
Closely related works that learn noise schedule emerged until very recently. San-Roman et al. (202I) proposed noise estimation (NE) method, which trained a neural net with regression loss to estimate the noise scale from the noisy sample at each time point, and then predicted the next noise scale. However, NE requires a prior assumption of the noise schedule following linear or Fibonacci rule. Most recently, concurrent work to ours by Kingma et al. (2021) jointly trained neural net to predict the signal-to-noke -o--o ratio (SNR) by maximizing the variational lower bound. The SNR was then used for noise scheduling. Different from ours, this scheduling neural net only took t as imput and is independent of the noisy sample generated during the loop of sampling process. Intrinsically, with limited information about the sampled data, the predicted SNR could deviate from the actual SNR of the noisy data during sampling.
3 BACKGROUND
3.1 DIFFUSION PROBABILISTIC MODELS (DPMS)
Given i..dd samples (wo E RD) from unknown data distribution Panaa pana(aro) (2o), diffusion prob- abilistic models (DPMs) (Sohl-Dickstein et al. 2015) define forward process q(#1:r
IT ( (1) that converts any complex data distribution into a simple, tractable distribution af- ter T steps of diffusion. reverse process po (zz-1 parameterized by is used to model the data distribution: po(= f (xx)) I (0 )d where n (ær) is the prior distribution for starting the reverse process. Then, the variational parameters 0 can be learned by maximizing the standard log evidence lower bound (ELBO):
Felbo E.g logro(z0 (2)) -
Σ DKa. 20) DKL. (00
Dea.
2
3.2 DENOISING DIFFUSION PROBABILISTIC MODELS (DDPMS)
As an extension to DPMs, denoising diffusion probabilistic models (DDPMs) (Ho et al. 2020
applied the score matching technique (Hyvarinen Dayan 2005) Song & Ermon 2019) define the reverse process. In particular, DDPMs considered Gaussian diffusion process parameterized by a noise schedule B R with 0 < ,1,...,,3T I:
T
q (E1.T (2)) == II ga, ( (x),),, where qs ( = N(V1 Br
(2)
Based on the nice property of isotropic Gaussians, one can express Vy directly conditioned on æp!
gs(x:(0o)) = Naaa (1 - (1-a7)44 a²) where at = II
V 3
(3)
i=1
To revert this forward process, DDPMs employ a score network
6o (0 a4)) to define
B o
pa (Z) =
(æi, ac)
Eg
(4)
Be
Here, co(#;, a:) is conditioned on the continuous noise scale ar, as Song et al. 2020b
Chen et all
2020
Alternatively, the score network can also be conditioned on a discrete time index cø (E,, 2). in (Song
a
202T Ho al. 2020). An approximate mapping of noise schedule to time schedule (Kong & Ping
2021
exists, therefore consider conditioning noise scales the general case.
3

---

Published as a conference paper at ICLR 2022
Reverse process
Forward process
X
AO
Xy- (Xelao
n-
y (+) Drz Pe Po lllgplae (xn- X2.
ll
for Score Network
(0) stap ($: B*) = Duz. ( po (3 18n x;) (8, (8n-1i for Schedule Network
Figure 1: bilateral denoising diffusion model (BDDM) introduces a junctional variable æy and schedule network Φb The schedule network can optimize the shortened noise schedule (.) if we know the score of the distribution at the junctionna step, using the KI. divergence to directly compare pa- (in-1 @n = xt) against the re-paramcterized forward process posteriors.
where Σ, is the co-variance matrix defined for the reverse process.
2020 showed that setting 2; 3.1 1-af_1 T-o? 3,I is optimal for deterministic xo, while setting 2, 3 is optimal
for a white noise 3g ~ (N I). Alternatively, Nichol & Dhariwal (2021) proposed learnable variances by interpolating the two optimals with jointly trained neural network, i.e., Ex,0(x)) =
(ixg(ex)p())(x)) log 3, (1 - vx)) log B,)), where (x))) E RP is a trainable network.
Note that the calculation of the complete ELBO in Eq. (I requires T forward passes of the score network, which would make the training computationally prohibitive for a large T. To feasibly train the score network, instead of computing the complete ELBO, |Ho al. (2020) proposed an efficient training mechanism by sampling from a discrete uniform distribution: ~~ U(I,..., T",
æo Pdata (2o), Et NO, I) at each training iteration compute the training loss:
ddpm (0) = 6: 60
(1:t0 + V of 04
(5)
which is a re-weighted form DKL (qa(x;-11 [zt, 2o (PP))lpP((p_ (xx)). Ho et al. (2000) reported that
the re-weighting worked effectively for learning 0. Yet, we demonstrate it is deficeent for learning the noise schedule ,3 in our ablation experiment in Section 6.2
BILATERAL DENOISING DIFFUSION MODELS (BDDMs)
4.1 PROBLEM FORMULATION
For fast sampling with DPMs, we strive for noise schedule for sampling that is much shorter than the noise schedule / for training. As shown in Fig. [ we define separate diffusion processes corresponding the noise schedules, and B, respectively. The upper diffusion pro- cess parameterized by 3 is the same as in Eq. ? whereas the lower process is defined as ( ( II. 93., (#t with much fewer diffusion steps (N < T). In our prob- lem formulation, , is given, but P is unknown. The goal is to find a 9 for the reverse process pa( 1/n (.)) such that do can be effectively recovered from & with N reverse steps.
4.2 MODEL DESCRIPTION
Although many prior arts (Ho et al. 2020 Chen et al. 2020 Song et al. 2021 San- Roman et al. 2021) directly applied shortened linear or Fibonacci noise schedule to the reverse process, we argue that these are sub-optimal solutions. Theoretically, the diffusion process specified by new shortened noise schedule essentially different from the one used to train the score network 0. Therefore, 0 is not guaranteed suitable for reverting the shortened diffusion process. This issue motivated novel modeling perspective to establish a link between the shortened schedule 3 and the score network 0, i.e., to have 3 optimized according 0.
4

---

Published as a conference paper at ICLR 2022
As starting point, consider an N [T/r], where < T' hyperparameter controlling the step size such that each diffusion step between two consecutive variables the shorter diffusion process corresponds to t diffusion steps in the longer one. Based on Eq. (2). define the following:
(1 al a
(1- citz at (1-0)))
)
1a (#n+1 [., xx)) (3( (x,) = N
Lic.
(6)
where 2, is an intermediate diffused variable we introduced to link the two differently indexed diffusion sequences. We call it junctional variable, which can be easily generated given æn and (
during training: x, = 0, + /1 -aff, of
Unfortunately, for the reverse process when Do is not given, the junctional variable is intractable. However, our key observation is that while using the score by score network 0 trained for the
long --parameterized diffusion process, a short noise schedule (()) can be optimized accordingly by introducing schedule network Φ. We provide its mathematical derivations in Appendix A.3
Next, we present a formal definition of BDDM and derive its training objectives, 2 (0) and c(u) (o; 8"), for the score network and the schedule network, respectively, in more detail.
4.3 SCORE NETWORK
Recall that a DDPM starts the reverse process with a white noise By ~ (O( and takes T steps to recover the data distribution:
Po (xo)
DDPM
Ex(o.)) (Epo(mirrilaz)) [Po (1r1))]
A BDDM, in contrast, starts from the junctional variable æt, and reverts shorter sequence of diffusion random variables with only n steps:
(0)) Bag 1
TE 2 1) [po -1m- ]]]]
2 S
(8)
where 9g (#n 1: Be, en) defined as re- ce-aarmmeterization the posterior:
2t
a en
1g Cn) 9=
In-1
En 21,
20
(9)
ân
1
Bn
1 - nn 10-100
-N
En
3nl
(10)
Bn
V(1 B1) 022)
0
where ån II V 1 B,, æ. 0p + /1 - aREn en is the junctional variable that maps a
to on given an approximate index ~ u((n 1T..... nT - 1, ner and sampled white noise
6n (0, I). Detailed derivation from Eq. 99) to 110 is provided in Appendix A.2
4.3.1 TRAINING OBJECTIVE FOR SCORE NETWORK
With the above definition, new form of lower bound to the log marginal likelihood can be derived such that log po (b)) 2 score (0) (( (0) Ro (æo, æ.),, where
(n) score (0) : DKL ( 16 = xr, (2))l 4,3 (n. (2n-1; 1; cn))
(11)
Ro xt.) : Epo(é: (x., [log 2o (00
(12)
See detailed derivation in Proposition in Appendix A.2 In the following Proposition we prove that via the junctional variable øt, the solution 8* for optimizing the objective Z (0), Vt {1. is also the solution for optimizing cindee (0), Vn E (2,..., Thereby, show that the score network can be trained with (()ppm (0) and re-used for reverting the short diffusion process over n. Although the newly derived lower bound result in the same objective as the conventional score network. it for the first time establishes link between the score network 0 and & N-u- The connection essential for learning B,, which will describe next.
(+)
5

---

Published as a conference paper at ICLR 2022
4.4 SCHEDULE NETWORK
In BDDMs, a schedule network is introduced to the forward process by re-parameterizing 3n as (o( fo zz: Bm+1). and recall that during training, we can use Dt an + / af and Bn+1= = - ..- Through the re-parameterization, the task of noise scheduling, i.c., searching for B. can now be reformulated training schedule network fo that ancestrally estimates data- dependent variances. The schedule network leams to predict B,, based on the current noisy sample 2; this makes our method fundamentally different from existing and concurrent work, includ- ing Kingma et al. (2021) as reveal that, aside from Bn+1, or that reflects diffusion step information, 2t is also essential for noise scheduling from a reverse direction at inference time.
Specifically, we adopt the ancestral step information (6x+1) ) to derive an upper bound for the current step while leaving the schedule network only take the current noisy sample æ; input predict relative change of noise scales against the ancestral step. First, derive upper bound of ß, Bn+1 But1) } in Appendix A.1 Then, by multiplying the upper
by proving 0 Bn min {1
bound by a ratio estimated by a neural network 0s :R + (0, 1),
we define
(a Bat1) == min { a2 Bnt1
Buti (2)()))
(13)
where the network parameter set é is learned to estimate the ratio between two consecutive noise scales (3, and Bn-1) from the current noisy input æt.
Finally, at inference time for noise scheduling, starting from maximum reverse steps (N) and two hyperparameters B.v, we ancestrally predict the noise scale (n( fo (ans in++) for from N 1, and cumulatively update the product á,
6n-1
VI-Bnt1
4.4.1 TRAINING OBJECTIVE FOR SCHEDULE NETWORK
Here we describe how to learn the network parameters & effectively. First, we demonstrated that should be trained after 0 is well-optimized, referring Proposition B in Appendix A.3 The
Proposition also shows that we are minimizing the gap between the lower bound F(()
(0*) and
log pg- (æo), i.c., log pa- (#o) - Flmn (0*), ), by minimizing the following objective
clup (ob; 0²) D (pp- (#n-1lăn, xx)) (e))l193, (2n ax)).a))))))))
(14)
which is defined as a KL divergence to directly compare Po- (#a-1 fr = xxe) against the re- parameterized forward process posteriors, which are tractable when conditioned on the junctional noise scale ae and Øo-
The detailed derivation Eq. formulas as shown in Step 8 10 in Alg.
also provided in the proof of Proposition to get its concrete
5 ALGORITHMS: TRAINING, NOISE SCHEDULING, AND SAMPLING
5.1 TRAINING SCORE AND SCHEDULE NETWORKS
Following the theoretical result Appendix A.3 should be optimized before learning db. Thereby first, to train the score network Eg, we refer to the settings in (Ho et al.
et al. 2020 Song ct al. 2021) to define as linear noise schedule: 3, = Bstart + $ [Beed BBtart), for I I < T, where Ben and Bend are hyperparameter that specifies the start value and the end value. This results in Algorithm I. which resembles the training algorithm in (Ho et al.) (020).
Next, based on the converged score network 6*, train the schedule network 6. We draw an n ~ 21(2..., NN at each training step, and then draw a ~ U((n 1)r,..,, nT). These together can be re-formulated directly drawing l U(T T 7} for finer-scale time step. Then,

---

Published as a conference paper at ICLR 2022
Algorithm Training Score Network (0)
Algorithm 2 Training Schedule Network (Ø)
1: Given T, {B
Given 8¹, ,T, T, (0,,,,
2: {or)i=1 III.-I -
2: repeat
3: repeat
3: at Dasua(oo)
4: Wo ~ (20)
4: L~ U [T...., T 7)
5: I~ 1~1((,....,
5: ò
ai
6: 6, 1 (0,,
6: En ~ NO I
7: mi = 0120 + 11
7: at a:30 Voeen
8: Calp ||e, es (2,, ax)ll
8: B. min dc, aite
o oa(we)
9: Take gradient descent step V Vo
9: C = 1 log dt Bn) + 2 [3n/d=
10: until converged
10: Sup 2 B.) cr.
8s es- (221, (22, an) ||2 +0
Algorithm 3 Noise Scheduling
1: Given 8*, âx, 8N, xx I)(
11: Take gradient descent step on V Volis
2: for n = N to 2 do
12: until converged
3: -n-1 pa- (2n-1 i dun, 3m) &n
Algorithm 4 Sampling
4: án-1 VI-
Given 0*, (Ba) J
n= Ex. 10,
5: Bn-1 min(1 an An Bntoo((nn1)
N.
6: if T. < B1 then
2: {
II V Bn
7: return Bn...., AN
for n. = NN do
8: end if
4: Øn-1 ~ pa- (#nn 1 (Zn; Ci, Bn)
9: end for
5: end for
10: return BN
6: return du
sequentially compute the variables needed for calculating <in (o: ), as presented in Algorithm We observed that, although linear schedule is used to define B, the noise schedule of 3 predicted by fo is not limited to but rather different from a linear one.
5.2 NOISE SCHEDULING FOR FAST AND HIGH- QUALITY SAMPLING
After the score network and the schedule network are trained, the inference procedure can divide into two phases: (1) the noise scheduling phase and (2) the sampling phase.
First, we run the noise scheduling process similarly mum. Different from training, where a₁ is forward-computed á, is instead backward computed variable (from N to 1) that may deviate from the forward one because ( )m- are unknown in the noise scheduling phase during inference. To start noise scheduling, we first set two hyperparame- ters: āN and By. We use 31, the smallest noise scale seen in training, as a threshold to early stop the noise scheduling process so that we can ignore small noise scales (< (81) that were never by the score network. Overall, the noise scheduling process presents in Algorithm 3
sampling process with N iterations maxi-
In practice, apply grid search algorithm of M bins to Algorithm 3 which takes O(M²) time, to find proper values for (åN, B.)). We used M 9 as (Chen al., 2020). The grid search for our noise scheduling algorithm can be evaluated on a small subset of the training samples, Empirically, even as few as sample for evaluation works well in our algorithm. Finally, given the predicted noise schedule / RR,,,, we generate samples with N, sampling steps, as shown in Algorithm 4
EXPERIMENTS
We conducted series of experiments on neural vocoding tasks evaluate the proposed BDDMs. First, we compared BDDMs against several strongest models that have been published: the mixture of logistics (Mol.) WaveNet (Oord et al., 2018) implemented (Yamamoto) 2020), the WaveGlow (Prenger et al. 22119) implemented in (Valle 22020). the MelGAN (Kumar et al. 20199 implemented in (Kumar 2019), the HiFi-GAN (Kong et al. (020b) implemented in (Kogg et al. 2020a) and
the two most recently proposed diffusion-bart based vocoders, i.e., WaveGrad (Chen et al. 2020) and
DiffWave (Kong et al. (002). both re- re-implemented in our code. The hyperparameter settings of BDDMs and all these models are detailed in Appendix(B]
In addition, we also compared BDDMs to a variety of scheduling and acceleration techniques ap- plicable to DDPMs, including the grid search (GS) approach in WaveGrad, the fast sampling (FS)
7

---

Published as a conference paper at ICLR 2022
Table I: Comparison of neural vocoders in terms of MOS with 95% confidence intervals, real-time factor (RTF) and model size in megabytes (MB) for inference. The highest score and the scores that not significantly different from the highest score (p-values 2 0.05) are bold-faced.
Neural Vocoder
MOS
RTF
Size
Ground-truth
4.64 = 0.08
WaveNet (MoL) Oord et al.
2018
3.52 = 0.16
318.6
282MB
WaveGlow (Prenger al. 2019)
3.03 - 0.15
0.0198
645MB
MelGAN Kumar et al. 2019)
3.48 = 0.14
0.00396
17MB
HiFi-GAN (Kong al.
2020b
4.33 = 0.12
0.0134
54MB
WaveGrad - TO00 steps (Chen et al.
2020
4.36 + 0.13
38.2
183MB
DiffWave 200 steps (Kong al. 2021)
4.49 + 0.13
0.13
7.30
27MB
BDDM 3 steps (&N 0.68, 8x 0.53)
3.64 + 0.13
0.110
27MB
BDDM 7 steps (&nv 0.62, 8N 0.42)
4.43 = 0.11
0.256
27MB
BDDM 12 steps (å.N 0.67, 3N 0.12)
4.48 + 0.12
0.438
27MB
Table 2: Comparison of sampling acceleration methods with the same score network and the same number of steps. The highest score and the scores that are not significantly different from the highest score (p-values 2 0.05) are bold-faced.
Steps
Acceleration Method
STOI
PESQ
MOS
GS Chen et al.
2020
0.965 0.009
3.66
3.61 16110011 0.12
FS (Kong al. 2021)
0.939 = 0.023
3.09
0.12
3 DDIM (Song et al., 2021)
0.943 0.015
0.22
0.13
NE (San-Koman et at. 2021
0.966 0.010
0.18
+
BDDM
0.966 0.011
3.63
0.13
FS (Kong et al. 2021)
0.981 + 0.006
3.68
3.70 0.14
DDIM (Song et al., 2021)
0.974 0.008
3.12
+
7
NE (San-Roman et al. 202
0.978 0.007
3.15
4.02
BDDM
0.983 + 0.006
0.00
4.43 0.11
DDIM (Song et al. 2021)
0.979 + 0.006
3.90
4.16 0.12
12
NE ISan-Roman et al. 2021
0.981 + 0.007
3.82 +
3.98 + 0.14
BDDM
0.987 + 0.006 3.98 0.12
4.48 + 0.12
approach based on a user-defined 6-step schedule in DiffWave, the DDIMs (Song al., 2021 and a noise estimation (NE) approach (San-Roman et al. 202T). For fair and reproducible comparison with other models and approaches. we used the LJSpeech dataset (Ito & Johnson| 2017). which consists of 13, 100100 22kHz audio clips of female speaker. All diffusion models were trained on the same training split as in (Chen et al. (2020). We also replicated the comparative experiment of neural vocoding using multi-speaker VCTK dataset (Yamagishi et al. 2019) as presented in Appendix C and obtained a result consistent with that obtained from the LJSpeech dataset.
6.1 SAMPLING QUALITY OBJECTIVE AND SUBJECTIVE METRICS
To assess the quality of each generated audio sample, used both objective and subjective mea- sures for comparing different neural vocoders given the same ground-truth spectrogram as the condition, i.c., ee (0 8, az)). Specifically, we used two scale-invariant metries: the perceptual evalu- ation of speech quality (PESQ) (Rix et al. 2001) and the short-time objective intelligibility (STOI) TTal et al., 2010 to measure the noisiness and the distortion of the generated speech relative the reference speech. Mean opinion score (MOS) was also used as a subjective metric for evaluating the naturalness of the generated speech. The assessment scheme of MOS included Appendix B
In Table we compared BDDMs against the state-of-the-art (SOTA) vocoders. To predict noise schedules with different sampling steps (3, 7, and 12), we set three pairs of {áN, 3N } for BDDMs by running on Algorithm 3 a quick hyperparameter grid search, which is detailed in Appendix B Among the 9 evaluated vocoders, only our proposed BDDMs with 7 and 12 steps and DiffWave with 200 steps showed no statistic- significat difference from the ground- -runh in terms of MOS. More- over, BDDMs significantly outspeeded DiffWave in terms of RTFs. Notably, previous diffusion-
8

---

Published as a conference paper at ICLR 2022
Predicted Noise Scale Ratio (ap)
0.8
Lowcr bounds to logoy(Xa)
0.6
0.8
0.4
L step elbo
0.2
0.4
0.0
0.2
0.2
F_ bddm
0.4
F elbo
0.0
20
40
100
50
75
100
125
150
175
Training Step
Figure 2: Different training losses for ds
Figure 3: Different lower bounds to log po
based vocoders achieved high MOS scores at the cost of unacceptable RTF for industrial deploy- ment. In contrast, BDDMs managed to achieve a high standard of generation quality with only sampling steps (143x faster than WaveGrad and 28.6x faster than DifiWave).
In Table 2 we evaluated BDDMs and alternative accelerated sampling methods, which used the same score network for a pair-to-pair comparison. The GS method performed stably when the step number was small (i.e., N < 6) but not scalable to more step numbers, which were therefore bypassed in the comparisons of 7 and 12 steps. The FS method by [Song et al. (202T) T) was linearly interpolated to and 7 steps for a fair comparison. Comparing its 7-step and 3-step results, we observed that the FS performance degraded drastically. Both the DDIM and the NE methods were stable across all the steps but were not performing competitively enough. In comparison, BDDMs consistently attained the leading scores across all the steps. This evaluation confirmed that BDDM was superior to other acceleration methods for DPMs in terms of both stability and quality.
6.2 ABLATION STUDY AND ANALYSIS
We attribute the primary advantage BDDMs to the newly derived objective c(n) for learning ø To better reason about this, we performed ablation study, where substituted the proposed loss with the standard negative ELBO for learning & as mentioned by Sohl-Dickstein et al. (2015). We plotted the network outputs with different training losses in Fig.
turned out that, when using (()) to learn o the network output rapidly collapsed to zero within several training steps; whereas, the network trained with cltp produced fluctuating outputs. The fluctuation desirable property showing the network properly predicts t-dependent noise scales, as t is a random time step drawn from a uniform distribution in training.
By setting , = 3, we empirically validated that F-(() == F + L(0) 2 F(t) with their respective
values at E [20, 180] using the same optimized 6* Each value provided with 95% confidence intervals, shown in Fig. In this experiment, used the LJ speech dataset and set T 200 and
T 20. Notably, we dropped their common entropy term Rø (0o, x; < 0 to mainly compare their KL divergences. This explains those positive lower bound values in the plot. The graph shows that
our proposed bound bolm is always a tighter lower bound than the standard one across all examined
(()
Moreover, we found that =(() attained low values with a relatively much lower variance for f(f I < 50, where F(t) was highly volatile. This implies that bidm better tackles the difficult training part, i.e., when the score becomes more challenging to estimate as /. -
CONCLUSIONS
BDDMs parameterize the forward and reverse processes with schedule network and a score net- work, of which the former's optimization is tied with the latter by introducing a junctional variable. We derived a new lower bound that leads to the same training loss for the score network as in DDPMs (Ho et al.) 2020), which thus enables inheriting any pre-trained score networks in DDPMs. We also showed that training the schedule network after a well-optimized score network can be viewed as tightening the lower bound. Followed from the theoretical results, efficient training algorithm and a noise scheduling algorithm were respectively designed for BDDMs. Finally, our experiments, BDDMs showed clear edge over the previous diffusion-based vocoders.

---

Published as a conference paper at ICLR 2022
REFERENCES
Mikolaj Binkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. International conference on learning representations, 2020.
Bond- Taylor, A Leach, Y Long, and CG Willcocks. Deep generative modelling: comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models. IEEE Transac- fions on Pattern Analysis and Machine Intelligence, 2021.
Miguel A Carreira- Perpinan and Geoffrey Hinton. On contrastive divergence learning. AISTATS, pp. 33 40, 2005.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In International conference on learning representations, 2020.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differ- ential equations. Advances in neural information processing systems, pp. 6571-6583, 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34, 2021.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint /605 10000000,,.08003,, 2016.
Brendan J Frey. Local probability propagation for factor analysis. Advances in neural information processing systems, 12:442-448, 1999.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
G. E. Hinton. Training products of experts by minimizing contrastive divergence. neural computa- tion. Neural computation, pp. 14(8):1771-1800, 2002.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. ICMI., 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
neural information processing systems, 33:6840-6$ 6851, 2020.
Aapo Hyvarinen and Peter Dayan. Estimation of nonmaazndl statistical models by score match- ing. Journal of Machine Learning Research, pp. 6(4), 2005.
Keith Ito and Linda Johnson. The Ij speech dataset.
2017.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pp. 2410-2419. PMLR, 2018.
Diederík Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, pp. 10215-10224, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. star, 1050:1, 2014.
Diederík Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In
Advances in neural information processing systems, 2021.
Jungil Kong. Jachyeon Kim, and Jackyoung Bac. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. 2020a.
Jungil Kong, Jachyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances neural information processing systems, 33, 2020b.
10

---

Published as a conference paper at ICLR 2022
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. International conference on learning representations, 2021.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
Rithesh Kumar. Official repository for the paper melgan: Generative adversarial networks for condi- tional waveform synthesis. https: /gg thub. 2019.
wtp.errtiiiic/mmeeeee neur ps
Max WY Lam, Jun Wang, Dan Su, and Dong Yu. Effective low-cost time-domain audio separation using globally attentive locally recurrent networks. In 202/ IEEE Spoken Language Technology Workshop (SLT), pp. 801-808. IEEE, 2021.
Lars Maaløc, Marco Fraccaro, Valentin Liévin, and Ole Winther, Biva: A very deep hierarchy of latent variables for generative modeling. Advances in neural information processing systems, pp- 6548-6558, 2019.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171, PMLR, 2021.
Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International conference machine learning, pp. 3918-3926. PMLR, 2018.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. JMLR, pp. 22(57):1- 64, 2021.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 3617-3621. IEEE, 2019.
Flavio Protasio Ribeiro, Dinei Florencio, Cha Zhang, and Mike Seltzer. CROWDMOS: An approach for crowdsourcing mean opinion score studies. In ICASSP. IEEE, 2011. Edition: ICASSP.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. ICML, pp. 1278-1286, 2014.
Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment telephone networks and codees. In 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing.
Proceedings (Cat. No. 01CH37221), volume 2, pp. 749-752. IEEE, 2001.
Robin San-Roman, Eliya Nachmani, and Lior Wolf. Noise estimation for generative diffusion mod- cls. arXiv preprint wrXiv:2104.02600 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
Jascha
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. Interna- fional conference learning representations, 2021.

---

Published as a conference paper at ICLR 2022
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 13:12433-124453 2020.
Yang Song. Sahaj Garg. Jiaxin Shi, and Stefano Ermon. Sliced score matching: scalable approach to density and score estimation. UAI, pp. 574-584, 2020a.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- fional conference learning representations, 2020b.
Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. A short-time objective intelligibility measure for time-frequency weighted noisy speech. In 2010 IEEE international conference on acoustics, speech and signal processing, pp. 4214-4217. IEEE, 2010.
Rafael Valle. Waveglow: a flow-based generative network for speech synthesis.
https://
github www/NVIDIAI wwwwwgg ow 2020.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. Proc. 9th ISCA Speech Synthesis Workshop, pp. 125-125, 2016.
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning efficiently sam- ple from diffusion probabilistic models. arXiv preprint arXiv:2106.00 200.i::200000000,2,, 2021.
Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald. CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92), 2019.
Ryuichi Yamamoto. Wavenet vocoder. httpp:: http://gtttt.... ctsttuub.omm/30/ wtt.:/c.cmyyyyy/ee wwwp:///////eeeeeeee http://ww....ce/....///aa/eeeeea 2020.
/ttpp:://ithub.cco
vocader
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel. Wavegan: fast waveform genera- tion model based on generative adversarial networks with multi-resolltion spectrogram. ICASSP, 2020.
Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, and Lei Xie. Multi-band melgan: Faster waveform generation for high-quality text-to-speech. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 492-498. IEEE, 2021.
12

---

Published as a conference paper at ICLR 2022
A THEORETICAL DERIVATIONS FOR BDDMS
In this section, we provide the theoretical supports for the following:
The derivation for upper bounding 3,, (see Appendix A.1
The score network trained with C (0) for the reverse process po(2t-1( can be re-used for the reverse process pe ( (see Appendix A.2)
The schedule network can be trained with C (0;; 0*) after the score network opti- mized. (see Appendix A.3)
A.1
DERIVING AN UPPER BOUND FOR NOISE SCALE
Since monotonic noise schedules have been successfully applied to in many prior arts including
DPMs (Ho et al. 2020; Kingma al. 2021) and score-based methods (Song al.) 2020a Song
& Ermon 2020 we also follow the monotonic assumption and derive an upper bound for B, as
below:
Remark 1. Suppose the noise schedule for sampling is monotonic, i.e, 0 << 31 <.... < 3N <1 then, for 1 < n < N, 3n satisfies the following inequality:
(1-
2n
0 < 3m min
Bnt1y
(15)
Bu+1
Proof. By the general definition of noise schedule, we know that < BI,...,,ax
3N < 1 (Note: no
inequality sign in between). Given that ân II v Bi, also have â dg 1.
22
First, we show that Bn <1
1-Ba+1
àn-1
ân
<1 Bn <1 a
a
(16)
V - 3n
1 Bn+1
Next, we show that Bn
< ān-1:
än / - Br
ā
ant1 < 214 + Br < < ant1.
(17)
V1 3n
Bn
3n
Now, we have B, < min {1
62 an+1l }
When -n+11
we can show
1-Bu+1
1-
that Bost < 1 ant1:
2
2
2nt1 <
GZ - ān+1 1 > à :
> an+1
(18)
1 Bn+1
à?
2 < 1 ān+1 + Br+1 < 21= ān+1.
(19)
By the assumption of monotonic sequence, we also have B, < Bn4I. Knowing that Bn+1 < 2
1-āa41 is always true, we obtain a tighter bound for 3,, 0 < 3n min
Br+t
1.. O
A.2 DERIVING THE TRAINING OBJECTIVE FOR SCORE NETWORK
First, followed from the data distribution modeling of BDDMs as proposed in Eq. (8):
Po (it) == Eg, 1hh--1n9p - (e.))
E (d 2/ [Po (b @i.n
TE
(20)
derive new lower bound the log marginal likelihood follows:
13

---

Published as a conference paper at ICLR 2022
Proposition 1. Given æt qg(2)) (o)) the following lower bound holds for (2....,N}
log po 2 F(m) Fecare(0) (0) ;- -C(co) (0) - Ro (oo x+);
(21)
where
[(a) (0) == DKL (po (2 (2 1 - x ( 6n))
(22)
Ro Ro(@o (Bo; xet) == -ppo ((t [log po (foo
(23)
Proof.
(og)o( = log / pol 19/ 1: 24 en d
(24)
log / pol@un-zlēn-
xe Cnl po (#i:n- @ xz.)
d
pe 12(111111) [n = x.,)
(25)
po ( a )g n T4 En)
loo Bpo(ti. 1/ m.)
(26)
pe [ Bnn xt)
pol (b 1 19 In 1 20 Enl
[Jensen's Inequality]
Bpa (tē) (i-
log
(27)
po (0 [nn, x24)
Eo (#: fa me))
[log pe ( [b)l - DEL (po(En-1 (2, x1)ll9 ( 3z Cnl)
(28)
C(a) (0)
Ry ( xx,)
(29)
(1) Next, we show that the score network 0 trained with ddpm (0) can be re-used in BDDMs. We first provide the derivation for Eq.
9 10 We have
1
a En
( Dz, 11: En) = 93
in-1 2n = Dc, 2n
(30)
ân
11 Bn
a 1n Dt
/
6 1)
2 11
=N
(31)
1- a
On
1 22
0
V 8n - -
ân ân-12p 11
δn 1
0
=N
ā (1 (1-11-0 - 6.)
1 6
Et
ān V
62
En
1
6
3
(32)
Bn
1- 2
=N
Ft
Cy:
3
(33)
3n
V(1 3m)(1 6%,
Proposition 2. Suppose x, ~ (x (zo)), then any solution satisfying 0" = argming (1) {1.... also satisfies 0" argming clade (0), Vn N3.
dilpm (0). Vt
Proof. By the definition in Eq. 44, we have
Bn
1 1 à
Pe P xx.) =N
Eo (a an))
3
(34)
VI-8. Bn
/1
Here, from the training objective in Eq. for the score network known to be 0t- Therefore,
5). since x, = a, + /1 of e,, the noise scale argument eg (Iz,, Oze) instead of eg (æ on.) for
14

---

Published as a conference paper at ICLR 2022
expanding Caaee(() (0). Since pa (#n æx)) and 033 (0 We; 6n) isotropic Gaussians with the same variance, the KL divergence is a scaled 22-norm of their means' difference:
(.core()) (0) DDI. (po (2 lann æt)llua [ 1i (t, 6n)
(35)
1 1 a
Bn v
Co (1t, ai)
(36)
2(1 an- 1)
Pn
8n
En
(37)
V1 Bn
V (1 3n) a%)
2
(1 81) 62t)
Bn
(En ED (Iir ait))
(38)
2(1 1 Bn 62,)3,,
V(1 B1)(1 0%)
(1 (1 2
B2
||5n Eo (I'e 02)12
(39)
2(1 3n 62,
a1) (1 3n)
In
1
6n 60
04:30 + v - og 0tt
(40)
2(1 - Bn ag)
which is proportional to ((t)
ddpm
=
Cn co
(1r20 + / 1 of at
|| as defined in Eq.
(5)
Thus,
argming ((t) (0) = argmin, ((n) (0).
(41)
Next, we can simplify (0 n..) reconstruction loss for abg: Ro (0o, 2x)) = -Epo (te, =w.) [log pa (zol x
(42)
= EEpolái =
log.V
1
B
Es
(1), @1), 8:),811
(43)
V1-
31
a
D
1
1
31
= Eppolár =x))
log 2m
Eo (0) an)
2
2,31
v
31
(44)
D log 2 + Epo =n))
-B
④1
2 231
3
(45)
where @o (2,- x.) can be efficiently sampled using the reverse process in (Song et al.
2021
Yet, in practice, similar to the training in (Song et al. 2021 Chen et al. 2020 Kong et al.
2021
we dropped Ro 24) when training In theory, we know that R2 (ao. x.) achieves its optimal (()
value at 0r = al ( - l12. which shares a similar objective as ddpm By minimizing ((t)
ddpp train score network 6* that best minimizes A := ||E: - Eg- (a.zo + / a7,,,,,,))) at)
for all < t < T. Since the first diffusion step has the smallest effect corrupting &y (i.e., 31 ss 0), it suffices to consider a di = VI Br = 0n, in which case we can jointly minimize Ra (0o, xx) by minimizing dddm
(1)
In this sense, during training. given 2; qo ( (o)) train the score network with the same training objective as in DDPMs and DDIMs. Practically, is beneficial for BDDMs as we can re-use the score network 0 of any well-trained DDPM or DDIM.
15

---

Published as a conference paper at ICLR 2022
A.3 DERIVING THE TRAINING OBJECTIVE FOR SCHEDULE NETWORK
Given that I can be trained to maximize the log evidence with the pre-specified noise schedule 3 for training, the consequent question of interest in BDDMs is how to find fast and good enough noise schedule 3 E RN for sampling given an optimized l". In BDDMs, this problem is reduced to how effectively learn the network parameters 6
Proposition 3. Suppose 0 has been optimized and hypothetically converged to the optimal 0", where by optimal means that with 8 we have pe- (#n-1 [ xx) 93 gg(#n-1;28 Rt, En) given ar qa (2g) When 3 is unknown but we have 2o Bo and on ar
minimize the gap between the optimal lower bound Fuome (0") and log pa- (zo). i.e, log pe- (2o) Fin) (f"), by minimizing the following objective with respect to Bn:
(3ni 0-) =DII (po (o-n(innléé [.n 1e)11gg, (in-1
(46)
An
(arazo + Vo
+C,
(47)
2(6, - -Ba)
dc
where
1
8
D
Bn
-1
== - a
C=
log
+
(48)
4
Bn
2
6t
Proof. Note that to = Zo. ân a₁ 2t 0,20 II of and pø- (in-1/ān = x2)
(g(ān,1i22 e.). When vy given pa-, we can express the probability as follows:
Pn- (0n-1 [.,, (200)) 2n)
(49)
- N22: 0(2;0.T)vv T)po- (#n-1 16 00,00 + v ozz)de
(50)
-NN(x: [[x 1g =0 + / azz 2, En 2)dz
(51)
/ /N((z IJN an-1i a130 + -=az
Br
0
1
3 I
dz
V1 3n
(1- - B1)(1 6%)
(52)
(See Eq. (2) in (Fryy 1999h
V1 a
3n
1 o (1
3n) 3n
=N in-1i 02.00
+
I
1 at
1 3n
/1 Bn
(11- Bn)(1 af))
(53)
1 - oz Bn
04200
=V[
Zn-1i
=: 93, (#n Lo. on)).
(54)
1 3r
3n
where, different from Po- (# b. = xx), from Eq. (49 to Eq. 50
instead of conditioning on a
specific æs, when 2p is given st can be generated using any N T).
16

---

Published as a conference paper at ICLR 2022
From this, can express the gap between log po- (0o) and F (0*) in the following form:
log pa- (0o wo) -F (0*)
(55)
Pa (0) ()gg())nn1))) 1i wes En.
(Bn
log pa- (30 xo) pp. (#1,n- 1166 =x))
log pa pu Inn 22,)
(56)
Po- (Do:n xn x3,)
log pa- (30 x0)
pp- (aimm 1/ =m))
log
(57)
Pp- (#) (2imn-
[n = xz,)
log po- (00 1 [., xz,)
-Bpg- (21m 1
(58)
pa- (d1:n-) 1 &n = De, 100 = 07
log pg. (mn 1 [nn xz)
-Epg- (6
p 1l0, =
(59)
18, ( 10 Oks)
=DKL (pg- (#n-1 1 m4)) 1133, (# wo ano)
(60)
Next, we evaluate the above KL divergence term. By definition, we have
Bn
1 1 a - B
po- (p- 2n xx)== =N
Et
ca. (It, dn)
)
(61)
3n
V
a
1 a
Together with Eq. (54). we have
clup (3n;0") DKL po- (in- 16, 22)1199, (2n- Tu. oe)
(62)
2
1 1- 3n
3n
at
20
8t
Eg. (2te, (tt)
+
(63)
2(1 3n a47)
1 3n
V1 1 - Bn
v1 at
2
1- Bn
a1
1
0::00 + V 1 - 1aajen
Bn
Ev- (Ii, (t) (x)
+C
2(1 3n 0
Bn
V 1 Bn
v1
a
2
(64)
1 Pm
1 of
8n
Cn
Co. (o-(2tt 0z)
+
(65)
2(1 - 3n at)
1 8n
V(1
Br (1
ag)
2
1- a?
In
En 1 ai
Eg- (2), as)
+
(66)
2(1 3n a?)
2
8,
Bn Eg.
En
(arao + V Vbien at
+C,
(67)
2(8, - Bn)
8r
where
1
8r
D
Bn
1
6, = 1 - ai
C=
log
(68)
4
8e
Pn
As we use a schedule network é estimate 8, from (an41, as defined in Eq. the final step loss for learning az
13 we obtain
cmpp (0;00)
(n)
6,
6n
Ba(
dz eg- (2i, at)
log
8z
D
(x(
2(6,
(o))
Bn(
8
(69)
17

---

Published as a conference paper at ICLR 2022
This proposed objective for training the schedule network can be interpreted to better model the data distribution (i.e., maximizing log pogpo))) by correcting the gradient scale 3n, for the next reverse step (from &n to 1) given the gradient vector cØ- estimated by the score network l".
B EXPERIMENTAL DETAILS
B.1 CONVENTIONAL GRID SEARCH ALGORITHM FOR DDPMS
We reproduced the grid search algorithm in (Chen et al. 2020). in which a 6-step noise schedule was searched. In our paper, we generalized the grid search algorithm by similarly sweeping the N-step noise schedule over the following possibilities with a bin width M = 9:
2, 6,888 9, & {10- (10-6-N/N 10 10 1- 1)/N 10 1.
(70)
where denotes the cartesian product applied on two sets. LS-MSE was used as a metric to select the solution during the search. When N = 6, we resemble the GS algorithm in (Chen et al.) (2020) Note that above searching method normally does not scale up to N > steps for its exponential computational cost (g9*).
B.2 HYPERPARAMETER SETTING IN BDDMS
Algorithm 2 took a skip factor T to control the stride for training the schedule network. The value of r would affect the coverage of step sizes when training the schedule network, hence affecting the predicted number of steps N for inference - the higher is, the shorter the predicted inference schedule tends be. We set 66 for training the BDDM vocoders in this paper.
For initializing Algorithm 3 for noise scheduling, we could take as few as 1 training sample for validation, perform a grid search the hyperparameters {(âx 0. BN 0.1j)} for i, = I,...,9, 9, i.e., 81 possibilities total, and use the PESQ measure as the selection metric. Then, the predicted noise schedule corresponding to the maximum PESQ was stored and applied to the online inference afterward, as shown in Algorithm ((M²) (c.g., MM 9 in this case), which is much more efficient than (MM') in the conventional grid search algorithm in (Chen et al. 2020), as discussed in Section B.T
Note that this searching has a complexity of only
B.3 IMPLEMENTATION DETAILS
Our proposed BDDMs and the baseline methods were all implemented with the Pytorch library. The score networks for the LJ and VCTK speech datasets were trained from scratch on a single NVIDIA Tesla P40 GPU with batch size 32 for about 1M steps, which took about 3 days.
For the model architecture, we used the same architecture as in DiffWave (Kong et al.) 2021) I for the score network with 128 residual channels; we adopted a lightweight GALR network (Lam et al.) 2021) for the schedule network. GALR was originally proposed for speech enhancement, we sidered well suited for predicting the noise scales. For the configuration of the GALR network, we used window length of 8 samples for encoding, a segment size of 64 for segmentation and only two GALR blocks of 128 hidden dimensions, and other settings were inherited from (Lam et al. (021)) To make the schedule network output with proper range and dimension, we applied sigmoid func- tion to the last block's 's output of the GALR network. Then the result was averaged over the segments and the feature dimensions to obtain the predicted ratio: (x(x)
AvgPool2D(r(GALLR))
where G-LR(-) denotes the GALR network, AvgPool2D(.) denotes the average pooling operation applied to the segments and the feature dimensions, and (x) = 1/(1 + e=*). The same net- work architecture was used for the NE approach for estimating of and was shown better than the ConVTASNet used in the original paper RSman et al. 2021). is also notable that the com- putational cost of schedule network is indeed fractional compared to the cost of score network, as predicting a noise scalar variable is intrinsically a relatively much easier task. Our GALR-bas based schedule network, while being able to produce stable and reliable results, about 3.6 times faster than the score network. The training of schedule networks for BDDMs took only 10k steps to converge, which consumed
than hour on single GPU.
18

---

Published as a conference paper at ICLR 2022
Table 3: Ratings that have been used in evaluation of speech naturalness of synthetic samples.
Rating
Naturalness
Definition
Unsatisfactory
Very annoying. distortion objectionable.
2
Poor
Annoying distortion, but not objectionable. Perceptible distortion, slightly annoying.
3
Fair
4
Good
Slight perceptible level of distortion, but not annoying.
5
Excellent
Imperceptible level of distortion.
Table 4: Performances of different noise schedules the multi-speaker VCTK speech dataset, each of which used the same score network (Chen et al. 2020) E that was trained on VCTK for about 1M iterations.
Noise schedule
LS-MSE (1)
MCD ())
STOI (1)
PESQ (1)
MOS (t)
DDPM (Ho al., 2020; 8 steps (Grid Search)
Chen al. 2020)
101
2.09
0.787
3.31
4.22 +: 0.04
1,000 steps (Linear)
85.0
2.02
0.798
3.39
4.40
0.05
DDIM (Song al. 2021
8 steps (Lmeerr)
553
3.20
0.701
2.81
3.83 + 0.04
16 steps (Linear)
412
2.90
0.724
3.04
3.88 0.05
21 steps (Linear)
355
2.79
0.739
3.12
4.12 = 0.05
100 steps (Linear)
259
2.58
0.759
3.30
4.27 = 0.04
NE (San-Roman et al. 2021
8 steps (Linear)
208
2.54
0.740
3.10
4.18 0.04
16 steps (Linear)
183
2.53
0.742
3.20
4.26 = 0.04
steps (Linear)
852
3.57
0.699
2.66
3.70 = 0.03
BDDM (åN, BN))
8 steps (0.2, 0.9)
98.4
2.11
0.774
3.18
4.20
0.04
16 steps (0.5, 0.5) 21 steps (0.5, 0.1)
73.6
1.93
0.813
3.39
4.35 0.05
76.5
1.83
0.827
3.43
4.48
0.06
Regarding the image generation task, to demonstrate the generalizability of our method, we directly adopted Iscore network pre-trained on the CIFAR-10 10 dataset implemented by third-party open- source repository Regarding the schedule network, to demonstrate that it does not have to use replaced GALR by the VGG11 (Simonyan & Zisserman 2014), which
specialized architecture, was also used by as noise estimator in (San-Roman et al. 2021). The output dimension (number of classes) of VGGI1 I was a sigmoid activation to the last layer to ensure [0, 1] output. Similar to the training in speech domain, we trained the VGG1 1-based schedule networks while freezing the score networks for 10k steps, which normally can be finished in about two hours.
Similar the setting for GALR in speech synthesis, we added
Our code for the speech vocoding and the image generation experiments will be uploaded Github after the final decision of ICLR is released.
B.4 CROWD-SOURCED SUBJECTIVE EVALUATION
All our Mean Opinion Score (MOS) tests crowd-sourced. We refer to the MOS scores in (Pro- tasio Ribeiro et al., 2011). and the scoring criteria have been included in Table 3 for completeness. The samples presented and rated one at time by the testers.
C ADDITIONAL EXPERIMENTS
demonstration page at
github. i shows some samples generated by BDDMs trained IJ speech and VCTK datasets.
19

---

Published as a conference paper at ICLR 2022
C.I MULTI-SPEAKER SPEECH SYNTHESIS
In addition to the single-speaker speech synthesis, we evaluated BDDMs on the multi-speaker speech synthesis benchmark VCTK (Yamagishi al. 2019). VCTK consists of utterances sam- pled at 48 KHz by 108 native English speakers with various accents. We split the VCTK dataset for training and testing: 100 speakers were used for training the multi-speaker model and speakers for testing. We trained on a 44257-utterance subset (40 hours) and evaluated on a held-out 100- utterance subset. For the score network, we used the Wavegrad architecture (Chen et al., (020) so as to examine whether the superiority of BDDMs remains in a different dataset and with a different score network architecture.
Results are presented in Table 4 For this multi-speaker VCTK dataset, we obtained consistent observations with that for the single-speaker LJ dataset presented the main paper. Again, the proposed BDDM with only 16 or 21 steps outperformed the DDPM with 1,000 steps. To the best of our knowledge, ours was the first work that reported this degree of superior. When reducing to 8 steps, BDDM obtained performance on par with (except for worse PESQ) the costly grid- searched 8 steps (which were unscalable to more steps) in DDPM. For NE, we could again observe degradation from its 16 steps to 21 steps, indicating the instability of NE for the VCTK dataset likewise. In contrast, BDDM gave continuously improved performance while increasing the step number.
C.2 COMPARING DIFFERENT REVERSE PROCESSES FOR BDDMS
This section demonstrates that BDDMs do not restrict the sampling procedure to a specialized re- verse process in Algorithm 4 In particular, we evaluated different reverse processes, including that of DDPMs as shown in Eq. (4) and DDIMs (Song et al. (021), for BDDMs and compared the ob- jective scores on the generated samples. DDIMs (Song et al.) 202T) formulate a non-Markovian gen- erative process that accelerates the inference while keeping the same training procedure as DDPMs. The original generative process in Eq. 4 in DDPMs is modified into
Pg (1) (xor)) == 1 (xTT)
II i=1
Pb (n)) (I (xn,))
II
(0) Pe (2 (z)) EE5
(71)
where y is sub-sequence length N of [1, ..., TJ with 7N = T, and 7 = (1, T) \ y is defined
as its complement; Therefore, only part of the models are used in the sampling process.
To achieve the above, DDIMs defined prediction function f(() (ax) that depends on eg to predict the observation zo given æ, directly:
1
(x+)
V at afEe(
(72)
Oie
By leveraging this prediction function, the conditionals in Eq. (11 are formulated as
(70) Pe (x =N
(an (x - < a oz ifi [N]
(73)
(1) P8 (xo x.) (z))
(z1).
otherwise.
(74)
where the detailed derivation of and can be referred to (Song et al. 2021). In the original DDIMs, the accelerated reverse process produces samples over the subsequence of (3 indexed by y:
& {B, E 7). In BDDMs, to apply the DDIM reverse process, we use the predicted by the schedule network in place of a subsequence of the training schedule f.
Note that the subjective evaluation (MOS) is omitted here since the other assessments above have shown that the MOS scores are highly corre- lated with the objective measures, including STOI and PESQ. They indicate that applying BDDMs to either DDPM or DDIM reverse process leads to comparable and competitive results. Meanwhile, the results show some subtle differences: BDDMs over a DDPM reverse process gave slightly better samples in terms of signal eтror and consistency metrics (i.e., LS-MSE and MCD), while BDDM over a DDIM reverse process tended to generate better samples in terms of intelligibiltty and per- ceptual metrics (i.e., STOI and PESQ).
Finally. the objective scores are given in Table 5
20

---

Published as a conference paper at ICLR 2022
Table Performances of different reverse processes for BDDMs on the VCTK speech dataset, each of which used the same score network (Chen et al. 2020 co(-) and the same noise schedule.
Noise schedule
LS-MSE (J)
MCD ())
STOI (1)
PESQ (1)
BDDM (DDPM reverse process)
steps (0.3,04 0.9, le 16
91.3
2.19
0.936
3,22
16 steps (0.7, 0.1,
73.3
1.88
0.949
3.32
21 steps (0.5, 0.1, 1e
72.2
1.91
0.950
3.33
BDDM (DDIM reverse process)
8 steps (0.3, 0.9, le 5)
91.8
2.19
0.938
3.26
16 steps (0.7,0.1, 0.1, 6)
77.7
1.96
0.953
3.37
21 steps (0.5, 0.1, 9)
77.6
1.96
0.954
3.39
Table 6: Comparing sampling methods for DDPM with different number of sampling steps in terms of FIDs in CIFAR1O.
Sampling method
Sampling steps
FID
DDPM (baseline)
(Ho et al.
2020)
1000
3.17
DDPM (sub-VP) Song et al.
2020b
100
3.69
128
5.24
DDPM (DP + reweighting)
Watson et al.
2021
6.74
100
4.16
DDIM (quadratic) (Song et
2021
50
4.67
100
2.86
FastDPM (approx. STEP)
(Kng & Ping
2021
50
3.20
100
4.63
mmrrved DDPM (hybrid) (Nichol & Dhariwal
2021
50
5.09
VDM (augmented) Kingma et al.
2021
1000
7.4 F
100
2.38
Ours BDDM
50
2.93
C.3
UNCONDITIONAL IMAGE GENERATION
For the unconditional image generation task, we evaluated the proposed BDDMs on the benchmark functions, including those initially proposed in DDPMs (Ho et al. 2020) or DDIMs (Song et al., 2021) and those pre-trained in the above third-party im- plementations, are all conditioned on discrete step-index. We estimated the noise schedule & in continuous space using the VGGI1 schedule network and then mapped discrete time schedule using the approximation method in (Kong & Ping 2021)
CIFAR-10 10 (32 32) dataset. The
Table O shows the performances of different sampling methods for DDPMs in CIFAR-10. By set- ting the maximum number of sampling steps (N) for noise scheduling, we can fairly compare the improvements achieved by BDDMs against related methods the literature terms of FID. Re- markably, BDDMs with 100 sampling steps not only surpassed the 1000-step DDPM baseline, but also produced the SOTA FID performance amongst all generative models using less than or equal to 100 sampling steps.
"Our implementation was hased on https:::p// : /ii thub. cat.uubbacommpeaaa///iaaaa "hhe authors of VDM claimed that they tuned the hyperparameters only for minimizing the likelihood and did pursue further tuning of the model to improve FID.
ffusion
21

---

Published as a conference paper at ICLR 2022
n = 3
n= 2
m mm
n=
n=
Figure 4: Spectrum plots of the speech samples produced by BDDM within 3 sampling steps. The first row shows the spectrum to the bottom, we show the spectrum of the resultant signal after each step of the reverse process performed by the BDDM. We also provide the corresponding WAV files on our demo page.
random signal for starting the reverse process. Then, from the top
22


Reviews:
[{"review_id": "WYW9F50ihDs", "reviewer": "Reviewer_8FYb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Although I just did a quick pass, I think the derivations in Appendix A are correct. \n- Results in Table 1 seem convincing.\n- Results for VCTK are nice. I'd suggest adding them to the main paper.\n- I found the ablation of ELBO vs BDDM appropriate.\n\nWeaknesses:\n- I had difficulty in understanding the paper. I think it is not accessible for a wide audience.\n- I personally find that the paper lacks motivation in every step. High-level interpretations are missing and quite often one has the impression that the corresponding formula \"just appears\" in there, somehow randomly or out of nowhere. After carefully reading it for hours I still do not understand the main intuition behind the proposed approach: Why this way?\n- The paper discusses and proposes generic approaches in a general setting, but then only empirically evaluates them in the task of speech vocoding, which is in a quite particular domain (speech) and, more importantly, corresponds to a strongly-conditioned task (that is, a task where the conditioning carries a lot of information; for instance, it is known that tasks like that require much less sampling steps for diffusion models). It leaves a lot of doubt of whether the proposed approach works with, say, unconditional image generation, or even a less strongly-conditioned task like category-based generation.\n- I'm not convinced about the significance of the improvement between the methods presented in Tables 1 and 2 (I'm talking about perceptual significance, not statistical significance here). Furthermore, for some methods and measures (and specially in Table 2), confidence intervals are quite wide, leaving the reader with the doubt of whether the approach is worth this complicated development when compared to, for instance, DDIM.\n\nOther/random:\n- I think the role of $\\tau$ is not discussed enough.\n- I'd have appreciated more discussion on the role of the architecture for the scheduling network.\n- Why GS becomes \"prohibitively\" for $N>6$ and not $N>5$ or $N>10$? Even with a reduced set of synthesis utterances?", "summary_of_the_paper": "The paper proposes to train a scheduling network in addition to the score network for fast sampling of diffusion models. To train such network a new loss is proposed. Results are presented for the task of neural vocoding (single-speaker and multi-speaker).", "summary_of_the_review": "I think the proposal of the paper is sound and the developments are correct. However, the choice of the task for which to provide empirical evidence is in a single domain (speech) and very specific (extremely strong conditioning). I personally find the paper hard to follow and lacking motivation/intuition.\n\n[Update: After authors' response and improved manuscript I decided to raise my score from 6 to 8]", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "U1Rpnm0wK6F", "reviewer": "Reviewer_vo24", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. The idea to find the optimal forward process is interesting and novel (modulo concurrect work like VDM https://arxiv.org/abs/2107.00630 which attempts to reach the same goal).\n2. The experimental results persuade that the method indeed works well with few inference steps.\n\n##########################################################################\n\nCons:\n\n1. There are a lot of technical details which make the paper hard to read. It would profit from moving some parts into the supplementary material. In particular, Section 3 first lists 2 pages of propositions and theorems - and only later the reader understands why those were necessary. It might be better to first state what is being done in simple language/terms, and only then explain it mathematically - or even move the math into the supplementary.\n2. The comparison methodology to DiffWave is a bit strange: the paper correctly states that confidence intervals for both proposed method and DiffWave cover ground truth under t-test, but why not perform paired test to determine which method is better? I.e. show two generations and ask the rater which is more realistic. Also, there exist paired tests with less strict assumptions than t-test. For example, Wilcoxon signed-rank test doesn't assume normal distribution of scores: https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test.\n\n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease address the cons above.\n\n#########################################################################\n\nMinor suggestions and typos: \n\n(1) Parallel WaveNet is worth mentioning as one of the earlier attempts to apply distillation for speeding up generation.\n\n(2) \"Bilateral\" in the paper name is slightly confusing: it made me think about https://en.wikipedia.org/wiki/Bilateral_filter, which is unrelated to the proposed method.", "summary_of_the_paper": "This speech synthesis paper proposes to learn the forward diffusion process and reach better ELBO values by doing that, as well as facilitate inference with fewer steps while keeping good generation quality. The experiments were conducted on LJSpeech and VCTK datasets. The paper compared the proposed method to some strong baselines, including WaveNet, GANs and two recent diffusion methods: WaveGrad and DiffWave. The metrics were STOI, PESQ and human evaluation MOS. The MOS results for the proposed method show no significant difference from the ground truth (same as DiffWave) starting from 7 steps while being and an order of magnitude faster than the best baseline DiffWave.", "summary_of_the_review": "I am inclined to accept this paper, because it contains novel ideas and the experiments are solid. Writing would profit from simplification though.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BTAX43BtkGD", "reviewer": "Reviewer_LPmT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n- The idea proposed by the paper, this is, learning a noise scheduling network based on an adapted objective such that learning this network corresponds to explicitly tightening the lower bound on the data log-likelihood, is new to the best of my knowledge (although related methods exist). Since the noise schedule is learnt specifically for synthesis with fewer steps, the work addresses an important problem of denoising diffusion models, their slow sampling speed.\n- The experimental results are promising. The method achieves almost state-of-the-art results, while requiring much fewer synthesis steps, and also seems to be superior to several previous approaches for accelerating sampling in these types of models.\n\n**Weaknesses and Questions/Suggestions**:\n\nOverall, I found the paper difficult to follow and lacking in clarity. I have several questions and concerns:\n- While the overall approach seems to be novel, it is closely related to [1]. [1] also learns the noise schedule and additionally also shows how one can perform synthesis with fewer steps. I think this suggests a more thorough, quantitative comparison to [1]. In particular, it would be interesting to learn the noise schedule in a similar manner as [1] and then also reduce the number of steps similarly ([1] works on images, but the method is general). I would be curious which method performs better. The current ablation is a bit inconclusive in that regard, as [1] uses a significantly different noise parametrization, which may not as easily collapse.\n- The paper has no thorough Related Work section, which has become a standard part of many papers to appropriately position the work in the literature. I would recommend including such a section in the paper.\n- Beginning of section 3; \"...which is used to reversely sample $x_n \\sim q_\\phi(x_n|x_{n+1}, x_0)$ using a scheduling network $\\phi$.\": Why do we even need an additional scheduling network here? When conditioning on both $x_{n+1}$ and $x_0$, as indicated in the distribution, the ideal denoising distribution, or posterior, is tractable [2].\n- In Figure 1, both $q_\\phi(x_n|x_{n-1})$ and $q_\\phi(x_n|x_{n+1}, x_0)$ have $\\phi$ indices and depend on the learnt scheduling network. However, these are different distributions (one defines a forward process, the other the backward posterior). It is unclear, which distribution exactly is learnt here and how the forward and backward processes depend on the learnt distribution. I would suggest to clarify.\n- Below Proposition 2; \"In this regard, the proposed lower bound $F^{(t)}_{score}(\\theta)$ allows us to consider only one $t$ at each training step for efficient training, which is practically more advantageous than the standard ELBO in Eq. (1) that entails computing a sum of T KL terms.\" -> In DDPM and related methods, we never actually calculate the whole sum either. Rather, $t$ is randomly sampled within a mini batch. Furthermore, my understanding is that BDDM in fact uses a DDPM objective to learn the score network as well. Hence, the advantage is not really clear to me.\n- Above Proposition 2, $\\pi(x_t)$ is defined as the \"prior\" at $t$, which, I think, we can interpret as the marginal distribution over all possible $x_t$ at $t$. But now in Proposition 3, we assume $\\pi(x_{t-1})=q_\\beta(x_{t−1}|x_t, x_0)$, which corresponds to the posterior over $x_{t−1}$, given specific conditionings $x_t$ and $x_0$. This seems very different compared to the $\\pi(x_t)$ described above Prop. 2 and might even be contradictory. I think this needs clarification.\n- What specifically is $p_{\\theta^{*}}(x_{1:t−1}|x_0)$ in Proposition 3? My understanding is that $p_{\\theta}$ generally is the learnt score model that defines a distribution of the form $p_\\theta(x_{t−1}|x_t)$. How does this distribution then predict $x_{1:t−1}$ given $x_0$, as indicated?\n- Theorem 1: In practice, it is highly unlikely that we ever have $\\theta=\\theta^*$. Hence, I believe the $\\theta\\neq\\theta^*$ case is the only practically relevant one. But in this case, it isn't clear to me why the bound will be tighter than the regular ELBO (because the $\\mathcal{L}_{step}$ term isn't there). The proof says it follows from Proposition 2, but it isn't clear. The proposition does not explicitly compare the derived bound with the regular ELBO bound from DDPMs in terms of how tight the bounds are, unless I am missing something.\n- Eq. 19 is supposed to define a linear noise schedule, but has $t$ in the denominator. Is there an error?\n- [3] also learns a noise schedule. I believe it also deserves to be discussed and cited.\n\n[1] Nichol and Dhariwal, \"Improved Denoising Diffusion Probabilistic Models\", 2021.\n\n[2] Ho et al., \"Denoising Diffusion Probabilistic Models\",  2020.\n\n[3] Kingma et al. \"Variational Diffusion Models\", 2021.", "summary_of_the_paper": "The paper aims at improving generative denoising diffusion models for audio synthesis. In particular, it proposes a method to learn a noise prediction or scheduling network that optimally chooses the noise scales (this is, the scales of the noise that is injected during the iterative synthesis process) at inference time. This allows for faster synthesis without much reduction in quality. The method, called Bilateral Denoising Diffusion Model (BDDM), relies on a slightly modified training objective compared to previous works, in which case learning the noise schedule corresponds to optimizing the objective towards a tighter bound on the data log-likelihood. The main score model is learned in a similar fashion like previous diffusion models, such that the objectives for the score model and the noise scheduling network decompose into two separate objectives that can be optimized essentially one after another. Experimentally, BDDM achieves audio generation with a quality almost as good as the current state-of-the-art, while requiring much less synthesis steps. It also outperforms other methods for accelerated sampling from diffusion models for audio synthesis.", "summary_of_the_review": "Overall, the paper presents a new idea for learning the inference noise schedule in generative diffusion models and the experimental results are promising. However, closely related methods exist [1] and a more detailed comparison isn't presented (it's not one of the baselines). My main concern is that I found the presentation and writing of the paper lacking in clarity and difficult to follow, as indicated by my questions above. This also makes it difficult to check the derivations and develop a solid intuition for the method. Related to this, the motivation for the specific approach and many steps in the derivations are a bit unclear. In conclusion, I think the paper is not ready for publication in its current form. However, the method does seem promising nonetheless. Hence, I would be willing to raise my score if the weaknesses and questions can be addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work suggests an extension of diffusion-based generative models, where both the forward and reverse process have learnable parameters (rather than just the reverse process). This is then applied to speech synthesis, with high-fidelity audio generated in very few sampling steps compared to what is typical for this class of models. The proposed model is specifically compared to other diffusion-based approaches for speech synthesis in terms of inference speed.

Reviewers highlighted the novelty of the idea and the convincing experimental results. Concerns were raised about the accessibility and clarity of the presentation (structure, too many technical details), lack of a related work section, and the methodology used to compare the proposed model against baselines. The authors have attempted to address these issues, and two reviewers raised their scores as a result. All reviewers now recommend acceptance.

I am therefore recommending acceptance as well, but I would like to encourage the authors to polish the presentation further, in order to make the work maximally accessible to a wide audience.
Decision: Accept (Poster)