DOSE: Diffusion Dropout with Adaptive Prior for Speech Enhancement
Authors: Wenxin Tai
Yue Lei
Fan Zhou
Goce Trajcevski
Ting Zhong
Venue: NeurIPS 2023 poster (2023)
Keywords: speech enhancement
diffusion models
adaptive prior
dropout
generalization
TL;DR: We identify that prevailing diffusion enhancement models are susceptible to condition collapse, and we propose an efficient method to address this issue.

Abstract: Speech enhancement (SE) aims to improve the intelligibility and quality of speech in the presence of non-stationary additive noise. Deterministic deep learning models have traditionally been used for SE, but recent studies have shown that generative approaches, such as denoising diffusion probabilistic models (DDPMs), can also be effective. However, incorporating condition information into DDPMs for SE remains a challenge. We propose a model-agnostic method called DOSE that employs two efficient condition-augmentation techniques to address this challenge, based on two key insights: (1) We force the model to prioritize the condition factor when generating samples by training it with dropout operation; (2) We inject the condition information into the sampling process by providing an informative adaptive prior. Experiments demonstrate that our approach yields substantial improvements in high-quality and stable speech generation, consistency with the condition factor, and inference efficiency. Codes are publicly available at https://github.com/ICDM-UESTC/DOSE.

DOSE: Diffusion Dropout with Adaptive Prior for Speech Enhancement
Wenxin Tai', Yue Lei', Fan Zhou 1.2 Goce Trajcevski Ting Zhong 1.2
University of Electronic Science and Technology of China Kashi Institute of Electronics and Information Industry Iowa State University
Abstract
Speech enhancement (SE) aims to improve the intelligibilitt and quality of speech in the presence of non-stationary additive noise. Deterministic deep learning models have traditionally been used for SE, but recent studies have shown that generative approaches, such as denoising diffusion probabilistic models (DDPMs), can also be effective. However, incorporating condition information into DDPMs for SE remains a challenge. We propose model-agnostic method called DOSE that employs two efficient condition-aute aagtiionttaummttttiioonn techniques to address this challenge, based on two key insights: (1) We force the model to prioritize the condition factor when generating samples by training it with dropout operation; (2) We inject the condition information into the sampling process by providing an informative adaptive prior. Experiments demonstrate that our approach yields substantial improvements in high-quality and stable speech generation, consistency with the condition factor, and inference efficiency. Codes publicly available https: //tpthu.///thbbb... COW.ICCDM-UBSTC//OOOO
Introduction
Speech enhancement (SE) aims to improve the intelligibility and quality of speech, particularly in scenarios where degradation is caused by non-stationary additive noise. It has significant practical implications in various fields such as telecommunications ①. medicine (2]. and entertainment (3) Modern deep learning models are often used to learn a deterministic mapping from noisy to clean speech. While deterministic models have long been regarded as more powerful in the field of SE, recent advancements in generative models [4] have significantly closed this gap.
One such generative approach is based on using denoising diffusion probabilistic models (DDPMs) 6 77 which have been shown to effectively synthesize natural-sounding speech. Several diffusion enhancement models have been developed 8). which try to learn probability distribution the data and then generate clean speech conditioned on the noisy input. A key challenge in using diffusion enhancement models is how effectively incorporate condition information into learning Previous works address this issue through designing specific
and generating faithful speech [9 10 g
condition-injecting strategies
devising complex network architectures
5)
We conduct a thorough examination to understand the limitation of diffusion- SE methods and find that diffiusion enhancement models susceptible to condition collapse, where the primary cause of inconsistent generation is the non-dominant position of the condition factor. We thus introduce paradigm to effectively incorporate condition information into the diffusion enhancement models. Specifically, we propose a Diffusion-dis drronn Speech Enhancement method (DOSE), which model- SE method (Figure I that employs two efficient condition augmentation tech-
"Correponning author: fan. www.wuestu.cccmmo edu.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Dropout
- Bernoulti(1 - p)
Adaptive prior
Step Ps(Ro 13
Ro
Nt y
Nakey Spoach
Residual
Step 2 Po(Ko|Rx2.y)
X
L1 loss
Clean Sperrih
Figure I: An illustration the proposed DOSE. DOSE consists two primary procedures: (1) training condition diffusion model using dropout operation, and (2) generating speech using conditional diffusion model equipped with the adaptive prior.
niques: (1) During training, we randomly drop out intermediate -enettttedeereeeddd samples. This dropout mechanism guides the model's attention toward the condition factors; (2) Instead of letting the model generate samples from scratch (Gaussian distribution), we employ an adaptive prior derived from the conditional factor to generate samples. Experiments on benchmark datasets demonstrate that our method surpasses recent diffusion enhancement models terms of both accuracy and efficiency. Additionally, DOSE produces more natural-sounding speech and exhibits stronger generalization capabilitiies compared to deterministic mapping- based methods using the same network architecture.
Related works
There are main categories diffusion-basse SE methods: (1) designing specific condition-
injecting strategies [ 9 5 or (2) generating speech with an auxiliary condition optimizer [12]
10 8.
The first category considerates noisy speech in the diffusion (or reverse) process, either by
linearly interpolating between clean and noisy speech along the process 4 9
or by defining such
transformation within the drift term stochastic differential equation (SDE)
Works from the second category rely on an auxiliary condition optimizer generator (diffusion model) synthesizes clean speech and condition optimizer informs what to generate [12 Both the generator and condition optimizer have the ability to denoise, with the latter undertaking the part. Given the challenges in leveraging condition information [8) diffusion-based SE methods within this category often necessitate specific network architecture design to guarantee the participation of condition factors.
TO g.
In paradigm sense, our method is quite similar but different to the second branch - unlike previous approaches that require additional auxiliary networks, DOSE is an end-to-ena tiffusion-basee SE method. In addition, DOSE model -ageosignottic that does not need any specific network design to guarantee consistency between the generated sample and its corresponding condition factor.
3 Preliminaries
We now provide brief introduction to the diffusion probabilistic model (diffusion models, for short), the definition of speech enhancement, and the condition collapse problem.
3.1 Diffusion models
A diffusion model 13 consists of forward (or, diffusion) process and reverse process. Given data point æ6 with probability distribution p(aro). the forward process gradually destroys its data structure by repeated application of the following Markov diffusion kernel:
p (a (._)) 1) N((x;; / V1 B, LE (1,2, ,
(1)
where :., 3T is pre-defined noise variance schedule. With enough diffusion step T, p(ær) converges to the unit spherical Gaussian distribution. Based on the Markov chain, the marginal distribution at arbitrary timestep has the following analytical form:
p(a No Vayso, (1 ad-a))D a)).). te (1,2. .T}.
(2)
where -I.'-1 (1 B.).

---

Loss curve
PESQ
0.12
0.10
Unnconiiton
-Condition
Cenod
Conoe
Conent
2.32
0.08
RCD
Rib
RDD
2.2
2.25
0.06
Rini
Kini
Bin)
2.0
1.96
0.04
1.79
0.02
1.8
0.00
Step
1.6
50k
100k
150k
200k
250k
a
(a)
(-)
(b)
(c)
Figure 2: Investigation of the condition collapse problem. From left to right: (I) comparison of loss curves between unconditional and conditional diffusion models; (2) three variants; (3) PESQ performance of different variants, (-) represent the unprocessed speech.
As for the reverse process, it aims to learn a transition kernel from æ; to 24-1, which is defined as the following Gaussian distribution 6
po x- xx)) - (N: Me (2e, 1), Exer, ()),
where 0 is the learnable parameter and p4g ( t) - (a. vi-a: co (x t)) denotes the mean
9e
of #t-1, which obtained by subtracting the estimated Gaussian noise cg ( t) in the Øy. With such learned transition kernel, can approximate the data distribution p(20)) via:
po()))) = -f po (xT))) II po (xe- - x )da
where po(xT) = (wr; 0)
3.2
Problem formulation
Speech enhancement refers to methods that try reduce distortions, make speech sounds more pleasant, and improve intelligibility. In real environments, the monaural noisy speech in the time domain can be modeled as:
y
(5)
where æ and n denote clean and noise signals. respectively. For human perception, the primary goal of speech enhancement is to extract from y. Mapping-based speech enhancement methods directly optimize po (zly), while diffusion enhancement methods generate clean samples through a Markov process pe (Zo:T-1 11:T. y).
3.3 Condition Collapse in diffusion enhancement models
The condition collapse problem in speech enhancement was first proposed in 8 and refers to the limited involvement of the condition factor during conditional diffusion training, resulting in inconsistencies between the generated speech and its condition factor.
In this work, we argue that the condition factor y indeed participates and helps the intermediate- generated sample 2; approximate p(0+-1122 a0) Our assertion supported by the experiment depicted in the left part of Figure _ the diffusion model equipped with the condition factor exhibits a lower loss curve compared to the unconditional on To better understand the condition collapse phenomenon, we devise two variants that explicitly modify the mutual information between the condition factor and the model's output (Figure 2 (middle)). We use skip connections to add the condition factor multiple layers, forcing the likelihood of maintaining strong connection between the condition factor and output features. Since the dependence of the output on any hidden state in the hierarchy becomes weaker
moves further away from the output in that hierarchy (cf. [14). using skip connections can explicitly enhance connections between the generated sample and condition factor.
"We use DiffWave 7 as basic architecture and use the same experimental settings as the only difference being the change in the way of condition- injecting since most speech enhancement methods will directly use noisy speech the condition factor, rather than Mel-spectrogram.

---

As shown in Figure 2 (right), an increase in mutual information (connections) leads to significant improvement in the consistency between the generated sample and the condition factor - 6). However, it requires meticulously designed model to guarantee its effectiveness (b c). While previous studies [5]
10 g focus on explicitly enhancing the consistency between the output speech and condition factor through specific network architecture design, we explore the possibility of solution independent of the model architecture. This would broaden the applicability of our method, as it enables slight modifications to existing deterministic mapping-based models to transform them into diffusion enhancement models.
Methodology
Considering the diffusion model provides transition function from 2; to Lt-1, typical condition generation process represented as:
pe (2oll) plar)
II t=
po 1 y) dæT
ΦT N(Tr; I).
(6)
Prior
Condition
Our experiments above indicate that pe [ee y) will easily collapse to pe p xx). resulting in the condition generation process degenerating into a vanilla unconditional process:
T
T II
/ po(2g)
II
po (1 (el.etyddee 2 + po(xrr)
(7)
As result, facilitating automatic learning of the joint distribution for both clean and noisy speech samples does work well for the speech enhancement task.
4.1 Condition augmentation I: Adaptive Prior
Let's revisit Eq. ( since we cannot easily inject the condition factor into the condition term, how about the prior term? For example, we can modify the condition generation process
po(aolly) =
p(a,-ly)
II (a 1 (xx) da
(8)
Conditional
Unconditional
where p is formulated as p(æ+ly) = Narr Vary, y T)) The following propositions verify the feasibility of our proposal.
Proposition 1. For any & > such that < < M for some finite positive value M, there exists positive value E {0, ... ,T, that satisfies:
DEL (p(a,) (=(c ST
where p(z,(c) (xx;;
Remark 1. This proposition indicates that, given a tolerable margin of error & and a well-trained always find suitable T such that we are able to recover the clean speech
diffusion model, from its noisy one y using Eq. (8)
While Proposition allows us to generate clean speech æ given the noisy speech using Eq. 8 it does not guarantee that our model will achieve successful recovery with high probability.
Proposition 2. Let be the clean sample, be it's corresponding noisy one, and x' be any neighbor from the neighbor set S(x)). Then diffusion enhancement models can recover a with high probability the following inequality is satisfied:
p(x)
log
20 (1
v11
||
-112),
Vas S(x),
(10)
p(x')
where of
1-āc Cc.
Remark 2. Assuming that the condition factor y is closer than to x', obtain non non-positive right-hand side (RHS). For a given x, the left-andd side (LHS) value is fixed, and to ensure the inequality always holds, smaller of is preferred.

---

As shown in Figure of will increase as the timestep t increases. Thus, according to Proposition 2, we should choose small т for Eq. (8) to maximize the probability of successfully recovering the clean speech from the noisy one. However, constrained by Proposition 1, cannot be too small. In other words, the clean speech distribution p(æ,) and the noisy speech distribution p(y-) will get closer over the forward diffusion process, and the gap |n| = y - ø between the noisy speech and the clean one will indeed be "washed out" by the increasingly added noise. Since the original semantic information will also be removed r is too large, there should be trade-off when set for the diffusion enhancement model.
2.5
0.0
50, 1o-4, 0.035
.2.
200, 1e-4, 0.05
200
Timestep
Figure 3: The change curves of logio of Elements in legend are T, B1, BT respectively.
Condition optimizer. We find that both propositions are corre-
lated with the condition factor y. If we can reduce the gap between the condition factor and clean speech, choose smaller effectively increasing the likelihood of recovering clean speech. One simple idea is to employ neural network fu to optimize the condition factor, as demonstrated in [15. Accordingly, we can rewrite Eq. (8) as:
po. (zolg)
/ P (
II po (2))
(11)
where Py (x Naxz; (x) (1-a)))
In practice, we should also consider failure cases of the condition optimizer in complex scenarios, especially the issue of excessive suppression that has been reported in recent literature [7 [8] To mitigate this issue, we use 0.5c + 0.5y (like simple residual layer) as a mild version of the condition factor:
[7
Pe (2, y) = ((x,;; (f + y) (1 a.)
(12)
We call Pip (yyll) the adaptive prior it varies with different noisy samples y.
4.2 Condition augmentation II: Diffusion Dropout
Aside from changing the prior p(æT) to conditional prior Pw (x, ly) we also optimize the condition term (0( i |
Instead of designing specific condition-injecting strategies (4 9 11 or devising complicated network architecture (0 5), we attempt to "do subtraction" by discarding some shared intermeddate--eeeeene generated samples & condition factor) and important (target-related) information from intermediate -egeerrtedd samples. Naturally, if we discard some information from H;, then the diffusion enhancement model is forced to the condition factor y to recover the speech. Taking a further step, we can even discard the entire 2t, as the condition factor y alone is sufficient for recovering the clean speech 2n (this is what deterministic models do). To this end, define neural network fo (d( xc, p). y. t) to approximate p( x; D2, pp):
(a Ft; y) A N(x. (o p), E(xr,
(13)
where d(æ p) is the dropout operation:
d(zr,p) p)
{
xt
if r===1 = 1
r <~ Bermouli(1 - p).
(14)
if r = 0
4.3 DOSE training
Ho et al. [6] and much of the following work choose to parameterize the denoising model through directly predicting with neural network Eo (z,, t), which implicitly sets:
1 (2.- a,60)
(15)
In this case, the training loss is also usually defined as the mean squared error in the 6-space || - (x,, 1)l.. Although this standard specification works well for training an unconditional diffusion model, it is not suited for DOSE for two reasons.
5

---

Algorithm DOSE Training
Algorithm DOSE Sampling
1: choose
choose T1. T2, T2 < T1 S
2: repeat
2: Step 1: Generate Drgg
3: To ~ p(x)
3: Yn ~ NN U, (1 Ūr kr, IT)
4: f Uniform( (() (L...., T})
4: do = fe (vr, M
5: 6 <0,,)
5: Ørg ~ ( @o - y), 0ra
6: Take gradient descent step on
Step 2: Generate $2
7:
Vellzu - Ie( (d (Ze, p) y, t
7: do = fe (t y:
8:
until converged
return &o
First, we cannot estimate e without the help of 2; because 6 and y are independent. Second, as discussed earlier, want DOSE to start with small timestep and we strive make т small. However, approaches zero, small changes in x-space have an increasingly amplified effect on the implied prediction in e-space (Eq. (15). other words, the efforts made by diffusion enhancement models become so negligible that diffusion models lose their ability to calibrate the speech at small timesteps.
So, we need to ensure that the estimation of ên remains flexible as the timestep gets smaller. Considering the equivalently of the c-space loss ||c - c0 t)ll to weighted reconstruction loss in x-space oz ||200 2z (2r, 1). we can directly estimate the clean speech æo at each timestep t:
C = (m)
[| fo((( p). t)12]
(16)
4.4 DOSE inference
After training, the ideal scenario is that pa p0 12,, y) approximates p(æ;-1 [2 æo) precisely, enabling to generate clean speech using Eq. (6). However, when applied in practice, it is difficult to completely eliminate errors (both sample error and true error). If these errors are not effectively managed or corrected during the generation process, the quality of the generated samples may deteriorate, leading to artifacts, blurriness, etc [19 20). This issue is particularly pronounced when using diffusion models for fine-grain -reeggranedd conditional generation tasks, as diffusion models require large number of steps to generate samples, which will significantly reduce the consistency between the generated sample and condition factor (see $5.3 Figure c
The adaptive prior (Sec ( provides an opportunity to address the error accumulation issue. Specifi- cally, we can select a suitable T smaller than T. conditioned on an adaptive prior, and generate speech in fewer steps. We can extend Eq. b transforming the unconditional diffusion enhancement model into a conditional one:
Po. (xoly) = Pup (xx, (y) II po (0,-11<0 y) dz t=1
(17)
and the number of sampling steps reduced from T to T +.
Readers familiar with diffusion models may recall that the standard process repeatedly applies a "single-step"" denoising operation æ;-1 denoise( (z;()) that aims to convert a noisy sample at some timestep to a (slightly less) noisy sample at the previous timestep ( - 1. In fact, each application of the one-step denoiser consists of two steps: (1) estimation of the fully denoised sample æ from the current timestep t, and (2) computing a (properly weighted, according to the diffusion model) average between this estimated denoised sample and the noisy sample the previous timestep t 1. Thus, instead of performing the entire r-step diffusion process to denoise sample, it is also possible run denoise once and simply output the estimated sample in one shot (2.. Accordingly, Eq. (17) can be further rewritten as:
Po. v (woly) = P p (ax ly)po (I ( y)
(18)
We can even achieve DOSE without the condition optimizer f.) using conditional diffusion enhancement model instead. For example, we can generate clean speech via:
Po (o = po @ Vy., v)po @r. y) dy
(19)

---

where 71. (T2 < S T) are two pre-defined nyper-parameters. The motivation behind Eq. 19 is
that, once we have trained a neural network fo (o 2) that can accurately estimate 2o (Eq. (T)).
according to the theoretical analysis in Sec 4.1 we can first choose a suitable value for 71 to ensure relatively good approximation of zo:
To = fo : T.) = fe( 71)
(20)
In the second step, once we have obtained good condition factor, we can choose a smaller timestep 72 < Ti to get a better estimation of 2o than &h generated in the first step.
Summary. DOSE has three important benefits: (1) By dropping d; entirely, we make the condition factor y the "protagonist", automatically enhancing the consistency between the generated sample and the condition factor. (2) By training the model with this modified training objective, DOSE can perform well not only on Gaussian noise (I;, - To) but also on various types of non-Gaussian noise (y + zz)) (3) DOSE is efficient (2 steps), faster than existing diffusion enhancement models.
5
Experiments
We compare DOSE with prevailing diffusion enhancement methods and deterministic mapping- based enhancement methods $5.1 We conduct a counterfactual verification to understand the intrinsic mechanism of DOSE in $5.2 We show two visual cases of excessive suppression and error accumulation in $5.3 While providing a self-contained version of our main results, we note that we also have additional quantitative observations reported in the Appendices. Specifically, we compare DOSE with other baselines via subjective evaluation (Appendix(A.. A.2); We investigate the significance of the proposed adaptive prior and explain why we need to use mild version of the condition factor (Appendix (::3): We examine the effect of our new training objective and demonstrate the necessity of using it (Appendix AA.4;; We explain why use steps in speech generation (Appendix A We provide parameter sensitivity experiments (Appendix () We show plenty of visual cases of excessive suppression and error accumulation (Appendix A.7 and A.8) To help readers better understand our research, we include a discussion subsection in Appendix A.9 Specifically,
we: (1) Analyze the reasons behind the superior generalizability of diffusion enhancement models compared to deterministic mapping- happing-based models (from the robust training perspective); (2) Explain why we use 0.5 in the mild version of the condition factor; (3) Discuss the broader impacts of speech enhancement methods.
Dataset and baselines. Following previous works 4 81, we use the VoiceBank-DEMAA
dataset (22 23 for performance evaluations. To investigate the generalization ability of models, we use CHiME-4 [24] as another test dataset following 19]. i.c., the models are trained on VoiceBank- DEMAND and evaluated on CHiME-4. We compare our model with recent open-sourced diffusion enhancement models such as DiffuSE [4) CDiffuSE 9. SGMSE [11], SGMSE+ [5), and DR- DiffuSE g. Since the only difference between SGMSE+ and SGMSE is their network architecture, compare our model with just one of them.
Evaluation metrics. We use the following metrics to evaluate SE performance: the perceptual evaluation of speech quality (PESQ) [25], short-time objective intelligibility (STOI) (26). segmental signal-to-noise ratio (SSNR), the mean opinion score (MOS) prediction of the speech signal distortion (CSIG) 227. the MOS prediction of the intrusiveness of background noise (CBAK) (27 and the MOS prediction of the overall effect (COVL) (27). Besides these metrics, we also design two MOS metrics (MOS and Similarity MOS) for subjective evaluation.
Configurations. To ensure fair comparison, we keep the model architecture exactly the same as that of the DiffWave model 7 for all method F DiftWave takes 50 steps with the linearly spaced training noise schedule 33 E 1 x 10 0.035] 44. We train all methods for 300,000 iterations using 1 NVIDIA RTX 3090 GPU with batch size of 16 audios. We select the best values for and T2 according to the performance on a validation dataset, small subset (10%) extracted from the training data. More experiment settings can be found in Appendix A.
"Since the focus our work is on studying the capabilities diffusion dropout and adaptive prior for consistency enhancement, we use off aot-the-shelf architectures to avoid confounding our findings with model improvements. This decision (using DiffWave) rests on both its widely validated effectiveness and the minimal changes it required in the baseline experimental setup.
7

---

Table 1: Comparison of different diffusion enhancement methods.
Method
Year
Efficiency
Dataset
STO)
PESQU
CSIG
CBAK
COVL
Unprocessed
92.1
1.97
3.35
2.44
2.63
Dive
2021
step (dis)
93.3
2.51
3.72
3.27
3.11
DiffuSE
2021
steps
93.5 18:98
2.39
3.71
8:85
3.04
3.03 1.10
CDiffuSE
6 steps
VB
93. Lans
2.4319011
3.77
3.09
2.
3.09
SGMSE
2022
50 steps
93.3
2.34
0.31
3.69
2.90
-037
3.00 10
=1
-8
DR-DiffuSE
2023
6 steps
92.9 +0.06
0.00
2.50 +0.02
3.68 =0.02
3.27
3.08 +0.02
+0.02
DOSE
steps
93.6 +0.05
2.56 20.01
3.83
20.01
3.27 20.00
3. 19 +0.01
Unprocessed
71.5
1.21
2.18
1.97
1.62
DiflWave
2021
step (dis)
72.3
1.22
2.21
1.95
1.63
DiffuSR
2021
steps
43.740.05
+.591
2.91
1.070
2.19
2.19
2.
2019
10.50
CDiffuSE
2022
steps
CHIME-4
82.8 110.3
1.58 1.5819.01
10.35
2.88 0.87
2.15 0.30
2.18 2.18
76.85
SGMSE
2022
50 steps
84.5
10.05
1.57
2.92 2.92
0.001
-9-1
2.18 10.02
2.18 2.18
+0.85
10.01
DR-DiffuSE
2023
6 steps
77.6 47.0400.0
1.29 20.04
2.40 2.40
2.04
20.00
1.78
4.70
DOSE
steps
86.6 20.05
1.52 20.00
2.71 20.01
2.15 20.01
2.06
20.01
5 step:
35 step:
Figure 4: Counterfactual visualization. The first two columns are associated with dropout probability of 0. while the last two columns are associated with a dropout probability of 0.9. In each row, the blue waveforms in the first and third columns are the counterfactual samples, and the blue waveforms the second and fourth columns are the normal samples. The orange waveforms are generated samples from the model.
5.1 Performance comparison
We compare our method with previous diffusion enhancement methods and summarize our ex- perimental results in Table [ We observe that: (1) Diffusion enhancement methods have better generalizability than deterministic methods. (2) Methods with specific cundition-injectinn strategies, such as DiffuSE, CDiffuSE, and SGMSE, have strong generalization but perform slightly worse than deterministic mapping based methods in matched scenarios. (3) Method (DR- RDDiifffSEE) with auxiliary condition optimizer, performs better in matched scenarios shows a slight improvement in mismatched scenarios. (4) Our method performs well in both matched and mismatched scenarios and is on par with state-of-the-an diffusion enhancement models while requiring fewer steps.
5.2
Counterfactual verification
We perform counterfactual verification to gain insights into the underlying mechanism DOSE. To verify whether dropout can increase the "discourse power" of the conditional factor, we keep the condition factor y fixed and reverse the intermediatt--eenrrii speech at a specific step (reverse(x;)) This reversed ceermeddiategennerate generated speech is more similar to the condition factor than the counterfactual speech, we can conclude that the condition factor plays dominant role in the generation process. Otherwise, say that the condition factor is less influential.
speech is called counterfactual sample. Notably, if the final
As shown in Figure compare the performance of two models with different dropout probabilities (0.1 vs. 0.9). We have two findings here: (1) A higher dropout probability encourages the model to prioritize the condition factor with small timestep t. (2) When timestep is large, DOSE ef-

---

Figure Excessive suppression visualization (unconditional diffusion enhancement model on CHIME-4). From left to right: (1) DiffWave (dis); (2) adaptive prior with the estimated condition; (3)
adaptive prior with the mild condition; (4) clean speech.
Figure 6: Error accumulation visualization (VB, DOSE). From left right: (1) noisy speech; (2) full
(50) steps; (3) 2 steps; (4) clean.
fectively captures condition information, ensuring the model's robustness to noise and maintaining consistency the early stages of inference.
5.3 Excessive suppression & error accumulation
We provide a visual case of excessive suppression in Figure and a visual case error accumulation in Figure [6] From Figure 5 we can that: (1) The deterministic model fails in mismatched scenarios and generates samples that lose speech details; (2) The diffusion enhancement model generate defective speech when directly using the estimated speech the condition factor; (3) The diffusion enhancement model equipped with a mild version of the condition factor can recover clean speech effectively. From Figure notice that: (1) Full-step generation can remove noise and generate natural-sounding speech. However, it can't guarantee the consistency between the generated speech and condition factor; (2) Two-step speech generation with adaptive prior can promise consistency and high quality simultaneously.
Conclusions
In this work, we present a new approach DOSE that effectively incorporates condition information into diffusion models for speech enhancement. DOSE uses two efficient cundition-augmentee techniques to address the condition collapse problem. Comprehensive experiments on benchmark datasets demonstrate the efficiency and effectiveness of method.
In our method, there are two groups of param-errmeterss:: the dropout probability p for the dropout operation and two timesteps TI, for the adaptive prior. These parameters are critical to model performance. For example, if the dropout probability set too high, the diffusion enhancement model will rely solely on the condition factor to estimate the speech. Then our diffusion enhancement model will degenerate into a deterministic model, losing its generalizability. We also need to make a trade-off when choosing the timestep (especially ri): ) On one hand, a large is needed to reduce the gap between the clean speech and condition factor. On the other hand, the original semantic information will also be removed if is set too large.
In practice, is necessary to evaluate the model on subset of data and then empirically set the hyperparameters. These manually defined hyper-parameters are selected based on the Empirical Risk Minimization (ERM) principle and may not be optimal for every individual sample. Thus, an important direction for future research is to develop methods that can adaptively choose hyper- parameters for different samples. also expected that the model can adaptively select appropriate coefficients when forming mild version of the conditioning factor.

---

Acknowledgement
This work was supported in part by National Natural Science Foundation of China (Grant No.62176043 and No.62072077). Natural Science Foundation of Sichuan Province (Grant No. No.2022NSFSC0505) Kashgar Science and Technology Bureau (Grant No. No.K32023,25), and Na- tional Science Foundation SWIFT (Grant No.2030249).
References
[1] Steven L Gay and Jacob Benesty. Acoustic signal processing for telecommunication, volume
551. Springer Science & Business Media, 2012.
[2] Tim Van den Bogaent, Simon Doclo, Jan Wouters, and Mare Moonen. Speech enhancement with multichannel wiener filter techniques in multimicrophone binaural hearing aids. The Journal of the Acoustical Society of America, 125(():360-33) -710, 2009.
[3] Ivan Tashev. Recent advances in human-machine interfaces for gaming and entertainment. International journal of information technologies and security, 3(3):69-76, 2011.
[4] Yen-Ju Lu, Yu Tsao, and Shinji Watanabe. A study speech enhancement based diffusion probabilistic model. In 2021 Stia-Pacific Signal and Information Processing Association Anmual Summit and Conference (APSIPA ASC), pages 659-666. IEEF, 2021.
[5] Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhancement and dereverberation with diffusion-based generative models. arXiv preprint arXiv:2208.0588. 05830, 2022.
[6] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
[7] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
[8] Wenxin Tai, Fan Zhou, Goce Trajcevski, and Ting Zhong, Revisiting denoising diffusion probabilistic models for speech enhancement: Condition collapse, efficiency and refinement. In AAAI, 2023.
[9] Yen-Ju Lu, Zhong-Qin Wang, Shinji Watanabe, Alexander Richard, Cheng Yu, and Yu Tsao. Conditional diffusion probabilistic model for speech enhancement. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7402-7406. IEEE, 2022.
[10] Joan Serrà, Santiago Pascual, Jordi Pons, R Oguz Araz, and Davide Scaini. Universal speech enhancement with score-based diffusion. arXiv preprint arXiv:2206.0 03065, 2022.
[11] Simon Welker, Julius Richter, and Timo Gerkmann. Speech enhancement with score-based generative models in the complex STFT domain. In Proc. Interspeech 2022, pages 2928-222 29322,,,,
2022. doi: 00.21447/nterspeecch222222 202. 10653.
[12] Jianwei Zhang, Suren Jayasuriya, and Visar Berisha. Restoring degraded speech via a modified diffusion model. In 22nd Anmual Conference of the International Speech Communication Association, INTERSPEECH 2021, pages 2753-2757. International Speech Communication Association, 2021.
[13] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[14] Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
[15] Zongsheng Yue and Chen Change Loy. Difface: Blind face restoration with diffused error contraction. arXiv preprint 2rXiv:2212.06512, 2022.
10

---

[16] Andong Li, Chengshi Zheng, Cunhang Fan, Renhua Peng, and Xiaodong Li. A recursive network with dynamic attention for monaural speech enhancement. In Proc. Interspeech 2020. pages 1422-24266 2020. doi: 0021143//nnerrseechh20000 1513. URL htl.tt/p://xx doi org/
10. 10.21437/Interspeee 1tt.eerrppeecc.2222111111 2020-1513
[17] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
[18] Santiago López-Tapia and Nicolás Pérez de la Blanca. Fast and robust cascade model for multiple degradation single image super-resolution. IEEE Transactions on Image Processing, 30:4747-4759, 2021.
[19] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[20] Jean-Marie Lemercier, Julius Richter, Simon Welker, and Timo Gerkmann. Storm: diffusion- based stochastic regeneration model for speech enhancement and dereverberation. arXiv preprint arXiv:2212.1185// 2022.
[21] Nicholas Carlini, Florian Tramer, Zico Kolter, et al. (certified!!) adversarial robustness for free! In International Conference Learning Representations, 2023.
[22] Christophe Veaux, Junichi Yamagishi, and Simon King. The voice bank corpus: Design, collection and data analysis of a large regional accent speech database. In International Conference Oriental COCOSDA, pages 1-4. IEEE, 2013.
[23] Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. The diverse environments multi- channel acoustic noise database (demand): A database of multichannel environmental noise recordings. In Proceedings of Meetings on Acoustics, volume 19, page 035081. Acoustical Society of America, 2013.
[24] Emmanuel Vincent, Shinji Watanabe, Aditya Arie Nugraha. Jon Barker, and Ricard Marxer. An analysis of environment, microphone and data simulation mismatches in robust speech recognition. Computer Speech Language, 46:535-557 2017.
[25] Antony W Rix, John G Beerends, Michael Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In IEEE international conference on acoustics, speech, and signal processing (ICASSP), pages 749-752. IEEE, 2001.
[26] Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. A short-time objec- tive intelligibility measure for frequeryy weighted noisy speech. In IEEE international conference acoustics, speech, und signal processing (ICASSP), pages 4214-4217. IEEE, 2010.
[27] Yi Hu and Philipos C Loizou. Evaluation objective quality measures for speech enhancement. IEEE Transactions audio, speech, and language processing,
2007.
[28) Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Progressive fast diffusion model for high-quality text-to-speech. In Proceedings of the 30th ACM International Conference Multimedia, pages 2595-2605, 2022.
11

---

A
Appendix
In the supplemental material,
A.1:
We provide the proofs for Propositions and 2.
A.2:
We compare DOSE with other baselines via subjective evaluation.
We investigate the significance of the proposed adaptive prior and explain why we need to use a mild version of the condition factor.
A.3
A.4
We examine the effect of our new training objective and demonstrate the necessity of using it.
A.5
We explain why
two steps in speech generation.
A.6:
We provide parameter sensitivity experiments.
A.7
We present several visual cases excessive suppression.
A.8:
We present several visual cases of error accumulation.
We analyze the reasons behind the superior generalizability of diffusion enhancement models compared to deterministic mapping-bus models (from the robust training perspective). explain why use 0.5 in the mild version of the condition factor, and discuss the broader impacts of speech enhancement methods.
A.9
A.10:
We include more information about speech processing and basic architecture.
A.1
Mathematical proofs
We now present the proofs of the Propositions stated in the main text.
Proposition 1. (cf. $4.1): For any 0 such that 0 & M for some finite positive value M,
there exists positive value E {0, (0, .T'} [''' that satisfies:
DKL [x) SE. Vr St
(9)
where p(æ = (ax; Vocc -1-04) T).
Proof. Given two Gaussian distributions P, Q defined over vector space R2, the KL divergence of multivariate Gaussian distributions is defined as follows:
1 DEL (PIIQ) - Tr Ei) (u2 m1) 11 m1) d In
det Σg det Σ₁
(21)
Here, u1 E Fd and Ej ddd are the mean and covariance matrix of distribution P, and μ1 Rd and Eg E dd are the mcan and covariance matrix of distribution Q. d is the dimensionality of the vectors (i.c.. the number of dimensions in the vector space), and Tr denotes the trace operator.
Note that when the two Gaussian distributions have diagonal covariance matrices (i.e., when the different dimensions are independent), the above formula simplifies to the sum of the KL divergences of each univariate Gaussian distribution. Thus, given two Gaussian distributions p(æ; æ²), p(x; yy) and Eq. (5). the KL divergence between these two distributions can be calculated as follows:
DEL (p(=:(=))lp(<=ll))
O2 Cz.
1
In
(22)
of
According to the definition of the diffusion model and Figure 3 DEL L (p(a, (a: (p(,,ly)
monotonically decreasing function. Ideally, for bounded error n with (almost) infinite timestep T. we have:
lim DEL (p(m)p))][p<<>)) = +00, lim DEL L (p(x, ()) 0.
(23)
t0
t-f
According to Bolzano's theorem, there exists at least one point r in the interval (0...., . such that DeL (p(z, ( lv)) E. Then, Eq. (9) holds for t < T.
D
12

---

Testing audio
0.00 00:05
Insruction:
What's your rating / of voice
natureliee. hurannsouncling: recording? examing quality and ignore the differences of style simbre,emotion and prosody)
Select foryour favonte
Excellent Completely natural. speech
4.5
Moad-MMttyyy natural speech
3.5
Fair Equally natural and unnatural speech 3
Poor-Mostly unnatural speech-2
1.5
Bad Completely unnatural speech 1
You selected Excellent-Complett natural speech--
Figure 7: Screenshot of MOS test.
Proposition (cf. $4.1y: Let x be the clean sample, y be it's corresponding noisy one, and x' be any neighbor from the neighbor set S(x). Then diffusion enhancement models can recover a with high probability if the following inequality is satisfied:
p(æ)
log
-1 || -
1-yyl
Vol E S(x),
(10)
p(xx)
207
where of 1-āe is the variance of the Gaussian noise added at timestep in the forward diffusion process.
5.
Proof. The main idea to prove that any point x'' quite similar but different to the ground-true speech should have lower density than in the conditional distribution that the diffusion enhancement models can recover æ with a high probability. other words, we should have:
p(æa- a/ y.) > p(æo- x' l' y.)
(24)
According to Bayes' theorem, we have:
p(0o- = x|x; = y.) p(10 x. Z.t = yc)
(25)
p(x; = y1)
p(20 = x)) p(x: Ye ye x))
pl a/ y.)
Applying Eq. 25
to Eq. 24
we obtain:
-|x y11 p(x')
1
||
y1
p(x)-
exp
exp
(26)
v /2TT²)dd
207
(2
20
p(x)
+ log
(1|=
y11
||
y v1)).
Val E S(xr)
p(xx)
20?
and the proof now complete.
A.2 Subjective evaluation
We conduct two types of Mean Opinion Score (MOS) tests verify the quality of synthesized audio through human evaluation.
13

---

Reference audio
0:00 / 0:35
4
Testing audio
0:00 0:35
40
Insruction:
What's your favorite audio:
What's your rating of voice
How similer's is this recordingto to the reference audia?
choose one audio
Select an rating for your ferorite
Please focus on similarity of the style (specker imettin and prosody) to the reference, and ignore the audio quality
Reference Audio
Excellent-Comlletely natural speech 5
Tosting
4.5
You selected Reference Audio
Good Mostly natural speech
3.5
Fair-Euullly netural and
unnatural speech
2.5
Mostly unnetural speech
Comp-leellllyy unnatural
speech
You selected Excellent-Complettly
natural speeth
Figure 8: Screenshot of Similarity MOS test.
Table 2: MOS tests under different scenarios.
Method
Scenarios
MOSt
Similarity MOS
Scenarios
MOSt
Similarity MOS
Unprocessed
3.60
3.33
3.30
3.10
DiffWave (dis)
3.8010.2
3.75
2.10 +0.25
1.00
DiffuSE
3.40 0:30
4.17 1.87
3.00 3.00 +0:27
0:30
3.33
CDiffuSE
Matched
3.85 +0.25
4.12
Mismatched
2.55 2.55 +0.25
3.42
2..250:352
+0.29
SGMSE
3.65 in 88
3.95 10:27
2.83 10-35
3.41
10:25
DR-DiffuSE
3.80
3.84
2.48
1.45 10.33
12:35
DOSE
4.05 10.29
4.35
3.48
Naturalness. For audio quality evaluation, conduct the MOS (mean opinion score) tests and explicitly instruct the raters to "focus on examining the audio quality and naturalness, and ignore the differences of style (timbre, emotion, and prosody)". The testers present and rate the samples, and each tester is asked to evaluate the subjective naturalness on a 1-5 Likert scale.
Consistency. For audio consistency evaluation, we explicitly instruct the raters to "focus on the similarity of the speech (content, timbre, emotion, and prosody) to the reference, and ignore the differences of audio quality". This is slightly different from the original definition of SMOS for speech synthesis.
the SMOS (similarity mean opinion score) tests, we pair each synthesized utterance with ground truth utterance evaluate how well the synthesized speech matches that of the target speaker. The testers present and rate the samples, and each tester asked to evaluate the subjective consistency 1-5 Likert scale.
14

---

Unprocessed
Dis
Noiey
Estimated
Mild
Clean
1.50
1.40
1.34
L
0.02
0.07
1.11
3.22
0.01
30
T Timestep
Timestep
Figure 9: Performance the unconditional diffusion enhancement model with adaptive prior. From left to right: (I) each step loss matched VB; (2) each step loss on mismatched CHIME-4; (3) PESQ comparison for different priors on mismatched CHIME-4
Our subjective evaluation tests crowd-sourced and conducted by 15 volunteers. The screenshots of instructions for testers have been shown in Figure and Figure 8 We paid $10 to participants hourly and totally spent about $300 participant compensation.
The MOS results with the 95% confidence interval are shown in Table 2 we observe that: (1) Our method surpasses all baselines, demonstrating the strong ability of the proposed framework in synthesizing natural speech; (2) Our model can synthesize consistent speech the golden speech, which is aligned with motivation for algorithm design. It's also exciting to see that DOSE yields similar scores to methods with specific condition-injeet iittttiinnnneiinng strategies (i.c., DiffuSE, CDiffuSE, and SGMSE) on Similarity MOS.
A.3 Adaptive prior analysis
We now investigate the significance of the proposed adaptive prior ($4.1) and show why we need to mild version (Eq. (12) of the condition factor.
We design three variants to investigate the effect of different condition optimizer settings on denoising performance. These variants are: (a) applying adaptive prior with the noisy speech; (b) applying adaptive prior with the estimated one (from the deterministic model); (c) applying adaptive prior with the mild condition (Eq. (2)). To control variables,
unsupervised diffusion model with adaptive priors. We conduct experiments on the matched VB dataset and mismatched CHIME-4 dataset respectively, and our results shown in Figure
As shown in Figure P (left), we plot the one-step loss on matched VB and obtain the following observations: (1) The trend of loss curves is in line with our analysis in $4.1 that we need to find trade-off timestep for better performance. (2) Equipping the unconditional diffusion enhancement model with the adaptive prior technique has certain denoising ability but is inferior to its counterpart discriminative model in matched scenarios. We attribute the second phenomenon to the limited denoising capacity of the unconditional diffusion enhancement models (cf. 99 (1)..
Although the performance of the unconditional diffusion enhancement model equipped with adaptive prior is mediocre in matched scenarios, as illustrated in Figure |9 (mid), it exhibits greater stability than discriminative models in mismatched scenarios. To verify the influence of different priors, we compare their PESQ on mismatched CHIME-4, shown in Figure 9 (right). We see that: (1) The deterministic model fails mismatched scenarios and generates samples that are even worse than the unprocessed ones; (2) The diffusion enhancement model has strong generalizability and performs significantly better than the deterministic model; (3) The diffusion enhancement model loses its capability when using the estimated speech as the condition factor, and (4) Although the estimated speech is worse than the unprocessed one, the diffusion enhancement model equipped with a mild version of the condition factor achieves the best performance. This implies that the estimated speech can provide additional complementary guidance to the diffusion model, and the model can adaptively "separate the wheat from the chaff" Thus, using mild version of the condition factor is important and necessary.
In summary, our research shows the strong generalizability of the diffusion enhancement model. However, we also find that the unconditional diffusion enhancement model has mediocre performance. This suggests that relying solely the adaptive prior technique is not be sufficient, further emphasiz-
15

---

Loss cuive
Condition on
Uncondition ori
0.12
Uncondition new
Condition new
0.1
0.08
0.06
0.04
0.02
Step
20k
40k
60k
80k
100k
120k
Figure 11: Investigation of the training objective. we compare the e-space and z--pace loss curves.
ing the importance of the diffusion dropout operation training conditional diffusion enhancement models in a supervised manner.
A.4 Training objective investigation
We now examine the effect of our new training objective and demonstrate the necessity of using it.
Let's recall the sampling process of DOSE. In the first step, we generate a relatively good estimation of the clean speech. In the second step, we use DOSE with small timestep to generate better one. We plot the relationship between Ae and Axg Figure 10 Specifically, we fix the Aro as a constant and use Eq. (15) to calculate the corresponding Ae. This experiment aims to show how effort for calibration in x-space is equivalent to that in c-space. From Figure 10 we see that, as approaches zero, small changes in x-space amplify the implied prediction in e-space. There is nearly 100-fold difference between the values of Ae and Awo. This implies that the efforts of the diffusion enhancement model at small timesteps become negligible, causing diffusion models to lose their ability to recover satural-sounding speech from the defective speech estimated in the small timestep.
10
10!
10
20
Timestep t
Figure 10:
The relationship between
Aro = agg so (orange) and Ae jē El (blue).
We also plot the training loss curves in Figure 11 After substituting the training objective from space to x-space, we observe that the loss change becomes more significant. This demonstrates more effective participation of conditioning factors in diffusion training: training diffusion enhancement model with this new objective allows for easier and more effective exploitation of conditioning factors.
A.5 Complexity analysis & why use two-steps generation?
In this subsection, we explain why we use two steps to generate speech.
Why not one-step speech generation? According to $4.4 we can generate speech in one shot using a trained conditional diffusion enhancement model to reduce error accumulation [21]. This one-step speech generation provides an appcaling performance (Table 3). However, if consider the diffusion model training as a multi-task paradigm, the denoising task at a smaller timestep is easier than that at larger timestep [2 Correspondingly. the primary estimation error occurs the large timestep area and directly estimating clean speech at a large timestep will result in sub-optimal performance (28). Meanwhile, we can't choose small timestep t as it will lead to the mismatched problem discussed in $4.1
Since the conditional diffusion model can learn vital information from both the intermediate-genenne and noisy speech the estimated speech can provide complementary guidance to the diffusion model (Appendix A.3 allowing us to further improve the result by generating speech with multiple steps.
16

---

Table 3: Ablation study for two-step speech generation.
Method
Scenarios
PESQT
STOI(%)
Scenarios
PESQ1
STOI
Unprocessed
1.97
92.1
1.21
71.5
DOSE (fixed step)
2.4710.50
93.0
0.90
1.38 0.17
82.81113
DOSE (handpicked step) DOSE (fixed 2 steps)
Matched
2.50 10.3a
+0.01
93.4
0.05
1.30
Mismatched
1.51 T0.31
20:01
86.4 11.77
+0.06
2.48 +0.01
93.
-0.05
1.44
20 +0.01
83.6 +0.04
DOSE (handpicked steps)
2.56
1.52
86.6
Unprocessed
Noisy
Mild
Clean
Los5
Loss
0010
0.010
0.010
0.00
0.005
20
10
Timestep
Timestep
Timestep
Figure 12: Performance of the conditional diffusion enhancement model with different dropout probability in VB. From left to right: (1) without dropout; (2) dropout 50%; (3) dropout 90%.
Hyper-parammter selection. Basically, we need to set optimal hyper- parameters by evaluating the performance with small batch of data. Suppose we choose the number of sampling steps as K (K < T), and the amount of test data as N, then the computational complexity of the grid search O (NT!/(T - K)!). Since always set as a large number in the diffusion model's setup, the complexity introduced by choosing large K is often unmanageable.
Trivial solution. We have simple alternative solution: defining the hyper- paree-parameterss empirically (e.g., equal intervals (7). We present speech quality comparisons between empirically defined (fixed) and handpicked pyper-parameters Table 3 As shown, the conditional diffusion enhancement model with handpicked hyper-parameters performs better than that with empirically defined hyper- parameters.
Although fixed hapery-yraamn parameters have inferior performance compared to handpicked ones, they still show appealing performance compared prevailing diffusion enhancement baselines. Therefore, in situations where we can't evaluate the model in advance, we can use empirically defined hyper- parameters instead.
A.6 Parameter sensitivity
There are two groups of typer-parameters that are critical to model performance: the dropout probability for model training and two timesteps TI, for model inference. In this subsection, we conduct parameter sensitivity experiments to investigate how these syper-parameterd affect the model's performance.
Dropout probability p. We vary the dropout probability p, setting it to (0.0, 0.5, 1.0). and plot the one-step loss on matched VB (Figure 12 and mismatched CHIME-4 (Figure [3] respectively. Please note that the figure with p has been excluded since it degenerates mapping-basee model and the performance remains unchanged across all timesteps.
deterministic
that: (1) The proposed conditional diffusion enhancement model works. Compared to the deterministic mapping-basee model, our model performs slightly better matched scenarios and significantly better mismatched scenarios; (2) When intermediate-genenne 2; is unreliable (large t), the condition factor plays dominant role. (3) As the dropout probability increases, the model focuses small, the loss curve oscillates when gets large.
From Figure 12
speech
the condition factor, while when the dropout probability is
From Figure 13, We find that: when using higher dropout probability (such as p 0.5 and 0.9). the model can generally achieve better generalizability. Note that if the dropout probability is set too high, the diffusion enhancement model will rely solely on the condition factor to estimate the clean
17

---

Unprocessed
Dis
Noisy
Esti
Mild
Clean
0,020
0.020
0.015
0.015
0.015
Loss
0.10
0.005
0.005
0.005
Timestep
Timestep
Timestep
Figure 13: Performance of the conditional diffusion enhancement model with different dropout
probability in CHIME-4, From left to right: (1) without dropout; (2) dropout 50%; ; (3) dropout 90%.
2.56
1.50
0.85
2.50
1.40
0.80
2.45
0.932
1.30
1.20
0.75
2,10
0.930
1.10
0.70
2.35
30
Figure 14: Performance of two-step speech generation. From left right: (1) PESQ on matched VB dataset; (2) STOI on matched VB dataset; (3) PESQ on mismatched CHIME-4 dataset; (4) STOI on mismatched CHIME-4 dataset.
speech and the diffusion enhancement model will degenerate into deterministic model, losing its generalizability.
Timesteps and 72. Considering the computational complexity of the "one-step" grid search, search optimal hyperparameters with slightly larger step. Specifically, we select both optimal Ti and 72 from the predefined set {1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50}. We show the PESQ and STOI performance different combinations in Figure 14 We can observe that 71 > T2 (excluding STOI on matched VB dataset, and the difference is negligible: 0.930 0.934) will lead to better performance, which is in line with our analysis ($4.4).
18

---

A.7 Visual case of excessive suppression
We present several visual cases of excessive suppression.
Figure 15: Excessive suppression visualization (CHIME-4, DOSE). From left right: (1) estimated
condition; (2) 2 steps; (3) clean; (4) noisy speech.
19

---

A.8 Visual case of error accumulation
We present several visual cases of error accumulation.
Figure 16: Error accumulation visualization (VB, DOSE). From left to right: (1) full (50) steps; (2)
steps; (3) clean; (4) noisy speech.
20

---

A.9 Discussion
To help readers better understand our approach, we analyze the reasons behind the better generaliz- ability of diffusion enhancement models compared to deterministic mapping- based models (from the robust training perspective), explain why we use 0.5 in the mild version of the condition factor, and discuss the broader impacts of speech enhancement methods.
A.9.1 Generalization analysis
Given that full-step diffusion enhancement models have better generalizability than deterministic models, delve into the following question: if we just using diffusion models one-step (or two-step) denoisers, what accounts for their enhanced performance compared to deterministic mapping-basee models?
To answer this question, we need to point out that training a diffusion model can considered equivalent to training multi-task paradigm. This involves training model with shared parameters on multiple levels of Gaussian noise concurrently. Recent research [21] has demonstrated that the full training process of diffusion models leads to significantly improved one-shot denoising capabilities, which are more generalizable compared to previous works that trained standalone denoisers on a single noise level. Please refer to 21 ($5.2) for more details.
A.9.2 Why use 0.5 in the mild version of the condition factor? (reviewer boQC)
Employing an equal weight provides stability, yet the performance during instances with low SNR would be compromised (albeit still superior to direct utilization of y condition factor, unless the condition optimizer falters). One prospective solution involves introducing an additional adaptive strategy, i.e.,
(y) (1 ayy. We can design an adaptive alpha predictor and hope can output a based on the quality of raw condition and samples from the condition optimizer. For example, when the condition optimizer produces lower-quality samples, giving more weight the original condition factor would make sense. Conversely, if the raw condition factor has low SNR, emphasizing the generated counterpart could be more effective. However, implementing this idea practically is intricate.
Given our strong reliance on diffusion-enhancee models enhance generalization, any new adaptive strategy must be generalizable. For instance, training an adaptive alpha predictor on the (seen) VB-dataset (high SNR & consistent condition optimizer performance) could lead the model to consistently output higher a values for fusion. Unfortunately, this auxiliary model might not effectively adapt to variations when evaluating the mismatched (unseen) CHiME-4 dataset (low SNR & potential condition optimizer challenges). To this end, we might need other techniques such as data augmentation and adversarial training to improve generalizability and robustness. This creates a dilemma: harnessing the speech diffusion model for overarching speech noise reduction generalization while simultaneously necessitating a pre-established generalized model facilitate its implementation. So far, despite our efforts to train strong alpha predictor, progress has been limited (the alpha predictor is still not generalizable, and the new system has no significant performance improvement over DOSE).
A.9.3 Broader impacts
As speech enhancement technology continues to advance and become more prevalent, it's important consider its broader impacts.
Positive impacts. The impact of speech enhancement technology on real-life situations, particularly for individuals with hearing impairments, cannot be overstated. Hcaring aids have long been the primary solution for those with hearing loss, but they are not always effective in noisy environments or for certain types of hearing loss. Speech enhancement technology can greatly improve speech intelligibility and communication for hearing aid users. For example, some hearing aids have Al- Al-powered speech enhancement that boosts speech quality.
In addition to the benefits for individuals with hearing impairments, speech enhancement technol- ogy also has significant implications for various applications. In transportation, clearer and more intelligible speech can improve communication between pilots and air traffic control, leading to safer and more efficient air travel. In industry, speech enhancement can improve communication
21

---

noisy factory floors, leading to increased productivity and safety. In educational settings, speech enhancement can improve student comprehension and engagement during lectures and presentations.
Negative impacts. While speech enhancement technology has the potential to greatly improve communication and speech intelligibiliiy, one potential concern is that speech enhancement could modify the semantic content of speech, potentially mislcading listeners. Thus, it's important for developers of speech enhancement technology consider this potential negative effect and work towards creating trustworthy systems.
A.10 Experimental details
Speech preprocessing. We process the speech waveform 16 kHz sampling rate. To maintain dimensionality consistency within mini-batches, we pad each utterance to 2 seconds (32000 points) using zero-padding technique.
Basic architecture. make a fair comparison, we use DiffWave 7 as the basic architecture following 9 the only difference being the change in the way of condition-injecting since most speech enhancement methods will directly use noisy speech as the condition factor, rather than -ee-spectrogram. We concatenate the condition factor with the intermediate seneeedditttggerrteeed sample along the channel dimension as the model's s input. Specifically, the network is composed of 30 residual bidirectional dilated convolution (Bi-DilConv) with
layers with residual channels 128. We kernel size in each layer. We sum the skip connections from all residual layers. The total number of trainable parameters is 2.31M, slightly smaller than naive Dave (2.64M). Please refer to ( and our code for more details.
22


Reviews:
[{"review_id": "P5Z9lncWim", "reviewer": "Reviewer_Zy9S", "summary": "The authors propose a model-agnostic method called DOSE that employs two efficient condition-augmentation techniques to incorporate condition information into DDPMs for SE. Experiments demonstrate that the approach yields substantial improvements in high-quality and stable speech generation.", "questions": "If there is a training-inference mismatch? You randomly drop x_t in training, I wonder if it causes the mismatch as the dropout usually does.", "limitations": "/", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. In-depth presentation on diffusion SE, including formulation and methodology.\n2. Good results. The authors compare different SE baselines and demonstrate the SOTA results.", "weaknesses": "1. It seems that the authors adopt the adaptive prior. Does it only use in the inference process? What is the difference from the adaptive prior in PriorGrad? \n2. Why use similarity MOS to evaluate the enhancement model, and what's the difference from MOS?\n3. How do you choose p for diffusion dropout? It lacks evaluation and ablation studies on p, which is an important parameter for the proposed diffusion dropout operation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Below we address specific questions.\n ***   \n> Q1: It seems that the authors adopt the adaptive prior. Does it only use in the inference process? What is the difference from the adaptive prior in PriorGrad?\n    \nYes, the adaptive prior is exclusively used in the inference process. Our method differs from PriorGrad [1] in three key aspects. \n\n* PriorGrad injects instance-level prior knowledge at the initial timestep $T$ and requires modifications to the training process (line 3-5 of Algorithm 1 in [1]). In contrast, our adaptive prior is independent of the training process, allowing our approach to be directly applied to arbitrary pre-trained diffusion enhancement models. \n\n* PriorGrad is designed to speed up the training convergence. Our adaptive prior is used to provide condition knowledge (it can also accelerate inference speed). PriorGrad requires a complete inference process, whereas our approach starts from the intermediate timestep, shortening the sampling trajectory, and thereby improving inference efficiency (i.g., generating clean speech in 2 steps).\n\n* PriorGrad is sensitive to the prior selection -- they have tried several sources of conditional information to compute the prior, but only the normalized frame-level energy of the mel-spectrogram worked (cf. Sec 4.1 in [1]). This means that it is hard for developers to choose an appropriate adaptive prior. In contrast, our adaptive prior, computed directly from noisy speech, is stable and effective.\n\nWe appreciate your question and hope this explanation addresses your concerns.\n***\n> Q2: Why use similarity MOS to evaluate the enhancement model, and what's the difference from MOS?\n\nAs the primary focus of our work is to tackle the condition collapse problem in diffusion enhancement models, it is crucial to assess the consistency of the generated speech with real speech. While MOS is commonly employed to rate the overall naturalness and fluency of synthesized audio in speech synthesis, we introduce another metric, called similarity MOS, which specifically evaluates the consistency (content, timbre, emotion, and prosody) between the generated speech and the real speech. We provided details about subjective human evaluation in Appendix A.3 (line 486-496).\n***\n> Q3: How do you choose $p$ for diffusion dropout? It lacks evaluation and ablation studies on $p$, which is an important parameter for the proposed diffusion dropout operation.\n\nSimilar to the process of selecting $\\tau_1$ and $\\tau_2$, we determine the optimal values for $p$ by evaluating the performance on a validation dataset. In this study, we pre-defined a relatively coarse candidate set $\\{0, 0.1, 0.5, 0.9\\}$ for $p$ and found that $p=0.5$ generated appealing results. We have included supplementary ablation studies for the hyper-parameter $p$ in the above table (in General Response). Note that we presented parameter sensitivity experiments in Appendix A.7, and the influence of $p$ (line 589-604) was shown in Figure 11 and Figure 12. We will mention this explicitly in the main text.\n***\n> Q4: If there is a training-inference mismatch? You randomly drop $x_t$ in training, I wonder if it causes the mismatch as the dropout usually does.\n\nGreat question! It is essential to clarify that our method is different from the conventional dropout technique typically applied to neural networks. In conventional dropout, random units (neurons) or their activations are dropped out during training to prevent overfitting and improve generalization. In contrast, our approach focuses on randomly dropping out $x_t$ (input features) to mitigate the condition collapse problem in diffusion enhancement models. This strategy would force the model to generate condition-consistent samples. While dropout might introduce a slight training-inference mismatch, we carefully validate our model's performance on benchmark datasets and ensure that it does not significantly affect the quality of the generated samples. We will include a more detailed explanation of the dropout technique and its implications in our revision. \n    "}, {"review_id": "Sx1dRSBXlV", "reviewer": "Reviewer_CtVJ", "summary": "This paper proposes a novel model-agnostic approach called DOSE for speech enhancement (SE) using denoising diffusion probabilistic models (DDPMs). In this paper, the authors focus on addressing the challenge of incorporating condition information into DDPMs with two efficient condition-augmentation techniques. Based on the experimental results, the authors claim that the proposed method obtain significant improvements in high-quality and stable speech generation, consistency with the condition factor, and efficiency. ", "questions": "Can authors conduct the ablation study to present and analyze the effectiveness of DOSE. ", "limitations": "The authors do not analyze the limitation of this paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method shows good generalization ability with good performance in both matched and mismatched scenarios. \n\n2. This paper shows detailed experimental results and a comprehensive comparison with existing diffusion enhancement methods and deterministic mapping-based method enhancement methods. \n\n3. This paper provides a proper introduction to the problem of condition collapse in generative speech enhancement. \n\n4. This paper is well-written and the flow of the writing is natural so it was easy to read and follow.\n", "weaknesses": "1. To replicate the experiments, more training details and configuration should be provided.  \n\n2. It would be better if there were some qualitative analysis in the experiment section.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Below we address specific questions.\n***    \n>Q1: To replicate the experiments, more training details and configuration should be provided.\n    \nWe reported our configurations in Sec 5, line 278-283. We added more experimental details including speech processing, basic architecture, and baseline description in Appendix A.12. As Reviewer k89b suggested, we will update the dataset details in the revision. Note that we also provided the code of DOSE in both supplementary material and anonymous GitHub (Appendix A.1, line 451-452) for replication.\n***\n>Q2: It would be better if there were some qualitative analysis in the experiment section.\n    \nFully appreciating the question, we'd like to note that there are multiple qualitative results discussed in the Appendix (which we'd be happy to refer to more explicitly in the revision). Specifically:\n\n* We showed visual cases of excessive suppression in A.8;\n* We presented visual cases of error accumulation in A.9;\n* We conducted a counterfactual verification to understand the intrinsic mechanism of DOSE in A.10. \n\n***\n>Q3: Can authors conduct the ablation study to present and analyze the effectiveness of DOSE.\n    \nThanks for the suggestion! We have conducted ablation studies to quantitatively show the significance of adaptive prior and dropout operation. The results are shown in General Response. We can observe that both of them are crucial for generating consistent samples. We also investigated the significance of adaptive prior and dropout (from other perspectives than metric scores) in Appendix A.6, A.7, and A.10.\n***\n>Q4: The authors do not analyze the limitation of this paper.\n\nAs Reviewer k89b pointed out, we had discussions about limitations in Sec 6 (line 325-338). We also discussed the border impacts in Appendix A.13 (line 675-697). We'd be happy to dedicate a (sub)section for discussing the limitations and the societal impact in the main text."}, {"review_id": "QwN3GqF8eC", "reviewer": "Reviewer_boQC", "summary": "This paper focuses on a new approach in the field of speech enhancement called DOSE, which effectively addresses the problem of conditional collapse by incorporating conditional information into a diffusion enhancement model.DOSE employs two effective conditional enhancement techniques that can significantly improve the performance of the model while ensuring its efficiency. The paper demonstrates the efficiency and effectiveness of the method through comprehensive experiments on a benchmark dataset.", "questions": "My detailed questions are as described above.", "limitations": "There are limitations to its use in real-time scenarios.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. In this paper, the authors propose an Adaptive Prior, aimed at incorporating conditioning information during the generation process, thereby ensuring greater consistency in the generated samples and augmenting the efficacy of speech enhancement.\n\n2. The paper elucidates that by employing dropout operations during the training phase, the model is compelled to prioritize conditioning elements, which efficaciously mitigates the conditioning collapse issue. This methodology engenders a dependency on conditioning information within the model during generation, culminating in the synthesis of more coherent speech.\n\n3. The authors undertake a comparative analysis between DOSE and extant diffusion-based speech enhancement models on two public datasets. Notably, DOSE attains superior performance with an exceedingly limited number of sampling steps, which substantiates the efficacy of the proposed method.", "weaknesses": "This work realizes a two-step sampling process. The designs in sampling process include:\n\n1. The parameter $T_1$ denotes an intersection point, enabling a shallow reverse process by leveraging the noisy speech y. Similar mechanism can be visited in text-to-speech synthesis, DiffSinger (AAAI, 2022) and image editing, SDEdit (ICLR 2022).\n2. The coarse estimation $\\hat{x}_{0}$ denoised from $T_1$ is first mixed with y as adaptive prior,  and then corrupted with a shallow forward process, which generates the latent representation at $T_2$.\n3. The high-quality estimation $\\hat{x}_{0}$ can recovered from $T_2$ in one-step.\n\nTwo designs in training include:\n\n1. The estimation target in training objective is set as clean waveform instead of noise.\n2. The $x_t$ is randomly dropped out to force the model to rely on the conditioning information $y$.\n\nQuestions:\n\n1. I can understand the dropout operation in training is helpful to utilizing the conditioner y. However, the adaptive prior is used to obtain the latent representation at T_2. I think it is manipulating the sampling trajectory. What is the relationship with condition optimizer? The unchanged noisy observation y has been provided as the condition. The abstract claims two condition-augmentation techniques.\n2. I do not understand the comparison study of adaptive prior analysis very well. What are the three variants when computing the adaptive prior \\hat{x}_{0}? Why is an unconditional diffusion model used? To compare the design of Eq. 12, the conditional generation should be fixed.\n3. This work mentions the error accumulation of diffusion models. However, the high-quality generation of diffusion models is usually guaranteed by its iterative refinement mechanism. In this work, 50 time steps are used in training, while 2 steps are used in sampling. I am curious about the results of increasing the number of sampling steps.\n\nExperiments:\n\n1. Subjective tests MOS and SMOS are conducted. But where is the demo page showing the generated samples?\n2. DiffWave does not claim one-step mapping for either waveform generation or denoising. Is it good to use it as a discriminative model?\n3. Generative baseline models such as DiffuSE, CDiffuSE, and SGMSE have been changed to keep the uniform model architecture and training method with this work. Will this cause performance decrease in their methods? What are the results of those unchanged baseline models?\n\nOthers:\n\n1. The thesis writing looks over-complicated. Moreover, the adaptive prior does not mean condition optimizer from my perspective. It is manipulating the sampling trajectory with observation y.\n2. I suggest showing the training and sampling algorithms in the main content instead of in appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Below we address specific questions.\n***   \n>Q1: I think adaptive prior is manipulating the sampling trajectory. What is the relationship with condition optimizer? The unchanged noisy observation $y$ has been provided as the condition. The abstract claims two condition-augmentation techniques.\n    \nYes, you are right, our adaptive prior is designed to explicitly incorporate condition knowledge by manipulating the sampling trajectory. Ideally, we can directly use the noisy speech $y$ to generate the adaptive prior like SDEdit [2]. However, when in low-SNR scenarios (speech signal is contaminated by noises severely), we have to choose a relatively large $t$ to guarantee an acceptable error bound (Proposition 1). According to Proposition 1 and 2, if we can narrow the gap between the condition factor and input $x$, we can opt for a smaller (better) $t$. This enables the model to prevent excessive removal of original semantic information in the condition factor (line 168-172). To this end, we can employ a condition optimizer to generate an enhanced adaptive prior (Eq. 12). And the second condition-augmentation technique involves employing a condition optimizer to generate an adaptive prior and explicitly injecting the condition knowledge at the inference stage. On the whole, it is a condition augmentation technique tailored for diffusion enhancement models,  aiming at alleviating the condition collapse problem. We appreciate your question and hope this explanation addresses your concerns. \n***\n>Q2: What are the three variants when computing the adaptive prior $\\hat{x}_0$? Why is an unconditional diffusion model used? To compare the design of Eq. 12, the conditional generation should be fixed.\n    \nWe explored using unconditional diffusion model with adaptive prior technique as a first attempt to address the condition collapse problem of conditional diffusion enhancement methods (Sec 4.1). We found it essential to consider failure cases of the condition optimizer, particularly in mismatched scenarios -- using the estimated speech directly from the condition optimizer (Eq. 11, similar to DifFace [3] and DiffSinger [4]) could lead to excessive suppression problem. To investigate this further, we defined three variants (cf. Appendix A.4, lines 504-506): \n* Applying the adaptive prior with the noisy speech (similar to SDEdit [2]); \n* Applying the adaptive prior with the estimated speech (similar to DifFace and DiffSinger, Eq. 11); \n* Applying the adaptive prior with a milder one (Eq. 12). \n\nIn our experiments (reported in Appendix A.4), we discovered that the mild condition is more stable in complex scenarios, while the unconditional diffusion model showed limited effectiveness in matched scenarios. Both of these insights were very valuable and used for designing DOSE (cf. Sec 4.2).\n***\n>Q3: This work mentions the error accumulation of diffusion models. However, the high-quality generation of diffusion models is usually guaranteed by its iterative refinement mechanism. In this work, 50 time steps are used in training, while 2 steps are used in sampling. I am curious about the results of increasing the number of sampling steps.\n    \nGreat question! We would like to emphasize that the definition of high-quality in image/speech synthesis is not consistent with the one in speech enhancement: the former focus on naturalness, while the latter focus on point-to-point consistency. We presented multiple visual cases in Fig. 5 and Fig. 15 which illustrate that increasing the number of sampling steps will lead to inconsistency (error accumulation) problem and subpar results. Additional experimental results are shown in the above table (in General Response). Note that several recent works [5, 6, 7] share the same findings as ours that the consistency gets worse when starting with a large $t$.\n***    \n>Q4: Subjective tests MOS and SMOS are conducted. But where is the demo page showing the generated samples?\n    \nWe have made the test page public. Due to the NIPS policy that \"rebuttal should not contain any links to external pages\", we have send an anonymized link to the AC in a separate comment.\n***\n>Q5: DiffWave does not claim one-step mapping for either waveform generation or denoising. Is it good to use it as a discriminative model?\n    \nWe use DiffWave as a discriminative model due to the following three reasons. \n\n* As stated in DiffWave [8], their network architecture is based on WaveNet [9], a speech synthesis model that has been successfully applied to **speech enhancement** [10] and separation [11];\n\n* Pioneer works [5, 12] in DDPM-based speech enhancement are all based on DiffWave. To ensure a fair comparison, we need to keep the model architecture exactly the same;\n\n* There exists a concurrent work [13] that does a similar thing to ours, i.e., they use NCSN++ [14], another generative model as the basic architecture.\n***\n>Q6: Generative baseline models such as DiffuSE, CDiffuSE, and SGMSE have been changed to keep the uniform model architecture and training method with this work. Will this cause performance decrease in their methods? What are the results of those unchanged baseline models?\n    \nAs we explained in Appendix A.12 (line 641-644), current SOTA speech enhancement methods directly use noisy speech as the condition factor, rather than Mel-spectrogram. We note that this slight modification also leads to an improved performance of these methods (please see the respective reported performance for more details).\n***\n>Q7: There are limitations to its use in real-time scenarios.\n    \nWe would like to emphasize that one of the attractive properties of our method is that speech can be generated in only 2 steps. We believe our method can shed light on the design of future fast diffusion enhancement models.\n***\n>Q8: Writing and paper organization advice.\n    \nThanks for the suggestion! We will reorganize the paper in the revision."}, {"review_id": "MDhwjcn8E4", "reviewer": "Reviewer_k89b", "summary": "This paper presents a solution to the problem of condition-collapse in denoising diffusion models for speech enhancement by introducing the adaptive prior and sample dropout techniques. The paper is well-written and provides valuable insights into the functioning of the denoising diffusion probabilistic model for speech enhancement. While the concept of adaptive priors for conditioning the generative process is not entirely novel and has been explored in vision-related tasks, the authors' application of this technique, along with theoretical analysis, is intriguing. The authors also justify their choices regarding noise scheduling and propose a faster method for sampling clean speech from the trained model based on intermediate approximation.\n\nThe experimental strategy adopted in this study assesses the generalization capability of the proposed model using objective metrics such as STOI and PESQ, as well as subjective scores like CBAK and COVL. The effectiveness of the mixed conditioning strategy is demonstrated through the analysis of spectrogram plots, which is an interesting observation. It is important to note that while the proposed technique may not consistently outperform baseline methods across all scenarios, it does excel in specific matched scenarios.", "questions": "None", "limitations": "The conclusion section mentions that the model is sensitive to the choice of dropout probability and the sampling time-indices. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In this paper, a novel approach is introduced to address the problem of condition collapse in diffusion models for speech enhancement. The authors propose the utilization of adaptive prior and sample dropout techniques, which offer an interesting and promising solution to this issue. Furthermore, the paper delves into the theoretical aspects of clean speech recovery, shedding light on the conditions and constraints necessary for successful restoration.\n\nOne notable contribution of this work is the development of a fast sampling technique, which not only proves effective in the context of speech enhancement but also holds potential for application in other conditional generation tasks. This aspect highlights the broader implications and versatility of the proposed approach.\n\nTo evaluate the efficacy of the proposed technique, the authors conduct comprehensive experiments and compare their approach against various diffusion-based models specifically designed for speech enhancement. The experiments are meticulously designed and executed, providing a thorough analysis of the results. This level of detail and scrutiny enhances the credibility of the proposed approach and contributes to a better understanding of its strengths and limitations.", "weaknesses": "One main weakness in my opinion is the understanding of Proposition 2. I do not understand how a diffusion model has high probability of recovering ground-truth if the inequality 23 from appendix holds. I might be missing some theoretical analysis on diffusion models but I am giving the benefit of doubt to the authors. \n\nThe dataset section lacks the details about the type and level of noise present in Chime and Voicebank corpora. Another issue is in the experiment section where the authors have shown impressive performance on a wide-range of metrics. I believe that WER can be easily calculated in the evaluation and is a very straight-forward way to compare the noise-reduction performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Below we address specific questions.\n***    \n>Q1: One main weakness in my opinion is the understanding of Proposition 2. I do not understand how a diffusion model has high probability of recovering ground-truth if the inequality 23 from appendix holds. I might be missing some theoretical analysis on diffusion models but I am giving the benefit of doubt to the authors.\n    \nUnlike directly maximizing the condition probability $p(x_0 = x|x_t = y_t)$ or the difference between the target speech and candidates $||p(x_0 = x|x_t = y_t) - \\max (p(x_0 = x^{\\prime}|x_t = y_t); x^{\\prime} \\in \\mathcal{S}(x))||_2^2$, Proposition 2 (Eq. 23) presents a relatively relaxed constraint. If Proposition 2 holds, considering the characteristics of unconditional diffusion models / score-based models [14, 15], particles starting at the adaptive prior $y_t$  are more likely to converge to the ground-truth objective $x$ through an iterative MCMC procedure (known as Langevin dynamics), rather than other natural but inconsistent candidates $\\forall x^{\\prime} \\in \\mathcal{S}(x)$. This constraint also suggests that we should select a smaller $t$ and narrow the gap between the condition factor and $x_t$, which provides guidance for the subsequent DOSE design. We appreciate your question and hope this explanation addresses your concerns.\n***\n>Q2: The dataset section lacks the details about the type and level of noise present in Chime and Voicebank corpora. Another issue is in the experiment section where the authors have shown impressive performance on a wide-range of metrics. I believe that WER can be easily calculated in the evaluation and is a very straight-forward way to compare the noise-reduction performance. \n    \nThanks for the suggestion! The VoiceBank-DEMAND dataset is a classical benchmark dataset for speech enhancement using clean speech from the VCTK corpus. The training utterances are artificially contaminated with eight real-recorded noise samples from the DEMAND database and two artificially generated noise samples (babble and speech shaped) at 0, 5, 10, and 15 dB SNR levels, amounting to 11,572 utterances. The testing utterances are mixed with different noise samples at 2.5, 7.5, 12.5, and 17.5 dB SNR levels, amounting to 824 utterances in total. The CHiME-4 simulated test data is created based on real-recorded noises from four real-world environments (including street, pedestrian areas, cafeteria and bus) based on four speakers, with a total of 1320 utterances. Following [12], we use the signals from the fifth microphone for evaluation. We will update the dataset details in the revision.\n\nWe have evaluated all speech enhancement methods using two public pre-trained ASR models (CRDNN-RNNLM and Conformer-Transducer) from huggingface. The result is shown in the table below. \n\n|  |  |  | VoiceBank |  |  |  |  |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---: |\n| Model | DOSE | DiffuSE | CDiffuSE | SGMSE | DR-DiffuSE | DiffWave(dis) |\n| CRDNN-RNNLM | **12.77%** | 14.28% | 12.97% | 14.81% | 13.01% | 14.31% |\n| Conformer-Transducer | **9.83%** | 10.83% | 9.96% | 11.67% | 9.89% | 10.90% |\n\n|  |  |  | CHIME-4 |  |  |  |  |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---: |\n| Model | DOSE | DiffuSE | CDiffuSE | SGMSE | DR-DiffuSE | DiffWave(dis) |\n| CRDNN-RNNLM | 39.66% | 38.10% | 37.59% | **37.17%** | 44.51% | 71.92% |\n|  Conformer-Transducer | 30.30% | 28.44% | **28.41%** | 28.62% | 31.07% | 59.76% |\n\n\nWe have the following observations: \n\n* On the VoiceBank-DEMAND dataset (matched scenario), the performance gap between diffusion enhancement models and deterministic model is not prominent. For instance, WER ranges from 0.127 to 0.148 with CRDNN and from 0.098 to 0.116 with Conformer-Transducer. \n\n* On the CHIME-4 dataset (mismatched scenario), diffusion enhancement models significantly outperform deterministic models in terms of performance, e.g., WER of diffusion enhancement models ranges from 0.371 to 0.445 with CRDNN and from 0.284 to 0.310 with Conformer-Transducer, while WER of the deterministic model is  0.719 with CRDNN and 0.597 with Conformer-Transducer. \n\n* We find our method has no significant differences with DiffuSE, CDiffuSE, and SGMSE on WER evaluation. "}, {"review_id": "24PzXKvdzZ", "reviewer": "Reviewer_RTAK", "summary": "This paper describes a new method for providing noisy-signal conditioning information (y) to the diffusion steps of a diffusion-based speech enhancement algorithm.  Three specific innovations are proposed: (1) improve dependence of x_0 on y by dropping out x_t, at random with Bernoulli probability p.  (2) In order to make innovation #3 possible, train each x_t explicitly using MSE of the implied x_0, rather than MSE of the error \\epsilon.  (3) For greater efficiency, generate x_0 in only two steps, selected from the T-step trained diffusion process using validation data.", "questions": "Significance:  Dropout enhances accuracy.  The two-step inference enhances efficiency.  Is there any interaction between these two things?  It seems like the dropout might reduce the accuracy degradation that two-step inference would otherwise incur; is that true?  Similarly, why two -- is two the optimum number of steps in any way?  Results show pretty clearly that two is better than one, but is there any theoretical reason for that?  The theory seems to predict simply that the more steps you have, the more accurate is the inference.\n\n\"Diffusion enhancement methods have better generalizability than deterministic methods\" -- By \"deterministic methods\" I think you mean DiffWave.  In what sense is that a deterministic method?\n\np. 6 Considering the equivalently -> Considering the equivalence\n\np. 8 We contribute -> We attribute\n\n", "limitations": "Limitations and ethical considerations are not explicitly addressed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a diffusion-based speech enhancement with both improved performance and (because of the two-step inference) improved efficiency.  Both the performance gains and the efficiency gains are theoretically well motivated and empirically demonstrated.", "weaknesses": "Clarity:  (1) The dropout described in Eq. (13) is then not referenced again for the rest of the paper.  I think that's because Eq. (13) affects the T-step training process, while equations (14)-(18) are about the proposed reduction of inference from T steps to 2 steps.  But the division into training and testing is never really made explicit.  I think this is because the derivations assume that the reader has fully understood Figure 1, but Figure 1 cannot be fully understood until one has first understood the algorithm; I had to go back and examine Figure 1 after reading the derivations in order to know what's going on.  (2) I think that Eq. (17) should have an integral over dx_\\tau.  The algorithm might use the one-point approximation, but you've done a great job up to this point of keeping the theoretically-required integral in your equations, it seems a shame to abandon it here.  (3) Shouldn't the first term on the RHS in Eq. (18) be p_\\theta(\\hat{x}_{\\tau_2}|...)?  Or are you trying to say that p(x_{\\tau_2}|...) = p(\\hat{x}_0|...)?  That doesn't seem correct, since it misses the interpolation step.  ... also, I think there should be an integration over d\\hat{x}_{\\tau_2}.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Below we address specific questions.\n***\n>Q1: The dropout described in Eq. (13) is then not referenced again for the rest of the paper. I think that's because Eq. (13) affects the $T$-step training process, while equations (14)-(18) are about the proposed reduction of inference from $T$ steps to 2 steps. But the division into training and testing is never really made explicit. I think this is because the derivations assume that the reader has fully understood Figure 1, but Figure 1 cannot be fully understood until one has first understood the algorithm; I had to go back and examine Figure 1 after reading the derivations in order to know what's going on. \n\nThank you for your valuable feedback and comments! We apologize for any confusion caused by the paper writing. We will move Figure 1 to Sec 4 and reorganize Sec 4.2 -- explicitly divide training and testing as your suggested.\n***\n>Q2: I think that Eq. (17) should have an integral over $dx_{\\tau}$. The algorithm might use the one-point approximation, but you've done a great job up to this point of keeping the theoretically-required integral in your equations, it seems a shame to abandon it here.\n\nYou are right that Eq. 17 should have an integral over $dx_{\\tau}$. We will correct these in the revision.\n***\n> Q3: Shouldn't the first term on the RHS in Eq. (18) be $p\\_\\theta(\\hat{x}\\_{\\tau\\_2}|...)$? Or are you trying to say that $p(x\\_{\\tau\\_2}|...) = p(\\hat{x}\\_0|...)$? That doesn't seem correct, since it misses the interpolation step. ... also, I think there should be an integration over $d\\hat{x}\\_{\\tau\\_2}$.\n\nThank you for pointing out these oversights in Eq. 18! The RHS in Eq. 18 should be $p\\_\\theta(\\hat{x}\\_{\\tau\\_2}|...)$ and it should be an integration over $d\\hat{x}\\_{\\tau_2}$. We will correct these in the revision.\n    \n>Q4: Significance: Dropout enhances accuracy. The two-step inference enhances efficiency. Is there any interaction between these two things? It seems like the dropout might reduce the accuracy degradation that two-step inference would otherwise incur; is that true? Similarly, why two -- is two the optimum number of steps in any way? Results show pretty clearly that two is better than one, but is there any theoretical reason for that? The theory seems to predict simply that the more steps you have, the more accurate is the inference.\n    \nGreat question! It's important to clarify that full-step generation doesn't always yield better results compared to few-step generation. This phenomenon was exemplified in DiffuSE [5], where generating speech in 6 steps outperforms 50-step generation. Similarly, in [6, 7], authors demonstrated that generating samples in 1/10 steps performs better than full-step image generation (purification). \n    \nThe notion that \"the more steps, the more accurate the inference\" holds true only if each step produces a better estimation than the preceding one. Now, suppose the model can always generate a better condition factor. According to Proposition 1, a smaller $t$ can always be chosen. From a holistic standpoint, this loop continues as each step generates a better condition factor, thus the more steps, the more accurate the result (Sec 4.1, lines 164-166). However, in practice, this ideal scenario is hindered by empirical and generalization errors. Various factors contribute to model errors, including complexity, architecture, data quality, optimization, and stochasticity. Empirical evidence [16, 17] indicates that 2-step generation tends to outperform 1-step generation. This is because model can always generate improved estimates compared to the initial condition $c=y$, making 2-step better than 1-step in most cases [16, 17]. However, we cannot guarantee that $K$-step ($K > 2$) is better than 2-step. We showed multiple visual cases in Fig. 5 and Fig. 15 that increasing the number of sampling steps will lead to inconsistency (error accumulation) problem and subpar results. Additional results for different $K$ (i.e., 2, 6, 50 steps) are attached in the table (in General Response). Note that research on speech enhancement based on progressive learning [18, 19] also shows that iterative learning over 5 steps often leads to performance degradation. Taking into account the computational complexity of optimizing hyperparameters for $K$ (Appendix A.6, lines 571-575), we opt to directly set $K=2$ for both efficiency and stability reasons. We appreciate your question and hope this explanation clarifies our approach and reasoning.\n***\n>Q5: \"Diffusion enhancement methods have better generalizability than deterministic methods\" -- By \"deterministic methods\" I think you mean DiffWave. In what sense is that a deterministic method?\n\nTo ensure a fair comparison, we kept the model architecture exactly the same as that of the DiffWave, but used $y$ as the $x_t$ (so the model's input is two noisy speech $y$ concatenated along the channel dimension). We used a zero vector as the time step embedding so that it did not contain any additional information. This common practice of comparing generative diffusion models with deterministic counterparts is prevalent in the literature [13, 20].\n***\n>Q6: p. 6 and p. 8 typos.\n    \nThanks for spotting the typos! We will correct these in the revision."}]

Author rebuttal:
We would like to thank all reviewers for providing high-quality reviews and insightful feedback. 

---

We are encouraged that reviewers think our paper "provides valuable insights into the functioning of the denoising diffusion probabilistic model for speech enhancement'' (R4), "an interesting and promising solution to condition collapse issue'' (R3, R4), "technically/theoretically in-depth'' (R1, R5), "comprehensive experiments and thorough analysis'' (R2, R3, R4, R5), "broader implications and versatility" (R4), and "well-written'' (R2, R4). 

(We abbreviate the reviewer Zy9S, CtVJ, boQC, k89b, RTAK to R1, R2, R3, R4, R5, respectively.)

---

We provide additional ablation results requested by R1, R2 and R3, shown in the table below -- $p$ denotes the dropout rate, $\epsilon$ and $x$ denote different training objectives, and the number of steps denotes how many steps are needed to generate speech during the inference stage.

|  |  |  | VoiceBank |  |  |  |  | CHIME-4 |  |  |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| Variable | STOI(%) | PESQ | CSIG | CBAK | COVL | STOI(%) | PESQ | CSIG   | CBAK | COVL |
| p = 0 ( $\epsilon$ ) 2 steps (to R2) | 92.7    | 2.13 | 3.44 | 2.58 | 2.76 | 86.5    | 1.39 | 2.69   | 2.04 | 1.98 |
| p = 0 ( $x$ ) 2 steps (to R1) | 93.3    | 2.49 | 3.74 | 3.03 | 3.10 | 80.6    | 1.37 | 2.59   | 2.06 | 1.92 |
| p = 0.1 ( $x$ ) 2 steps (to R1) | 93.5 | 2.50 | 3.66 | 3.24 | 3.08 | 82.0    | 1.44 | 2.69   | 2.10 | 2.00 |
| p = 0.5 ( $x$ ) 2 steps (to R1) | **93.6** | **2.56** | **3.83** | **3.27** | **3.19** | **86.6** | **1.52** | **2.71** | **2.15** | **2.06** |
| p = 0.9 ( $x$ ) 2 steps (to R1) | 92.6 | 2.33 | 3.54 | 3.01 | 2.93 | 83.3 | 1.36 | 2.67 | 2.02 | 1.95 |
| p = 0.5 ( $x$ ) 6 steps (to R3) | 93.1 | **2.56** | 3.78 | 3.03 | 3.16 | 82.2 | 1.43 | 2.70 | 2.12| 2.01 |
| p = 0.5 ( $x$ ) 50 steps (to R2 and R3) | 93.2 | 2.48 | 3.66 | 3.18 | 3.06 | 82.1 | 1.39 | 2.49 | 2.01 | 1.87 |

---    

We list all needed references here to facilitate the subsequent point-to-point rebuttals.
    
[1] PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior, ICLR 2021.

[2] SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations, ICLR, 2021.

[3] DifFace: Blind Face Restoration with Diffused Error Contraction, Arxiv, 2022.

[4] Diffsinger: Singing voice synthesis via shallow diffusion mechanism, AAAI, 2022.

[5] A Study on Speech Enhancement Based on Diffusion Probabilistic Model, APSIPA, 2021.

[6] (Certified!!) Adversarial Robustness for Free!, ICLR, 2023.

[7] DensePure: Understanding Diffusion Models for Adversarial Robustness, ICLR, 2023.

[8] DiffWave: A Versatile Diffusion Model for Audio Synthesis, ICLR, 2021.

[9] Wavenet: A Generative Model for Raw Audio, Arxiv, 2016.

[10] A Wavenet for Speech Denoising, ICASSP, 2018.

[11] End-to-End Music Source Separation: Is It Possible in the Waveform Domain?, Interspeech, 2019.

[12] Conditional Diffusion Probabilistic Model for Speech Enhancement, ICASSP, 2022.

[13] Analysing Diffusion-based Generative Approaches Versus Discriminative Approaches for Speech Restoration, ICASSP, 2023.

[14] Score-Based Generative Modeling through Stochastic Differential Equations, ICLR, 2021.

[15] Generative Modeling by Estimating Gradients of the Data Distribution, NeurIPS, 2019.

[16] A Recursive Network with Dynamic Attention for Monaural Speech Enhancement, Interspeech, 2020.

[17] A Time-domain Monaural Speech Enhancement with Feedback Learning, APSIPA, 2020.

[18] Densely Connected Progressive Learning for LSTM-based Speech Enhancement, ICASSP, 2018.

[19] A Multi-Target SNR-Progressive Learning Approach to Regression Based Speech Enhancement, TASLP, 2020.

 [20] DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability, ICASSP, 2023.



Meta-review: All reviewers see the merit of the paper and recommend acceptance. The approach is simple and intuitive, and the experiments are sound. The writing, however, is overly complicated.  Some revision to improve the readability and accessibility of the paper, such as front-loading the topic sentence in each paragraph, is recommended.
Decision: Accept (poster)