Title: DOSE: Diffusion Dropout with Adaptive Prior for Speech Enhancement
Venue: NeurIPS 2023
Accepted: yes
Keywords: speech enhancement

diffusion models

adaptive prior

dropout

generalization
Abstract: Speech enhancement (SE) aims to improve the intelligibility and quality of speech in the presence of non-stationary additive noise. Deterministic deep learning models have traditionally been used for SE, but recent studies have shown that generative approaches, such as denoising diffusion probabilistic models (DDPMs), can also be effective. However, incorporating condition information into DDPMs for SE remains a challenge. We propose a model-agnostic method called DOSE that employs two efficient condition-augmentation techniques to address this challenge, based on two key insights: (1) We force the model to prioritize the condition factor when generating samples by training it with dropout operation; (2) We inject the condition information into the sampling process by providing an informative adaptive prior. Experiments demonstrate that our approach yields substantial improvements in high-quality and stable speech generation, consistency with the condition factor, and inference efficiency. Codes are publicly available at https://github.com/ICDM-UESTC/DOSE.

DOSE: Diffusion Dropout with Adaptive Prior for Speech Enhancement

Abstract

Speech enhancement (SE) aims to improve the intelligibility and quality of speech in the presence of non-stationary additive noise. Deterministic deep learning models have traditionally been used for SE, but recent studies have shown that generative approaches, such as denoising diffusion probabilistic models (DDPMs), can also be effective. However, incorporating condition information into DDPMs for SE remains a challenge. We propose a model-agnostic method called DOSE that employs two efficient condition-augmentation techniques to address this challenge, based on two key insights: (1) We force the model to prioritize the condition factor when generating samples by training it with dropout operation; (2) We inject the condition information into the sampling process by providing an informative adaptive prior. Experiments demonstrate that our approach yields substantial improvements in high-quality and stable speech generation, consistency with the condition factor, and inference efficiency. Codes are publicly available at https://github.com/ICDM-UESTC/DOSE .

1 Introduction

Speech enhancement (SE) aims to improve the intelligibility and quality of speech, particularly in scenarios where degradation is caused by non-stationary additive noise. It has significant practical implications in various fields such as telecommunications [1], medicine [2], and entertainment [3]. Modern deep learning models are often used to learn a deterministic mapping from noisy to clean speech. While deterministic models have long been regarded as more powerful in the field of SE, recent advancements in generative models [4, 5] have significantly closed this gap.

One such generative approach is based on using denoising diffusion probabilistic models (DDPMs) [6, 7], which have been shown to effectively synthesize natural-sounding speech. Several diffusion enhancement models have been developed [4, 5, 8], which try to learn a probability distribution over the data and then generate clean speech conditioned on the noisy input. A key challenge in using diffusion enhancement models is how to effectively incorporate condition information into learning and generating faithful speech [9, 10, 8]. Previous works address this issue through designing specific condition-injecting strategies [4, 9, 11] or devising complex network architectures [10, 5].

We conduct a thorough examination to understand the limitation of diffusion-based SE methods and find that diffusion enhancement models are susceptible to condition collapse , where the primary cause of inconsistent generation is the non-dominant position of the condition factor . We thus introduce a new paradigm to effectively incorporate condition information into the diffusion enhancement models. Specifically, we propose a Diffusion-drOpout Speech Enhancement method (DOSE), which is a model-agnostic SE method (Figure 1) that employs two efficient condition-augmentation tech-

Dropout

r ~Bernoulli(1 -p)

Adaptiveprior

Step1

RO

xt

X

+

→xO

P(xoly1y)

y

Residual-

Step

2

T2

XT

pe(xolx2,y)

L1loss

ICleanSpeech

niques: (1) During training, we randomly drop out intermediate-generated samples. This dropout mechanism guides the model's attention toward the condition factors; (2) Instead of letting the model generate samples from scratch (Gaussian distribution), we employ an adaptive prior derived from the conditional factor to generate samples. Experiments on benchmark datasets demonstrate that our method surpasses recent diffusion enhancement models in terms of both accuracy and efficiency. Additionally, DOSE produces more natural-sounding speech and exhibits stronger generalization capabilities compared to deterministic mapping-based methods using the same network architecture.

2 Related works

There are two main categories of diffusion-based SE methods: (1) designing specific conditioninjecting strategies [4, 9, 11, 5], or (2) generating speech with an auxiliary condition optimizer[12, 10, 8]. The first category considerates noisy speech in the diffusion (or reverse) process, either by linearly interpolating between clean and noisy speech along the process [4, 9], or by defining such a transformation within the drift term of a stochastic differential equation (SDE) [11, 5].

Works from the second category rely on an auxiliary condition optimizer - a generator (diffusion model) synthesizes clean speech and a condition optimizer informs what to generate [12, 10, 8]. Both the generator and condition optimizer have the ability to denoise, with the latter undertaking the core part. Given the challenges in leveraging condition information [8], diffusion-based SE methods within this category often necessitate specific network architecture design to guarantee the participation of condition factors.

In a paradigm sense, our method is quite similar but different to the second branch - unlike previous approaches that require additional auxiliary networks, DOSE is an end-to-end diffusion-based SE method. In addition, DOSE is model-agnostic that does not need any specific network design to guarantee consistency between the generated sample and its corresponding condition factor.

3 Preliminaries

We now provide a brief introduction to the diffusion probabilistic model (diffusion models, for short), the definition of speech enhancement, and the condition collapse problem.

3.1 Diffusion models

A diffusion model [13, 6] consists of a forward (or, diffusion) process and a reverse process. Given a data point x 0 with probability distribution p ( x 0 ) , the forward process gradually destroys its data structure by repeated application of the following Markov diffusion kernel:

where β 1 , · · · , β T is a pre-defined noise variance schedule. With enough diffusion step T , p ( x T ) converges to the unit spherical Gaussian distribution. Based on the Markov chain, the marginal distribution at arbitrary timestep t has the following analytical form:

Loss curve

xt

y

xt

y

xt

y

PESQ

0.12

0.10

--Uncondition

--Condition

Concat

Concat

Concat

2.4

2.32

0.08

R(1)

R(2)

R(2)

R(1)

R(2)

R(1)

2.2

2.25

0.06

R(n).

：

R(n).

R(n).

2.0

1.96

0.04

0.02

RELU

+

+

1.8

1.79

0.00

Step

3

3

1.6

50k

100k150k200k250k

(a)

(b)

(c)

(a)

(-)

(b) (c)

As for the reverse process, it aims to learn a transition kernel from x t to x t -1 , which is defined as the following Gaussian distribution [6]:

where θ is the learnable parameter and µ θ ( x t , t ) = 1 √ 1 -β t ( x t -β t √ 1 -¯ α t ϵ θ ( x t , t )) denotes the mean of x t -1 , which is obtained by subtracting the estimated Gaussian noise ϵ θ ( x t , t ) in the x t . With such a learned transition kernel, one can approximate the data distribution p ( x 0 ) via:

where p θ ( x T ) = N ( x T ; 0 , I ) .

3.2 Problem formulation

Speech enhancement refers to methods that try to reduce distortions, make speech sounds more pleasant, and improve intelligibility. In real environments, the monaural noisy speech y in the time domain can be modeled as:

where x and n denote clean and noise signals, respectively. For human perception, the primary goal of speech enhancement is to extract x from y . Mapping-based speech enhancement methods directly optimize p θ ( x | y ) , while diffusion enhancement methods generate clean samples through a Markov process p θ ( x 0: T -1 | x 1: T , y ) .

3.3 Condition Collapse in diffusion enhancement models

The condition collapse problem in speech enhancement was first proposed in [8] and it refers to the limited involvement of the condition factor during conditional diffusion training, resulting in inconsistencies between the generated speech and its condition factor.

In this work, we argue that the condition factor y indeed participates and helps the intermediategenerated sample x t approximate p ( x t -1 | x t , x 0 ) . Our assertion is supported by the experiment depicted in the left part of Figure 2 - the diffusion model equipped with the condition factor exhibits a lower loss curve compared to the unconditional one 2 . To better understand the condition collapse phenomenon, we devise two variants that explicitly modify the mutual information between the condition factor and the model's output (Figure 2 (middle)). We use skip connections to add the condition factor to multiple layers, forcing the likelihood of maintaining a strong connection between the condition factor and output features. Since the dependence of the output on any hidden state in the hierarchy becomes weaker as one moves further away from the output in that hierarchy (cf. [14]), using skip connections can explicitly enhance connections between the generated sample and condition factor.

As shown in Figure 2 (right), an increase in mutual information (connections) leads to a significant improvement in the consistency between the generated sample and the condition factor ( a → b ). However, it requires a meticulously designed model to guarantee its effectiveness ( b → c ). While previous studies [5, 10, 8] focus on explicitly enhancing the consistency between the output speech and condition factor through specific network architecture design, we explore the possibility of a solution independent of the model architecture. This would broaden the applicability of our method, as it enables slight modifications to existing deterministic mapping-based models to transform them into diffusion enhancement models.

4 Methodology

Considering the diffusion model provides a transition function from x t to x t -1 , typical condition generation process is represented as:

Our experiments above indicate that p θ ( x t -1 | x t , y ) will easily collapse to p θ ( x t -1 | x t ) , resulting in the condition generation process degenerating into a vanilla unconditional process:

As a result, facilitating automatic learning of the joint distribution for both clean and noisy speech samples does not work well for the speech enhancement task.

4.1 Condition augmentation I: Adaptive Prior

Let's revisit Eq. (6): since we cannot easily inject the condition factor into the condition term, how about the prior term? For example, we can modify the condition generation process as:

where p ( x τ | y ) is formulated as p ( x τ | y ) = N ( x τ ; √ ¯ α τ y , (1 -¯ α τ ) I ) . The following propositions verify the feasibility of our proposal.

Proposition 1. For any ξ > 0 such that 0 < ξ < M for some finite positive value M , there exists a positive value τ ∈ { 0 , · · · , T } that satisfies:

where p ( x t | c ) = N ( x t ; √ ¯ α t c , (1 -¯ α t ) I ) .

Remark 1. This proposition indicates that, given a tolerable margin of error ξ and a well-trained diffusion model, we can always find a suitable τ such that we are able to recover the clean speech x from its noisy one y using Eq. (8) .

While Proposition 1 allows us to generate clean speech x given the noisy speech y using Eq. (8), it does not guarantee that our model will achieve successful recovery with a high probability.

Proposition 2. Let x be the clean sample, y be it's corresponding noisy one, and x ′ be any neighbor from the neighbor set S ( x ) . Then diffusion enhancement models can recover x with a high probability if the following inequality is satisfied:

where σ 2 t = 1 -¯ α t ¯ α t .

Remark 2. Assuming that the condition factor y is closer to x than to x ′ , we obtain a non-positive right-hand side (RHS). For a given x , the left-hand side (LHS) value is fixed, and to ensure the inequality always holds, a smaller σ 2 t is preferred.

As shown in Figure 3, σ 2 t will increase as the timestep t increases. Thus, according to Proposition 2 , we should choose a small τ for Eq. (8) to maximize the probability of successfully recovering the clean speech from the noisy one. However, constrained by Proposition 1 , τ cannot be too small. In other words, the clean speech distribution p ( x τ ) and the noisy speech distribution p ( y τ ) will get closer over the forward diffusion process, and the gap | n | = | y -x | between the noisy speech and the clean one will indeed be 'washed out' by the increasingly added noise. Since the original semantic information will also be removed if τ is too large, there should be a trade-off when we set τ for the diffusion enhancement model.

Condition optimizer. We find that both propositions are corre- lated with the condition factor y . If we can reduce the gap between the condition factor and clean speech, we can choose a smaller τ , effectively increasing the likelihood of recovering clean speech. One simple idea is to employ a neural network f ψ to optimize the condition factor, as demonstrated in [15]. Accordingly, we can rewrite Eq. (8) as:

where p ψ ( x τ | y ) = N ( x τ ; √ ¯ α τ f ψ ( y ) , (1 -¯ α τ ) I ) .

In practice, we should also consider failure cases of the condition optimizer in complex scenarios, especially the issue of excessive suppression that has been reported in recent literature [16, 17, 18]. To mitigate this issue, we use 0 . 5 c + 0 . 5 y (like a simple residual layer) as a mild version of the condition factor:

We call p ψ ( x τ | y ) the adaptive prior as it varies with different noisy samples y .

4.2 Condition augmentation II: Diffusion Dropout

Aside from changing the prior p ( x T ) to conditional prior p ψ ( x τ | y ) , we also optimize the condition term p θ ( x t -1 | x t , y ) . Instead of designing specific condition-injecting strategies [4, 9, 11] or devising complicated network architecture [8, 10, 5], we attempt to 'do subtraction' by discarding some shared (intermediate-generated samples & condition factor) and important (target-related) information from intermediate-generated samples. Naturally, if we discard some information from x t , then the diffusion enhancement model is forced to use the condition factor y to recover the speech. Taking a further step, we can even discard the entire x t , as the condition factor y alone is sufficient for recovering the clean speech x 0 (this is what deterministic models do). To this end, we define a neural network f θ ( d ( x t , p ) , y , t ) to approximate p ( x t -1 | x t , x 0 ) :

where d ( x t , p ) is the dropout operation:

4.3 DOSE training

Ho et al. [6] and much of the following work choose to parameterize the denoising model through directly predicting ϵ with a neural network ϵ θ ( x t , t ) , which implicitly sets:

In this case, the training loss is also usually defined as the mean squared error in the ϵ -space ∥ ϵ -ϵ θ ( x t , t ) ∥ 2 2 . Although this standard specification works well for training an unconditional diffusion model, it is not suited for DOSE - for two reasons.

0

100

200

Timestep

t

-2.5

0.0

2.5

log

10

(

σ

2

t

)

50, 1e-4, 0.035

200, 1e-4, 0.05

First, we cannot estimate ϵ without the help of x t because ϵ and y are independent. Second, as discussed earlier, we want DOSE to start with a small timestep and we strive to make τ small. However, as τ approaches zero, small changes in x -space have an increasingly amplified effect on the implied prediction in ϵ -space (Eq. (15)). In other words, the efforts made by diffusion enhancement models become so negligible that diffusion models lose their ability to calibrate the speech at small timesteps.

So, we need to ensure that the estimation of ˆ x 0 remains flexible as the timestep t gets smaller. Considering the equivalently of the ϵ -space loss ∥ ϵ -ϵ θ ( x t , t ) ∥ 2 2 to a weighted reconstruction loss in x -space 1 σ 2 t ∥ x 0 -x θ ( x t , t ) ∥ 2 2 , we can directly estimate the clean speech x 0 at each timestep t :

4.4 DOSE inference

After training, the ideal scenario is that p θ ( x t -1 | x t , y ) approximates p ( x t -1 | x t , x 0 ) precisely, enabling us to generate clean speech using Eq. (6). However, when applied in practice, it is difficult to completely eliminate errors (both sample error and true error). If these errors are not effectively managed or corrected during the generation process, the quality of the generated samples may deteriorate, leading to artifacts, blurriness, etc [19, 20]. This issue is particularly pronounced when using diffusion models for fine-grained conditional generation tasks, as diffusion models require a large number of steps to generate samples, which will significantly reduce the consistency between the generated sample and its condition factor (see §5.3, Figure 6).

The adaptive prior (Sec 4.1) provides an opportunity to address the error accumulation issue. Specifically, we can select a suitable τ smaller than T , conditioned on an adaptive prior, and generate speech in fewer steps. We can extend Eq. 11 by transforming the unconditional diffusion enhancement model into a conditional one:

and the number of sampling steps is reduced from T to τ +1 .

Readers familiar with diffusion models may recall that the standard process repeatedly applies a 'single-step' denoising operation x t -1 = denoise ( x t ; t ) that aims to convert a noisy sample at some timestep t to a (slightly less) noisy sample at the previous timestep t -1 . In fact, each application of the one-step denoiser consists of two steps: (1) an estimation of the fully denoised sample x 0 from the current timestep t , and (2) computing a (properly weighted, according to the diffusion model) average between this estimated denoised sample and the noisy sample at the previous timestep t -1 . Thus, instead of performing the entire τ -step diffusion process to denoise a sample, it is also possible to run denoise once and simply output the estimated sample in one shot [21]. Accordingly, Eq. (17) can be further rewritten as:

We can even achieve DOSE without the condition optimizer f ψ ( · ) - using conditional diffusion enhancement model instead. For example, we can generate clean speech via:

where τ 1 , τ 2 ( τ 2 < τ 1 ≤ T ) are two pre-defined hyper-parameters. The motivation behind Eq. (19) is that, once we have trained a neural network f θ ( x t , y , t ) that can accurately estimate x 0 (Eq. (16)), according to the theoretical analysis in Sec 4.1, we can first choose a suitable value for τ 1 to ensure a relatively good approximation of x 0 :

In the second step, once we have obtained a good condition factor, we can choose a smaller timestep τ 2 < τ 1 to get a better estimation of x 0 than ˆ x 0 generated in the first step.

Summary. DOSE has three important benefits: (1) By dropping x t entirely, we make the condition factor y the 'protagonist', automatically enhancing the consistency between the generated sample and the condition factor. (2) By training the model with this modified training objective, DOSE can perform well not only on Gaussian noise ( x t → x 0 ) but also on various types of non-Gaussian noise ( y → x 0 ). (3) DOSE is efficient (2 steps), faster than existing diffusion enhancement models.

5 Experiments

We compare DOSE with prevailing diffusion enhancement methods and deterministic mappingbased enhancement methods in §5.1. We conduct a counterfactual verification to understand the intrinsic mechanism of DOSE in §5.2. We show two visual cases of excessive suppression and error accumulation in §5.3. While providing a self-contained version of our main results, we note that we also have additional quantitative observations reported in the Appendices. Specifically, we compare DOSE with other baselines via subjective evaluation (Appendix A.2); We investigate the significance of the proposed adaptive prior and explain why we need to use a mild version of the condition factor (Appendix A.3); We examine the effect of our new training objective and demonstrate the necessity of using it (Appendix A.4); We explain why we use two steps in speech generation (Appendix A.5); We provide parameter sensitivity experiments (Appendix A.6); We show plenty of visual cases of excessive suppression and error accumulation (Appendix A.7 and A.8). To help readers better understand our research, we include a discussion subsection in Appendix A.9. Specifically, we: (1) Analyze the reasons behind the superior generalizability of diffusion enhancement models compared to deterministic mapping-based models (from the robust training perspective); (2) Explain why we use 0.5 in the mild version of the condition factor; (3) Discuss the broader impacts of speech enhancement methods.

Dataset and baselines. Following previous works [4, 9, 8], we use the VoiceBank-DEMAND dataset [22, 23] for performance evaluations. To investigate the generalization ability of models, we use CHiME-4 [24] as another test dataset following [9], i.e., the models are trained on VoiceBankDEMAND and evaluated on CHiME-4. We compare our model with recent open-sourced diffusion enhancement models such as DiffuSE [4], CDiffuSE [9], SGMSE [11], SGMSE+ [5], and DRDiffuSE [8]. Since the only difference between SGMSE+ and SGMSE is their network architecture, we compare our model with just one of them.

Evaluation metrics. We use the following metrics to evaluate SE performance: the perceptual evaluation of speech quality (PESQ) [25], short-time objective intelligibility (STOI) [26], segmental signal-to-noise ratio (SSNR), the mean opinion score (MOS) prediction of the speech signal distortion (CSIG) [27], the MOS prediction of the intrusiveness of background noise (CBAK) [27] and the MOS prediction of the overall effect (COVL) [27]. Besides these metrics, we also design two MOS metrics (MOS and Similarity MOS) for subjective evaluation.

Configurations. To ensure a fair comparison, we keep the model architecture exactly the same as that of the DiffWave model [7] for all methods 3 . DiffWave takes 50 steps with the linearly spaced training noise schedule β t ∈ [ 1 × 10 -4 , 0 . 035 ] [4]. We train all methods for 300,000 iterations using 1 NVIDIA RTX 3090 GPU with a batch size of 16 audios. We select the best values for τ 1 and τ 2 according to the performance on a validation dataset, a small subset (10%) extracted from the training data. More experiment settings can be found in Appendix A.10.

Method

Year

Efficiency

Dataset

STOI

(%)

↑

PESQ

↑

CSIG

↑

CBAK

↑

COVL

↑

Unprocessed

-

-

VB

92.1

1.97

3.35

2.44

2.63

DiffWave

2021

1 step (dis)

93.3

2.51

3.72

3.27

3.11

DiffuSE

2021

6 steps

93

.

5

+0

.

20

±

0

.

05

2

.

39

-

0

.

12

±

0

.

01

3

.

71

-

0

.

01

±

0

.

01

3

.

04

-

0

.

23

±

0

.

01

3

.

03

-

0

.

08

±

0

.

01

CDiffuSE

2022

6 steps

93

.

7

+0

.

40

±

0

.

05

2

.

43

-

0

.

08

±

0

.

01

3

.

77

+0

.

05

±

0

.

01

3

.

09

-

0

.

18

±

0

.

01

3

.

09

-

0

.

02

±

0

.

01

SGMSE

2022

50 steps

93

.

3

+0

.

00

±

0

.

08

2

.

34

-

0

.

17

±

0

.

01

3

.

69

-

0

.

03

±

0

.

01

2

.

90

-

0

.

37

±

0

.

01

3

.

00

-

0

.

11

±

0

.

01

DR-DiffuSE

2023

6 steps

92

.

9

-

0

.

04

±

0

.

06

2

.

50

-

0

.

01

±

0

.

02

3

.

68

-

0

.

04

±

0

.

02

3

.

27

+0

.

00

±

0

.

02

3

.

08

-

0

.

03

±

0

.

02

DOSE

-

2 steps

93

.

6

+0

.

30

±

0

.

05

2

.

56

+0

.

05

±

0

.

01

3

.

83

+0

.

11

±

0

.

01

3

.

27

+0

.

00

±

0

.

01

3

.

19

+0

.

08

±

0

.

01

Unprocessed

-

-

CHIME-4

71.5

1.21

2.18

1.97

1.62

DiffWave

2021

1 step (dis)

72.3

1.22

2.21

1.95

1.63

DiffuSE

2021

6 steps

83

.

7

+11

.

4

±

0

.

05

1

.

59

+0

.

36

±

0

.

01

2

.

91

+0

.

70

±

0

.

01

2

.

19

+0

.

24

±

0

.

01

2

.

19

+0

.

56

±

0

.

01

CDiffuSE

2022

6 steps

82

.

8

+10

.

5

±

0

.

05

1

.

58

+0

.

36

±

0

.

01

2

.

88

+0

.

67

±

0

.

01

2

.

15

+0

.

20

±

0

.

01

2

.

18

+0

.

55

±

0

.

01

SGMSE

2022

50 steps

84

.

5

+12

.

2

±

0

.

05

1

.

57

+0

.

34

±

0

.

02

2

.

92

+0

.

71

±

0

.

01

2

.

18

+0

.

23

±

0

.

02

2

.

18

+0

.

55

±

0

.

01

DR-DiffuSE

2023

6 steps

77

.

6

+5

.

30

±

0

.

06

1

.

29

+0

.

07

±

0

.

04

2

.

40

+0

.

19

±

0

.

02

2

.

04

+0

.

09

±

0

.

01

1

.

78

+0

.

15

±

0

.

01

DOSE

-

2 steps

86

.

6

+14

.

3

±

0

.

05

1

.

52

+0

.

30

±

0

.

01

2

.

71

+0

.

50

±

0

.

01

2

.

15

+0

.

20

±

0

.

01

2

.

06

+0

.

43

±

0

.

01

5 step:

+ t+

4ttt-

35 step:

++..

+++++..

Figure 4: Counterfactual visualization. The first two columns are associated with a dropout probability of 0.1, while the last two columns are associated with a dropout probability of 0.9. In each row, the blue waveforms in the first and third columns are the counterfactual samples, and the blue waveforms in the second and fourth columns are the normal samples. The orange waveforms are generated samples from the model.

5.1 Performance comparison

We compare our method with previous diffusion enhancement methods and summarize our experimental results in Table 1. We observe that: (1) Diffusion enhancement methods have better generalizability than deterministic methods. (2) Methods with specific condition-injecting strategies, such as DiffuSE, CDiffuSE, and SGMSE, have strong generalization but perform slightly worse than deterministic mapping-based methods in matched scenarios. (3) Method (DR-DiffuSE) with auxiliary condition optimizer, performs better in matched scenarios and shows a slight improvement in mismatched scenarios. (4) Our method performs well in both matched and mismatched scenarios and is on par with state-of-the-art diffusion enhancement models while requiring fewer steps.

5.2 Counterfactual verification

We perform a counterfactual verification to gain insights into the underlying mechanism of DOSE. To verify whether dropout can increase the 'discourse power' of the conditional factor, we keep the condition factor y fixed and reverse the intermediate-generated speech at a specific step ( reverse ( x t ) ). This reversed intermediate-generated speech is called a counterfactual sample. Notably, if the final generated speech is more similar to the condition factor than the counterfactual speech, we can conclude that the condition factor plays a dominant role in the generation process. Otherwise, we can say that the condition factor is less influential.

As shown in Figure 4, we compare the performance of two models with different dropout probabilities (0.1 vs. 0.9). We have two findings here: (1) A higher dropout probability encourages the model to prioritize the condition factor even with a small timestep t . (2) When timestep t is large, DOSE ef-

fectively captures condition information, ensuring the model's robustness to noise and maintaining consistency in the early stages of inference.

5.3 Excessive suppression & error accumulation

We provide a visual case of excessive suppression in Figure 5 and a visual case of error accumulation in Figure 6. From Figure 5, we can see that: (1) The deterministic model fails in mismatched scenarios and generates samples that lose speech details; (2) The diffusion enhancement model generate defective speech when directly using the estimated speech as the condition factor; (3) The diffusion enhancement model equipped with a mild version of the condition factor can recover clean speech effectively. From Figure 6, we notice that: (1) Full-step generation can remove noise and generate natural-sounding speech. However, it can't guarantee the consistency between the generated speech and condition factor; (2) Two-step speech generation with adaptive prior can promise consistency and high quality simultaneously.

6 Conclusions

In this work, we present a new approach DOSE that effectively incorporates condition information into diffusion models for speech enhancement. DOSE uses two efficient condition-augmentation techniques to address the condition collapse problem. Comprehensive experiments on benchmark datasets demonstrate the efficiency and effectiveness of our method.

In our method, there are two groups of hyper-parameters: the dropout probability p for the dropout operation and two timesteps τ 1 , τ 2 for the adaptive prior. These parameters are critical to model performance. For example, if the dropout probability is set too high, the diffusion enhancement model will rely solely on the condition factor to estimate the speech. Then our diffusion enhancement model will degenerate into a deterministic model, losing its generalizability. We also need to make a trade-off when choosing the timestep τ (especially τ 1 ): On one hand, a large τ is needed to reduce the gap between the clean speech and condition factor. On the other hand, the original semantic information will also be removed if τ is set too large.

In practice, it is necessary to evaluate the model on a subset of data and then empirically set the hyperparameters. These manually defined hyper-parameters are selected based on the Empirical Risk Minimization (ERM) principle and may not be optimal for every individual sample. Thus, an important direction for future research is to develop methods that can adaptively choose hyperparameters for different samples. It is also expected that the model can adaptively select appropriate coefficients when forming a mild version of the conditioning factor.

References

Steven L Gay and Jacob Benesty. Acoustic signal processing for telecommunication , volume 551. Springer Science & Business Media, 2012.

Tim Van den Bogaert, Simon Doclo, Jan Wouters, and Marc Moonen. Speech enhancement with multichannel wiener filter techniques in multimicrophone binaural hearing aids. The Journal of the Acoustical Society of America , 125(1):360-371, 2009.

Ivan Tashev. Recent advances in human-machine interfaces for gaming and entertainment. International journal of information technologies and security , 3(3):69-76, 2011.

Yen-Ju Lu, Yu Tsao, and Shinji Watanabe. A study on speech enhancement based on diffusion probabilistic model. In 2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) , pages 659-666. IEEE, 2021.

Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhancement and dereverberation with diffusion-based generative models. arXiv preprint arXiv:2208.05830 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations , 2021.

Wenxin Tai, Fan Zhou, Goce Trajcevski, and Ting Zhong. Revisiting denoising diffusion probabilistic models for speech enhancement: Condition collapse, efficiency and refinement. In AAAI , 2023.

Yen-Ju Lu, Zhong-Qiu Wang, Shinji Watanabe, Alexander Richard, Cheng Yu, and Yu Tsao. Conditional diffusion probabilistic model for speech enhancement. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 7402-7406. IEEE, 2022.

Joan Serrà, Santiago Pascual, Jordi Pons, R Oguz Araz, and Davide Scaini. Universal speech enhancement with score-based diffusion. arXiv preprint arXiv:2206.03065 , 2022.

Simon Welker, Julius Richter, and Timo Gerkmann. Speech enhancement with score-based generative models in the complex STFT domain. In Proc. Interspeech 2022 , pages 2928-2932, 2022. doi: 10.21437/Interspeech.2022-10653.

Jianwei Zhang, Suren Jayasuriya, and Visar Berisha. Restoring degraded speech via a modified diffusion model. In 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021 , pages 2753-2757. International Speech Communication Association, 2021.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

Thomas M Cover. Elements of information theory . John Wiley & Sons, 1999.

Zongsheng Yue and Chen Change Loy. Difface: Blind face restoration with diffused error contraction. arXiv preprint arXiv:2212.06512 , 2022.

Andong Li, Chengshi Zheng, Cunhang Fan, Renhua Peng, and Xiaodong Li. A recursive network with dynamic attention for monaural speech enhancement. In Proc. Interspeech 2020 , pages 2422-2426, 2020. doi: 10.21437/Interspeech.2020-1513. URL http://dx.doi.org/ 10.21437/Interspeech.2020-1513 .

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Santiago López-Tapia and Nicolás Pérez de la Blanca. Fast and robust cascade model for multiple degradation single image super-resolution. IEEE Transactions on Image Processing , 30:4747-4759, 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Jean-Marie Lemercier, Julius Richter, Simon Welker, and Timo Gerkmann. Storm: A diffusionbased stochastic regeneration model for speech enhancement and dereverberation. arXiv preprint arXiv:2212.11851 , 2022.

Nicholas Carlini, Florian Tramer, J Zico Kolter, et al. (certified!!) adversarial robustness for free! In International Conference on Learning Representations , 2023.

Christophe Veaux, Junichi Yamagishi, and Simon King. The voice bank corpus: Design, collection and data analysis of a large regional accent speech database. In International Conference Oriental COCOSDA , pages 1-4. IEEE, 2013.

Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. The diverse environments multichannel acoustic noise database (demand): A database of multichannel environmental noise recordings. In Proceedings of Meetings on Acoustics , volume 19, page 035081. Acoustical Society of America, 2013.

Emmanuel Vincent, Shinji Watanabe, Aditya Arie Nugraha, Jon Barker, and Ricard Marxer. An analysis of environment, microphone and data simulation mismatches in robust speech recognition. Computer Speech Language , 46:535-557, 2017.

Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In IEEE international conference on acoustics, speech, and signal processing (ICASSP) , pages 749-752. IEEE, 2001.

Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. A short-time objective intelligibility measure for time-frequency weighted noisy speech. In IEEE international conference on acoustics, speech, and signal processing (ICASSP) , pages 4214-4217. IEEE, 2010.

Yi Hu and Philipos C Loizou. Evaluation of objective quality measures for speech enhancement. IEEE Transactions on audio, speech, and language processing , 16(1):229-238, 2007.

Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Progressive fast diffusion model for high-quality text-to-speech. In Proceedings of the 30th ACM International Conference on Multimedia , pages 2595-2605, 2022.

A Appendix

In the supplemental material,

A.1: We provide the proofs for Propositions 1 and 2.

A.2: We compare DOSE with other baselines via subjective evaluation.

A.3: We investigate the significance of the proposed adaptive prior and explain why we need to use a mild version of the condition factor.

A.4: We examine the effect of our new training objective and demonstrate the necessity of using it.

A.5: We explain why we use two steps in speech generation.

A.6: We provide parameter sensitivity experiments.

A.7: We present several visual cases of excessive suppression.

A.8: We present several visual cases of error accumulation.

A.9: We analyze the reasons behind the superior generalizability of diffusion enhancement models compared to deterministic mapping-based models (from the robust training perspective), explain why we use 0.5 in the mild version of the condition factor, and discuss the broader impacts of speech enhancement methods.

A.10: We include more information about speech processing and basic architecture.

A.1 Mathematical proofs

We now present the proofs of the Propositions stated in the main text.

Proposition 1. (cf. §4.1): For any ξ > 0 such that 0 < ξ < M for some finite positive value M , there exists a positive value τ ∈ { 0 , · · · , T } that satisfies:

where p ( x t | c ) = N ( x t ; √ ¯ α t c , (1 -¯ α t ) I ) .

Proof. Given two Gaussian distributions P , Q defined over a vector space R d , the KL divergence of multivariate Gaussian distributions is defined as follows:

Here, µ 1 ∈ R d and Σ 1 ∈ R d × d are the mean and covariance matrix of distribution P , and µ 2 ∈ R d and Σ 2 ∈ R d × d are the mean and covariance matrix of distribution Q . d is the dimensionality of the vectors (i.e., the number of dimensions in the vector space), and Tr denotes the trace operator.

Note that when the two Gaussian distributions have diagonal covariance matrices (i.e., when the different dimensions are independent), the above formula simplifies to the sum of the KL divergences of each univariate Gaussian distribution. Thus, given two Gaussian distributions p ( x t | x ) , p ( x t | y ) and Eq. (5), the KL divergence between these two distributions can be calculated as follows:

According to the definition of the diffusion model and Figure 3, D KL ( p ( x t | x ) ∥ p ( x t | y )) is a monotonically decreasing function. Ideally, for a bounded error n with (almost) infinite timestep T , we have:

According to Bolzano's theorem, there exists at least one point τ in the interval { 0 , · · · , T } such that D KL ( p ( x τ | x ) ∥ p ( x τ | y )) = ξ . Then, Eq. (9) holds for τ ≤ t ≤ T .

Testing audio

0:00/0:05

D

Insruction:

What's your rating of voice

How nature(i.e.human-sounding)is this recording?

Select an rating for yourfavorite audio

OExcellent-Completely natural speech-5

Please focus on examing the audio quality and naturalness, andignorethedifferencesofstyle(timbre,emotionand prosody)

O 4.5

O Good-Mostly natural speech -4

○3.5

O Fair-Equally natural and unnatural speech -3

O 2.5

OPoor-Mostly unnatural speech-2

O 1.5

OBad-Completelyunnatural speech-1

Youselected Excellent-Completelynatural speech-5

Proposition 2. (cf. §4.1): Let x be the clean sample, y be it's corresponding noisy one, and x ′ be any neighbor from the neighbor set S ( x ) . Then diffusion enhancement models can recover x with a high probability if the following inequality is satisfied:

where σ 2 t = 1 -¯ α t ¯ α t is the variance of the Gaussian noise added at timestep t in the forward diffusion process.

Proof. The main idea is to prove that any point x ′ quite similar but different to the ground-true speech x should have a lower density than x in the conditional distribution so that the diffusion enhancement models can recover x with a high probability. In other words, we should have:

According to Bayes' theorem, we have:

Applying Eq. (25) to Eq. (24), we obtain:

A.2 Subjective evaluation

We conduct two types of Mean Opinion Score (MOS) tests to verify the quality of synthesized audio through human evaluation.

Referenceaudio

?

0:00/0:05

D

Testing audio

0:00/0:05

：

Insruction:

What's your favorite

What's your rating.

audio:

of voice

How similar is this recording to the

reference audio?

choose one audio

Select anrating foryour favorite audio

Pleasefocusonthe similarity of the

Reference Audio

Excellent-Completelynatural

style(speckeridentity,emotion and

O Testing Audio

speech-5

prosody)to the reference,and ignore

O 4.5

the audio quality

You selected ReferenceAudio

O Good-Mostly natural speech -4

O3.5

O Fair-Equally natural and

unnatural speech-3

O 2.5

OPoor-Mostly unnatural speech-2

O 1.5

O Bad-Completely unnatural

speech -1

You selected Excellent-Completely

natural speech-5

Naturalness. For audio quality evaluation, we conduct the MOS (mean opinion score) tests and explicitly instruct the raters to 'focus on examining the audio quality and naturalness, and ignore the differences of style (timbre, emotion, and prosody)'. The testers present and rate the samples, and each tester is asked to evaluate the subjective naturalness on a 1-5 Likert scale.

Consistency. For audio consistency evaluation, we explicitly instruct the raters to 'focus on the similarity of the speech (content, timbre, emotion, and prosody) to the reference, and ignore the differences of audio quality'. This is slightly different from the original definition of SMOS for speech synthesis. In the SMOS (similarity mean opinion score) tests, we pair each synthesized utterance with a ground truth utterance to evaluate how well the synthesized speech matches that of the target speaker. The testers present and rate the samples, and each tester is asked to evaluate the subjective consistency on a 1-5 Likert scale.

0

10

20

30

40

50

Timestep

0.01

0.02

0.03

Loss

0

10

20

30

40

50

Timestep

0.01

0.02

0.03

Loss

1.00

1.10

1.20

1.30

1.40

1.50

PESQ

1.21

1.22

1.34

1.15

1.46

Unprocessed

Dis

Noisy

Estimated

Mild

Clean

Our subjective evaluation tests are crowd-sourced and conducted by 15 volunteers. The screenshots of instructions for testers have been shown in Figure 7 and Figure 8. We paid $10 to participants hourly and totally spent about $300 on participant compensation.

The MOS results with the 95% confidence interval are shown in Table 2. we observe that: (1) Our method surpasses all baselines, demonstrating the strong ability of the proposed framework in synthesizing natural speech; (2) Our model can synthesize consistent speech to the golden speech, which is aligned with our motivation for algorithm design. It's also exciting to see that DOSE yields similar scores to methods with specific condition-injecting strategies (i.e., DiffuSE, CDiffuSE, and SGMSE) on Similarity MOS.

A.3 Adaptive prior analysis

We now investigate the significance of the proposed adaptive prior (§4.1) and show why we need to use a mild version (Eq. (12)) of the condition factor.

We design three variants to investigate the effect of different condition optimizer settings on denoising performance. These variants are: (a) applying adaptive prior with the noisy speech; (b) applying adaptive prior with the estimated one (from the deterministic model); (c) applying adaptive prior with the mild condition (Eq. (12)). To control variables, we use an unsupervised diffusion model with adaptive priors. We conduct experiments on the matched VB dataset and mismatched CHIME-4 dataset respectively, and our results are shown in Figure 9.

As shown in Figure 9 (left), we plot the one-step loss on matched VB and obtain the following observations: (1) The trend of loss curves is in line with our analysis in §4.1 that we need to find a trade-off timestep for better performance. (2) Equipping the unconditional diffusion enhancement model with the adaptive prior technique has a certain denoising ability but is inferior to its counterpart discriminative model in matched scenarios. We attribute the second phenomenon to the limited denoising capacity of the unconditional diffusion enhancement models (cf. [9, 21]).

Although the performance of the unconditional diffusion enhancement model equipped with adaptive prior is mediocre in matched scenarios, as illustrated in Figure 9 (mid), it exhibits greater stability than discriminative models in mismatched scenarios. To verify the influence of different priors, we compare their PESQ on mismatched CHIME-4, shown in Figure 9 (right). We see that: (1) The deterministic model fails in mismatched scenarios and generates samples that are even worse than the unprocessed ones; (2) The diffusion enhancement model has strong generalizability and performs significantly better than the deterministic model; (3) The diffusion enhancement model loses its capability when using the estimated speech as the condition factor; and (4) Although the estimated speech is worse than the unprocessed one, the diffusion enhancement model equipped with a mild version of the condition factor achieves the best performance. This implies that the estimated speech can provide additional complementary guidance to the diffusion model, and the model can adaptively 'separate the wheat from the chaff'. Thus, using a mild version of the condition factor is important and necessary.

In summary, our research shows the strong generalizability of the diffusion enhancement model. However, we also find that the unconditional diffusion enhancement model has mediocre performance. This suggests that relying solely on the adaptive prior technique is not be sufficient, further emphasiz-

20k

40k

60k

80k

100k

120k

Step

0

0.02

0.04

0.06

0.08

0.1

0.12

Loss

curve



Condition_ori



Uncondition_ori



Uncondition_new



Condition_new

ing the importance of the diffusion dropout operation - training conditional diffusion enhancement models in a supervised manner.

A.4 Training objective investigation

We now examine the effect of our new training objective and demonstrate the necessity of using it.

Let's recall the sampling process of DOSE. In the first step, we generate a relatively good estimation of the clean speech. In the second step, we use DOSE with a small timestep to generate a better one. We plot the relationship between ∆ ϵ and ∆ x 0 in Figure 10. Specifically, we fix the ∆ x 0 as a constant and use Eq. (15) to calculate the corresponding ∆ ϵ . This experiment aims to show how effort for calibration in x -space is equivalent to that in ϵ -space. From Figure 10 we see that, as t approaches zero, small changes in x -space amplify the implied prediction in ϵ -space. There is a nearly 100-fold difference between the values of ∆ ϵ and ∆ x 0 . This implies that the efforts of the diffusion enhancement model at small timesteps become negligible, causing diffusion models to lose their ability to recover natural-sounding speech from the defective speech estimated in the small timestep.

0

20

40

Timestep

t

10 0

10 1

10 2

We also plot the training loss curves in Figure 11. After substituting the training objective from ϵ -space to x -space, we observe that the loss change becomes more significant. This demonstrates more effective participation of conditioning factors in diffusion training: training diffusion enhancement model with this new objective allows for easier and more effective exploitation of conditioning factors.

A.5 Complexity analysis & why use two-steps generation?

In this subsection, we explain why we use two steps to generate speech.

Why not one-step speech generation? According to §4.4, we can generate speech in one shot using a trained conditional diffusion enhancement model to reduce error accumulation [21]. This one-step speech generation provides an appealing performance (Table 3). However, if we consider the diffusion model training as a multi-task paradigm, the denoising task at a smaller timestep t is easier than that at a larger timestep [21]. Correspondingly, the primary estimation error occurs at the large timestep area and directly estimating clean speech at a large timestep will result in sub-optimal performance [28]. Meanwhile, we can't choose a small timestep t as it will lead to the mismatched problem discussed in §4.1.

Since the conditional diffusion model can learn vital information from both the intermediate-generated and noisy speech - the estimated speech can provide complementary guidance to the diffusion model (Appendix A.3) - allowing us to further improve the result by generating speech with multiple steps.

0

10

20

30

40

50

Timestep

0.005

0.010

0.015

Loss

0

10

20

30

40

50

Timestep

0.005

0.010

0.015

Loss

0

10

20

30

40

50

Timestep

0.005

0.010

0.015

Loss

Unprocessed

Dis

Noisy

Esti

Mild

Clean

Hyper-parameter selection. Basically, we need to set optimal hyper-parameters by evaluating the performance with a small batch of data. Suppose we choose the number of sampling steps as K ( K < T ), and the amount of test data as N , then the computational complexity of the grid search is O ( NT ! / ( T -K )!) . Since T is always set as a large number in the diffusion model's setup, the complexity introduced by choosing a large K is often unmanageable.

Trivial solution. We have a simple alternative solution: defining the hyper-parameters empirically (e.g., equal intervals [7]). We present speech quality comparisons between empirically defined (fixed) and handpicked hyper-parameters in Table 3. As shown, the conditional diffusion enhancement model with handpicked hyper-parameters performs better than that with empirically defined hyperparameters.

Although fixed hyper-parameters have inferior performance compared to handpicked ones, they still show appealing performance compared to prevailing diffusion enhancement baselines. Therefore, in situations where we can't evaluate the model in advance, we can use empirically defined hyperparameters instead.

A.6 Parameter sensitivity

There are two groups of hyper-parameters that are critical to model performance: the dropout probability p for model training and two timesteps τ 1 , τ 2 for model inference. In this subsection, we conduct parameter sensitivity experiments to investigate how these hyper-parameters affect the model's performance.

Dropout probability p . We vary the dropout probability p , setting it to { 0 . 0 , 0 . 5 , 0 . 9 , 1 . 0 } , and plot the one-step loss on matched VB (Figure 12) and mismatched CHIME-4 (Figure 13) respectively. Please note that the figure with p = 1 has been excluded since it degenerates to a deterministic mapping-based model and the performance remains unchanged across all timesteps.

From Figure 12, we see that: (1) The proposed conditional diffusion enhancement model works. Compared to the deterministic mapping-based model, our model performs slightly better in matched scenarios and significantly better in mismatched scenarios; (2) When intermediate-generated speech x t is unreliable (large t ), the condition factor y plays a dominant role. (3) As the dropout probability increases, the model focuses more on the condition factor, while when the dropout probability is small, the loss curve oscillates when t gets large.

From Figure 13, We find that: when using a higher dropout probability (such as p = 0 . 5 and p = 0 . 9 ), the model can generally achieve better generalizability. Note that if the dropout probability is set too high, the diffusion enhancement model will rely solely on the condition factor to estimate the clean

0

10

20

30

40

50

Timestep

0.005

0.010

0.015

0.020

Loss

0

10

20

30

40

50

Timestep

0.005

0.010

0.015

0.020

Loss

0

10

20

30

40

50

Timestep

0.005

0.010

0.015

0.020

Loss

Unprocessed

Dis

Noisy

Esti

Mild

Clean

50

40

30

20

10

1

50

40

30

20

10

1

50

40

30

20

10

1

50

40

30

20

10

1

50

40

30

20

10

1

50

40

30

20

10

1

50

40

30

20

10

1

50

40

30

20

10

1

2.35

2.40

2.45

2.50

2.55

1.10

1.20

1.30

1.40

1.50

0.70

0.75

0.80

0.85

0.930

0.932

0.934

speech - and the diffusion enhancement model will degenerate into a deterministic model, losing its generalizability.

Timesteps τ 1 and τ 2 . Considering the computational complexity of the 'one-step' grid search, we search optimal hyperparameters with a slightly larger step. Specifically, we select both optimal τ 1 and τ 2 from the predefined set { 1 , 5 , 10 , 15 , 20 , 25 , 30 , 35 , 40 , 45 , 50 } . We show the PESQ and STOI performance of different combinations in Figure 14. We can observe that τ 1 > τ 2 (excluding STOI on matched VB dataset, and the difference is negligible: 0 . 930 ∼ 0 . 934 ) will lead to better performance, which is in line with our analysis (§4.4).

A.7 Visual case of excessive suppression

We present several visual cases of excessive suppression.

A.8 Visual case of error accumulation

We present several visual cases of error accumulation.

W

A.9 Discussion

To help readers better understand our approach, we analyze the reasons behind the better generalizability of diffusion enhancement models compared to deterministic mapping-based models (from the robust training perspective), explain why we use 0.5 in the mild version of the condition factor, and discuss the broader impacts of speech enhancement methods.

A.9.1 Generalization analysis

Given that full-step diffusion enhancement models have better generalizability than deterministic models, we now delve into the following question: if we are just using diffusion models as one-step (or two-step) denoisers, what accounts for their enhanced performance compared to deterministic mapping-based models?

To answer this question, we need to point out that training a diffusion model can be considered equivalent to training a multi-task paradigm. This involves training a model with shared parameters on multiple levels of Gaussian noise concurrently. Recent research [21] has demonstrated that the full training process of diffusion models leads to significantly improved one-shot denoising capabilities, which are more generalizable compared to previous works that trained standalone denoisers on a single noise level. Please refer to [21] (§5.2) for more details.

A.9.2 Why use 0.5 in the mild version of the condition factor? (reviewer boQC)

Employing an equal weight provides stability, yet the performance during instances with low SNR would be compromised (albeit still superior to direct utilization of y as a condition factor, unless the condition optimizer falters). One prospective solution involves introducing an additional adaptive strategy, i.e., c = αf θ ( y ) + (1 -α ) y . We can design an adaptive alpha predictor and hope it can output α based on the quality of raw condition and samples from the condition optimizer. For example, when the condition optimizer produces lower-quality samples, giving more weight to the original condition factor would make sense. Conversely, if the raw condition factor has a low SNR, emphasizing the generated counterpart could be more effective. However, implementing this idea practically is intricate.

Given our strong reliance on diffusion-enhanced models to enhance generalization, any new adaptive strategy must be generalizable. For instance, training an adaptive alpha predictor on the (seen) VB-dataset (high SNR & consistent condition optimizer performance) could lead the model to consistently output higher α values for fusion. Unfortunately, this auxiliary model might not effectively adapt to variations when evaluating the mismatched (unseen) CHIME-4 dataset (low SNR & potential condition optimizer challenges). To this end, we might need other techniques such as data augmentation and adversarial training to improve its generalizability and robustness. This creates a dilemma: harnessing the speech diffusion model for overarching speech noise reduction generalization while simultaneously necessitating a pre-established generalized model to facilitate its implementation. So far, despite our efforts to train a strong alpha predictor, progress has been limited (the alpha predictor is still not generalizable, and the new system has no significant performance improvement over DOSE).

A.9.3 Broader impacts

As speech enhancement technology continues to advance and become more prevalent, it's important to consider its broader impacts.

Positive impacts. The impact of speech enhancement technology on real-life situations, particularly for individuals with hearing impairments, cannot be overstated. Hearing aids have long been the primary solution for those with hearing loss, but they are not always effective in noisy environments or for certain types of hearing loss. Speech enhancement technology can greatly improve speech intelligibility and communication for hearing aid users. For example, some hearing aids have AI-powered speech enhancement that boosts speech quality.

In addition to the benefits for individuals with hearing impairments, speech enhancement technology also has significant implications for various applications. In transportation, clearer and more intelligible speech can improve communication between pilots and air traffic control, leading to safer and more efficient air travel. In industry, speech enhancement can improve communication on

noisy factory floors, leading to increased productivity and safety. In educational settings, speech enhancement can improve student comprehension and engagement during lectures and presentations.

Negative impacts. While speech enhancement technology has the potential to greatly improve communication and speech intelligibility, one potential concern is that speech enhancement could modify the semantic content of speech, potentially misleading listeners. Thus, it's important for developers of speech enhancement technology to consider this potential negative effect and work towards creating trustworthy systems.

A.10 Experimental details

Speech preprocessing. We process the speech waveform at a 16 kHz sampling rate. To maintain dimensionality consistency within mini-batches, we pad each utterance to 2 seconds (32000 points) using a zero-padding technique.

Basic architecture. To make a fair comparison, we use DiffWave [7] as the basic architecture following [4, 9] - the only difference being the change in the way of condition-injecting since most speech enhancement methods will directly use noisy speech as the condition factor, rather than Mel-spectrogram. We concatenate the condition factor with the intermediate-generated sample along the channel dimension as the model's input. Specifically, the network is composed of 30 residual layers with residual channels 128. We use a bidirectional dilated convolution (Bi-DilConv) with kernel size 3 in each layer. We sum the skip connections from all residual layers. The total number of trainable parameters is 2.31M, slightly smaller than naive DiffWave (2.64M). Please refer to [7] and our code for more details.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a new method for providing noisy-signal conditioning information (y) to the diffusion steps of a diffusion-based speech enhancement algorithm.  Three specific innovations are proposed: (1) improve dependence of x_0 on y by dropping out x_t, at random with Bernoulli probability p.  (2) In order to make innovation #3 possible, train each x_t explicitly using MSE of the implied x_0, rather than MSE of the error \\epsilon.  (3) For greater efficiency, generate x_0 in only two steps, selected from the T-step trained diffusion process using validation data.\n\nSTRENGTHS:\nThis paper proposes a diffusion-based speech enhancement with both improved performance and (because of the two-step inference) improved efficiency.  Both the performance gains and the efficiency gains are theoretically well motivated and empirically demonstrated.\n\nWEAKNESSES:\nClarity:  (1) The dropout described in Eq. (13) is then not referenced again for the rest of the paper.  I think that's because Eq. (13) affects the T-step training process, while equations (14)-(18) are about the proposed reduction of inference from T steps to 2 steps.  But the division into training and testing is never really made explicit.  I think this is because the derivations assume that the reader has fully understood Figure 1, but Figure 1 cannot be fully understood until one has first understood the algorithm; I had to go back and examine Figure 1 after reading the derivations in order to know what's going on.  (2) I think that Eq. (17) should have an integral over dx_\\tau.  The algorithm might use the one-point approximation, but you've done a great job up to this point of keeping the theoretically-required integral in your equations, it seems a shame to abandon it here.  (3) Shouldn't the first term on the RHS in Eq. (18) be p_\\theta(\\hat{x}_{\\tau_2}|...)?  Or are you trying to say that p(x_{\\tau_2}|...) = p(\\hat{x}_0|...)?  That doesn't seem correct, since it misses the interpolation step.  ... also, I think there should be an integration over d\\hat{x}_{\\tau_2}.\n\nQUESTIONS:\nSignificance:  Dropout enhances accuracy.  The two-step inference enhances efficiency.  Is there any interaction between these two things?  It seems like the dropout might reduce the accuracy degradation that two-step inference would otherwise incur; is that true?  Similarly, why two -- is two the optimum number of steps in any way?  Results show pretty clearly that two is better than one, but is there any theoretical reason for that?  The theory seems to predict simply that the more steps you have, the more accurate is the inference.\n\n\"Diffusion enhancement methods have better generalizability than deterministic methods\" -- By \"deterministic methods\" I think you mean DiffWave.  In what sense is that a deterministic method?\n\np. 6 Considering the equivalently -> Considering the equivalence\n\np. 8 We contribute -> We attribute\n\nLIMITATIONS:\nLimitations and ethical considerations are not explicitly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a solution to the problem of condition-collapse in denoising diffusion models for speech enhancement by introducing the adaptive prior and sample dropout techniques. The paper is well-written and provides valuable insights into the functioning of the denoising diffusion probabilistic model for speech enhancement. While the concept of adaptive priors for conditioning the generative process is not entirely novel and has been explored in vision-related tasks, the authors' application of this technique, along with theoretical analysis, is intriguing. The authors also justify their choices regarding noise scheduling and propose a faster method for sampling clean speech from the trained model based on intermediate approximation.\n\nThe experimental strategy adopted in this study assesses the generalization capability of the proposed model using objective metrics such as STOI and PESQ, as well as subjective scores like CBAK and COVL. The effectiveness of the mixed conditioning strategy is demonstrated through the analysis of spectrogram plots, which is an interesting observation. It is important to note that while the proposed technique may not consistently outperform baseline methods across all scenarios, it does excel in specific matched scenarios.\n\nSTRENGTHS:\nIn this paper, a novel approach is introduced to address the problem of condition collapse in diffusion models for speech enhancement. The authors propose the utilization of adaptive prior and sample dropout techniques, which offer an interesting and promising solution to this issue. Furthermore, the paper delves into the theoretical aspects of clean speech recovery, shedding light on the conditions and constraints necessary for successful restoration.\n\nOne notable contribution of this work is the development of a fast sampling technique, which not only proves effective in the context of speech enhancement but also holds potential for application in other conditional generation tasks. This aspect highlights the broader implications and versatility of the proposed approach.\n\nTo evaluate the efficacy of the proposed technique, the authors conduct comprehensive experiments and compare their approach against various diffusion-based models specifically designed for speech enhancement. The experiments are meticulously designed and executed, providing a thorough analysis of the results. This level of detail and scrutiny enhances the credibility of the proposed approach and contributes to a better understanding of its strengths and limitations.\n\nWEAKNESSES:\nOne main weakness in my opinion is the understanding of Proposition 2. I do not understand how a diffusion model has high probability of recovering ground-truth if the inequality 23 from appendix holds. I might be missing some theoretical analysis on diffusion models but I am giving the benefit of doubt to the authors. \n\nThe dataset section lacks the details about the type and level of noise present in Chime and Voicebank corpora. Another issue is in the experiment section where the authors have shown impressive performance on a wide-range of metrics. I believe that WER can be easily calculated in the evaluation and is a very straight-forward way to compare the noise-reduction performance.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nThe conclusion section mentions that the model is sensitive to the choice of dropout probability and the sampling time-indices.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on a new approach in the field of speech enhancement called DOSE, which effectively addresses the problem of conditional collapse by incorporating conditional information into a diffusion enhancement model.DOSE employs two effective conditional enhancement techniques that can significantly improve the performance of the model while ensuring its efficiency. The paper demonstrates the efficiency and effectiveness of the method through comprehensive experiments on a benchmark dataset.\n\nSTRENGTHS:\n1. In this paper, the authors propose an Adaptive Prior, aimed at incorporating conditioning information during the generation process, thereby ensuring greater consistency in the generated samples and augmenting the efficacy of speech enhancement.\n\n2. The paper elucidates that by employing dropout operations during the training phase, the model is compelled to prioritize conditioning elements, which efficaciously mitigates the conditioning collapse issue. This methodology engenders a dependency on conditioning information within the model during generation, culminating in the synthesis of more coherent speech.\n\n3. The authors undertake a comparative analysis between DOSE and extant diffusion-based speech enhancement models on two public datasets. Notably, DOSE attains superior performance with an exceedingly limited number of sampling steps, which substantiates the efficacy of the proposed method.\n\nWEAKNESSES:\nThis work realizes a two-step sampling process. The designs in sampling process include:\n\n1. The parameter $T_1$ denotes an intersection point, enabling a shallow reverse process by leveraging the noisy speech y. Similar mechanism can be visited in text-to-speech synthesis, DiffSinger (AAAI, 2022) and image editing, SDEdit (ICLR 2022).\n2. The coarse estimation $\\hat{x}_{0}$ denoised from $T_1$ is first mixed with y as adaptive prior,  and then corrupted with a shallow forward process, which generates the latent representation at $T_2$.\n3. The high-quality estimation $\\hat{x}_{0}$ can recovered from $T_2$ in one-step.\n\nTwo designs in training include:\n\n1. The estimation target in training objective is set as clean waveform instead of noise.\n2. The $x_t$ is randomly dropped out to force the model to rely on the conditioning information $y$.\n\nQuestions:\n\n1. I can understand the dropout operation in training is helpful to utilizing the conditioner y. However, the adaptive prior is used to obtain the latent representation at T_2. I think it is manipulating the sampling trajectory. What is the relationship with condition optimizer? The unchanged noisy observation y has been provided as the condition. The abstract claims two condition-augmentation techniques.\n2. I do not understand the comparison study of adaptive prior analysis very well. What are the three variants when computing the adaptive prior \\hat{x}_{0}? Why is an unconditional diffusion model used? To compare the design of Eq. 12, the conditional generation should be fixed.\n3. This work mentions the error accumulation of diffusion models. However, the high-quality generation of diffusion models is usually guaranteed by its iterative refinement mechanism. In this work, 50 time steps are used in training, while 2 steps are used in sampling. I am curious about the results of increasing the number of sampling steps.\n\nExperiments:\n\n1. Subjective tests MOS and SMOS are conducted. But where is the demo page showing the generated samples?\n2. DiffWave does not claim one-step mapping for either waveform generation or denoising. Is it good to use it as a discriminative model?\n3. Generative baseline models such as DiffuSE, CDiffuSE, and SGMSE have been changed to keep the uniform model architecture and training method with this work. Will this cause performance decrease in their methods? What are the results of those unchanged baseline models?\n\nOthers:\n\n1. The thesis writing looks over-complicated. Moreover, the adaptive prior does not mean condition optimizer from my perspective. It is manipulating the sampling trajectory with observation y.\n2. I suggest showing the training and sampling algorithms in the main content instead of in appendix.\n\nQUESTIONS:\nMy detailed questions are as described above.\n\nLIMITATIONS:\nThere are limitations to its use in real-time scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes a novel model-agnostic approach called DOSE for speech enhancement (SE) using denoising diffusion probabilistic models (DDPMs). In this paper, the authors focus on addressing the challenge of incorporating condition information into DDPMs with two efficient condition-augmentation techniques. Based on the experimental results, the authors claim that the proposed method obtain significant improvements in high-quality and stable speech generation, consistency with the condition factor, and efficiency.

STRENGTHS:
1. The proposed method shows good generalization ability with good performance in both matched and mismatched scenarios. 

2. This paper shows detailed experimental results and a comprehensive comparison with existing diffusion enhancement methods and deterministic mapping-based method enhancement methods. 

3. This paper provides a proper introduction to the problem of condition collapse in generative speech enhancement. 

4. This paper is well-written and the flow of the writing is natural so it was easy to read and follow.

WEAKNESSES:
1. To replicate the experiments, more training details and configuration should be provided.  

2. It would be better if there were some qualitative analysis in the experiment section.

QUESTIONS:
Can authors conduct the ablation study to present and analyze the effectiveness of DOSE.

LIMITATIONS:
The authors do not analyze the limitation of this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
The authors propose a model-agnostic method called DOSE that employs two efficient condition-augmentation techniques to incorporate condition information into DDPMs for SE. Experiments demonstrate that the approach yields substantial improvements in high-quality and stable speech generation.

STRENGTHS:
1. In-depth presentation on diffusion SE, including formulation and methodology.
2. Good results. The authors compare different SE baselines and demonstrate the SOTA results.

WEAKNESSES:
1. It seems that the authors adopt the adaptive prior. Does it only use in the inference process? What is the difference from the adaptive prior in PriorGrad? 
2. Why use similarity MOS to evaluate the enhancement model, and what's the difference from MOS?
3. How do you choose p for diffusion dropout? It lacks evaluation and ablation studies on p, which is an important parameter for the proposed diffusion dropout operation.

QUESTIONS:
If there is a training-inference mismatch? You randomly drop x_t in training, I wonder if it causes the mismatch as the dropout usually does.

LIMITATIONS:
/
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10