SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 2. (cf. §4.1): Let  $\pmb{x}$  be the clean sample,  $\pmb{y}$  be its corresponding noisy one, and  $\pmb{x}'$  be any neighbor from the neighbor set  $S(\pmb{x})$ . Then diffusion enhancement models can recover  $\pmb{x}$  with a high probability if the following inequality is satisfied:
where  $\sigma_t^2 = \frac{1 - \bar{\alpha}_t}{\bar{\alpha}_t}$  is the variance of the Gaussian noise added at timestep  $t$  in the forward diffusion process.
Proof. The main idea is to prove that any point  $x'$  quite similar but different to the ground-true speech  $x$  should have a lower density than  $x$  in the conditional distribution so that the diffusion enhancement models can recover  $x$  with a high probability. In other words, we should have:
$$
p \left(\boldsymbol {x} _ {0} = \boldsymbol {x} \mid \boldsymbol {x} _ {t} = \boldsymbol {y} _ {t}\right) > p \left(\boldsymbol {x} _ {0} = \boldsymbol {x} ^ {\prime} \mid \boldsymbol {x} _ {t} = \boldsymbol {y} _ {t}\right) \tag {24}
$$
According to Bayes' theorem, we have:
$$
\begin{array}{l} p \left(\boldsymbol {x} _ {0} = \boldsymbol {x} \mid \boldsymbol {x} _ {t} = \boldsymbol {y} _ {t}\right) = \frac {p \left(\boldsymbol {x} _ {0} = \boldsymbol {x} , \boldsymbol {x} _ {t} = \boldsymbol {y} _ {t}\right)}{p \left(\boldsymbol {x} _ {t} = \boldsymbol {y} _ {t}\right)} \tag {25} \\ = p (\boldsymbol {x} _ {0} = \boldsymbol {x}) \cdot \frac {p (\boldsymbol {x} _ {t} = \boldsymbol {y} _ {t} | \boldsymbol {x} _ {0} = \boldsymbol {x})}{p (\boldsymbol {x} _ {t} = \boldsymbol {y} _ {t})}. \\ \end{array}
$$
Applying Eq. (25) to Eq. (24), we obtain:
$$
\begin{array}{l} p (\boldsymbol {x}) \cdot \frac {1}{\sqrt {\left(2 \pi \sigma_ {t} ^ {2}\right) ^ {d}}} \exp \frac {- \| \boldsymbol {x} - \boldsymbol {y} \| _ {2} ^ {2}}{2 \sigma_ {t} ^ {2}} > p \left(\boldsymbol {x} ^ {\prime}\right) \cdot \frac {1}{\sqrt {\left(2 \pi \sigma_ {t} ^ {2}\right) ^ {d}}} \exp \frac {- \| \boldsymbol {x} ^ {\prime} - \boldsymbol {y} \| _ {2} ^ {2}}{2 \sigma_ {t} ^ {2}} \tag {26} \\ \Leftrightarrow \log \left(\frac {p (\boldsymbol {x})}{p (\boldsymbol {x} ^ {\prime})}\right) > \frac {1}{2 \sigma_ {t} ^ {2}} \left(\| \boldsymbol {x} - \boldsymbol {y} \| _ {2} ^ {2} - \| \boldsymbol {x} ^ {\prime} - \boldsymbol {y} \| _ {2} ^ {2}\right), \quad \forall \boldsymbol {x} ^ {\prime} \in S (\boldsymbol {x}), \\ \end{array}
$$
and the proof is now complete.
A.2 Subjective evaluation
We conduct two types of Mean Opinion Score (MOS) tests to verify the quality of synthesized audio through human evaluation.
Figure 8: Screenshot of Similarity MOS test.
How similar is this recording to the reference audio?
Please focus on the similarity of the style (specker identity, emotion and prosody) to the reference, and ignore the audio quality
What's your favorite audio:
choose one audio
$\mathbb{O}$  Reference Audio
$\bigcirc$  Testing Audio
You selected Reference Audio
What's your rating/ of voice
Excellent-Completely natural speech -5
4.5
Good-Mostly natural speech -4
3.5
Fair-Equally natural and unnatural speech -3
2.5
Poor-Mostly unnatural speech -2
1.5
Bad-Completely unnatural speech -1
Table 2: MOS tests under different scenarios.
Method	Scenarios	MOS↑	Similarity MOS↑	Scenarios	MOS↑	Similarity MOS↑
Unprocessed	3.60±0.31	3.33±0.34	3.30±0.30	3.10±0.32
DiffWave (dis)	Matched	3.80+0.20	3.75+0.42	Mismatched	2.10-1.20	1.00-2.10
DiffuSE	3.40±0.20	4.17+1.84	3.00-0.30	3.33+0.23
CDiffuSE	3.85+0.25	4.12+1.79	2.55-0.75	3.42+0.32
SGMSE	3.65+0.05	3.95+0.62	2.83+0.33	3.41+0.25
DR-DiffuSE	3.80+0.20	3.84+0.51	2.48-0.82	1.45-1.65
DOSE	4.05±0.29	4.35±0.21	3.48±0.26	3.17±0.23
Naturalness. For audio quality evaluation, we conduct the MOS (mean opinion score) tests and explicitly instruct the raters to "focus on examining the audio quality and naturalness, and ignore the differences of style (timbre, emotion, and prosody)". The testers present and rate the samples, and each tester is asked to evaluate the subjective naturalness on a 1-5 Likert scale.
Consistency. For audio consistency evaluation, we explicitly instruct the raters to "focus on the similarity of the speech (content, timbre, emotion, and prosody) to the reference, and ignore the differences of audio quality". This is slightly different from the original definition of SMOS for speech synthesis. In the SMOS (similarity mean opinion score) tests, we pair each synthesized utterance with a ground truth utterance to evaluate how well the synthesized speech matches that of the target speaker. The testers present and rate the samples, and each tester is asked to evaluate the subjective consistency on a 1-5 Likert scale.
Figure 9: Performance of the unconditional diffusion enhancement model with adaptive prior. From left to right: (1) each step loss on matched VB; (2) each step loss on mismatched CHIME-4; (3) PESQ comparison for different priors on mismatched CHIME-4.
Our subjective evaluation tests are crowd-sourced and conducted by 15 volunteers. The screenshots of instructions for testers have been shown in Figure 7 and Figure 8. We paid $10 to participants hourly and totally spent about $300 on participant compensation.
The MOS results with the  $95\%$  confidence interval are shown in Table 2. we observe that: (1) Our method surpasses all baselines, demonstrating the strong ability of the proposed framework in synthesizing natural speech; (2) Our model can synthesize consistent speech to the golden speech, which is aligned with our motivation for algorithm design. It's also exciting to see that DOSE yields similar scores to methods with specific condition-injecting strategies (i.e., DiffuSE, CDiffuSE, and SGMSE) on Similarity MOS.
A.3 Adaptive prior analysis
We now investigate the significance of the proposed adaptive prior (§4.1) and show why we need to use a mild version (Eq. (12)) of the condition factor.
We design three variants to investigate the effect of different condition optimizer settings on denoising performance. These variants are: (a) applying adaptive prior with the noisy speech; (b) applying adaptive prior with the estimated one (from the deterministic model); (c) applying adaptive prior with the mild condition (Eq. (12)). To control variables, we use an unsupervised diffusion model with adaptive priors. We conduct experiments on the matched VB dataset and mismatched CHIME-4 dataset respectively, and our results are shown in Figure 9.
As shown in Figure 9 (left), we plot the one-step loss on matched VB and obtain the following observations: (1) The trend of loss curves is in line with our analysis in §4.1 that we need to find a trade-off timestep for better performance. (2) Equipping the unconditional diffusion enhancement model with the adaptive prior technique has a certain denoising ability but is inferior to its counterpart discriminative model in matched scenarios. We attribute the second phenomenon to the limited denoising capacity of the unconditional diffusion enhancement models (cf. [9, 21]).
Although the performance of the unconditional diffusion enhancement model equipped with adaptive prior is mediocre in matched scenarios, as illustrated in Figure 9 (mid), it exhibits greater stability than discriminative models in mismatched scenarios. To verify the influence of different priors, we compare their PESQ on mismatched CHIME-4, shown in Figure 9 (right). We see that: (1) The deterministic model fails in mismatched scenarios and generates samples that are even worse than the unprocessed ones; (2) The diffusion enhancement model has strong generalizability and performs significantly better than the deterministic model; (3) The diffusion enhancement model loses its capability when using the estimated speech as the condition factor; and (4) Although the estimated speech is worse than the unprocessed one, the diffusion enhancement model equipped with a mild version of the condition factor achieves the best performance. This implies that the estimated speech can provide additional complementary guidance to the diffusion model, and the model can adaptively "separate the wheat from the chaff". Thus, using a mild version of the condition factor is important and necessary.
In summary, our research shows the strong generalizability of the diffusion enhancement model. However, we also find that the unconditional diffusion enhancement model has mediocre performance. This suggests that relying solely on the adaptive prior technique is not be sufficient, further emphasizing
Figure 11: Investigation of the training objective. we compare the  $\epsilon$ -space and  $x$ -space loss curves.
ing the importance of the diffusion dropout operation - training conditional diffusion enhancement models in a supervised manner.
A.4 Training objective investigation
We now examine the effect of our new training objective and demonstrate the necessity of using it.
Let's recall the sampling process of DOSE. In the first step, we generate a relatively good estimation of the clean speech. In the second step, we use DOSE with a small timestep to generate a better one. We plot the relationship between  $\Delta \epsilon$  and  $\Delta x_0$  in Figure 10. Specifically, we fix the  $\Delta x_0$  as a constant and use Eq. (15) to calculate the corresponding  $\Delta \epsilon$ . This experiment aims to show how effort for calibration in  $\mathbf{x}$ -space is equivalent to that in  $\epsilon$ -space. From Figure 10 we see that, as  $t$  approaches zero, small changes in  $\mathbf{x}$ -space amplify the implied prediction in  $\epsilon$ -space. There is a nearly 100-fold difference between the values of  $\Delta \epsilon$  and  $\Delta x_0$ . This implies that the efforts of the diffusion enhancement model at small timesteps become negligible, causing diffusion models to lose their ability to recover natural-sounding speech from the defective speech estimated in the small timestep.
Figure 10: The relationship between  $\Delta \pmb{x}_0 = |\hat{\pmb{x}}_0 - \pmb{x}_0|$  (orange) and  $\Delta \epsilon = |\hat{\epsilon} -\epsilon |$  (blue).
We also plot the training loss curves in Figure 11. After substituting the training objective from  $\epsilon$ -space to  $\mathbf{x}$ -space, we observe that the loss change becomes more significant. This demonstrates more effective participation of conditioning factors in diffusion training: training diffusion enhancement model with this new objective allows for easier and more effective exploitation of conditioning factors.
A.5 Complexity analysis & why use two-steps generation?
In this subsection, we explain why we use two steps to generate speech.
Why not one-step speech generation? According to §4.4, we can generate speech in one shot using a trained conditional diffusion enhancement model to reduce error accumulation [21]. This one-step speech generation provides an appealing performance (Table 3). However, if we consider the diffusion model training as a multi-task paradigm, the denoising task at a smaller timestep  $t$  is easier than that at a larger timestep [21]. Correspondingly, the primary estimation error occurs at the large timestep area and directly estimating clean speech at a large timestep will result in sub-optimal performance [28]. Meanwhile, we can't choose a small timestep  $t$  as it will lead to the mismatched problem discussed in §4.1.
Since the conditional diffusion model can learn vital information from both the intermediate-generated and noisy speech – the estimated speech can provide complementary guidance to the diffusion model (Appendix A.3) – allowing us to further improve the result by generating speech with multiple steps.
Table 3: Ablation study for two-step speech generation.
Method	Scenarios	PESQ↑	STOI(%)↑	Scenarios	PESQ↑	STOI(%)↑
Unprocessed	1.97	92.1	1.21	71.5
DOSE (fixed 1 step)	2.47±0.50	93.0±0.90	1.38±0.17	82.8±11.3
DOSE (handpicked 1 step)	2.50±0.53	93.4±1.30	Mismatched	1.51±0.31	86.4±15.7
DOSE (fixed 2 steps)	2.48±0.51	93.1±0.05	1.44±0.23	83.6±12.1
DOSE (handpicked 2 steps)	2.56±0.59	93.6±1.50	1.52±0.32	86.6±15.1
Figure 12: Performance of the conditional diffusion enhancement model with different dropout probability in VB. From left to right: (1) without dropout; (2) dropout  $50\%$ ; (3) dropout  $90\%$ .
Hyper-parameter selection. Basically, we need to set optimal hyper-parameters by evaluating the performance with a small batch of data. Suppose we choose the number of sampling steps as  $K$  ( $K < T$ ), and the amount of test data as  $N$ , then the computational complexity of the grid search is  $\mathcal{O}(NT! / (T - K)!)$ . Since  $T$  is always set as a large number in the diffusion model's setup, the complexity introduced by choosing a large  $K$  is often unmanageable.
Trivial solution. We have a simple alternative solution: defining the hyper-parameters empirically (e.g., equal intervals [7]). We present speech quality comparisons between empirically defined (fixed) and handpicked hyper-parameters in Table 3. As shown, the conditional diffusion enhancement model with handpicked hyper-parameters performs better than that with empirically defined hyperparameters.
Although fixed hyper-parameters have inferior performance compared to handpicked ones, they still show appealing performance compared to prevailing diffusion enhancement baselines. Therefore, in situations where we can't evaluate the model in advance, we can use empirically defined hyperparameters instead.
A.6 Parameter sensitivity
There are two groups of hyper-parameters that are critical to model performance: the dropout probability  $p$  for model training and two timesteps  $\tau_{1},\tau_{2}$  for model inference. In this subsection, we conduct parameter sensitivity experiments to investigate how these hyper-parameters affect the model's performance.
Dropout probability  $p$ . We vary the dropout probability  $p$ , setting it to  $\{0.0, 0.5, 0.9, 1.0\}$ , and plot the one-step loss on matched VB (Figure 12) and mismatched CHIME-4 (Figure 13) respectively. Please note that the figure with  $p = 1$  has been excluded since it degenerates to a deterministic mapping-based model and the performance remains unchanged across all timesteps.
From Figure 12, we see that: (1) The proposed conditional diffusion enhancement model works. Compared to the deterministic mapping-based model, our model performs slightly better in matched scenarios and significantly better in mismatched scenarios; (2) When intermediate-generated speech  $\boldsymbol{x}_t$  is unreliable (large  $t$ ), the condition factor  $y$  plays a dominant role. (3) As the dropout probability increases, the model focuses more on the condition factor, while when the dropout probability is small, the loss curve oscillates when  $t$  gets large.
From Figure 13, We find that: when using a higher dropout probability (such as  $p = 0.5$  and  $p = 0.9$ ), the model can generally achieve better generalizability. Note that if the dropout probability is set too high, the diffusion enhancement model will rely solely on the condition factor to estimate the clean
Figure 13: Performance of the conditional diffusion enhancement model with different dropout probability in CHIME-4. From left to right: (1) without dropout; (2) dropout  $50\%$ ; (3) dropout  $90\%$ .
Figure 14: Performance of two-step speech generation. From left to right: (1) PESQ on matched VB dataset; (2) STOI on matched VB dataset; (3) PESQ on mismatched CHIME-4 dataset; (4) STOI on mismatched CHIME-4 dataset.
speech – and the diffusion enhancement model will degenerate into a deterministic model, losing its generalizability.
Timesteps  $\tau_{1}$  and  $\tau_{2}$ . Considering the computational complexity of the "one-step" grid search, we search optimal hyperparameters with a slightly larger step. Specifically, we select both optimal  $\tau_{1}$  and  $\tau_{2}$  from the predefined set  $\{1,5,10,15,20,25,30,35,40,45,50\}$ . We show the PESQ and STOI performance of different combinations in Figure 14. We can observe that  $\tau_{1} > \tau_{2}$  (excluding STOI on matched VB dataset, and the difference is negligible:  $0.930 \sim 0.934$ ) will lead to better performance, which is in line with our analysis (§4.4).
A.7 Visual case of excessive suppression
We present several visual cases of excessive suppression.
Figure 15: Excessive suppression visualization (CHIME-4, DOSE). From left to right: (1) estimated condition; (2) 2 steps; (3) clean; (4) noisy speech.
A.8 Visual case of error accumulation
We present several visual cases of error accumulation.
Figure 16: Error accumulation visualization (VB, DOSE). From left to right: (1) full (50) steps; (2) 2 steps; (3) clean; (4) noisy speech.
A.9 Discussion
To help readers better understand our approach, we analyze the reasons behind the better generalizability of diffusion enhancement models compared to deterministic mapping-based models (from the robust training perspective), explain why we use 0.5 in the mild version of the condition factor, and discuss the broader impacts of speech enhancement methods.
A.9.1 Generalization analysis
Given that full-step diffusion enhancement models have better generalizability than deterministic models, we now delve into the following question: if we are just using diffusion models as one-step (or two-step) denoisers, what accounts for their enhanced performance compared to deterministic mapping-based models?
To answer this question, we need to point out that training a diffusion model can be considered equivalent to training a multi-task paradigm. This involves training a model with shared parameters on multiple levels of Gaussian noise concurrently. Recent research [21] has demonstrated that the full training process of diffusion models leads to significantly improved one-shot denoising capabilities, which are more generalizable compared to previous works that trained standalone denoisers on a single noise level. Please refer to [21] (\$5.2) for more details.
A.9.2 Why use 0.5 in the mild version of the condition factor? (reviewer boQC)
Employing an equal weight provides stability, yet the performance during instances with low SNR would be compromised (albeit still superior to direct utilization of  $\pmb{y}$  as a condition factor, unless the condition optimizer falters). One prospective solution involves introducing an additional adaptive strategy, i.e.,  $c = \alpha f_{\theta}(\pmb{y}) + (1 - \alpha)\pmb{y}$ . We can design an adaptive alpha predictor and hope it can output  $\alpha$  based on the quality of raw condition and samples from the condition optimizer. For example, when the condition optimizer produces lower-quality samples, giving more weight to the original condition factor would make sense. Conversely, if the raw condition factor has a low SNR, emphasizing the generated counterpart could be more effective. However, implementing this idea practically is intricate.
Given our strong reliance on diffusion-enhanced models to enhance generalization, any new adaptive strategy must be generalizable. For instance, training an adaptive alpha predictor on the (seen) VB-dataset (high SNR & consistent condition optimizer performance) could lead the model to consistently output higher  $\alpha$  values for fusion. Unfortunately, this auxiliary model might not effectively adapt to variations when evaluating the mismatched (unseen) CHIME-4 dataset (low SNR & potential condition optimizer challenges). To this end, we might need other techniques such as data augmentation and adversarial training to improve its generalizability and robustness. This creates a dilemma: harnessing the speech diffusion model for overarching speech noise reduction generalization while simultaneously necessitating a pre-established generalized model to facilitate its implementation. So far, despite our efforts to train a strong alpha predictor, progress has been limited (the alpha predictor is still not generalizable, and the new system has no significant performance improvement over DOSE).
A.9.3 Broader impacts
As speech enhancement technology continues to advance and become more prevalent, it's important to consider its broader impacts.
Positive impacts. The impact of speech enhancement technology on real-life situations, particularly for individuals with hearing impairments, cannot be overstated. Hearing aids have long been the primary solution for those with hearing loss, but they are not always effective in noisy environments or for certain types of hearing loss. Speech enhancement technology can greatly improve speech intelligibility and communication for hearing aid users. For example, some hearing aids have AI-powered speech enhancement that boosts speech quality.
In addition to the benefits for individuals with hearing impairments, speech enhancement technology also has significant implications for various applications. In transportation, clearer and more intelligible speech can improve communication between pilots and air traffic control, leading to safer and more efficient air travel. In industry, speech enhancement can improve communication on
noisy factory floors, leading to increased productivity and safety. In educational settings, speech enhancement can improve student comprehension and engagement during lectures and presentations.
Negative impacts. While speech enhancement technology has the potential to greatly improve communication and speech intelligibility, one potential concern is that speech enhancement could modify the semantic content of speech, potentially misleading listeners. Thus, it's important for developers of speech enhancement technology to consider this potential negative effect and work towards creating trustworthy systems.
A.10 Experimental details
Speech preprocessing. We process the speech waveform at a  $16\mathrm{kHz}$  sampling rate. To maintain dimensionality consistency within mini-batches, we pad each utterance to 2 seconds (32000 points) using a zero-padding technique.
Basic architecture. To make a fair comparison, we use DiffWave [7] as the basic architecture following [4, 9] – the only difference being the change in the way of condition-injecting since most speech enhancement methods will directly use noisy speech as the condition factor, rather than Mel-spectrogram. We concatenate the condition factor with the intermediate-generated sample along the channel dimension as the model's input. Specifically, the network is composed of 30 residual layers with residual channels 128. We use a bidirectional dilated convolution (Bi-DilConv) with kernel size 3 in each layer. We sum the skip connections from all residual layers. The total number of trainable parameters is 2.31M, slightly smaller than naive DiffWave (2.64M). Please refer to [7] and our code for more details.