SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenxin Tai $^{1}$ , Yue Lei $^{1}$ , Fan Zhou $^{1,2*}$ , Goce Trajcevski $^{3}$ , Ting Zhong $^{1,2}$ $^{1}$ University of Electronic Science and Technology of China  
 $^{2}$ Kashi Institute of Electronics and Information Industry  
 $^{3}$ Iowa State University
Speech enhancement (SE) aims to improve the intelligibility and quality of speech in the presence of non-stationary additive noise. Deterministic deep learning models have traditionally been used for SE, but recent studies have shown that generative approaches, such as denoising diffusion probabilistic models (DDPMs), can also be effective. However, incorporating condition information into DDPMs for SE remains a challenge. We propose a model-agnostic method called DOSE that employs two efficient condition-augmentation techniques to address this challenge, based on two key insights: (1) We force the model to prioritize the condition factor when generating samples by training it with dropout operation; (2) We inject the condition information into the sampling process by providing an informative adaptive prior. Experiments demonstrate that our approach yields substantial improvements in high-quality and stable speech generation, consistency with the condition factor, and inference efficiency. Codes are publicly available at https://github.com/ICDM-UESTC/DOSE.
Speech enhancement (SE) aims to improve the intelligibility and quality of speech, particularly in scenarios where degradation is caused by non-stationary additive noise. It has significant practical implications in various fields such as telecommunications [1], medicine [2], and entertainment [3]. Modern deep learning models are often used to learn a deterministic mapping from noisy to clean speech. While deterministic models have long been regarded as more powerful in the field of SE, recent advancements in generative models [4, 5] have significantly closed this gap.
One such generative approach is based on using denoising diffusion probabilistic models (DDPMs) [6, 7], which have been shown to effectively synthesize natural-sounding speech. Several diffusion enhancement models have been developed [4, 5, 8], which try to learn a probability distribution over the data and then generate clean speech conditioned on the noisy input. A key challenge in using diffusion enhancement models is how to effectively incorporate condition information into learning and generating faithful speech [9, 10, 8]. Previous works address this issue through designing specific condition-injecting strategies [4, 9, 11] or devising complex network architectures [10, 5].
We conduct a thorough examination to understand the limitation of diffusion-based SE methods and find that diffusion enhancement models are susceptible to condition collapse, where the primary cause of inconsistent generation is the non-dominant position of the condition factor. We thus introduce a new paradigm to effectively incorporate condition information into the diffusion enhancement models. Specifically, we propose a Diffusion-drOpout Speech Enhancement method (DOSE), which is a model-agnostic SE method (Figure 1) that employs two efficient condition-augmentation tech
*Corresponding author: fan.zhou@uestc.edu.cn
Figure 1: An illustration of the proposed DOSE. DOSE consists two primary procedures: (1) training a condition diffusion model using dropout operation, and (2) generating speech using a conditional diffusion model equipped with the adaptive prior.
niques: (1) During training, we randomly drop out intermediate-generated samples. This dropout mechanism guides the model's attention toward the condition factors; (2) Instead of letting the model generate samples from scratch (Gaussian distribution), we employ an adaptive prior derived from the conditional factor to generate samples. Experiments on benchmark datasets demonstrate that our method surpasses recent diffusion enhancement models in terms of both accuracy and efficiency. Additionally, DOSE produces more natural-sounding speech and exhibits stronger generalization capabilities compared to deterministic mapping-based methods using the same network architecture.
There are two main categories of diffusion-based SE methods: (1) designing specific condition-injecting strategies [4, 9, 11, 5], or (2) generating speech with an auxiliary condition optimizer[12, 10, 8]. The first category considers noisy speech in the diffusion (or reverse) process, either by linearly interpolating between clean and noisy speech along the process [4, 9], or by defining such a transformation within the drift term of a stochastic differential equation (SDE) [11, 5].
Works from the second category rely on an auxiliary condition optimizer – a generator (diffusion model) synthesizes clean speech and a condition optimizer informs what to generate [12, 10, 8]. Both the generator and condition optimizer have the ability to denoise, with the latter undertaking the core part. Given the challenges in leveraging condition information [8], diffusion-based SE methods within this category often necessitate specific network architecture design to guarantee the participation of condition factors.
In a paradigm sense, our method is quite similar but different to the second branch – unlike previous approaches that require additional auxiliary networks, DOSE is an end-to-end diffusion-based SE method. In addition, DOSE is model-agnostic that does not need any specific network design to guarantee consistency between the generated sample and its corresponding condition factor.
We now provide a brief introduction to the diffusion probabilistic model (diffusion models, for short), the definition of speech enhancement, and the condition collapse problem.
3.1 Diffusion models
A diffusion model [13, 6] consists of a forward (or, diffusion) process and a reverse process. Given a data point  $\pmb{x}_0$  with probability distribution  $p(\pmb{x}_0)$ , the forward process gradually destroys its data structure by repeated application of the following Markov diffusion kernel:
$$
p \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {t - 1}\right) = \mathcal {N} \left(\boldsymbol {x} _ {t}; \sqrt {1 - \beta_ {t}} \boldsymbol {x} _ {t - 1}, \beta_ {t} \boldsymbol {I}\right), \quad t \in \{1, 2, \dots , T \}, \tag {1}
$$
where  $\beta_{1},\dots ,\beta_{T}$  is a pre-defined noise variance schedule. With enough diffusion step  $T$ $p(\boldsymbol {x}_T)$  converges to the unit spherical Gaussian distribution. Based on the Markov chain, the marginal distribution at arbitrary timestep  $t$  has the following analytical form:
$$
p \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {0}\right) = \mathcal {N} \left(\boldsymbol {x} _ {t}; \sqrt {\bar {\alpha}} _ {t} \boldsymbol {x} _ {0}, (1 - \bar {\alpha} _ {t}) \boldsymbol {I}\right), \quad t \in \{1, 2, \dots , T \}, \tag {2}
$$
where  $\bar{\alpha}_t = \prod_{s=1}^t (1 - \beta_s)$ .
Figure 2: Investigation of the condition collapse problem. From left to right: (1) comparison of loss curves between unconditional and conditional diffusion models; (2) three variants; (3) PESQ performance of different variants, (-) represent the unprocessed speech.
As for the reverse process, it aims to learn a transition kernel from  $\pmb{x}_t$  to  $\pmb{x}_{t-1}$ , which is defined as the following Gaussian distribution [6]:
$$
p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t - 1} \mid \boldsymbol {x} _ {t}\right) = \mathcal {N} \left(\boldsymbol {x} _ {t - 1}; \boldsymbol {\mu} _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t}, t\right), \boldsymbol {\Sigma} \left(\boldsymbol {x} _ {t}, t\right)\right), \tag {3}
$$
where  $\pmb{\theta}$  is the learnable parameter and  $\pmb{\mu}_{\pmb{\theta}}(\pmb{x}_t,t) = \frac{1}{\sqrt{1 - \beta_t}} (\pmb{x}_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}}\pmb{\epsilon}_{\pmb{\theta}}(\pmb{x}_t,t))$  denotes the mean of  $\pmb{x}_{t-1}$ , which is obtained by subtracting the estimated Gaussian noise  $\pmb{\epsilon}_{\pmb{\theta}}(\pmb{x}_t,t)$  in the  $\pmb{x}_t$ . With such a learned transition kernel, one can approximate the data distribution  $p(\pmb{x}_0)$  via:
$$
p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {0}\right) = \int p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {T}\right) \prod_ {t = 1} ^ {T} p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t - 1} \mid \boldsymbol {x} _ {t}\right) d \boldsymbol {x} _ {1: T}, \tag {4}
$$
where  $p_{\pmb{\theta}}(\pmb{x}_T) = \mathcal{N}(\pmb{x}_T; \pmb{0}, \pmb{I})$ .
3.2 Problem formulation
Speech enhancement refers to methods that try to reduce distortions, make speech sounds more pleasant, and improve intelligibility. In real environments, the monaural noisy speech  $\mathbf{y}$  in the time domain can be modeled as:
$$
\boldsymbol {y} = \boldsymbol {x} + \boldsymbol {n} \tag {5}
$$
where  $\pmb{x}$  and  $\pmb{n}$  denote clean and noise signals, respectively. For human perception, the primary goal of speech enhancement is to extract  $\pmb{x}$  from  $\pmb{y}$ . Mapping-based speech enhancement methods directly optimize  $p_{\theta}(\pmb{x}|\pmb{y})$ , while diffusion enhancement methods generate clean samples through a Markov process  $p_{\theta}(\pmb{x}_{0:T-1}|\pmb{x}_{1:T}, \pmb{y})$ .
3.3 Condition Collapse in diffusion enhancement models
The condition collapse problem in speech enhancement was first proposed in [8] and it refers to the limited involvement of the condition factor during conditional diffusion training, resulting in inconsistencies between the generated speech and its condition factor.
In this work, we argue that the condition factor  $\pmb{y}$  indeed participates and helps the intermediate-generated sample  $\pmb{x}_t$  approximate  $p(\pmb{x}_{t-1} | \pmb{x}_t, \pmb{x}_0)$ . Our assertion is supported by the experiment depicted in the left part of Figure 2 – the diffusion model equipped with the condition factor exhibits a lower loss curve compared to the unconditional one<sup>2</sup>. To better understand the condition collapse phenomenon, we devise two variants that explicitly modify the mutual information between the condition factor and the model's output (Figure 2 (middle)). We use skip connections to add the condition factor to multiple layers, forcing the likelihood of maintaining a strong connection between the condition factor and output features. Since the dependence of the output on any hidden state in the hierarchy becomes weaker as one moves further away from the output in that hierarchy (cf. [14]), using skip connections can explicitly enhance connections between the generated sample and condition factor.
2We use DiffWave [7] as basic architecture and use the same experimental settings as [4, 9] - the only difference being the change in the way of condition-injecting since most speech enhancement methods will directly use noisy speech as the condition factor, rather than Mel-spectrogram.
As shown in Figure 2 (right), an increase in mutual information (connections) leads to a significant improvement in the consistency between the generated sample and the condition factor  $(a\to b)$ . However, it requires a meticulously designed model to guarantee its effectiveness  $(b\rightarrow c)$ . While previous studies [5, 10, 8] focus on explicitly enhancing the consistency between the output speech and condition factor through specific network architecture design, we explore the possibility of a solution independent of the model architecture. This would broaden the applicability of our method, as it enables slight modifications to existing deterministic mapping-based models to transform them into diffusion enhancement models.
Considering the diffusion model provides a transition function from  $\pmb{x}_t$  to  $x_{t-1}$ , typical condition generation process is represented as:
$$
p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {0} \mid \boldsymbol {y}\right) = \int_ {\text {P r i o r}} \underbrace {p \left(\boldsymbol {x} _ {T}\right)} \prod_ {t = 1} ^ {T} \underbrace {p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t - 1} \mid \boldsymbol {x} _ {t} , \boldsymbol {y}\right)} _ {\text {C o n d i t i o n}} d \boldsymbol {x} _ {1: T}, \quad \boldsymbol {x} _ {T} \sim \mathcal {N} \left(\boldsymbol {x} _ {T}; \boldsymbol {0}, \boldsymbol {I}\right). \tag {6}
$$
Our experiments above indicate that  $p_{\pmb{\theta}}(\pmb{x}_{t-1}|\pmb{x}_t, \pmb{y})$  will easily collapse to  $p_{\pmb{\theta}}(\pmb{x}_{t-1}|\pmb{x}_t)$ , resulting in the condition generation process degenerating into a vanilla unconditional process:
$$
\int p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {T}\right) \prod_ {t = 1} ^ {T} p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t - 1} \mid \boldsymbol {x} _ {t}, y\right) d \boldsymbol {x} _ {1: T} \Rightarrow \int p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {T}\right) \prod_ {t = 1} ^ {T} p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t - 1} \mid \boldsymbol {x} _ {t}\right) d \boldsymbol {x} _ {1: T}. \tag {7}
$$
As a result, facilitating automatic learning of the joint distribution for both clean and noisy speech samples does not work well for the speech enhancement task.
4.1 Condition augmentation I: Adaptive Prior
Let's revisit Eq. (6): since we cannot easily inject the condition factor into the condition term, how about the prior term? For example, we can modify the condition generation process as:
$$
p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {0} \mid \boldsymbol {y}\right) = \int_ {\text {C o n d i t i o n a l}} \underbrace {p \left(\boldsymbol {x} _ {\tau} \mid \boldsymbol {y}\right)} _ {t = 1} \prod_ {\substack {t = 1}} ^ {\tau} \underbrace {p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t - 1} \mid \boldsymbol {x} _ {t}\right)} _ {\text {U n c o n d i t i o n a l}} d \boldsymbol {x} _ {1: \tau}, \tag{8}
$$
where  $p(\boldsymbol{x}_{\tau}|\boldsymbol{y})$  is formulated as  $p(\boldsymbol{x}_{\tau}|\boldsymbol{y}) = \mathcal{N}(\boldsymbol{x}_{\tau}; \sqrt{\bar{\alpha}_{\tau}}\boldsymbol{y}, (1 - \bar{\alpha}_{\tau})\boldsymbol{I})$ . The following propositions verify the feasibility of our proposal.
Proposition 1. For any  $\xi >0$  such that  $0 < \xi < M$  for some finite positive value  $M$ , there exists a positive value  $\tau \in \{0,\dots ,T\}$  that satisfies:
$$
D _ {K L} \left(p (\boldsymbol {x} _ {t} | \boldsymbol {x}) \| p (\boldsymbol {x} _ {t} | \boldsymbol {y})\right) \leq \xi , \quad \forall \tau \leq t \leq T, \tag {9}
$$
where  $p(\boldsymbol{x}_t|\boldsymbol{c}) = \mathcal{N}(\boldsymbol{x}_t;\sqrt{\bar{\alpha}_t}\boldsymbol{c},(1 - \bar{\alpha}_t)\boldsymbol{I})$ .
Remark 1. This proposition indicates that, given a tolerable margin of error  $\xi$  and a well-trained diffusion model, we can always find a suitable  $\tau$  such that we are able to recover the clean speech  $\mathbf{x}$  from its noisy one  $\mathbf{y}$  using Eq. (8).
While Proposition 1 allows us to generate clean speech  $\pmb{x}$  given the noisy speech  $\pmb{y}$  using Eq. (8), it does not guarantee that our model will achieve successful recovery with a high probability.
Proposition 2. Let  $\mathbf{x}$  be the clean sample,  $\mathbf{y}$  be its corresponding noisy one, and  $\mathbf{x}'$  be any neighbor from the neighbor set  $S(\mathbf{x})$ . Then diffusion enhancement models can recover  $\mathbf{x}$  with a high probability if the following inequality is satisfied:
$$
\log \left(\frac {p (\boldsymbol {x})}{p \left(\boldsymbol {x} ^ {\prime}\right)}\right) > \frac {1}{2 \sigma_ {t} ^ {2}} \left(\| \boldsymbol {x} - \boldsymbol {y} \| _ {2} ^ {2} - \| \boldsymbol {x} ^ {\prime} - \boldsymbol {y} \| _ {2} ^ {2}\right), \quad \forall \boldsymbol {x} ^ {\prime} \in \mathcal {S} (\boldsymbol {x}), \tag {10}
$$
where  $\sigma_t^2 = \frac{1 - \bar{\alpha}_t}{\bar{\alpha}_t}$
Remark 2. Assuming that the condition factor  $\mathbf{y}$  is closer to  $\mathbf{x}$  than to  $\mathbf{x}'$ , we obtain a non-positive right-hand side (RHS). For a given  $\mathbf{x}$ , the left-hand side (LHS) value is fixed, and to ensure the inequality always holds, a smaller  $\sigma_t^2$  is preferred.
As shown in Figure 3,  $\sigma_t^2$  will increase as the timestep  $t$  increases. Thus, according to Proposition 2, we should choose a small  $\tau$  for Eq. (8) to maximize the probability of successfully recovering the clean speech from the noisy one. However, constrained by Proposition 1,  $\tau$  cannot be too small. In other words, the clean speech distribution  $p(\boldsymbol{x}_{\tau})$  and the noisy speech distribution  $p(\boldsymbol{y}_{\tau})$  will get closer over the forward diffusion process, and the gap  $|\boldsymbol{n}| = |\boldsymbol{y} - \boldsymbol{x}|$  between the noisy speech and the clean one will indeed be "washed out" by the increasingly added noise. Since the original semantic information will also be removed if  $\tau$  is too large, there should be a trade-off when we set  $\tau$  for the diffusion enhancement model.
Figure 3: The change curves of  $\log_{10}\sigma_t^2$ . Elements in legend are  $T, \beta_1, \beta_T$  respectively.
Condition optimizer. We find that both propositions are corre-
lated with the condition factor  $\pmb{y}$ . If we can reduce the gap between the condition factor and clean speech, we can choose a smaller  $\tau$ , effectively increasing the likelihood of recovering clean speech. One simple idea is to employ a neural network  $f_{\psi}$  to optimize the condition factor, as demonstrated in [15]. Accordingly, we can rewrite Eq. (8) as:
$$
p _ {\boldsymbol {\theta}, \psi} (\boldsymbol {x} _ {0} | \boldsymbol {y}) = \int p _ {\psi} (\boldsymbol {x} _ {\tau} | \boldsymbol {y}) \prod_ {t = 1} ^ {\tau} p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t - 1} | \boldsymbol {x} _ {t}\right) d \boldsymbol {x} _ {1: \tau}, \tag {11}
$$
where  $p_{\psi}(\boldsymbol{x}_{\tau}|\boldsymbol{y}) = \mathcal{N}(\boldsymbol{x}_{\tau};\sqrt{\bar{\alpha}_{\tau}} f_{\psi}(\boldsymbol{y}),(1 - \bar{\alpha}_{\tau})\boldsymbol{I})$
In practice, we should also consider failure cases of the condition optimizer in complex scenarios, especially the issue of excessive suppression that has been reported in recent literature [16, 17, 18]. To mitigate this issue, we use  $0.5c + 0.5y$  (like a simple residual layer) as a mild version of the condition factor:
$$
p _ {\psi} \left(\boldsymbol {x} _ {\tau} \mid \boldsymbol {y}\right) = \mathcal {N} \left(\boldsymbol {x} _ {\tau}; 0. 5 \sqrt {\bar {\alpha} _ {\tau}} \left(f _ {\psi} (\boldsymbol {y}) + \boldsymbol {y}\right), (1 - \bar {\alpha} _ {\tau}) \boldsymbol {I}\right). \tag {12}
$$
We call  $p_{\psi}(\boldsymbol{x}_{\tau}|\boldsymbol{y})$  the adaptive prior as it varies with different noisy samples  $\boldsymbol{y}$ .
4.2 Condition augmentation II: Diffusion Dropout
Aside from changing the prior  $p(\pmb{x}_T)$  to conditional prior  $p_{\psi}(\pmb{x}_{\tau}|\pmb{y})$ , we also optimize the condition term  $p_{\theta}(\pmb{x}_{t-1}|\pmb{x}_t, \pmb{y})$ . Instead of designing specific condition-injecting strategies [4, 9, 11] or devising complicated network architecture [8, 10, 5], we attempt to "do subtraction" by discarding some shared (intermediate-generated samples & condition factor) and important (target-related) information from intermediate-generated samples. Naturally, if we discard some information from  $\pmb{x}_t$ , then the diffusion enhancement model is forced to use the condition factor  $\pmb{y}$  to recover the speech. Taking a further step, we can even discard the entire  $\pmb{x}_t$ , as the condition factor  $\pmb{y}$  alone is sufficient for recovering the clean speech  $\pmb{x}_0$  (this is what deterministic models do). To this end, we define a neural network  $f_{\theta}(d(\pmb{x}_t, p), \pmb{y}, t)$  to approximate  $p(\pmb{x}_{t-1}|\pmb{x}_t, \pmb{x}_0)$ :
$$
p _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {t - 1} \mid \boldsymbol {x} _ {t}, \boldsymbol {y}\right) = \mathcal {N} \left(\boldsymbol {x} _ {t - 1}; f _ {\boldsymbol {\theta}} \left(d \left(\boldsymbol {x} _ {t}, p\right), \boldsymbol {y}, t\right), \boldsymbol {\Sigma} \left(\boldsymbol {x} _ {t}, t\right)\right), \tag {13}
$$
where  $d(\pmb{x}_t, p)$  is the dropout operation:
$$
d \left(\boldsymbol {x} _ {t}, p\right) = \left\{ \begin{array}{l l} \boldsymbol {x} _ {t} & \text {i f} r = 1 \\ \epsilon & \text {i f} r = 0 \end{array} , \quad r \sim \text {B e r n o u l l i} (1 - p). \right. \tag {14}
$$
4.3 DOSE training
Ho et al. [6] and much of the following work choose to parameterize the denoising model through directly predicting  $\epsilon$  with a neural network  $\epsilon_{\theta}(\boldsymbol{x}_t,t)$ , which implicitly sets:
$$
\hat {\boldsymbol {x}} _ {0} = \frac {1}{\sqrt {\bar {\alpha} _ {t}}} \left(\boldsymbol {x} _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}}\right). \tag {15}
$$
In this case, the training loss is also usually defined as the mean squared error in the  $\epsilon$ -space  $\| \pmb{\epsilon} - \pmb{\epsilon}_{\pmb{\theta}}(\pmb{x}_t,t)\| _2^2$ . Although this standard specification works well for training an unconditional diffusion model, it is not suited for DOSE – for two reasons.
Algorithm 1 DOSE Training
1: choose p
2: repeat
3: x0 ~ p(x)
4: t ~ Uniform{1,...,T}
5: ε ~ N(0,I)
6: Take gradient descent step on
7: ∇θ||x0 - xθ(d(xt,p),y,t)||2
8: until converged
Algorithm 2 DOSE Sampling
1: choose τ1, τ2, τ2 < τ1 ≤ T
2: Step 1: Generate xτ2
3: yτ1 ∼ N(√ατ1y, (1 - ατ1)I)
4: x̂0 = fθ(yτ1, y, τ1)
5: x̂τ2 ∼ N(0.5√ατ2(x̂0 + y), (1 - ατ2)I)
6: Step 2: Generate x̂0
7: x̂0 = fθ(xτ2, y, τ2)
8: return x̂0
First, we cannot estimate  $\epsilon$  without the help of  $x_{t}$  because  $\epsilon$  and  $y$  are independent. Second, as discussed earlier, we want DOSE to start with a small timestep and we strive to make  $\tau$  small. However, as  $\tau$  approaches zero, small changes in  $\mathbf{x}$ -space have an increasingly amplified effect on the implied prediction in  $\epsilon$ -space (Eq. (15)). In other words, the efforts made by diffusion enhancement models become so negligible that diffusion models lose their ability to calibrate the speech at small timesteps.
So, we need to ensure that the estimation of  $\hat{x}_0$  remains flexible as the timestep  $t$  gets smaller. Considering the equivalently of the  $\epsilon$ -space loss  $\| \epsilon - \epsilon_{\theta}(\pmb{x}_t, t) \|_2^2$  to a weighted reconstruction loss in  $\mathbf{x}$ -space  $\frac{1}{\sigma_t^2} \| \pmb{x}_0 - \pmb{x}_{\theta}(\pmb{x}_t, t) \|_2^2$ , we can directly estimate the clean speech  $\pmb{x}_0$  at each timestep  $t$ :
$$
\mathcal {L} = \mathbb {E} _ {\boldsymbol {x} _ {0} \sim p (\boldsymbol {x}), t \in \{1, \dots , T \}} \left[ \| \boldsymbol {x} _ {0} - f _ {\boldsymbol {\theta}} (d (\boldsymbol {x} _ {t}, p), \boldsymbol {y}, t) \| _ {2} ^ {2} \right] \tag {16}
$$
4.4 DOSE inference
After training, the ideal scenario is that  $p_{\theta}(\pmb{x}_{t-1}|\pmb{x}_t,\pmb{y})$  approximates  $p(\pmb{x}_{t-1}|\pmb{x}_t,\pmb{x}_0)$  precisely, enabling us to generate clean speech using Eq. (6). However, when applied in practice, it is difficult to completely eliminate errors (both sample error and true error). If these errors are not effectively managed or corrected during the generation process, the quality of the generated samples may deteriorate, leading to artifacts, blurriness, etc [19, 20]. This issue is particularly pronounced when using diffusion models for fine-grained conditional generation tasks, as diffusion models require a large number of steps to generate samples, which will significantly reduce the consistency between the generated sample and its condition factor (see §5.3, Figure 6).
The adaptive prior (Sec 4.1) provides an opportunity to address the error accumulation issue. Specifically, we can select a suitable  $\tau$  smaller than  $T$ , conditioned on an adaptive prior, and generate speech in fewer steps. We can extend Eq. 11 by transforming the unconditional diffusion enhancement model into a conditional one: