<figure><img src="image_1.png" /></figure>

HAL
open science

# Posterior Transition Modeling for Unsupervised
Diffusion-Based Speech Enhancement

Mostafa Sadeghi, Jean-Eudes Ayilo, Romain Serizel, Xavier Alameda-Pineda

<figure><img src="image_2.png" /></figure>

## To cite this version:

Mostafa Sadeghi, Jean-Eudes Ayilo, Romain Serizel, Xavier Alameda-Pineda. Posterior Transition Modeling for Unsupervised Diffusion-Based Speech Enhancement. IEEE Signal Processing Letters, 2025, 32, pp.2694-2698. 10.1109/LSP.2025.3583967. hal-05135495

HAL Id: hal-05135495

https://hal.science/hal-05135495v1

Submitted on 30 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Posterior Transition Modeling for Unsupervised
Diffusion-Based Speech Enhancement

Mostafa Sadeghi, Jean-Eudes Ayilo, Romain Serizel, and Xavier Alameda-Pineda, *Senior Member, IEEE*

***Abstract***—We explore unsupervised speech enhancement using diffusion models as expressive generative priors for clean speech. Existing approaches guide the reverse diffusion process using noisy speech through an approximate, noise-perturbed likelihood score, combined with the unconditional score via a trade-off hyperparameter. In this work, we propose two alternative algorithms that directly model the conditional reverse transition distribution of diffusion states. The first method integrates the diffusion prior with the observation model in a principled way, removing the need for hyperparameter tuning. The second defines a diffusion process over the noisy speech itself, yielding a fully tractable and exact likelihood score. Experiments on the WSJ0-QUT and VoiceBank-DEMAND datasets demonstrate improved enhancement metrics and greater robustness to domain shifts compared to both supervised and unsupervised baselines.

***Index Terms***—Unsupervised learning, speech enhancement, diffusion models, conditional transition modeling.

## I. INTRODUCTION

DIFFUSION models have recently emerged as a powerful class of generative models, achieving impressive results in image, audio, and video generation [1], [2]. These models learn expressive generative priors for complex natural data by gradually adding noise through a forward process, then training a reverse denoising process to remove it. Beyond their success in generation tasks, diffusion models have been extended to solve inverse problems [3], owing to their ability to perform both unconditional and conditional data generation.

In the field of speech processing, diffusion models have been successfully leveraged for speech enhancement (SE), an inverse problem aimed at recovering clean speech from a noisy recording [4]–[9]. While SE can encompass tasks such as dereverberation, we focus on single-channel speech denoising. Traditionally, deep learning-based SE has been dominated by predictive methods, which map noisy speech directly to clean speech via supervised training with paired data. Examples include regression-based convolutional or recurrent architectures that optimize mean-squared error, time-frequency masking, or perceptual loss functions [10]–[12]. In contrast, diffusion-based SE incorporates a prior distribution over clean speech with a generative training objective, rather than directly learning a noisy-to-clean mapping.

Two main approaches exist for applying diffusion models to SE. *Supervised* methods learn the conditional distribution of

clean speech given noisy input [4], [6]–[9], enabling reverse diffusion conditioned on the noisy signal. In contrast, *unsupervised* methods train only on clean speech to learn a prior [13]–[16], combining it at test time with a noise model via Bayes’ rule. Inference alternates posterior-guided denoising steps with noise parameter updates. Compared to supervised approaches, this unsupervised setting allows for reusing the trained prior for other speech inverse problems and may provide robustness to noise domain mismatch [13], [14], [17].

While the unsupervised framework is well studied in image processing [3], [18], its application to SE remains limited [13], [14]. In existing methods, the posterior score is obtained by combining the learned prior score with the gradient of a noisy-speech likelihood. However, the likelihood term is time-dependent and intractable, typically approximated under simplified assumptions such as an uninformative speech prior [19]. Its influence is further controlled by a trade-off hyperparameter. Moreover, these methods update only the mean of the prior (unconditional) reverse diffusion process using the posterior score, while keeping the variance fixed, which can lead to inaccurate posterior modeling and requires tuning of the guidance strength.

In this paper, we propose a novel unsupervised SE framework that performs posterior sampling by explicitly modeling the conditional (reverse) transition distribution of the diffusion states. Our first algorithm integrates the diffusion prior and observation model in a principled way, eliminating the need for a guidance hyperparameter. The second algorithm extends this approach by introducing a diffusion process over the noisy speech itself, enabling exact computation of the time-dependent likelihood. We evaluate both methods on the WSJ0-QUT and VoiceBank-DEMAND benchmarks and observe consistent improvements (mainly with the second algorithm) over existing unsupervised approaches across standard enhancement metrics. Notably, the second algorithm exhibits robustness to domain shifts between training and test conditions, often matching or surpassing supervised baselines.

## II. BACKGROUND AND RELATED WORK

We review the diffusion-based unsupervised SE framework [13], [14], which forms the foundation of our approach. Speech signals are first represented in the short-time Fourier transform (STFT) domain, resulting in a complex-valued 2D array of shape $F \times L$ ($F$ frequency bins, $L$ time frames). For notational convenience, this array is vectorized into $s \in \mathbb{C}^{FL}$. Speech enhancement is performed using a prior distribution over clean speech signals learned by a diffusion model.

M. Sadeghi, J.-E. Ayilo, and R. Serizel are with the Multispeech team, Université de Lorraine, CNRS, Inria, Loria, Nancy, France.

X. Alameda-Pineda is with the RobotLearn team, Université Grenoble Alpes, Inria, Grenoble, France.

This work was supported by the French National Research Agency (ANR) under the project REAVISE (ANR-22-CE23-0026-01).
This prior is defined via a continuous-time diffusion process {$s_t$}$_{t\in[0,T]}$, which gradually corrupts clean data $s_0 = s$ into Gaussian noise as the diffusion time $t$ increases (not to be confused with the time index of the STFT representation). The corruption is governed by a forward stochastic differential equation (SDE) [1], [13]:

$$ds_t = f(s_t)dt + g_t dw, \quad (1)$$

where $f(s_t) = -\gamma s_t$ is the drift coefficient ($\gamma > 0$), $g_t$ controls noise variance, and $w$ is a standard Wiener process. The solution of (1) defines a diffusion process whose conditional marginal distributions are characterized by the perturbation kernel $p_{0t}(s_t|s) = N_C(e^{-\gamma t}s, \sigma_t^2 I)$, where $\sigma_t^2$ is determined from the SDE. Under mild conditions [20], this process is reversible via a reverse SDE:

$$ds_t = [f(s_t) - g_t^2 \nabla_{s_t} \log p_t(s_t)] dt + g_t d\bar{w}, \quad (2)$$

where $\bar{w}$ is a backward Wiener process, and $\nabla_{s_t} \log p_t(s_t)$ (the score function) is approximated by a deep neural network (DNN)-based score model $S_\theta(s_t, t)$. The model parameters are learned by solving the following problem [21]:

$$\theta^* = \underset{\theta}{\operatorname{argmin}} \mathbb{E}_{t,s,\zeta,s_t|s} [\|\sigma_t S_\theta(s_t, t) + \zeta\|_2^2], \quad (3)$$

where $\zeta \sim N_C(0, I)$ represents zero-mean Gaussian noise. The trained score model $S_{\theta^*}(s_t, t)$ then replaces the score function in (2), and the resulting reverse SDE is solved numerically to generate clean speech samples starting from Gaussian noise.

To perform speech enhancement, the noisy speech signal is modeled as $x = s + n$, where $n \sim N_C(0, \operatorname{diag}(v_\phi))$ represents complex Gaussian noise with structured variance $v_\phi = \operatorname{vec}(WH)$, parameterized by low-rank non-negative matrices $W, H$. The parameters $\phi = \{W, H\}$ are optimized using the expectation-maximization (EM) algorithm [22]. The M-step maximizes the expected log-likelihood $\mathbb{E}_{p(s|x)}\{\log p_\phi(x|s)\}$, with the expectation approximated by Monte Carlo sampling from the posterior $p(s|x) \propto p_\phi(x|s)p(s)$ in the E-step.

Posterior sampling is performed by replacing the unconditional score in (2) with a conditional one: $\nabla_{s_t} \log p(s_t|x) = \nabla_{s_t} \log p_\phi(x|s_t) + \nabla_{s_t} \log p_t(s_t)$. The second term is approximated by the trained score model $S_{\theta^*}(s_t, t)$. Since $p_\phi(x|s_t)$ is intractable for $t > 0$, it is approximated under an uninformative prior assumption on $s$ [13], yielding

$$\tilde{p}_\phi(x|s_t) \sim N_C(e^{\gamma t}s_t, \sigma_t^2 e^{2\gamma t} I + \operatorname{diag}(v_\phi)). \quad (4)$$

The resulting conditional reverse SDE is then expressed as

$$ds_t = [f(s_t) - g_t^2 (S_{\theta^*}(s_t, t) + \lambda_t \nabla_{s_t} \log \tilde{p}_\phi(x|s_t))] dt + g_t dw, \quad (5)$$

where $\lambda_t > 0$ balances the likelihood's impact. This gradient guidance approach enables the reverse process to generate speech signals that are consistent with $x$. The above reverse SDE is numerically solved to estimate clean speech for the current value of $\phi$, which is then used to update $\phi$. The EM steps are repeated until convergence, typically within five iterations [13]. A more efficient algorithm was recently proposed [14], in which $\phi$ is updated at each reverse iteration using an intermediate speech estimate derived from Tweedie's

formula [23]:

$$\hat{s}_{0,t} = \mathbb{E}_{p_{t0}(s|s_t)}[s] \approx \frac{s_t + \sigma_t^2 S_{\theta^*}(s_t, t)}{e^{-\gamma t}}. \quad (6)$$

This method avoids a full reverse SDE process at each step to generate speech samples for updating $\phi$, significantly improving computational efficiency. These unsupervised approaches stand in contrast to supervised frameworks [4], [5], which directly learn the conditional score $\nabla_{s_t} \log p(s_t|x)$ during training from paired clean and noisy speech data.

### III. PROPOSED FRAMEWORK

We present DEPSE (Diffusion-based Explicit Posterior Sampling for Speech Enhancement), which integrates the diffusion prior and the observation model by deriving the conditional transition distribution. The proposed framework concerns only the E-step of the EM algorithm, while the M-step remains identical to that used in prior work [14]. Section III-A revisits diffusion-based prior sampling as used in [13], [14]. Section III-B introduces DEPSE-IL, which removes the need for the balancing hyperparameter $\lambda_t$ while still approximating the intractable likelihood (IL). Section III-C presents DEPSE-TL, which defines a diffusion SDE over noisy speech, resulting in a tractable likelihood (TL).

#### A. Diffusion prior sampling formulation

We adopt the Euler-Maruyama method as a numerical solver for the reverse SDE (2) with time discretization. This yields a finite sequence of latent variables, denoted by $\{s_i\}_{i=0}^N$, which form a Markov chain $s_0 \to s_1 \to \dots \to s_N$, with

$$p(s_0, \dots, s_N) = p(s_N) \prod_{i=1}^{N} p(s_{i-1}|s_i). \quad (7)$$

Discretizing (2) and inserting the score model leads to:

$$s_{i-1} = s_i - f_i \Delta \tau + g_{\tau_i}^2 S_{\theta^*}(s_i, \tau_i) \Delta \tau + g_{\tau_i} \sqrt{\Delta \tau} \zeta, \quad (8)$$

with $s_N \sim N_C(0, I)$. Here, $\{\tau_1, \dots, \tau_N\}$ is an equally spaced sequence of time steps in $[0, T]$ with a discretization width of $\Delta \tau = T/N$, $\zeta \sim N_C(0, I)$, and $f_i = -\gamma s_i$. By applying a similar discretization to the forward SDE in (1), we obtain:

$$p(s_i|s_0) = N_C(e^{-\gamma \tau_i} s_0, \sigma_{\tau_i}^2 I), \quad i \ge 1. \quad (9)$$

Since we have access to an approximate score function, we can improve the numerical SDE solver by incorporating a score-based Markov Chain Monte Carlo (MCMC) method, such as Langevin MCMC, used as a corrector step in prior works [1], [13], [14]. Combined with (8), unconditional sampling is then characterized by a backward transition distribution:

$$p(s_{i-1}|s_i) = N_C(\mu^{\text{back}}(s_i), \Sigma_i^{\text{back}}), \quad (10)$$

with

$$\left\{ \begin{array}{l} \mu^{\text{back}}(s_i) = h(s_i) - f_i \Delta \tau + g_{\tau_i}^2 S_{\theta^*}(h(s_i), \tau_i) \Delta \tau \\ \Sigma_i^{\text{back}} = g_{\tau_i}^2 \Delta \tau I. \end{array} \right. \quad (11)$$

where $h(s_i)$ is the result of one-step Langevin MCMC:

$$h(s_i) = s_i + \epsilon_{\tau_i} S_{\theta^*}(s_i, \tau_i) + \sqrt{2\epsilon_{\tau_i}} \zeta, \quad \zeta \sim N_C(0, I). \quad (12)$$

Here, $\epsilon_{\tau_i} = (\sigma_{\tau_i} \cdot r)^2$ denotes the Langevin step size ($r > 0$) [24]. Starting from Gaussian noise, i.e., $s_N \sim N_C(0, I)$, the above conditional sampling procedure is iterated to ultimately generate a speech signal $s_0$ from the learned distribution.
**Algorithm 1 DEPSE-IL**

1: **Input:** x, {τ₁, ..., τₙ}

2: sₙ ~ N<sub>C</sub>(x, σ<sup>2</sup><sub>τN</sub> I)

3: **for** i = N, ..., 1 **do**

4: Update {μ<sup>back</sup><sub>φ</sub>(s<sub>i</sub>), Σ<sup>back</sup><sub>i</sub>} using (11)

5: Update {μ<sup>IL</sup><sub>φ</sub>(s<sub>i</sub>, x), Σ<sup>IL</sup><sub>s<sub>i</sub>,i</sub>(φ)} using (18)

6: s<sub>i-1</sub> = μ<sup>IL</sup><sub>φ</sub>(s<sub>i</sub>, x) + √(Σ<sup>IL</sup><sub>s<sub>i</sub>,i</sub>(φ)ζ), ζ ~ N<sub>C</sub>(0, I)

7: ŷ<sub>0,i</sub> = (s<sub>i</sub> + σ<sup>2</sup><sub>τi</sub> S<sup>θ*</sup>(s<sub>i</sub>, τ<sub>i</sub>))/e<sup>-γτi</sup>

8: φ = {W, H} ← NMF(|x - ŷ<sub>0,i</sub>|²)

9: **end for**

10: **Output:** ŷ = s<sub>0</sub>

**Algorithm 2 DEPSE-TL**

1: **Input:** x, {τ₁, ..., τₙ}

2: sₙ ~ N<sub>C</sub>(xₙ, σ<sup>2</sup><sub>τN</sub> I)

3: **for** i = N, ..., 1 **do**

4: Update {μ<sup>back</sup><sub>φ</sub>(s<sub>i</sub>), Σ<sup>back</sup><sub>i</sub>} using (11)

5: x<sub>i-1</sub> = e<sup>-γτ<sub>i-1</sub></sup>x + σ<sub>τ<sub>i-1</sub></sub>ζ, ζ ~ N<sub>C</sub>(0, I)

6: Update {μ<sup>TL</sup><sub>φ</sub>(s<sub>i</sub>, x<sub>i-1</sub>), Σ<sup>TL</sup><sub>s<sub>i</sub>,i</sub>(φ)} using (22)

7: s<sub>i-1</sub> = μ<sup>TL</sup><sub>φ</sub>(s<sub>i</sub>, x<sub>i-1</sub>) + √(Σ<sup>TL</sup><sub>s<sub>i</sub>,i</sub>(φ)ζ), ζ ~ N<sub>C</sub>(0, I)

8: ŷ<sub>0,i</sub> = (s<sub>i</sub> + σ<sup>2</sup><sub>τi</sub> S<sup>θ*</sup>(s<sub>i</sub>, τ<sub>i</sub>))/e<sup>-γτi</sup>

9: φ = {W, H} ← NMF(|x - ŷ<sub>0,i</sub>|²)

10: **end for**

11: **Output:** ŷ = s<sub>0</sub>

## B. Explicit posterior sampling for speech enhancement

While diffusion prior sampling effectively captures the distribution of clean speech s, it does not incorporate any information about the noisy observation x. Therefore, we target the conditional distribution for SE, expressed as follows:

$$p_{\phi}(s_0, \dots, s_N | \mathbf{x}) = p(s_N) \prod_{i=1}^{N} p_{\phi}(s_{i-1} | s_i, \mathbf{x}). \quad (13)$$

We aim to approximate the conditional transition distribution $p_{\phi}(s_{i-1}|s_i, \mathbf{x})$ given the current $\phi$, which is expressed as

$$p_{\phi}(s_{i-1}|s_i, \mathbf{x}) \propto p_{\phi}(\mathbf{x}|s_{i-1}) p(s_{i-1}|s_i). \quad (14)$$

The second term is provided by the prior transition distribution in (10), and for the likelihood term, we write

$$p_{\phi}(\mathbf{x}|s_{i-1}) = \int p_{\phi}(\mathbf{x}|s_0) p(s_0|s_{i-1}) ds_0. \quad (15)$$

Similar to [13], [14], we make an uninformative prior assumption for $s_0$, leading to $p(s_0|s_{i-1}) \propto p(s_{i-1}|s_0)$. Combined with (9), we obtain:

$$p_{\phi}(\mathbf{x}|s_{i-1}) \approx N_{C} \left( e^{\gamma \tau_{i-1}} s_{i-1}, \Sigma_{x,i}^{\mathrm{IL}}(\phi) \right), \quad (16)$$

where $\Sigma_{x,i}^{\mathrm{IL}}(\phi) = \frac{\sigma_{\tau_{i-1}}^2}{e^{-2\gamma\tau_{i-1}}} I + \mathrm{diag}(v_{\phi})$. The conditional transition distribution then takes the following form

$$p_{\phi}(s_{i-1}|s_i, \mathbf{x}) = N_{C} \left( \mu_{\phi}^{\mathrm{IL}}(s_i, \mathbf{x}), \Sigma_{s,i}^{\mathrm{IL}}(\phi) \right) \quad (17)$$

with

$$\left\{ \begin{array}{l} \mu_{\phi}^{\mathrm{IL}}(s_i, \mathbf{x}) = \Sigma_{s,i}^{\mathrm{IL}}(\phi) \left( \frac{\mu_{\mathrm{back}}^{\mathrm{IL}}(s_i)}{\Sigma_{s,i}^{\mathrm{back}}} + \frac{\mathbf{x}}{e^{-\gamma \tau_{i-1}} \Sigma_{s,i}^{\mathrm{IL}}(\phi)} \right), \\ \Sigma_{s,i}^{\mathrm{IL}}(\phi) = \frac{e^{-2\gamma \tau_{i-1}} \Sigma_{s,i}^{\mathrm{IL}}(\phi) \Sigma_{s,i}^{\mathrm{back}}}{e^{-2\gamma \tau_{i-1}} \Sigma_{s,i}^{\mathrm{IL}}(\phi) + \Sigma_{s,i}^{\mathrm{back}}}. \end{array} \right. \quad (18)$$

Speech enhancement is performed through iterative sampling from the conditional transition distribution (17), with the parameter $\phi$ updated at each step using the Tweedie-based speech estimate given in (6). The complete DEPSE-IL procedure is summarized in Algorithm 1. In contrast to prior methods [13], [14], which perform posterior sampling by modifying only the mean of the reverse transition distribution (11) using a posterior score in (5) while keeping the variance fixed, our method explicitly derives and samples from the conditional transition distribution (17). This leads to more accurate posterior modeling and eliminates the need for the balancing hyperparameter $\lambda_t$, whose tuning is often non-trivial [25].

## C. Incorporating noisy speech diffusion process

We extend DEPSE-IL by introducing a diffusion process over the noisy speech signal x, inspired by [1], [24], [26], which define a measurement-space diffusion to simplify likelihood computation in the reverse process. Unlike DEPSE-IL and earlier methods [13], [14], which approximate the time-dependent likelihood with an uninformative prior assumption (see (4), (16)), this second approach, DEPSE-TL, yields a tractable likelihood and avoids such approximations. We define the set of latent variables {$x_i$}$_{i=0}^N$ ($x_0 = x$), following the same forward SDE as in (1). Similar to (9), we can write $p(\mathbf{x}_{i-1}|\mathbf{x}) = N_C(e^{-\gamma\tau_{i-1}}\mathbf{x}, \sigma_{\tau_{i-1}}^2\mathbf{I})$. Given the observation model $\mathbf{x} = \mathbf{s} + \mathbf{n}$, the time-dependent likelihood distribution can be computed in closed-form:

$$p_{\phi}(\mathbf{x}_{i-1}|s_{i-1}) = N_C(s_{i-1}, \Sigma_{x,i}^{\mathrm{TL}}(\phi)), \quad (19)$$

with $\Sigma_{x,i}^{\mathrm{TL}}(\phi) = e^{-2\gamma\tau_{i-1}} \mathrm{diag}(v_{\phi})$. The corresponding posterior distribution takes the following form:

$$p_{\phi}(s_{i-1}|s_i, \mathbf{x}_{i-1}) \propto p_{\phi}(\mathbf{x}_{i-1}|s_{i-1}) p(s_{i-1}|s_i). \quad (20)$$

Substituting the expressions for the two distributions on the right, the conditional transition distribution is derived as:

$$p_{\phi}(s_{i-1}|s_i, \mathbf{x}_{i-1}) = N_C \left( \mu_{\phi}^{\mathrm{TL}}(s_i, \mathbf{x}_{i-1}), \Sigma_{s,i}^{\mathrm{TL}}(\phi) \right), \quad (21)$$

where the mean and covariance are given by:

$$\left\{ \begin{array}{l} \mu_{\phi}^{\mathrm{TL}}(s_i, \mathbf{x}_{i-1}) = \Sigma_{s,i}^{\mathrm{TL}}(\phi) \left( \frac{\mu_{\mathrm{back}}^{\mathrm{TL}}(s_i)}{\Sigma_{s,i}^{\mathrm{back}}} + \frac{\mathbf{x}_{i-1}}{\Sigma_{s,i}^{\mathrm{TL}}(\phi)} \right), \\ \Sigma_{s,i}^{\mathrm{TL}}(\phi) = \frac{\Sigma_{s,i}^{\mathrm{TL}}(\phi) \Sigma_{s,i}^{\mathrm{back}}}{\Sigma_{s,i}^{\mathrm{TL}}(\phi) + \Sigma_{s,i}^{\mathrm{back}}}. \end{array} \right. \quad (22)$$

Algorithm 2 summarizes the overall DEPSE-TL algorithm.

## IV. EXPERIMENTS

**Baselines.** We compare the proposed algorithms with UD-iffSE+ (unsupervised-generative) [14], Conv-TasNet (supervised, adapted for SE and trained with the SNR objective) [12], and SGMSE+ (supervised-generative) [4]. These are not state-of-the-art (SOTA), as we focus on robustness under domain mismatch rather than SOTA performance.

**Model architectures.** For the unsupervised and SGMSE+ algorithms, we use a lightweight variant of the Noise Conditional Score Network (NCSN++) [4], with 5.2 M parameters. All networks, including Conv-TasNet, are trained from scratch.
<table><caption>TABLE I
AVERAGE RESULTS ON WSJ0-QUT UNDER MATCHED (WSJ0-QUT) AND MISMATCHED (VB-DMD) TRAINING CONDITIONS (OVERALL BEST, UNSUPERVISED BEST). "SUP." INDICATES SUPERVISION TYPE: S = SUPERVISED, U = UNSUPERVISED.</caption><thead><tr><th></th><th>Method</th><th>Sup.</th><th>SI-SDR</th><th>SI-SIR</th><th>SI-SAR</th><th>PESQ</th><th>ESTOI</th></tr></thead><tbody><tr><td></td><td>Input</td><td>-</td><td>-2.60</td><td>-2.60</td><td>46.66</td><td>1.83</td><td>0.50</td></tr><tr><td rowspan="4">Matched</td><td>SGMSE+ [4]</td><td>S</td><td>8.53</td><td>24.65</td><td>8.72</td><td>2.61</td><td>0.76</td></tr><tr><td>Conv-TasNet [12]</td><td>S</td><td><b>12.63</b></td><td><b>29.88</b></td><td><b>12.75</b></td><td><b>2.86</b></td><td><b>0.83</b></td></tr><tr><td>UDiffSE+ [14]</td><td>U</td><td>3.34</td><td><u>12.34</u></td><td>4.34</td><td><u>2.19</u></td><td>0.58</td></tr><tr><td>DEPSE-IL</td><td>U</td><td>3.41</td><td>10.61</td><td>4.83</td><td><u>2.19</u></td><td>0.59</td></tr><tr><td></td><td>DEPSE-TL</td><td>U</td><td><u>3.89</u></td><td>6.55</td><td><u>7.93</u></td><td>2.17</td><td><u>0.61</u></td></tr><tr><td rowspan="4">Mismatched</td><td>SGMSE+ [4]</td><td>S</td><td>3.02</td><td>12.79</td><td>3.86</td><td>2.08</td><td>0.61</td></tr><tr><td>Conv-TasNet [12]</td><td>S</td><td><b>5.83</b></td><td><b>17.07</b></td><td><b>6.36</b></td><td><b>2.21</b></td><td><b>0.63</b></td></tr><tr><td>UDiffSE+ [14]</td><td>U</td><td>0.06</td><td>8.72</td><td>1.21</td><td>2.05</td><td>0.52</td></tr><tr><td>DEPSE-IL</td><td>U</td><td>0.18</td><td>6.55</td><td>1.86</td><td>2.05</td><td>0.53</td></tr><tr><td></td><td>DEPSE-TL</td><td>U</td><td><u>2.66</u></td><td>5.96</td><td><u>6.06</u></td><td><u>2.17</u></td><td><u>0.59</u></td></tr></tbody></table>

**Hyperparameter setting.** Speech data are sampled at 16 kHz. STFT is computed using a Hann window of size 510 with 75% overlap. SDE hyperparameters follow [13]. The reverse SDE is discretized into $N = 30$ steps. In UDiffSE+, we set $\lambda_{\tau_i} = 1.5$ for even $i$, and 0 otherwise, as suggested in [14].

**Dataset.** We use WSJ0-QUT [27] and VoiceBank-DEMAND (VB-DMD) [28]. WSJ0-QUT is constructed by mixing clean speech from the Wall Street Journal corpus (WSJ0) [29] with noise from QUT-Noise [30] (e.g., car, café, street, living room, kitchen). VB-DMD combines clean speech from the Voice Bank corpus (VB) with noise from the DEMAND dataset [31] (e.g., living room, office space, bus, open-area cafeteria, public square, restaurant, subway). SE models are trained either on the WSJ0-QUT training set (25 hours) or on the VB-DMD corpus (9.39 hours), and evaluated on the test sets of WSJ0-QUT (1.48 hours) or VB-DMD test set (0.58 hour). Unsupervised models are trained only on the corresponding clean speech data. Noisy speech signals in WSJ0-QUT are generated with SNRs of {-5, 0, 5} dB (for both training and testing). In VB-DMD, SNRs are {0, 5, 10, 15} dB for training and {2.5, 7.5, 12.5, 17.5} dB for testing.

**Evaluation metrics.** We report standard SE metrics: scale-invariant signal-to-distortion, interference, and artifact ratios (SI-SDR, SI-SIR, SI-SAR) in dB [32]—though these are not ideal for evaluating generative models—as well as the extended short-term objective intelligibility (ESTOI) [33] ([0, 1]) and the perceptual evaluation of speech quality (PESQ) score [34] ([−0.5, 4.5]). Higher values indicate better performance.

**Results.** Tables I and II report the average metric values for evaluation on the WSJ0-QUT and VB-DMD test sets, respectively, both with matched and mismatched training sets. Boldfaced and underlined scores denote statistically significant improvements ($p < 0.05$) based on paired t-tests for related samples. Among the unsupervised algorithms, DEPSE-TL consistently outperforms DEPSE-IL and UDiffSE+ in terms of SI-SDR, PESQ, and ESTOI, across almost all evaluation setups. For instance, DEPSE-TL achieves a notable SI-SDR improvement over UDiffSE+ when trained and evaluated on VB-DMD (13.87 dB vs. 10.40 dB); similarly, it outperforms UDiffSE+ on PESQ (2.17 vs. 2.05) and ESTOI (0.59 vs. 0.52) when trained on VB and tested on WSJ0-QUT. These

<table><caption>TABLE II
AVERAGE RESULTS ON VB-DMD UNDER MATCHED (VB-DMD) AND MISMATCHED (WSJ0-QUT) TRAINING CONDITIONS (OVERALL BEST, UNSUPERVISED BEST). "SUP." INDICATES SUPERVISION TYPE: S = SUPERVISED, U = UNSUPERVISED.</caption><thead><tr><th></th><th>Method</th><th>Sup.</th><th>SI-SDR</th><th>SI-SIR</th><th>SI-SAR</th><th>PESQ</th><th>ESTOI</th></tr></thead><tbody><tr><td></td><td>Input</td><td>-</td><td>8.45</td><td>8.45</td><td>47.51</td><td>3.02</td><td>0.79</td></tr><tr><td rowspan="4">Matched</td><td>SGMSE+ [4]</td><td>S</td><td>17.16</td><td>28.89</td><td>17.65</td><td><b>3.51</b></td><td><b>0.85</b></td></tr><tr><td>Conv-TasNet [12]</td><td>S</td><td><b>19.26</b></td><td><b>32.36</b></td><td><b>19.80</b></td><td>3.40</td><td><b>0.85</b></td></tr><tr><td>UDiffSE+ [14]</td><td>U</td><td>10.40</td><td><u>22.45</u></td><td>11.04</td><td>3.30</td><td>0.77</td></tr><tr><td>DEPSE-IL</td><td>U</td><td>10.49</td><td>19.73</td><td>11.54</td><td>3.36</td><td>0.79</td></tr><tr><td></td><td>DEPSE-TL</td><td>U</td><td><u>13.87</u></td><td>16.40</td><td><u>18.38</u></td><td><u>3.41</u></td><td><u>0.82</u></td></tr><tr><td rowspan="4">Mismatched</td><td>SGMSE+ [4]</td><td>S</td><td>10.06</td><td>12.43</td><td><b>17.13</b></td><td>3.29</td><td>0.82</td></tr><tr><td>Conv-TasNet [12]</td><td>S</td><td>11.13</td><td>15.22</td><td>15.90</td><td>3.28</td><td><b>0.83</b></td></tr><tr><td>UDiffSE+ [14]</td><td>U</td><td>10.87</td><td><b>28.27</b></td><td>11.02</td><td>3.30</td><td>0.77</td></tr><tr><td>DEPSE-IL</td><td>U</td><td>11.06</td><td>24.58</td><td>11.41</td><td>3.31</td><td>0.78</td></tr><tr><td></td><td>DEPSE-TL</td><td>U</td><td><b>14.01</b></td><td>17.56</td><td><b>17.37</b></td><td><b>3.34</b></td><td><b>0.81</b></td></tr></tbody></table>

results show that performance improves without a trade-off hyperparameter or likelihood approximation.

We also observe that DEPSE-IL performs comparably to or slightly better than UDiffSE+ across all settings. DEPSE-TL, on the other hand, exhibits more performance boost by providing a tractable likelihood. Noteworthy, the superiority of DEPSE-TL in SI-SDR does not imply better SI-SIR and SI-SAR. In fact, DEPSE-TL tends to introduce less artifacts (higher SI-SAR) than UDiffSE+ and DEPSE-IL, but retains slightly more residual interference (lower SI-SIR). Overall, the gain in SI-SAR outweighs the loss in SI-SIR, resulting in higher SI-SDR and thus improved perceptual quality and intelligibility, as reflected by the PESQ and ESTOI metrics.

The supervised baselines, SGMSE+ and Conv-TasNet, achieve the highest performance across most metrics, except when trained on WSJ0-QUT and evaluated on VB-DMD, where DEPSE-TL outperforms them. When training and testing are conducted on different corpora, supervised models exhibit significant performance drops. For example, training on VB-DMD and evaluating on WSJ0-QUT leads to a 64.6% SI-SDR decrease for SGMSE+ (8.53 → 3.02) and a 53.8% drop for Conv-TasNet (12.63 → 5.83). In contrast, DEPSE-TL experiences an SI-SDR drop of 31.6% (3.89 → 2.66), highlighting the higher robustness of DEPSE-TL to training-test domain mismatch. This robustness of unsupervised algorithms is more pronounced in Table II.

V. CONCLUSION

We introduced two diffusion-based, unsupervised speech enhancement algorithms that sample from an explicitly derived conditional transition distribution. Unlike previous approaches, which rely on posterior-score guidance with approximated likelihoods and adjust only the mean of the prior and require a tuned hyperparameter, our first method computes both the conditional mean and variance without tuning. Our second method further introduces a forward diffusion over noisy speech, enabling exact likelihood computation. Experiments show that this method consistently outperforms prior unsupervised baselines in speech quality, intelligibility, and robustness under mismatched conditions. The results also demonstrate greater robustness to shifts between training and test conditions compared to supervised baselines.
REFERENCES

[1] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” in International Conference on Learning Representations (ICLR), 2021.

[2] F.-A. Croitoru, V. Hondru, R. T. Ionescu, and M. Shah, “Diffusion models in vision: A survey,” *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 45, no. 9, pp. 10850–10869, 2023.

[3] G. Daras, H. Chung, C.-H. Lai, Y. Mitsufuji, J. C. Ye, P. Milanfar, A. G. Dimakis, and M. Delbracio, “A survey on diffusion models for inverse problems,” *arXiv preprint arXiv:2410.00083*, 2024.

[4] J. Richter, S. Welker, J.-M. Lemercier, B. Lay, and T. Gerkmann, “Speech enhancement and dereverberation with diffusion-based generative models,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 2023.

[5] J.-M. Lemercier, J. Richter, S. Welker, and T. Gerkmann, “StoRM: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 31, pp. 2724–2737, 2023.

[6] P. Gonzalez, Z.-H. Tan, J. Østergaard, J. Jensen, T. S. Alstrøm, and T. May, “Diffusion-based speech enhancement in matched and mismatched conditions using a Heun-based sampler,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2024, pp. 10431–10435.

[7] J. Serrà, S. Pascual, J. Pons, R. O. Araz, and D. Scaini, “Universal speech enhancement with score-based diffusion,” *arXiv preprint arXiv:2206.03065*, 2022.

[8] Y.-J. Lu, Z.-Q. Wang, S. Watanabe, A. Richard, C. Yu, and Y. Tsao, “Conditional diffusion probabilistic model for speech enhancement,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2022, pp. 7402–7406.

[9] H. Yen, F. G. Germain, G. Wichern, and J. Le Roux, “Cold diffusion for speech enhancement,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2023, pp. 1–5.

[10] D. Wang and J. Chen, “Supervised speech separation based on deep learning: An overview,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 26, no. 10, pp. 1702–1726, 2018.

[11] Y. Xu, J. Du, L.-R. Dai, and C.-H. Lee, “A regression approach to speech enhancement based on deep neural networks,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 23, no. 1, pp. 7–19, 2014.

[12] Y. Luo and N. Mesgarani, “Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation,” *IEEE/ACM transactions on audio, speech, and language processing*, vol. 27, no. 8, pp. 1256–1266, 2019.

[13] B. Nortier, M. Sadeghi, and R. Serizel, “Unsupervised speech enhancement with diffusion-based generative models,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2024.

[14] J.-E. Ayilo, M. Sadeghi, R. Serizel, and X. Alameda-Pineda, “Diffusion-based unsupervised audio-visual speech enhancement,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2025.

[15] A. Iashchenko, P. Andreev, I. Shchekotov, N. Babaev, and D. Vetrov, “UnDiff: Unsupervised Voice Restoration with Unconditional Diffusion Model,” in *Proc. INTERSPEECH 2023*, 2023, pp. 4294–4298.

[16] E. Moliner, J.-M. Lemercier, S. Welker, T. Gerkmann, and V. Välimäki, “BUDDY: Single-channel blind unsupervised dereverberation with diffusion models,” in *18th International Workshop on Acoustic Signal Enhancement (IWAENC)*, 2024, pp. 120–124.

[17] J.-M. Lemercier, J. Richter, S. Welker, E. Moliner, V. Välimäki, and T. Gerkmann, “Diffusion models for audio restoration: A review,” *IEEE Signal Processing Magazine*, vol. 41, no. 6, pp. 72–84, 2025.

[18] H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye, “Diffusion posterior sampling for general noisy inverse problems,” in *International Conference on Learning Representations*, 2023. [Online]. Available: https://openreview.net/forum?id=OnD9zGAGT0k

[19] X. Meng and Y. Kabashima, “Diffusion model based posterior sampling for noisy linear inverse problems,” in *The 16th Asian Conference on Machine Learning (Conference Track)*, 2024. [Online]. Available: https://openreview.net/forum?id=A07d0PMRqc

[20] B. D. Anderson, “Reverse-time diffusion equation models,” *Stochastic Processes and their Applications*, vol. 12, no. 3, pp. 313–326, 1982. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0304414982900515

[21] D. P. Kingma and R. Gao, “Understanding diffusion objectives as the ELBO with simple data augmentation,” in *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.

[22] C. M. Bishop and N. M. Nasrabadi, *Pattern recognition and machine learning*. Springer, 2006, vol. 4, no. 4.

[23] B. Efron, “Tweedie’s formula and selection bias,” *Journal of the American Statistical Association*, vol. 106, no. 496, pp. 1602–1614, 2011.

[24] Y. Song, L. Shen, L. Xing, and S. Ermon, “Solving inverse problems in medical imaging with score-based generative models,” in *International Conference on Learning Representations*, 2022.

[25] B. Nortier, M. Sadeghi, and R. Serizel, “UDiffSE Supplementary Material,” https://github.com/joanne-b-nortier/UDiffSE/blob/main/docs/UDiffSE_Supplementary.pdf, 2024.

[26] Z. Dou and Y. Song, “Diffusion posterior sampling for linear inverse problem solving: A filtering perspective,” in *The Twelfth International Conference on Learning Representations*, 2024.

[27] S. Leglaive, X. Alameda-Pineda, L. Girin, and R. Horaud, “A recurrent variational autoencoder for speech enhancement,” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2020.

[28] C. V. Botinhao, X. Wang, S. Takaki, and J. Yamagishi, “Investigating rnn-based speech enhancement methods for noise-robust text-to-speech,” in *9th ISCA speech synthesis workshop*, 2016, pp. 159–165.

[29] J. Garofolo, D. Graff, D. Paul, and D. Pallett, “CSR-I (WSJ0) complete LDC93S6B,” *Web Download*. Philadelphia: Linguistic Data Consortium, vol. 83, 1993.

[30] D. Dean, A. Kanagasundaram, H. Ghaemmaghami, M. H. Rahman, and S. Sridharan, “The QUT-NOISE-SRE protocol for the evaluation of noisy speaker recognition,” in *Proceedings of Interspeech*, 2015, pp. 3456–3460.

[31] J. Thiemann, N. Ito, and E. Vincent, “The diverse environments multi-channel acoustic noise database (DEMAND): A database of multichannel environmental noise recordings,” in *Proceedings of Meetings on Acoustics*, vol. 19, no. 1. AIP Publishing, 2013.

[32] J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR-half-baked or well done?” in *IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2019.

[33] J. Jensen and C. H. Taal, “An algorithm for predicting the intelligibility of speech masked by modulated noise maskers,” *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 24, no. 11, pp. 2009–2022, 2016.

[34] A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs,” in *IEEE international conference on acoustics, speech, and signal processing. Proceedings (ICASSP)*, vol. 2, 2001, pp. 749–752.