Universal Speech Enhancement with Score-based Diffusion
Authors: Joan Serrà
Santiago Pascual
Jordi Pons
Recep Oguz Araz
Davide Scaini
Venue: Submitted to ICLR 2023 (2023)
Keywords: Speech enhancement
audio
score matching
diffusion
mixture density networks

Abstract: Removing background noise from speech audio has been the subject of considerable effort, especially in recent years due to the rise of virtual communication and amateur recordings. Yet background noise is not the only unpleasant disturbance that can prevent intelligibility: reverb, clipping, codec artifacts, problematic equalization, limited bandwidth, or inconsistent loudness are equally disturbing and ubiquitous. In this work, we propose to consider the task of speech enhancement as a holistic endeavor, and present a universal speech enhancement system that tackles 55 different distortions at the same time. Our approach consists of a generative model that employs score-based diffusion, together with a multi-resolution conditioning network that performs enhancement with mixture density networks. We show that this approach significantly outperforms the state of the art in a subjective test performed by expert listeners. We also show that it achieves competitive objective scores with just 4-8 diffusion steps, despite not considering any particular strategy for fast sampling. We hope that both our methodology and technical contributions encourage researchers and practitioners to adopt a universal approach to speech enhancement, possibly framing it as a generative task.

Under review as a conference paper at ICLR 2023
UNIVERSAL SPEECH ENHANCEMENT WITH SCORE- BASED DIFFUSION
Anonymous authors Paper under double-blind review
ABSTRACT
Removing background noise from speech audio has been the subject of consider- able effort, especially in recent years due to the rise of virtual communication and amateur recordings. Yet background noise is not the only unpleasant disturbance that can prevent intelligibility: reverb, clipping, codec artifacts, problematic equal- ization, limited bandwidth, or inconsistent loudness are equally disturbing and ubiquitous. In this work, we propose to consider the task of speech enhancement as holistic endeavor, and present a universal speech enhancement system that tackles 55 different distortions at the same time. Our approach consists of generative model that employs score-based diffusion, together with a multi-resolution condi- tioning network that performs enhancement with mixture density networks. We show that this approach significantly outperforms the state of the art in subjective test performed by expert listeners. We also show that it achieves competitive objective scores with just 4-8 diffusion steps, despite not considering any particular strategy for fast sampling- We hope that both our methodology and technical contributions encourage researchers and practitioners to adopt a universal approach to speech enhancement, possibly framing it as a generative task.
INTRODUCTION
Real-world recorded speech almost inevitably contains background noise, which can be unpleasant and prevent intelligibility. Removing background noise has traditionally been the objective of speech enhancement algorithms (Loizou, 2013). Since the 1940s (Kolmogorov, 1941; Wiener, 1949), myriad of denoising approaches based on filtering have been proposed, with a focus on stationary noises. With the advent of deep learning, the task has been dominated by neural networks, often outperforming more classical algorithms and generalizing to multiple noise types (Lu et al., 2013; Pascual et al., 2017; Rethage et al., 2018; Défossez et al., 2020; Fu et al., 2021). Besides recent progress, speech denoising still presents room for improvement, especially when dealing with distribution shift or real-world recordings.
Noise however is only one of the many potential disturbances that can be present in speech recordings. If recordings are performed in closed room, reverberation ubiquitous. With this in mind, number of works have recently started to zoom out the focus in order to embrace more realistic situations and tackle noise and reverberation at the same time (Su al., 2021: 2019; Polyak al., 2021). Some of these works adopt a generation or re- re-generation strategy (Maiti & Mandel, 2019), in which a two-stage approach is employed to first enhance and then synthesize speech signals. Despite the relative success of this strategy, it is still an open question whether such approaches can perceptually outperform the purely supervised ones, especially terms of realism and lack of voice artifacts.
Besides noise and reverberation, a few works propose to go one step further by considering additional distortions. Pascual et al. (2019) introduce a broader notion of speech enhancement by considering whispered speech, bandwidth reduction, silent gaps, and clipping. More recently, Nair & Koishida (2021) consider silent gaps, clipping, and codec artifacts, and Zhang et al. (2021a) consider clipping and codec artifacts. In concurrent work, Liu et al. (2021) deal with bandwidth reduction and clipping in addition to noise and reverberation. Despite the recent efforts to go beyond pure denoising, we are not aware of any speech enhancement system that tackles more than 2 distortions the same time.
In this work, we take a holistic approach and regard the task of speech enhancement as a universal endeavor. We believe that, for realistic speech enhancement situations, algorithms need not only face

---

Under review as a conference paper at ICLR 2023
and improve upon background noise and possibly reverberation, but also to correct large number of typical but usually neglected distortions that are present in everyday recordings or umateur-producce audio, such bandwidth reduction, clipping. codec artifacts, silent gaps, excessive dynamics compression//epansionn sub-optimal equalization, noise gating, and others (in total, we deal with 55 distortions, which can be grouped into 10 different families).
Our solution relies on an end-to-end approach, in which a generator network synthesizes clean speech and a conditioner network informs of what to generate. The idea that the generator learns from clean speech and both generator and conditioner have the capability of enhancing representations, with the latter undertaking the core part of this task. For the generator, we put together a number of known and less known advances in score-based diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020). For the conditioner, we develop a number of improved architectural choices, and further propose the usage of auxiliary, out-of-path mixture density networks for enhancement in both the feature and the waveform domains. We quantify the relative importance of these main development steps using objective metrics, and show how the final solution outperforms the state of the art in all considered distortions using a subjective test with expert listeners (objective metries for the denoising task are also reported in the Appendix). Finally, of diffusion steps needed for performing high-quality universal speech enhancement, and find to be on par with the fastest diffusion-based neural vocoders without the need for any specific tuning.
also study the number
2 RELATED WORK
Our approach is based on diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020). While diffusion models have been more extensively studied for unconditional or weakly- conditioned image generation, our work presents a number of techniques for croniily-conditioned speech re- re-generation or enhancement. Diffusion- multiple generative tasks, in different domains. In the audio domain, they have been particularly successful in speech synthesis (Chen et al., 2021; Kong et al., 2021), text-to-speech (Jeong et al., 2021; Popov et al., 2021), bandwidth extension (Lee & Han, 2021), or drum sound synthesis (Rouard & Hadjeres, 2021). An introduction to diffusion models is given in Appendix A
models achieve state-of-the-ar quality on
Diffusion-based models have also recently been used for speech denoising. Zhang et al. (2021a) expand the DiffWave vocoder (Kong et al., 2021) with a convolutional conditioner, and train that separately with an LI loss for matching latent representations. Lu al. (2021) study the potential of DiffWave with noisy mel band inputs for speech denoising and, later, Lu et al. (2022) and Welker al. (2022) propose formulations of the diffusion process that can adapt (non- (non-Gaussian) real audio noises. These studies with speech denoising show improvement over the considered baselines, but do not reach the objective scores achieved by state-of-theear approaches (see also Appendix E). Our work stems from the WaveGrad architecture (Chen et al., 2021), introduces number of crucial modifications and additional concepts, and pioneers universal enhancement by tackling an unprecedented amount of distortions.
The state of the art for speech denoising and dereverberation is dominated by regression and adver- sarial approaches (Défossez et al., 2020; Fu et al., 2021; Su et al., 2021; Isik et al., 2020; Hao et al., 2021; Kim & Seo, 2021; Zheng et al., 2021; Kataria et al., 2021). However, one considers further degradations of the signal like clipping, bandwidth removal, or silent gaps, it is intuitive to think that generative approaches have great potential (Polyak et al., 2021; Pascual et al., 2019; Zhang et al., 2021b), such degradations require generating signal where, simply, there is none. Yet, the best of our knowledge, this intuition has not been convincingly demonstrated through subjective tests involving human judgment. Our work sets a milestone showing that a generative approach can outperform existing supervised and adversarial approaches when evaluated by expert listeners.
3 DIFFUSION-BASED UNIVERSAL SPEECH ENHANCEMENT
3.1 METHODOLOGY
Data To train our model,
data set of clean and programmatically-dittoro speech recordings. To obtain the clean speech, we sample h,500 h of audio from an internal pool
pairs of
of data sets and convert it to 16 kHz mono. The speech sample consists of about 1.2 M utterances
2

---

Under review as a conference paper at ICLR 2023
of between 3.5 and s, from thousands of speakers, in more than 10 languages, and with 50 different recording conditions (clean speech is chosen to be the most dry possible and of high quality). To validate our model, et al., 2019) and Harvard sentences (Henter et al., 2014), together with noises/backgrounds from DEMAND (Thiemann al., 2013) and FSIDnissy 18k (Fonseca al., 2019). Train data does not overlap with the validation partition nor with other data used for evaluation or subjective testing.
1.5 h of clean utterances sampled from VCTK (Yamagishi
To programmatically generate distorted speech, we consider 10 distortion families: band limiting, codec artifacts, signal distortion, loudness dynamics, equalization, recorded noise, reverb, spectral manipulation, synthetic noise, and transmission. Each family includes a variety of distortion algo- rithms. which we generically call "types'. For instance, types of synthetic noise include colored noise, electricity tones, non-stationary noise bursts, etc., types of codecs include OPUS, Vorbis, MP3, EAC3, etc., types of reverb include algorithmic reverb, delays, and both real and simulated room impulse responses, and so on. total, we leverage 55 distortion types. Distortion type parameters such as strength, frequency, or gain are set randomly within reasonable bounds. A more comprehensive list of distortion families, types, and parameters can be found Appendix
Evaluation - To measure relative improvement, we use objective speech quality metrics reflecting different criteria. On the one hand, we employ speech enhancement metrics COVL (Loizou, 2013) and STOI (Taal et al., 2010), which are widely used for denoising dereverberation tasks. On the other hand, we employ the codec quality metric WARP-Q (Jassim et al., 2021) and an internal semi- supervise metrie imitating SESQA (Serrà et al., 2021b), which should perform well with generative algorithms with perceptually -valid outputs that do not necessarily perfectly align with the target signal. We also consider the composite measure COMP that results from normalizing the previous four metrics between () and 10 and taking the average.
To compare with the state of the art, we perform subjective preference test. In it, listeners presented with a reference distorted signal plus two enhanced versions of it: one by an existing approach and one by the proposed approach. Then, they are asked to choose which of the two enhanced signals they prefer, based on the presence of the original nuisance, voice distortion, audible artifacts, etc. The test was voluntarily performed by 22 expert listeners, each of them listening to randomly-chosen pairs of distorted and enhanced signals, taken from the online demo/example pages of 12 existing approaches, plus the corresponding version enhanced by our approach. Further details of our evaluation methodology are provided in Appendix D)
3.2 BASE APPROACH
Score-based diffusion - In this work, we use a variance exploding (VE) diffusion approach (Song al., 2021). We train score network S following denoising score matching paradigm (Song & Ermon, 2019; 2020), using
[ 2 |1o,5(xo ozzi, c,)) + +2,
ESCORE E,, Ex,
where t 2(00 1), Z, (0, 1). No Pdata c is the conditioning signal, and values 0, follow geometric noise schedule (Song & Ermon, 2019). In all our experiments we use To = 5 10 and σ1 = 5, which we find sufficient for audio signals between -1 and (cf. Song & Ermon, 2020). We consider different approaches to obtain but all of them have No. the distorted version of xo, as the main and only input (Xo and Kn are both in the waveform domain).
To sample. we follow noise-consisten Langevin dynamics Pooicoeur-Martines et al., 2021). which corresponds to the recursion
Nt,- Xt, + 1oo S(xt., ).,.) + _o,,,, 2
over N uniformly discretized time steps tu E [0, 1], where we set 17 and / with the help of a hyper- parameter oo). An extensive derivation of our approach to both training and sampling can be found in Appendix A, including an introduction to score-based diffusion.
General model description We use comvolutional blocks and couple of bi-directional recurrent layers. Convolutional blocks are formed by three ID convolutional layers, each one preceded by a multi-parametric ReLU (PReL.U) activation, and all of them under residual connection. If needed,
3

---

Under review as a conference paper at ICLR 2023
A: Generator network
B: Conditioning network
S
Linear PReLU
MDN
MDN
MDN
ConvBlock
Lin
ConvBlock
ConvBlock
ConvBlock
Lin
GRU(1)
ConvBlock
Lin
ConvBlock
GRU(2)
ConvBlock
Lin
ConvBlock
SigmaBlock
Conv
l0g10 (o)
X OZ
PReLU
ConvBlock
ConvBlock
ConvBlock
Conv
StConv PReLU
Mel bands
ConvBlock
Conv
Figure 1: Block diagram of the proposed approach. Individual blocks are depicted in Appendix B.
up- or down-sampling is applied before or after the residual link, respectively (Appendix B). We perform up-/down-sampling with transposed/stridedd convolutions, halving/doubling the number of channels at every step. The down-sampling factors are (2,4,4,5), which yield 100 Hz latent representation for a 16 kHz input.
The model consists of generator and conditioning network (Fig. 1). The generator network is formed by UNet-like structure with skip connections and gated recurrent unit (GRU) in the middle (Fig. 1-A). Convolutional blocks in the generator receive adaptor signals g. which inform the network about the noise level a, and conditioning signals which provide the necessary speech cues for synthesis. Signals g and e are mixed with the UNet activations using FiLM (Perez et al., 2018) and summation, respectively. To obtain g, we process the logarithm of a with random Fourier feature embeddings and an MLP as in Rouard & Hadjeres (2021) (see also Appendix B). The conditioning network processes the distorted signal X with convolutional blocks featuring skip connections to down-sampled latent that further exploits log-mel features extracted from x (Fig. 1-B). The middle and decoding parts of the network are formed by two-layer GRU and multiple convolutional blocks, with the decoder up-sampling the latent representation to provide multi-resolution conditionings to the generator. Multiple heads and target information are exploited to improve the latent representation and provide a better e (see below). We call our approach UNIVERSE, for universal speech enhancer.
3.3 DEVELOPING UNIVERSE
In the following, explain all the steps we took to arrive at the above-mentioned structure, motivating each decision and quantifying its impact with objective metrics (schematic diagrams are depicted in Fig. 2). Unless stated otherwise, all models are trained with Adam and weight decay for 1 M iterations, with two-second long audio frames and a batch size of 32. We use cosine learning rate schedule with linear warm-up that spans the first 5 % of the iterations. Under these specifications,
4

---

Under review as a conference paper at ICLR 2023
B2
$1,52
C
4
Decoder
Decoder
Dec
C
Dec
C
Dec
Dec
Encoder
Encoder
Enc
Enc
Enc
Enc
Enc
Enc
$4
S5
$6,57,
A3
CNN
MPN
Dec
Dec
Dec
Dec
Dec
Dec
Dec
CNN
MDN
Enc
Enc*
Enc
Enc²
Enc
Enc*
Enc
Enc
Figure 2: Diagrams of the followed steps: L indicates auxiliary losses and + capacity increase.
training UNIVERSE model with 49 M parameters takes less than 5 days using two Tesla V100 GPUs and PyTorch 's native automatic mixed precision (Paszke et al., 2019).
Initial baselines (B1-B3) We start with structure inspired by WaveGrad (Chen al., 2021), featuring UNet-like architecture with the aforementioned convolutional blocks, each with 64, 128, 256, and 512 channels (Sec. 3.2 and Appendix B). To condition we use the distorted speech x. from which we compute 100 Hz log-mel representation with 80 bands that is summed to the UNet latent after linear projection (Fig. 2-B1). This approach turns out lack enhancement capabilities, probably hampered by the coarse mel representation of X. It also presents considerable speech distortion, probably due to the distortions in the mel representation (Table B1).
We next consider an approach inspired by NU-Wave (Lee & Han, 2021). We employ the same convolutional UNet structure as before, but inject the conditioning signal at the input (Fig. 2-B2), concatenated with x oz (sce Eq. 1). We find that this approach yields less speech distortion and better scores than the previous one (Table 1, B2). However, after listening to some excepts, we notice that a lot of the audio is 'bypassed' from input to output. This leads to, for example, input noises present at the output, or silent gaps not being reconstructed. We hypothesize that the UNet's sss skip connections the cuiprit for this behavior (and removing them affected training in dramatic way).
Another baseline approach we consider is inspired by ModDW (Zhang et al., 2021a). In it, following DifiWave (Kong et al., 2021), an auxiliary encoder and loss are used to learn conditioning latent (Fig. 2-B3). This encoder uses the same blocks as the generative one, but with progressively down- sampled skip connections. Also different from ModDW, we use mean squared error (MSE) instead of an L1 loss, and compare directly with the clean band representation instead of pre-learnt latent. With these two modifications, we find no problems in training the model end-to-end, in contrast to the original ModDW, which had to be trained using separate stages (Zhang et al., 2021a). The scores for this approach are in the middle of the previous baselines, closer to the second one (Table 1, B3).
Capacity and losses (S1-S7) With the three baselines above, we decide to add processing capacity in order obtain better conditioning signal. This decision was motivated by preliminary analysis showing that B1 and B3 were capable to synthesize high-quality speech in vocoder setting (that is, when a clean mel-band conditioning was provided). Hence, with more capacity in the conditioning network, one should come closer to such clean mel-band scenario (an observation also made by Zhang et al. (2021a)). Unfortunately, increasing the capacity for the best scoring model, B2, could not avoid the bypass problems outlined above. Therefore, focus improving B3, which had the best scores after B2 and a similar listening quality, with less noise but more speech distortion.
We first add extra log mel input the conditioner's encoder, with convolutional block after it, and sum its output to the skip connections coming from the distorted waveform processing (mels are extracted from the same distorted signal that is input the waveform encoder, Fig. 1-B). This
5

---

Under review as a conference paper at ICLR 2023
Table 1: Test set objective scores for the steps in developing UNIVERSE. Increments A are calculated with respect to the COMP value of the previous row, except for Al-A3, which take S7 as reference.
Description
COVL. f
STOI f
WARP-Q 4
SESQA f
COMP f
A
B1
Inspired by WaveGirad
1.55
0.755
0.952
4.90
3.77
B2
Inspired by NU-Wave
2.02
0.829
0.873
5.57
4.69
B3
Inspired by ModDW
1.76
0.811
0.902
5.35
4.34
S1
Extra mel input
1.86
0.820
0.872
5.51
4.54
+0,20
$2
Latent RNN+CNN
1.99
0.844
0.834
5.41
4.75
+0.21
S3
Auxiliary decoder & loss
2.37
0.880
0.830
5.81
5.25
+0,50
S4
Multi-resolution cond.
2.53
0.888
0.811
6.18
5.54
+0.29
S5
Out-of-path losses
2.58
0.893
0.787
6.29
5,66
+0.12
S6
Mixture density
2.75
0.909
0.748
6.47
5.95
+0.29
S7
Extra latent targets
2.73
0.909
0.751
6.52
5.96
+0.01
Scaling parameters iterations
3.12
0.930
0.679
6.82
6.50
+0.54
A1
Two-stage training
2.63
0.904
0.783
6.35
5.76
0.20
A2
No SigmaBlock
2.53
0.897
0.816
6.42
5.64
-0.2
A3 No auxiliary losses
2.55
0.872
0.767
6.27
5.59
-0.37
already results in an improvement with respect to B3, with objective scores that are very close to B2 (Table 1, S1). Next, we add two-layer GRU and convolutional blocks after the skip summation. We also add a one-layer GRU to the generator latent. Both GRUs should provide the model with a larger receptive field and better sequential processing capabilities. With this additions, the model increases the COMP score to 4.75 (Table 1, S2).
After improving the encoders' capacity, we decide add waveform loss to the existing MSE on log-mel latents. To do so, we also need to include additional decoder blocks for up-sampling the latent (Fig. 2-S3). For the waveform loss, we use MSE together with a multi-resolution STFT loss (Yamamoto et al., 2020). The result is noticeable improvement (Table 1, S3). Although this is largest improvement in objective scores, we do not observe such a large difference throughout informal listening. Better low-level details are present, but we suspect a large part of the score improvement is due to the objective measures paying too much attention to (sometimes irrelevant) low-level detail, which is now induced by the losses in the waveform domain.
Now that we have parallel up-sampling blocks in the generator and conditioning decoders, we condition at multiple resolutions between 100 Hz and 16 kHz (Fig. 2-S4). This provides improvement (Table S4). However, after careful listening, we have the impression that loss errors have a strong effect on the conditioning, provoking alterations that difficult the task of the generator (for instance, we hear a bit of muffled speech, presumably resulting from using MSE in both mel-band and waveform domains). To try to alleviate these issues, we decouple the loss calculations from the main signal path of the conditioning network. To do so, we use two convolutional heads (Fig. 2-S5), which include layer normalization, PReL.Us and convolution with kernel width of With that, we observe an improvement not only in objective metrics (Table 1, S5), but also in subjective quality.
Decoupled auxiliary heads allow us to further think of alternative loss functions that might work better for latent and waveform representations. Interestingly, such loss functions do not need to be restricted to regression or adversarial approaches, as used the literature, but can be probabilistic and model the representations' distribution explicitly. Now that losses are out of the signal path, nothing prevents us from using approaches that otherwise would require to sample from a probabilistic prediction in order to proceed with the signal flow. With the idea of better modeling the distribution of both log-mels and waveforms, we employ a mixture density network (MDN) approach (Bishop, 1994), with the same head architecture before, but with 3 Gaussian components, each calculated from different convolutional layers (Fig. 2-S6). Assuming independent time steps, the negative log-likelihood of each time step using ls components of dimensionality d and diagonal covariance is
d
(4)
2
yj
m
LMNN -In
exp
1=1
d/2
T==1
2
==1

---

Under review as a conference paper at ICLR 2023
where is the target representation and a (9), m and stO) denote the output of the convolutional layer corresponding to the mixing probability, the mean, and the standard deviation of the i-th Gaussian, respectively. Replacing standard losses by MDNs has positive effect on the objective scores (Table 1, S6). Together with moving the loss computation out of the signal path (S5), they provide a COMP increment of 0.41, the largest one besides using an auxiliary decoder and loss (S3).
After introducing the MDNs, our last step consists of including more targets in the latent predictions. These additional targets are pitch and harmonicity, as provided by crepe (Kim et al., 2018), voice activity detection (VAD) and loudness, provided by simple internal algorithms, and the deltas of all of them. We consider a separate MDN for each target type. That is, one for mel bands and their deltas, one for pitch/harmonicit and their deltas, and one for VAD/loudness and their deltas. The result only provides a marginal improvement in objective scores (Table 1, S7), but we find such improvement to be consistent after listening to number of enhanced signals.
Scaling up (F) - -inally. after defining our base model, we can train a larger model for longer time. We increase the model size by doubling the number of channels and we reduce the learning rate by two. The resulting model has 189 M parameters, and is trained for 2.5 M iterations using a batch size of 64. This takes less than 14 days using 8 Tesla V100 GPUs. The result of scaling up parameters and training is a large improvement, both objectively and subjectively (Table I, F). This is the model we will use in our final evaluation, with 50 diffusion steps (see Sec. 4.3).
Further ablations (A1-A3) - Starting with S7, we can further assess the effect of some design choices. For instance, it is common in speech enhancement to train multi-part models using multiple stages, or taking some pre-trained (frozen) parts (Polyak et al., 2021; Maiti & Mandel, 2019; Nair & Koishida, 2021: Liu et al., 2021). The equivalent of this strategy for our two-part model would be to first train the conditioner using all LMDN losses, freeze it, and then train the generator with LSCORE- An intuition for this could be that, this way, the generator network is 'decoupled' from the enhancement/condiiion commitment (generating and cleaning, respectively). Nonetheless, this intuition seems be at least partially wrong, as we obtain worse scores (Table 1, AI). In fact, this result points towards 'coupling' situation, in which part of the generator performs some enhancement and part of the conditioner shapes the generator input.
one, and that therefore the two networks can fully concentrate in their
Another design choice we can question is the strategy to provide the diffusion model with the noise level a (SigmaBlock,, Fig. 1). This strategy is used by some audio generation models (like CRASH by Rouard & Hadjeres (2021), from which we borrow it), but other models employ other strategies or none. We observe that the use of this strategy is important for our generator network (Table 1, A2). Finally, we can also quantify the effect of the additional losses LMDN. By removing them and training the whole model only with LSCORE, we observe clear decrease in objective scores (Table 1, A3). Thus, we conclude that both auxiliary noise levels and losses are important.
4 RESULTS
4.1 COMPARISON WITH EXISTING APPROACHES
To compare with the state of the art, a common approach is use objective metrics and well- established test sets. However, since the task of universal speech enhancement has not been formally addressed before, an established test set does not exist. Furthermore, it is not yet clear if common objective metrics provide a reasonable measurement for the enhancement of other distortions beyond additive noise and reverberation. An alternative is to evaluate on separate, individual test sets, each of them focused on a particular task (for example denoising, declipping, codec artifact removal, and so on). However, to the best of our knowledge, there do not exist well-established test sets nor metrics for other tasks beyond denoising. Therefore, to be the most fair possible to existing approaches, and in order to skip potentially flawed objective metrics, we think the best approach is to conduct subjective test with expert listeners (Sec. 3.1 and Appendix D). Nonetheless, in Appendix E we also report objective scores for the two most common denoising test sets and show that the proposed approach achieves competitive results despite being generative and not favored by objective metrics.
In our subjective test, we compare against 12 existing approaches on different combinations of enhancement tasks (Table 2). We find that UNIVERSE outperforms all existing approaches by
7

---

Under review as a conference paper at ICLR 2023
Table 2: Preference test results, including indication of the model class (regression, adversarial, generative) and the considered distortions (noise, reverb, bandwidth reduction, clipping, codec artifacts, and others). Subjects' preference (other/cxisting approach or UNIVERSE) shown on the right, together with statistical significance (binomial test, 0.05, Holm-Bonferroni adjustment).
Approach
Class
Distortions
Preference (%
Noise
Rev
BW
Clip
Codee
Others
Other
UNIVERSE
Demucs (Défossez et al., 2020)
Reg
2.3
97.7
(Fu MetrieGAN+
2021)
Adv
2.3
97.7
PERL-AE (Kataria et al., 2021)
Reg
4.5
95.5"
Speech Reg. (Polyak et 2021)
Gen
4.5
95.5
SPEC-GAN (Su
2019)
Adv
6.8
93.2
HiFi-GAN-2 et al., 2021)
Adv
22.7
77.3*
WSRGlow (Zhang ct al., 2021b) 202
Gen
6.8
93.2
SEANet (Li et al., 2021b)
Adv
27.3
72.7
GSEGAN (Pascual al., 2019)
Gen
0.0
100.0
DNN-S (Mack & Habets, 2019)
Reg
2.3
92.7
CT TT--Ut (Nair & Koishida, 2021)
Reg
4,5
95.5
VoiceFixer (Liu et al., 2021)
Gen
29.5
70.5
UNIVERSE
Gen
n/a
n/a
UNIVERSE-Ree -RRggrss
Reg
11.4
88.6
UNIVERSE-Denee Denoise
Gen
43.2
56.8
Ground truth oracle
n/a
86.4
13.6
significant margin (Table top). In all considered distortions, UNIVERSE is preferred by expert listeners when compared to the corresponding competitor. The closest competitors are HiFi-GAN-2, which considers denoising, dereverb, and equalization, SEANet, which only considers bandwidth extension, and VoiceFixer, which considers denoising, dereverb, bandwidth extension, and declipping. We encourage the reader to listen to the UNIVERSE-enhande examples in the companion website
4.2 VARIATIONS AND INSIGHTS
Another interesting set of results stems from comparing against ablations ground truth data (Table 2, bottom). In particular, we try to answer the following questions:
1. Do we find clear gain from using generative, diffusion- -hased approach compared to using classical regression losses? To answer this question, we performed several preliminary experi- ments with different regression-bast based alternatives, and concluded that the best candidate for the subjective test which, instead of a score matching loss for the generator network and diffusion- based sampling, MSE and STFT losses for direct waveform prediction (the conditioner network was still found to be superior with the MDN losses). The result of this approach was not at the level the generative version (Table UNIVERSE-Regrest especially with regard speech distortion and artifacts. Listeners preferred the generative version 88 % of the time.
version of UNIVERSE with exactly the same configuration and capacity
2. Can it be a problem for the model to consider more distortions beyond additive noise? To assess this, we trained exactly the same version of UNIVERSE with 500 train set that consisted of only additive noise mixtures, and evaluated only in the additive noise case. The reason for using 1/3 of the hours used for the universal enhancement case is that estimate that this number not far from the amount of real-world additive noise in the full multi-distortion train set. In this case, the results indicate that adding extra data and extra distortions does not affect performance (Table 2, 57 % preference), albeit not a significant one.
In fact, there seems to be slight advantage in doing so (43 vs.
3. How far we from the ideal targets? To gain intuition about this question, included recordings from the target speech to the subjective test (with input references featuring all considered distortions). In this case, listeners preferred the ideal targets 86 % of the time (Table 2, Ground truth oracle). This, beyond confirming that listeners were able to spot the clean
Link removed to preserve anonymity. Examples attached to the current submission in a 7IP file. To foster future comparison, we will also provide the first (random) 100 pairs of our validation the website.
8

---

Under review as a conference paper at ICLR 2023
10
6
COMP
RTF
0.2
3
e=
2.3
0.1
-0-
6-1.
631
2
0.05
4
8
16
32
64
128
256
512
N
Figure 3: Speed-quality trade-off when varying sampling parameters. Speed is measured by the real-time factor on Tesla V100 GPU (RTF; gray dashed line) and quality is measured by COMP (colored solid lines). Sampling parameters are the number of iterations N and the constant (Sec. 3.2). Results for the other objective metries are in Appendix E.
references, indicates that there still some room for improvement for UNIVERSE. Informal listening by the authors indicates that two of the most problematic cases are with very loud noises and strong (and usually long) reverbs. The former tends to yield some babbling while the latter yields noticeable speech distortions and, in some cases, also babbling.
4.3 SPEED-QUALITY TRADEOFF
Score-based diffusion models require performing multiple denoising steps for high-quality sampling, and efficient strategies tackle this issue have been the subject of recent research. In the literature. we find that high-quality sampling can be obtained with relatively few steps for tasks that have rich conditioning signal such as speech vocoding (for example, less than 10 steps as Chen et al. (2021); Kong et al. (2021)). Speech enhancement, as formulated here, should also be considered task with a rich conditioning signal (essentially, low-level speech details are contained the input, except for the distorted parts). Therefore, hypothesize that achieved with relatively few steps. Our results confirm this hypothesis and show that we can obtain good quality synthesis with few 4 denoising steps, with speed above 10 times real-timo on GPU (Fig. 3 and Appendix E). Importantly, this holds for a variety of values of e, and without the of any specific strategy nor any search for an appropriate schedule of (we plain geometric scheduling).
synthesis
also be
CONCLUSION
In this work, we consider the task of speech enhancement as holistic endeavor, and propose universal speech enhancer that makes use of score-based diffusion for generation and MDN auxiliary heads for conditioning. We show that this approach, UNIVERSE, outperforms 12 state-of-the-ard approaches as evaluated by expert listeners enhancement with only 4-8 diffuuson steps. Regarding the potential societal impact, we do not foresee any serious implications at this initial research stage. Despite being a generative model, the nature of the task enforces to enhance what is being input to the system without changing major characteristics, keeping content and intent absolutely unaltered. Thus, for example, the system should not change the speaker's identity or words unless attacked by third party (we explicitly ask expert listeners to consider identity/word changes in their judgment). Finally, it is also worth noting that data-driven models highly depend on the training data characteristics. Accordingly, before deploying such models, one should ensure that target languages and use cases are sufficiently represented the train set (we explicitly include multiple languages and recording conditions in our training set).
subjective test, and that it can achieve high-quality

---

Under review as a conference paper at ICLR 2023
REFERENCES
B. D. O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their
Applications, 12:313-323, 1982.
C. M. Bishop. Mixture density networks. Technical report, Aston University, UK, 1994. 6, 18
O. Chang, D. N. Tran, and Koishida. Single-channel speech enhancement using learnable loss mixup. In Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPEECH), pp. 2696-2700 2021. 23
N. Chen, Y. Zhang, H. Zen, R. Weiss, M. Norouzi, and W. Chan. WaveGrad: estimating gradients for waveform generation. Proc. of the Int. Conf. on Learning Representations (ICLR), 2021. 5, 9, 16
Dang, H. Chen, and Zhang. DPT-FSNet: dual-path transformer based full-band and sub-band fusion network for speech enhancement. In Proc. of the IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), 2022. 23
A. Défossez, G. Synnaeve, and Adi. Real time speech enhancement the waveform domain. In Proc. of the Int. Speech Comm, Assoc. Conf. (INTERSPEECH)), pp. 3291-3295, 2020. 1, 2, 8, 23
R. Fejgin, J. Klejsa, L. Villemoes, and C. Zhou, Source coding of audio signals with generative model. In Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 341-345, 2020. 23
E. Fonseca, M. Plakal, D. P. W. E. Ellis, F. Font, X. Favory, and X. Serra. Learning sound event clas- sifiers from web audio with noisy labels. In Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 21-25, 2019. URL WtLLhtt::///wwtt www.::wwwur.oooooeeeeeooo..oo.ee FSDno sy 3
S.-W. Fu, C. Yu, T.-A. Hsieh, P. Plantinga, M. Ravanelli, X. Lu, and Y. Tsao. MetricGAN+: an improved version of MetricGAN for speech enhancement. In Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPEECHI, pp. 201-205, 2021. 1, 8,
X. Hao, X. Su, R. Horaud, and X. Li. FullSubNet: a full-band and sub-band fusion model for real-time single-channel speech enhancement. In Proc. of the IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), pp. 6633-6637, 2021. 2, 23
G. E. Henter, T. Merritt, M. Shannon, C. Mayo, and S. King. Repeated Harvard sentence prompts corpus version 0.5. Technical report, University of Edinburgh, The Centre for Speech and Technology Research (CSTR), 2014. URL https::/aoi.org/// 00..488/////9. ds/39.
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 6840-6851. Curran Associates. Inc., 2020. 2, 15
A. Hyvärinen. Estimation of non-normalieed statistical models by score matching. Journal of Machine Learning Research, 6(24))695-779 2005. ISSN 1532-4435. 15
U. Isik, Giri, Phansalkar, J.-M. Valin, K. Helwani, and Krishnaswamy. PoCoNet: better speech enhancement with frequency- positional and biased loss. In Proc, of the Int. Speech Comm. Assoc. Conf. (INTÉRSPEECH), pp. 2487-2491, 2020. 2, 23
frequency-positionaa embeddings, semi semi-supervised conversational data,
W. A. Jassim, J. Skoglund, M. Chinen, and A. Hines. Warp-Q: quality prediction for generative neural speech codecs. In Proc. of the IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), pp. 401-405, 2021. 1. 3, 21, 23
M. Jeong, H. Kim, S. J. Cheon, B. J. Choi, and N. S. Kim. Diff-TTS: denoising diffusion model for text-to-speech. In Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPEECH), pp. 3605-3609, 2021. 2
10

---

Under review as a conference paper at ICLR 2023
A.
R. Piché-Taillefer R. T. des Combes, and I. Mitliagkas. Adversarial score matching and improved sampling for image generation. In Proc. of the Int. Conf. on Learning Representations (ICLR), 2021. 3, 15, 16
S. Kataria, J. Villalba, and N. Dehak. Perceptual loss based speech denoising with an ensemble of audio pattern recognition and self-supervised models. In Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 7118-7122, 2021. 2, 8, 23
E. Kim and Seo. SE-Conformer: time-domain speech enhancement using conformer. In Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPEECTI), pp. 2736-2740, 2021. 2, 23
J. W. Kim, Salamon, P. Li, and J. P. Bello. Crepe: a convolutional representation for pitch estimation. In Proc. the 2018. 7
the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 161-165,
A. Kolmogorov. Interpolation and extrapolation of stationary random sequences. Izv. Akad. Nauk SSSR Ser. Mat., 5:3-14, 1941. 1
Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro. DiffWave: a versatile diffusion model for audio synthesis. In Proc. of the Int. Conf. on Learning Representations (ICLR), 2021. 2, 5,5,
J. Lee and S. Han. NU-Wave: a diffusion probabilistic model for neural audio upsampling.
arXiv:2104.022 023212 2021. 2,5
A. Li, W. Liu, C. Zheng, C. Fan, and X. Li. Two heads are better than one: a two-stage complex spectral mapping approach for monaural speech enhancement. IEEE/ACM Transactions Audio, Speech, and Language Processing, 29:18 1e:188991843 2021a. 23
Y. Li, M. Tagliasacchi, O. Rybakov, V. Ungureanu, and D. Roblek. Real-time speech frequency bandwidth extension. In Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 691-699, 2021b. 8
H. Liu, Q. Kong, Q. Tian, Y. Zhao, D. Wang, C. Huang, and Y. Wang. VoiceFixer: toward general speech restoration with neural vocoder. ArXiv: 1109.13731,, 2021. 7,
C. Loizou. Speech enhancement. CRC Press, 2013. 1, 21
X. Lu, Y. Tsao, S. Matsuda, and C. Hori. Speech enhancement based on deep denoising autoencoder. In Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPPEECH), pp. 436-440, 2013.
Y.-J. LU, Tsao, and S. Watanabe. A study on speech enhancement based on diffusion probabilistic model. In Proc. of Asia Pucific Signal and Information Proc. Assoc. Annual Submit and Conf. (APSIPA), 2021. 2, 23
Y.J. Lu, Z.-Q. Wang, S. Watanabe, A. Richard, C. Yu, and Y. Tsao. Conditional diffusion probabilistic model for speech enhancement. ArXiv: 2202.05256, 2022. 2, 23
Lv, Y. Hu, S. Zhang, and Xie. DCCRN+: channel-wise subband DCCRN with SNR estimation for speech enhancement. In Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPEEC)), pp. 2816 2820, 2021. 23
W. Mack and E. A. P. Habets. Declipping speech using deep filtering. In Proc. of the IEEE Workshop on Appl. of Signal Proc. to Audio and Acoustics (WASPAA), pp. 200-204, 2019, 8
S. Maiti and M. I. Mandel. Parametric resynthesis with neural vocoders. In Proc. of the IEEE Workshop on Appl. of Signal Proc. to Audio and Acoustics (WASPAA), pp. 303-307, 2019. 1,7
S. Maiti and M. I. Mandel. Speaker independence of neural vocoders and their effect on parametric resynthesis speech enhancement. Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP). pp. 206-210, 2020. 23
A. A. Nair and K. Koishida. Cascaded time + time- frequeenr UNet for speech enhancement: jointly addressing clipping, codec distortions, and gaps. In Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 7153-7157 2021. 7,8
11

---

Under review as a conference paper at ICLR 2023
S. Pascual, A. Bonafonte, and J. Serrà. SEGAN: speech enhancement generative adversarial network. In Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPEECH), pp. 8642-3646, 2017. 1, 23
S. Pascual, J. Serrà, and A. Bonafonte. Towards generalized speech enhancement with generative adversarial networks. Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPEECH), pP. 161-165, 2019. 2,
A. Paszke, S. Gross, Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, Bai, and S. Chintala. PyTorch: an imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems (NeurlPS), volume 32, pp. 0024-8035. Curran Associates, Inc., 2019. 5,
E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville. FiLM: visual reasoning with general conditioning layer. In Proc. of the AAAI Conf. on Artificial Intelligence (AAAI), volume 32, pp. 3942-3951, 2018. 4, 18
H. Phan, V. McLoughlin, L. Pham, O. Y. Chén, P. Koch, M. de Vos, and A. Mertins. Improving GANs for speech enhancement. IEEE Signal Processing Letters, 27:1700-1704, 2020. 23
A. Polyak, L. Wolf, Y. Adi, O. Kabeli, and Y. Taigman. High fidelity speech regeneration with application speech enhancement. In Proc. of the IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), pp. 7143-7147, 2021. 1, 7,8
J. Pons, S. Pascual, G. Cengarle, and J. Serrà. Upsampling artifacts in neural audio synthesis. In Proc. of the IEEE lnt. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 3005-3009, 2021. 17
V. Popov, Vovk, V. Gogoryan, T. Sadekova, and M. Kudinov, Grad-TTS: diffusion probabilistic model for text-to-speech. arXiv:2105.003: 2021.
C. K. A. Reddy, V. Gopal, R. Cutler, E. Beyrami, R. Cheng, Dubey, S. Matusevych, R. Aichner, A. Aazami, S. Braun, P. Rana, S. Srinivasan, and J. Gehrke. The INTERSPEECH 2020 deep noise suppression challenge: datasets, subjective testing framework, and challenge results. In Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPEECH), pp. 2492 2496, 2020. URL https: m/asttrr........ 22
github.com//c www.oo/mmmccooooiooomoooooomoooe DDNS
D. Rethage, Pons, and X. Serra. A WaveNet for speech demoising. In Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 5069-5073, 2018.
A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra. Perceptual evaluation of speech quality (PESQ) - new method for speech quality assessment of telephone networks and codecs. IEEE Int. Conf. on Acoustics, Speech, and Signal Processing (ICASSP), volume 2, pp. 749-752, 2001. 21
S. Rouard and G. Hadjeres. CRASH: raw audio score-based generative modeling for controllable high-resolution drum sound synthesis. In Proc. of the Int. Soc. for Music Information Retrieval Conf. (ISMIR), pp. 579-585, 2021. 4,
S. Routray and Q. Mao. Phase sensitive masking-based single channel speech enhancement using conditional generative adversarial network. Computer Speech & Language, 71:101270, 2022. 23
J. Serrà, S. Pascual, and J. Pons. On tuming consistent annealed sampling for denoising score matching. arXiv: 2104.00 03720, 2021a. 16
Serrà, J. Pons, and S. Pascual. SESQA: sumi-super learning for speech quality assessment. In Proc. of the IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), pp. 381-385, 2021b. 21
J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proc. of the Int. Conf. on Machine learning (ICML), pp. 2256-2265, 2015. 2
12

---

Under review as a conference paper at ICLR 2023
Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, pp. Curran Associates, Inc., 2019. 2, 15, 16
Y. Song and S. Ermon. Improved techniques for training score-based generative models. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 12438-12448, Curran Associates, Inc., 2020. 3, 15, 16
Y. Song, Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based gener- ative modeling through stochastic differential equations. In Proc. of the Int. Conf. on Learning Representations (ICLR), 2021. 3, 15, 16
M. Strauss and B. Edler. A flow-based neural network for time domain speech enhancement. In Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 5754-5758, 2021. 23
Su, A. Finkelstein, and Z.. Jin. pureetuulll-mmotivatee environment-specciii speech enhancement. In Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 7015-7019, 2019. 1,8 8
J. Su, Z. Jin, and A. Finkelstein. HiFi-GAN-2: studio-quality speech enhancement via generative adversarial networks conditioned acoustic features. In Proc. of the IEEE Workshop on Appl. of Signal Proc. to Audio and Acoustics (WASPAA), 2021. 1, 2, 8, 23
S. Särkkä and A. Solin. Applied stochastic differential equations, volume 10, Cambridge University Press, 2019. 15
H. Taal, R. C. Hendriks, R. Heusdens, and Jensen. A short-time objective intelligibility measure for time-frey weighted noisy speech. In Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 4214 4217, 2010. 3,21
Thiemann, N. Ito, and Vincent. DEMAND: a collection of multi-channel recordings of acoustic noise in diverse environments (1.0). In Proc. of the Int. Congress on Acoustics (ICA), 2013. URL https: / www.odd.oo///ecorr//222222
C.
Noisy speech database for training speech enhancement algorithms and TTS models. Technical report, University of Edinburgh, The Centre for Speech and Technology Research (CSTR), 2017. URL https: 22
P. Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 13(7):1661-1674 2011. 15, 16
S. Welker, J. Richter, and T. Gerkmann. Speech enhancement with score-based generative models in the complex STFT domain. ArXiv: 2203. / 17004, 2022. 2
N. Wiener. Interpolation, extrapolation, and smoothing of stationary time series. MIT Press, Cambridge: USA, 1949.
Y. Yamagishi, C. Veaux, and K. MacDonald. CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice cloning toolkit (version 0.92). Technical report, University of Edinburgh, The Centre for Speech and Technology Research (CSTR), 2019. URL https: http:///o..oooo.////////1 ds/ 2645. 3
R. Yamamoto, F. Song, and J.-M. Kim. Parallel WaveGAN: a fast waveform generation model based generative adversarial networks with multi-resolution spectrogram. In Proc. of the IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 6199-6203, 2020. 6
G. Yu, Y. Wang, H. Wang, Q. Zhang, and C. Zheng. A two-stage complex network using cycle- consistent generative adversarial networks for speech enhancement. Speech Communication, 134: 42-54, 2021. 23
J. Zhang, S. Jayasuriya, and V. Berisha. Restoring degraded speech via a modified diffusion model. In Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPFEECH) pp. 221-225, 2021a. 1, 2,
13

---

Under review as a conference paper at ICLR 2023
K. Zhang, Y. Ren, C. Xu, and Z. Zhao. WSRGlow: Glow-based waveform generative model for audio super-resolution. In Proc. of the Int. Speech Comm. Assoc. Conf. (INTERSPEECH), pp- (649-1653, 2021b. 2,
C. Zheng, X. Peng, Y. Zhang, S. Srinivasan, and Y. Lu. Interactive speech and noise modeling for speech enhancement. In Proc. the the AAAI Conf. on Artificial Intelligence (AAAI), pp. 14549-14557, 2021. 2, 23
14

---

Under review as a conference paper at ICLR 2023
APPENDIX
A SCORE-BASED DIFFUSION MODELS
A.1 THEORY
Diffusion-based models are defined through a forward process where we progressively add noise to samples from the data distribution, Xo Pdata- until we obtain result that is indistinguishable from a prior tractable distribution, X1 ~ Pknown- In the case of Gaussian noise, Pknown also becomes approximately Gaussian, and this process can be modeled (Song et al., 2021) through a stochastic differential equation (SDE):
dx f
g())dw,
where f.g IR R and w is the standard Wiener process (Brownian motion) indexed by continuous time variable E [0, 1]. Different definitions of f and g yield different but equivalent processes (Song et al., 2021). To model the backward process where we go from Pknown to Pdata we can then employ the reverse-time SDE (Anderson, 1982)
dx fx. t) V log Px)) dt + g(t)dw,
(3)
where w is the standard Wiener process in which time flows backward, and V, Px()) corresponds to the score of pt. the marginal distribution at time (Hyvärinen, 2005). Thus, once f and are defined, in order to obtain Xo Pdata we only need to know the score function Vx log pe (x) sample X1 ~ Pknown and simulate the process of Eq. 3.
Since does not typically have direct access to neither pt Vx Px(), the solution is to approximate the latter with a neural network S(x, t). In order to train S, Vincent (201 1) showed that, for given I, minimizing the score matching objective
Ex,
|S(x,, t) V. log
equivalent to minimizing the denoising objective
C = 1x0 Exo
[[( t) V, 1xgo)(x)/x0)11
2
where (x, Xu) corresponds to Gaussian kernel (Song al., 2021; Särkkä & Solin, 2019), the tran- sition kernel for the forward SDE (Eq. 2). For all 6, one can use the continuous generalization (Song & Ermon, 2019)
de || t) Vx (x [x))
L= E,EEE,INe No Ex,
(4)
with ~ L/(0 1), where X an appropriately chosen weight that depends on (Ho al., 2020; Song & Ermon, 2020).
Sampling with score-based diffusion models is done by simulating or solving the reverse-time SDE (Eq. 3) with finite (discrete) time schedule. This can be done in many ways, for instance by using numerical SDE and ODE solvers, as introduced by Song et al. (2021). Other schemes that have shown competitive performance include predictor-corrector schemes (Song al., 2021), ancestral sampling (Ho et al., 2020), and variations of Langevin dynamics (Song Ermon, 2020; Molicoeur-Martinear al., 2021).
A.2 IN PRACTICE
In our work, we employ the so-called variance exploding schedule (VE; Song et al., 2021), which corresponds to choosing
(t (
fxx t) 0 and
g(t)= -
(5)
dt
in Eq. 2. Then, the associated transition kernel for the forward process (Särkkä & Solin, 2019) is Pe (x, (x)== N (xx; Xo, [o? - -).... which in practice approximated by
pc (x, (x) N (x:; Xo, of
(6)
15

---

Under review as a conference paper at ICLR 2023
since du 0 (see also Song et al., 2021). The intuition is that Pt=0 becomes indistinguishable from Pdata= and that Pt=1 becomes indistinguishable from Pknown (a Gaussian distribution). In other words, perturbation and signal should become imperceptible at 0 and 1, respectively. In the VE schedule, the scale the signal Xo is kept intact, and then it corresponds to g to fulfill that notion through variance schedule (see Eq. 5). Therefore, do should be negligible while di should be large, compared to the variability of Xo. That is, of < [x [[xo])²] < of.
The of the approximated transition kernel (Eq. 6) implies that x, No JzZe, 2 (, I), and that
X log (x((Xo))) = Xxo C
(x: xo)²
X, Xo
20
o
where C' is a constant (see also Vincent, 20).)). Substituting these into Eq. and operating yields
Ae S(x0 0.Z2, at 2
f = E,, Exo
2
From Eq. 5, now remains to set how a evolves. Song & Ermon (2019; 2020) choose a geometric progression for di,
maxx
0t miin
mmin
and provide some generic guidance on how select mmin and 0max- They also justify setting proportional to the noise variance at time L: A o7. Using this weighting yields
C R, Exx
1 1
Io, + ozz. +
Finally, instead of using directly as input for S, we use and train S with continuum of noise scales (Chen et al., 2021). In addition, we need to use some conditioning information as an indication of what to generate (in our case, signal derived from the distorted speech x). With that, our training loss becomes
C C E,E Exo
1o,5(zx +0,,, =1122
This is the loss denoted as LSCORE the main paper.
For sampling with the VE schedule, we resort to consistent annealed sampling (Mliiceuu-Maartnneau et al., 2021). We use the discretization of E [0, 1] into N uniform steps tn = (n - 1) - 1). {1.... N). which implies discretized progressions for and o; Starting N, we initialize
Niy = Oe y, and then recursively compute
Nt.. + -oo S(x,,. C, ot..) ) Bo Zn-17
for n = N,N - 1.... Finally, at = 1 (f1 = 0), we take
Xo X + Sxx. Co).)
which corresponds to the empirically denoised sample dolicceur-Marrinean et al., 2021). The values of 1 and B are determined through the parameterization (Serrà al., 2021a)
2
17
1
=
3 V
where 7 E (0, 1) is the ratio of the geometric progression of the noise, 7 = / and the Note that the value of changes with the chosen
constant [1, 00) is left number of discretization steps N, thus facilitating hyper-parameter search for continuous noise scales different N (Serrà et al., 2021a).
16

---

Under review as a conference paper at ICLR 2023
B IMPLEMENTATION DETAILS
Architecture overview - The architecture contains two main parts that are jointly trained: the generator network, which is tasked with estimating the score of the perturbed speech distributions, and the conditioner network, which creates the conditioning signal required for synthesizing speech with preserved speaker characteristics. Both the generator and the conditioner networks are essentially encoder-decoder architectures, enhanced with various conditioning signals and skip connections (see figure in main paper). The encoding and decoding is done by down- and up-sampling, respectively, using convolutional blocks, which are the main building blocks of UNIVERSE. Throughout the architecture, the marti-pa rectified linear unit (PReI.U) activation function is used prior to all the layers (except where we specify otherwise), and all convolutional layers are one dimensional. Unless stated otherwise, we use Py Py'oorch's ' defaults (Paszke et al., 2019, version 1.10.I).
Convolutional block The convolutional block consists of core part which is common between all blocks, and an optional prior or posterior part that can exist depending on the block's s functionality (Fig. 4-B). The core part contains a convolutional layer of kernel width 5, followed by two convolu- tional layers of kernel width 3, and we use residual sum between its input and output. All residual sums and skip connections are weighted by 1/VF, where r is the number of elements in the sum.
If the convolutional block will be used for up-sampling, the core's input is processed initially by transposed convolutional layer of kernel width and stride equal to the up-sampling ratio, without padding (Pons et al., 2021). On the contrary, if the block will be used for down-sampling the output of the core is processed by a strided convolutional layer of kernel width and stride equal to the fown-sampling ratio, also without padding. Every down-sampling operation is accompanied by channel expansion of factor 2, and up-sampling by a reduction of 2. We start with 32 channels in the encoder and have 512 in the latent. Depending on block's position in the architecture we use skip connections and a maximum of two types of conditioning signals, all injected to a dedicated part in the convolutional block (Fig. 4-B). If a signal will be extracted with skip connection, we take from the output of the residual sum, and insert it to the target block before the residual signal.
A: SigmaBlock
B: ConvBlock
MDN
hn
{a, m,
PReLU
Conv
Linear
StConv
PReLU
PReLU
Vo
Linear
h
Conv
PReLU
log
g
C;
Co
Conv
PReLU
V;
TrConv
PReLU
h
Figure 4: Diagram of the individual blocks not depicted in the main paper. Dashed connections and blocks are optional, depending on the functionality of the block.
17

---

Under review as a conference paper at ICLR 2023
In order to preserve speaker characteristics that are consistent throughout a frame, we use conditioning signals between the generator and the conditioner. These conditioning signals are taken from the output of the first convolutional layer of the core, and inserted to the same position in the target block's core. Furthermore, a secondary conditioning signal containing information about the noise level is injected to all the convolutional blocks of the generator. This conditioning is done with FiLM (Perez et al., 2018).
The conditioner network contains stem cell, down-sampling encoder, recursive middle part, and an up-sampling decoder. Both the encoder and decoder are made up of convolutional blocks that perform only down-sampling and up-sampling, respectively. The distorted signal is down-sampled to a 100 sampling rate, with the encoder following a (2,4,4.5) factor progression, recursively processed in the middle part, and up-sampled the original sampling rate with the decoder, mirroring the encoder's 's progression. At each encoder block, we use skip connections, process them with adaptor networks, and sum the adapted signals with the output of the last encoder block. The adaptor networks consist of strided convolutions in order to match the sampling rate of the summation inputs.
Conditioner network-
Alongside this main path, we extract and normalize 80 mel band features from the distorted signal, process them with a convolutional layer followed by a convolutional block, and add the result to the summation of the down-sampled signals. The hop size is equal to the total down-sampling rate, preserving the sampling rate of the features during processing. We process the feature-enriched signal with the conditioner's middle part, consisting of convolutional block, two-layer bi-directional GRU with a residual sum, and a final convolutional block, all of them preserving the sampling rate. We finally up-sample the latent representation to the original sampling rate with the decoder, using an initial convolutional block preserving the sampling rate and numerous blocks performing up-sampling. We obtain hierarchical conditioning for the generator network by taking conditioning signals from each block of the decoder, corresponding to all the sampling rates. In order to adapt these representations to the ones of the generator network, we process each conditioning signal with linear layers.
Generator network The generator network also an encoder-decoder architecture with UNet- like skip connections, where there is a direct connection between each encoder and decoder block pair sharing the same sampling rate. We use the same down- and up-sampling progression as in the conditioner network. hence achieving parallel up-sampling blocks between the generator and conditioner decoders. However, we limit the middle part to only a single bi-directional GRU layer without residual sums. The output of the final block is processed by convolutional layer to reduce its dimension back to the score's dimension.
The generator is conditioned on two signals: Fourier features depending on the current noise level and the hierarchical conditioning extracted from the distorted speech. For the former, we embed the noise level with logarithmic compression and use it to modulate frequencies of random Fourier features (RFFs) sampled from a standard normal distribution. Following the extraction of RFFs, we process them with fully connected network of 3 times repeated linear layers followed by PReL.S to obtain noise level embeddings. This block, which we call SigmaBlock, expands 32 pairs of Fourier coefficients into 256 channels. (Fig. 4-A). After the embeddings are extracted, they are adapted to each block's 's frame rate and dimension with a linear projection layer. The conditioning signals taken from the conditioner network's decoder are injected to the generator network's 's decoder at each block with the same sampling rate, using summation. This way, the estimated score is conditioned with multiple sampling rates starting from the lowest (100 Hz), until the original (16 kHz).
Training objective - The main loss is the denoising score matching loss outlined in the main paper and derived in Appendix A.2. Additionally, we introduce auxiliary losses in order to structure the conditioner network's 's latent representation and prime its final block's 's output to perform speech enhancement. We use separate mixture density networks (MDN; Bishop. 1994) to model the distributions of the clean signal waveform and features. Each feature and its delta distribution is jointly estimated with an MDN that models the distribution using multivariate Gaussian components with diagonal covariance. The parameters of the mixture components are learned with convolutional layers of kernel width 3 and layer normalization (Fig. 4-C). We calculate the negative log-likelihodd (NLL) of the feature MDNs and average them. Then, the total objective is to minimize the sum of the denoising score matching loss, waveform NLL, and feature NLLs. Here, we would like to underline
18

---

Under review as a conference paper at ICLR 2023
that auxiliary losses generative, as they model continuous data distributions, making UNIVERSE a -elle-ggeerttive model (that is, not using any classification or regression losses). Moreover, as the auxiliary losses are taken out of the main signal path, we do not need any sampling procedure neither in training nor in validation stages.
Optimization - The resulting model contains about 49 M parameters. We train it with a batch size of 32, where each sample is an approximately two-scconds long frame with 16 kHz sampling rate, and using automatic mixed precision. We apply 1 M parameter updates, which takes approximately four and a half days using 2 Tesla V100 GPUs in parallel. The large/final model has 189 M parameters, is trained for 2.5 M iterations using a batch size of 64, and takes less than 14 days using 8 Tesla V100 GPUs. We use the Adam optimizer with a maximum learning rate of 2 10-4 and schedule the learning rate with cosine scheduler. The schedule includes warm-up of 50 k iterations, increasing the learning rate from starting point 1.6 - 10 We additionally implement a manual weight decay, excluding biases and the PReLU weights, and set its coefficient to 0.01.
C DATA AND DISTORTIONS
To create our training data, we sample chunks of 3.5-5.5 seconds from speech recordings of an internal pool of data sets, featuring multiple speakers, languages, prosody/emotions, and diverse recording conditions. Speech is pre-selected to be clean, but nonetheless can contain minimal background noise and reverb. We down-sample 16 kHz mono and follow two different signal paths to create input and target pairs. With distortions that introduce delays we automatically time-align at the sample level using a custom correlation function.
Input- - To programmatically generate distorted versions, we randomly choose the number of distortions between (1, 2, 5) with probabilities (0.35, 0.45, 0.15, 0.04, 0.01), respectively. Next, we cascade distortion types with random parameters (bounds for distortion parameters are selected such that distortions are perceptually noticeable by expert listeners; random selection is typically uniform, except for parameters that have an intuitive logarithmic behavior, such as frequency, in which case we sample uniformly in logarithmic space). Table summarizes the distortion families and types we consider, together with their weights (distortion type weights will define the probability of selecting each distortion type). In total, we consider 55 distortions, which we group into 10 families.
The ranges we consider for the distortion parameters include: SNRs between -5 and 25 dB for all added noises, slopes between 0.01 and 0.7 for colored noise, DC components between 10-6 and 10-1 non-stationary noise lengths between 20 and 350 ms, frequency ranges between 100 and 7500 Hz for filters and tones (except low and high pass, with a minimum and maximum of 1 kHz, respectively), Qs between 0.1 and 2 for filters, gains between - 12 and d for filters and random equalization, between and 20 mel bands for random equalization, gap lengths between 20 and 80 ms, temporal probabilities between 0.5 and 3 times/sec for transmission distortions, percentiles between 0.5 and 50 for elipping, resample frequencies of 4, 6, 8, 11, 12, and 14 kHz, wet proportions between 0.2 and for reverbs, augmentation of early and tail reflections for room impulse responses, room sizes between 3 and 1000 m² for algorithmic reverbs, bit rates between 2 and 96 kbps for codecs, compression ratios between 2 and 10, window lengths between 2 and 212 for spectral manipulations, amounts between 0.5 and 0.8 for spectral manipulation probability, wet proportions between 0.1 and 0.75 for other effects, and linear gains between and for plosive and sibilance addition.
Target - Given that the design of UNIVERSE does not rely on assumptions regarding the nature of the distortions (for instance, good example would be the assumption that noise is additive), nothing prevents us from using a different target than the original signal that was used as input to the distortion chain. In addition, we want to ensure that the generated signal is of the top/highest quality, which is a characteristic that is not shared by some of the original clean speech signals that we sample. Therefore, we decide to apply a small enhancement chain to clean the signals to be used as target. In particular, we apply four consecutive steps?:
1. Denoising We employ a denoiser to remove any (minimal) amount of noise that might be present in the recording. Since the pre-selected speech is already of good quality, we expect that
All of these algorithms are available https: //d.lby.ic////////ooooo
19

---

Under review as a conference paper at ICLR 2023
Table 3: Considered distortion types, grouped by family.
ID
Family
Type
Weight Randomized parameters
Band limiting
Band pass filter
Frequencies. filter characteristics.
High pass filter
Frequency, filter characteristies.
Low pass lilter
20
Frequency, filter churactemsticss Frequency, method.
4
Down-sample
30
Codec
AC3 codec
Bit rate, codec configuration. Bit rate, codec configuration. Bit rate, codec configuration.
EAC3 codec
MDCT codec
15
MP2 codec
Bit coder configuration.
MP3 codec
20
Bit rate. codec configuration. Mu.
Mu-law guantization
OGG/Vorbis codec
Bit rate, codec configuration. Bit rate, codec configuration.
12
OPUS codec
15
13
OPUS codec 2
Bit rate, codec configuration.
14
Distortion
More plosiveness
10
Gain,
15
More sibilance
10
Gain,
Overdrive
Gain, harmonicity.
17
Threshold elipping
Gain.
Loudness dynamnis
Compressor
10
Ratio, compressor churacteristics,
19
Destroy levels
20
Gains, durations, temporal probability.
20
Noise gating
10
Gate characteristics.
Simple compressor
Ratio.
22
Simple expansor
Ratio.
23
Tremolo
Tremolo characteristics.
24
Equalization
Band reject filter
Frequencies, filter characteristics. Number bands, gains. Frequency, filter characteristics.
Random equalizer
15
Two-pole filter
10
Recorded noise
Additive noise
SNR, noise type (cafeteria, traffic, nature, class- room, keyboard, plane, music, ...).
28
Impulsional additive noise
30
SNR, noise type, temporal probability.
29
Reverb/delay
Algorithmic reverb
30 Gain, reverb characteristics.
Algorithmic reverb
Gain, reverb characteristics.
Chorus
Gain, chorus characteristics.
32
Phaser
Gain, phaser characteristies.
33
RIR convolution
120 Gain, room impulse response (RIR), augment. Gain, delay time.
Very short delay
35
Spectral manipulation
Convolved spectrogram
Window, amount.
Griffin-Lim
Window, amount.
37
Phase randomization
Window, amount.
38
Phase shuffle
Window, amount.
Spectral holes
Window, amount.
Spectral noise
Window, amount.
41
Synthetic noise
Colored noise
15 SNR, slope.
42
DC component
Amplitude.
43
Electricity tone
SNR. frequency. type of waveform. SNR, slope, duration, temporal probability. Amplitude, duration, temporal probubility. SNR, frequency, duration, temporal prob. SNR, frequency, duration, temporul prob. SNR, frequency, type of waveform.
Non-stationary colored noise
45
Non-stutionary component
46
Non-stationary electricity tone
47
Non-stutionary rundom tone
Random tone
Transmission
Frame shuffle
10 Length, temporal probubility. Length, temporal probubility, gain. Length, temporal probubility, SNR. Length, temporal probubility, gain. Length, temporal prohability.
Insert attenuation
51
Inseri noise
52
Perturb amplitude
Sample duplicate
54
Silent gap (packet loss)
15 Length, temporal probubility.
55
Telephonic speech
10
Frequencies, compression ratio, filter type.
any decent denoiser will have no problems and will not introduce any noticeable distortion. We carefully verified that by listening several examples.
2. Deplosive -
We employ a deplosive algorithm, which is composed of detection and processing stages. The processing acts only the level of the plosive bands.
20

---

Under review as a conference paper at ICLR 2023
3. Deesser- We employ a basic deesser algorithm, which is composed of detection and processing stages. The processing acts only on the level of the sibilant bands.
4. Dynamic EQ - We also employ a signal processing tool that aims at bringing the speech to a predefined equalization target. The target is level-dependent, that the algorithm can act compressor and/or expander depending on the characteristics of the input speech.
Interestingly, passing the original clean recordings through this chain forces the model to not rely on bypassing input characteristics (especially low-level characteristics), which we find particularly relevant to remove other non- pon-desirblle input characteristics. In addition, the chain provides a more homogeneous character to the target speech, which should translate into some characteristic 'imprint' or "signature sound' of UNIVERSE.
D EVALUATION
D.1 OBJECTIVE MEASURES
We report results with COVL (Loizou, 2013) and STOI (Taal et al., 2010) as computed by the pysepm' package using default parameters (version June 30, 2021). WARP-Q (Jassim et al., 2021) used also with default parameters* (version March 14, 2021) and SESQA is reference-based version of the reference-free speech quality measure presented in Serrà et al. (2021b), trained on comparable data and losses as explained by the authors. These four objective measures are only used to aid in the assessment of the development steps presented the main paper. PESQ (Rix et al., 2001), COVL, and STOI are also used to compare with the state of the art in the task of speech denoising in Table 4 below. Calculation of PESQ is also done using pysepm².
D.2 SUBJECTIVE TEST
We perform a subjective test considering 15 competitor approaches (12 existing ones plus 3 ablations of the proposed approach, as reported in the main paper). For each existing approach, we download distorted and enhanced pairs of speech recordings from the corresponding demo websites, and use those as the main materials for the test. We randomly select at least 25 pairs per system, and remove few ones for which we consider the distorted signal is too simple/casy to enhance (for instance, signals that have very high SNR when evaluating denoising system, almost no clipping when evaluating a de-cplipping system, and so forth). This is done to reduce potentially ambivalent results in the test, as the two systems would presumably perform very well and the listener would be confused on which to choose (removed pairs were never more than 5 per approach). For a few systems that do not have enough material on their demo page, we download the code and enhance a random selection of distorted input signals from other systems. That is the case of Demucs, MetricGAN+ and VoiceFixer. For the ablations of the proposed approach, we take the first (random) 25 pairs of our validation set. In total, we have over (12+3)xX2 x enhanced signals from which to judge preference. All distorted signals are enhanced by UNIVERSE using N 64 diffusion steps and hyper-parameter = 2.3.
A total of 22 expert listeners voluntarily participated in the test. The test features set of instructions and then shows list of triplets from which to perform judgment (Fig. 5). Every time a listener lands on the test page, triplets of signals are uniformly randomly selected for each approach: input-distorted signal (Input), competitor-enhanced signal (A or B), and UNIVERSFE-nhame order of or as well as the order of the triplets is also uniformly randomly selected when landing to the web page. Preference for an enhanced signal can only be A or B lforced-choice test). Every subject listens two triplets per competitor system, performing total of 30 preference choices for 15 approaches (two per approach, yielding total of 22x2 preference judgments per system).
signal (A or B). The
The results of the test are based on counting preference choices per competitor approach (% of preference). Statistical significance is determined with a binomial test using < 0.05 and the Holm nolf--onnerrooi adjustment to compensate for multiple comparisons. the results table, we also
'https: //gi thub.com/ schm ph2 ppseem 'https: / / chub.com/ wwjsssi WARP-Q
'We always use wide-hand PESQ (also for computing COVI).. The only exception are the results of Table 4, where we report both narrow- and wide-hand PESQ (COVI. is still computed with wine-hand PESQ).
21

---

Under review as a conference paper at ICLR 2023
SPEECH ENHANCEMENT PREFERENCE TEST
Welcoma!
In this prefenence tast you'll have listen to trpletts spaech recordings. These riplets formed by an "Inpur" reference audio and two nupputs, corsisting erhanced versions inpul, produced by system system You need choces which outpul you profer,
INSTRUCTIONS:
Do the environment. Use Alll in the questoomaire the end the page and click the submit buttonn Othenwise the results are not sent.
headphones high-end bouispeskers.
pay unnatural insiiintioon
your
criteria:
background noise, background artfactss, revarberation, clipping, oodec distortions,
Voice quality This includes absence voice artlactss, good inteligbiity.. naturafness the speech, regarding identity, Intoration,
Identity and prosody includes changes
Thank you for your collaboration.
Preference:
Input
6:00 (0:00
0600 2005
0.00 02:6
Preference:
Input
Preference:
Figure 5: Screenshot of the subjective test interface. On the top, instructions are given to expert listeners. On the bottom, random test triplets are provided the listeners. For every test, subjects listen to two triplets comparing UNIVERSE with an existing approach, and enter preference for 15 of such approaches.
display which distortions were tackled by the competitor approach. Note that, since testing materials have different distortions and contents per approach, one cannot infer ranking of systems based on preference % (that is, one cannot compare preferences between rows of the table; actually they even correspond to different tasks: denoising, de-clipping. restoring codec artifacts, etc.). Preference % only represents a pairwise comparison between UNIVERSE and the corresponding competitor approach listed on the left of the table.
E ADDITIONAL RESULTS
E. DENOISING TASK WITH OBJECTIVE METRICS
To have comparison on well-established benchmark, can consider the task of speech denoising, which has long tradition in the speech community and, in the last years, has seen a consolidation of test data sets and objective metrics (this something that, the best our knowledge, has not yet happened with other tasks like dereverberation. declipping, bandwidth extension, and so on). Two widely-used used data sets with an established test partition are BhisebankDDEMANN
2017) and IS20 DNS- DNS-Challenge (Reddy et al., 2020). Since the standard objective metrics are PESQ. COVL, and STOI, and given that UNIVERSE produces audio with further enhancements like leveling/dynamies specifically for the denoising task. This is important because, otherwise, UNIVERSE would be performing additional enhancements besides denoising and, more importantly, the standard objective metrics would not find an agreement between the ground truth and the estimated clean speech, what would result in the whole evaluation being unfair UNIVERSE.
specific equalization, we need to train new version of UNIVERSE
22

---

Under review as a conference paper at ICLR 2023
Table 4: Comparison with the state of the art the speech denoising task using objective metrics (for all metrics, the higher the better). The first block of the table contains results for existing generative approaches (upper part), while the second block of the table contains results for regression/alversaraal approaches (middle part). Bottom rows correspond to the proposed approach UNIVERSE.
Approach
DoiceBan--DEMAR
IS20 DNS Challenge (no rev)
PESQNB
PESQ
PESQNB
PESQ
COVL
SEGAN (Pascual al., 2017)
2.16
2,80
DSEGAN (Phan
2020)
2.39
2.90
0.93
SE-Flow (Strauss & Edler, 2021)
2,43
3.09
DitfuSE (Lu ct ul.,20023 2021)
2.44
3,03
CDiffuSE (Lu al., 2022)
2.52
PR- PR-WaveGlow (Maiti & Mandel, 2020)
3.10
0.91
CycleGAN-DCD (Yu al. 2021)
2,90
3,49
0.94
PSMGAN (Routray & Mao, 2022)
2,92
3,52
PoCoNet (Isik
2020)
2.75
3.42
FullSubNet (Hao et al., 2021)
3.31
2.78
0.96
DCCRN+ (Lv
2021)
2.84
3.33
CTS-Net (Li et al., 2021a)
2.92
3.59
3.42
2.94
0.97
Demues (Dffosscz ct 2020)
3,07
3.63
0.95
SN-Net (Zheng et al., 2021)
3.12
3.39
SE-Conformer (Kim Sco, 2021)
3.12
3.82
0.95
MetrioGAN+ (Fu ct al., 2021)
3.15
3.64
PERL-AE (Kataria et al., 2021)
3.17
3.83
0.95
HiFi-GAN-2 (Su et al., 2021)
3.18
3,84
Loss Mixup (Chang al., 2021)
3,26
3.91
DPT-FSNet (Dang al.. 2022)
3,33
4.00
0.96
3,26
0.98
SAIVERSE-Denolse
3.85
3,21
3.68
0.95
3.58
3,01
3.60
0.97
UNIVERSE-Denone
3.94
3.33
3.82
0.96
3.73
3.17
3.75
0.98
In addition, because these standard metrics known to disfavor generative approaches due to lack of waveform alignment or minor spectral nuances (see Jassim et al., 2021, for a discussion and further pointers), we need to devise method to produce outputs that are 'less generative' and can therefore better coincide with what standard metrics measure. Inspired by Fejgin et al. (2020), we decide use an expectation of the enhanced waveform. Intuitively, that expectation should minimize to certain extent different nuances introduced by the generative model, especially regarding small misalignments and minor spectral nuances. To compute such expectation, we sample times using the same conditioning (distorted) signal, and take the sample average of the resulting waveforms. While this has some audible effect such as lowering the presence of high frequencies, clearly shows boost in the standard objective metrics, probably because they focus more small misalignments and low frequencies. We use E to denote this version of the approach. We also want to stress that we solely use this expectation version for the result in the corresponding row of Table 4.
Table shows the results for the denoising task. In it, we can observe the difference in standard metrics between generative and regression/adversarial approaches (first blocks of the table). The best- performing approaches in the generative block struggle to get the numbers of the worse-performin approaches in the vegeesiion/adversaraaal block (by listening some examples from both blocks, we believe this is metrics issue since we could not find a clear perceptual difference). Interestingly, we observe that even the generative version of the proposed approach SANIVERE--Denoise clearly surpasses all existing generative approaches in terms of standard metrics. In addition, the 'less generative' variant using an expectation over multiple realizations (UNIVERSE-Denoisee clear improvement for those metrics, with values that become competitive with the non- non-generative state-of-the-aa The results of our subjective test, which include some of the competing approaches in Table 4, suggest that UNIVERSE outperforms those in such more realistic evaluation.
shows a
E.2 SPEED-QUALITY TRADE-OFF
For completeness, provide the speed-quality plots for every considered objective metric in Fig, 6. As in the main paper, synthesis parameters are the number of denoising iterations N and the hyper- parameter (Sec. A.2). The real-time factor (RTF) is defined the time to process recording using
23

---

Under review as a conference paper at ICLR 2023
10
3.0
0.925
0,900
2.5
COVI.
1
RTF
STO)
1
2.0
0.5
0,850
1.5
0.2
0.2
+
[-13
1-23
0.1
1-23
1.0
m111
0,800
0.05
0.05
8
16
32
128
256
512
2
8
16
32
64
128
256
512
N
N
1.3
10
10
-3
1.2
1.1
-2
22
WARP-Q
SESQA
1.0
RIT
0.5
0.5
0.9
0.8
0.2
0.2
0.7
0.1
11
1-13
0.05
0.05
N
Figure 6: Speed-quality trade-off when varying synthesis parameters. Speed measured by the real-time factor on a Tesla V100 GPU (RTF; gray dashed line) and quality is measured with the considered objective metrics on the validation set (colored solid lines).
a single Tesla V100 GPU divided by the duration of that recording (for instance, if UNIVERSE takes seconds for enhancing 20-second recording, RTF=0.1 and say is 10 times faster than real time).
24


Reviews:
[{"review_id": "oHNQWzdg8J", "reviewer": "Reviewer_yPpC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on various types of speech distortions (a total of 55) and aims to realize universal speech enhancement based on score-diffusion-based models. The paper carefully describes their complicated architecture based on the various attempts motivated by related studies and incrementally improves the architecture described in Section 3. Overall, their proposed neural network architecture and training methods are novel. The paper also shows the effectiveness of the proposed method by comparing various other state-of-the-art methods with their speech distortion setups based on the subjective preference test.", "summary_of_the_review": "The paper has sufficient novelties in the problem setup and modeling parts. However, I'm concerned about the lack of reproducibility of this method and the fair comparisons with the existing methods in the conventional metrics.\n\nOther suggestions:\n- It is better to discuss speech separation since it is regarded as another critical speech distortion in the broad sense.\n- Section 2, the last paragraph: I'm not very sure about the effectiveness of the generative model method for this setup. Since we could simulate most of these distortions, we can make a discriminate model with the artificially created pair data. Please clarify this point.\n- Why does the evaluation metric not include PESQ?\n- Figure 1(a): I'm very curious how we predict \"g\" since $\\log (\\sigma)$ seems to be a scale, and it may not have enough information to predict it. Probably, I misunderstand $\\log (\\sigma)$. It would be great if the authors clarify it.\n", "strength_and_weaknesses": "Strength\n- the first part (abstract and introduction) of this paper is very well written, and I can easily understand the motivation of this work.\n- the related studies for diffusion-based models are adequate.\n- although the proposed architecture and training method are complicated, the authors provide detailed explanations of how they evolved their architecture and method with the experimental evidence.\n- task setup and their modeling are novel\n\nWeaknesses\n- The paper has weak reproducibility (see below).\n- The explanation of the model (especially Section 3.2) and some experimental discussions (e.g., Table 4) are not self-consistent (see my comments below).\n- The modeling efforts and task setup should be separately evaluated. For example, it is better to show the effectiveness of the proposed model with the same training data for the VCTK+demand experiments since the training data is different. We could also train the conventional models with the same training data.\n- another significant distortion comes from interference speakers. I expect the paper to have bubble noise, but speech separation should also be considered if we call this method \"universal\" speech enhancement.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "koQFyO17Bvh", "reviewer": "Reviewer_XfqC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a \"universal\" speech enhancement model that handles 55 different distortions including background noise, reverberation, codec artifacts, etc. The model uses a UNet-like generator that has been trained with a diffusion loss. The paper further conducts a side-by-side evaluation with 22 expert listeners, who preferred the model's output compared to competing models.", "summary_of_the_review": "As written, I do not think this paper is suitable for ICLR. While the authors claim impressive results on subjective side-by-side evaluations on an impressive number of speech distortions, it is hard to see how others in the ICLR community can build upon this work since there is no new ML technique introduced and the empirical findings are generally not reproducible. At ICLR, there is similar precedent for a well-engineered visual speech recognition system that also did not introduce new ML techniques and whose empirical findings are generally not reproducible (https://openreview.net/forum?id=HJxpDiC5tX). The reviewers recommended that the paper would be more suitable at an applications-oriented venue. I would make the same recommendation in this case.\n\nIf the authors would like to revise their work to make it more suitable for ICLR, I would suggest the following:\n1) Create a benchmark task for evaluating \"universal\" speech enhancement. This would involve open-sourcing code for programmatically generating their 55 speech distortions.\n2) Propose new automated metrics that holistically evaluates \"universal\" speech enhancement, and show that they correlate well with subjective side-by-side listener evaluations. This builds upon a long precedent in the speech enhancement literature (PESQ, COVL, STOI, etc) and would be an immensely valuable contribution for the community.\n3) Use a simpler and less \"hacky\" deep learning architecture that is easy for others to replicate by themselves, or release code for it to aid others in their reproducibility efforts.\n\nFinally, I would like to point the authors' attention to several related work in the literature for Table 4. [1], [2], [3], [4] outperforms Universe-Denoise on VoiceBank-Demand. [1], [3] outperforms Universe-Denoise-E on VoiceBank-Demand. [1] outperforms both Universe-Denoise and Universe-Denoise-E on IS20 DNS Challenge. Given that Universe-Denoise-E is an expectation rather than the output of a deterministic model, I think it also helps to provide distribution statistics like median and std to help readers understand its performance. Unfortunately, the related work does cast some doubt on whether a \"universal\" speech enhancement model can perform as competitively as specialized speech enhancement models, but in my opinion, the presence of better-performing specialized models would not detract from the contribution of a \"universal\" model.\n\n[1] DPT-FSNet: Dual-Path Transformer Based Full-Band and Sub-Band Fusion Network for Speech Enhancement. Dang et al.\n\n[2] Single-Channel Speech Enhancement using Learnable Loss Mixup. Chang et al.\n\n[3] CMGAN: Conformer-based Metric GAN for Speech Enhancement. Cao et al.\n\n[4] Dual-Branch Attention-In-Attention Transformer for Single-Channel Speech Enhancement. Yu et al.\n ", "strength_and_weaknesses": "Strengths:\n- To the best of my knowledge, this paper is the first in the speech enhancement literature to handle this many speech distortions. 55 distortions from 10 different families are considered: bandwidth reduction, clipping, codec artifacts, silent gaps, excessive dynamics\ncompression/expansion, sub-optimal equalization, noise gating, and others.\n- While it is not the first paper that proposes to use diffusion for speech enhancement, the authors show that it performs better than existing approaches on subjective side-by-side evaluations, as well as automated metrics on the VoiceBank-Demand test set.\n\nWeaknesses:\n- While an impressive engineering feat, the paper contributes little to the community's theoretical or empirical understanding of speech enhancement or diffusion.\n- The findings in the paper are also generally not reproducible. The authors use internal training data and programmatically generate distorted speech, but neither the training data nor code for the programmatic distortions are made available. The paper proposes a very \"hacky\" U-Net architecture with numerous non-trivial and non-standard modifications, but while the ablation study is useful, it is hard to see how others can easily replicate this without the authors releasing the code for it. Finally, the side-by-side evaluations are also subjective and not generally reproducible. I think it is a huge missed opportunity, because the authors could have taken the chance to establish a new benchmark task and associated metrics for \"universal\" speech enhancement.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gX6FiphmCuI", "reviewer": "Reviewer_BhNH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes diffusion models based speech enhancement. In particular, it extends enhancement to a large number of degradations beyond additive noise. The degradation includes common speech degradations such as codec artifacts, bandwidth reduction, reverb etc. The proposed approach puts together different pieces of diffusion-based learning in the context of speech enhancement and then different variations on the top of the base approach are applied to achieve improved performances. ", "summary_of_the_review": "This work proposes speech enhancement through diffusion based approaches. It trains a single model to learn from a large number of speech degradations. \n\n\n1. There is a waveform direct waveform based loss too - the significance of which seems to be very high. Doesn’t that default the whole process to so many other speech enhancement models where MSE  on waveform or spectrograms is common. Moreover, there are other latent targets which are also added like pitch, VAD, loudness. One of the versions uses noise level information too. I wonder how much some of the other prior methods can be improved just by adding these additional latent targets and  these auxiliary information. \n\n\n\n2. In Section 4.2.1 authors discuss a regression based approach with Universe architecture. Along with waveform and STFT losses are other losses  (as in UNIVERSE like latent targets etc, ) used here ? \n\n\n3. I am not sure how fair the comparison is with prior works here. \nFirst of all the proposed model is likely orders of magnitude bigger than several of the prior works. \nIn fact, some of them might be causal as well (which proposed model is not) and causality can have significant impact on performance.\nMoreover, the training data is different for different algorithms whereas here a large corpus of private data is used. But importantly, several of these prior works were never trained on such degradations.  I wonder what performance would look like if we just take them and train them with all these other degradations. \nI think decoupling the point above is necessary because it will tell us whether variety in speech degradations is important or diffusion-based learning is bringing something to the table. \n\n\n\n3. It would be good to show how this model performs in really difficult conditions. For example, a large number of state of the art methods do a really good job on enhancement but struggle in low-SNRs.  A model like the current one (very large, non-causal) are clearly not the most practical speech enhancement systems which often have real-time uses. Nevertheless, this work puts together a whole bunch of pieces to build the enhancement system. Considering this, stronger results would have been expected, and especially in situations where current state of the art struggles. How does this model compare against prior works in really difficult quality conditions?\n\n\n\n4. While I like subjective tests in the paper, I think adding MOS subjective scores would add a lot of value. It will clarify a lot of different aspects. Comparative tests do not give an idea of absolute performance of UNIVERSE or others. \n\n-- updates after rebuttal --\nupdated the score and comments. ", "strength_and_weaknesses": "Strengths\n\n\n1. Diffusion based approaches have been very successful in generative tasks in other domains. While their applications to the audio domain are starting to show up, the success is perhaps not yet at the level in other domains like images. To this end, the authors' attempt on using diffusion based learning for speech enhancement is a good idea. Extending it to degradations beyond additive noise also makes sense. \n\n\n2. I liked that authors did subjective tests as well. \n\n\nWeaknesses\n\n\n1. The experimental section leaves quite a few things unanswered. More on it in the review summary below - for which I would like to see the author's response. \n\n\n2. I think the novelty of the diffusion-learning process is limited. It largely follows prior works. \n\n\n3. Some questions below are part of the weaknesses. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VzjvxlXrQu", "reviewer": "Reviewer_5RnM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a universal diffusion model for speech enhancement for a wide variety of distortions (Table 3 in the appendix), in contrast to prior studies which focused on a limited set of distortion types (cf Table 2). The authors first presented a series of exploration of model architectures and auxiliary losses based on which the authors arrived at their final model. The authors then demonstrated that the proposed model, UNIVERSE, performs better than prior models developed for specific types of distortion as well as ablated models that are trained with regression loss or trained for a specific type of distortion. ", "summary_of_the_review": "It is exciting to see a universal model for speech enhancement based on diffusion with very strong performance. The quality of the work is good. However, it is hard to assess exactly why the proposed model is superior to prior work and how future work can be compared with the proposed model due to the use of non-public datasets.", "strength_and_weaknesses": "Strengths\n- This paper appears to be the first general model for speech enhancement based on diffusion. Empirical results are very strong. The samples provided in the supplementary materials are impressive especially when tested on the real world samples.\n- The presentation of how the authors arrived at their final model architecture and training losses is excellent. The authors motivated the design well. Results of intermediate models are included, serving as a nice reference for future development of audio enhancement models\n- It is also informative to include a baseline that is trained with regression loss.\n\nWeaknesses\n- The comparison with prior work may not be entirely fair considering that the proposed model is trained on a different dataset. Hence, it cannot be concluded from the paper that UNIVERSE is better than prior work because diffusion is a better objective of enhancement and/or the proposed model architecture is better. In addition, the target UNIVERSE is trained to predict is also enhanced as described in Appendix C, which might be another advantage of the proposed model over baselines. This question can be answered if a) the authors retrain the strongest baseline (e.g.,HiFi-GAN-v2) using the same dataset, or b) train the proposed model using the same public data as the prior work.\n- While the authors share the type of distortions used for corrupting clean speech as input, the details of the parameter used in each distortion are not provided (e.g., SNR for colored noise). How clean speech was selected /what noise dataset was used / how many hours of noise are used for training are not specified. It would be hard to reproduce the data setup given the paper.\n- The authors mentioned in the appendix that the target was enhanced (Sec C) but results comparing using original speech versus enhanced speech are not provided.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject