FINALLY: fast and universal speech enhancement with studio-like quality
Authors: Nicholas Babaev
Kirill Tamogashev
Azat Saginbaev
Ivan Shchekotov
Hanbin Bae
Hosang Sung
WonJun Lee
Hoon-Young Cho
Pavel Andreev
Venue: NeurIPS 2024 poster (2024)
Keywords: speech enhancement; generative models
Primary area: speech_and_audio
TL;DR: 2024 SoTA for speech enhancement

Abstract: In this paper, we address the challenge of speech enhancement in real-world recordings, which often contain various forms of distortion, such as background noise, reverberation, and microphone artifacts.
We revisit the use of Generative Adversarial Networks (GANs) for speech enhancement and theoretically show that GANs are naturally inclined to seek the point of maximum density within the conditional clean speech distribution, which, as we argue, is essential for speech enhancement task.
We study various feature extractors for perceptual loss to facilitate the stability of adversarial training, developing a methodology for probing the structure of the feature space.
This leads us to integrate WavLM-based perceptual loss into MS-STFT adversarial training pipeline, creating an effective and stable training procedure for the speech enhancement model.
The resulting speech enhancement model, which we refer to as FINALLY, builds upon the HiFi++ architecture, augmented with a WavLM encoder and a novel training pipeline.
Empirical results on various datasets confirm our model's ability to produce clear, high-quality speech at 48 kHz, achieving state-of-the-art performance in the field of speech enhancement. Demo page: https://samsunglabs.github.io/FINALLY-page/

FINALLY: fast and universal speech enhancement with studio-like quality
Nicholas Babaev" Samsung Research
Kirill Tamogashey" Samsung Research
Azat Saginbaev Samsung Research
Ivan Shchekotov Samsung Research
Hanbin Bae Samsung Research
Hosang Sung Samsung Research
WonJun Lee Samsung Reseach
Hoon- Young Cho Samsung Research
Hoon-Young
Pavel Andreev" Samsung Research
Abstract
In this paper, we address the challenge of speech enhancement in real-world recordings, which often contain various forms of distortion, such as background noise, reverberation, and microphone artefacts. We revisit the use of Generative Adversarial Networks (GANs) for speech enbancement and theoretically show that GANs are naturally inclined to seek the point of maximum density within the conditional clean speech distribution, which, as we argue, is essential for the speech enhancement task. We study various feature extractors for perceptual loss to facilitate the stability of adversarial training, developing methodology for probing the structure of the feature space. This leads us to integrate WavL.Mbaased perceptual loss into MS-STFT adversarial training pipeline, creating effecttvve and stable training procedure for the speech enhancement model. The resulting speech enhancement model, which we refer to as FINALLY, builds upon the Hili++ architecture, augmented with a WavLM encoder and novel training pipeline. Empirical results various datasets confirm our model's 's ability to produce clear, high-quality speech at 48 kHz, achieving state- of the-art performance in the field speech enbancement. Demo page: http://aanuunnaabbauuuuu.oo/o/oa
Introduction
Speech recordings often contaminated with background noise, reverberation, reduced frequency bandwidth, and other distortions. Unlike classical speech enhancement (Ephraim & Malah 1984 Pascual al. (017). which considers each task separately, universal speech enhancement (Serr et al. 2022: Su et al., 2021 Liu et al. 2022) aims to restore speech from all types of distortions simultaneously. Thus, universal speech enhancement seeks generalize across wide range of distortions, making it more suitable for real-world applications where multiple distortions may coexist.
Recent studies have categorized the problem speech enhancement as task of learning the clean speech distribution conditioned on degraded signals (Lemercier et al. [003]] Serrà et al. 2022 |0
Richter
al. 2023). This problem often addressed using diffusion models (Ho et al. 2020 Song et al. 2020), which are renowned for their exceptional ability to learn distributions. Diffusion models have recently achieved state-of-the-art results universal speech enhancement (Serrà el al. 2022).
"Equal contribution. Correspondence p.andreev @ samsung.com.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, the impressive performance of diffusion models comes with the high computational cost of their iterative inference process.
is important to note that the speech enhancement problem does not require the model to learn the entire conditional distribution. In practice, when presented with a noisy speech sample, the goal is often to obtain the most probable clean speech sample that retains the lexical content and voice of the original. This contrasts with applications such as text-lo-image synthesis (Ramesh al. 2022 Lee et al.
2024 Rombach et al. 2022), where the objective is to generate a variety of images for each text prompt due to the higher level of uncertainty and the need for diverse options to select the best image. For most speech enhancement applications, such as voice calls and compensation for poor recording conditions, capturing the entire conditional distribution is not necessary. Instead, it is more important to retrieve the most likely sample of this distribution (the main mode), which might be simpler task.
Diffusion models' main advantage over generative adversarial networks (GANs) (Goodfellow et al.) 2014) is their ability to capture different modes of the distribution. However, we argue that this prop- erty is not typically required for the task of speech enhancement and may unnecessarily complicate the operation of the neural network. Conversely, we show that GANs tend to retrieve the main mode of the distribution-preciiie
what speech enhancement should typically do.
Therefore, in this work, we revisit the GAN framework for speech enhancement and demonstrate that it provides rapid and high-quality universal speech enhancement. Our model outperforms both diffusion models and previous GAN- based models, achieving an unprecedented level of quality on both simulated and real-world data.
Our main contributions are as follows:
1.
We theoretically analyse the adversarial training with the least squares GAN (LS-GAN) loss and demonstrate that generator predicting single sample per input condition (producing a conditional distribution that is a delta function) is incentivized to select the point of maximum density. Therefore, we establish that LS-GAN training can implicitly regress for the main mode of the distribution, aligning with the objectives of the speech enhancement problem.
2.
We investigate various feature extractors as backbones for perceptual loss and propose criteria for selecting an extractor based on the structure of its feature space. These criteria are validated by empirical results from a neural vocoding task, indicating that the convolutional features of the WavLM neural network et al., 2022b) are well-suited for perceptual loss in speech generation.
3. We develop novel model for universal speech enhancement that integrates the proposed
perceptual loss with MS-STFT discriminator training (Défossez et al.) al.]] (023) and enhances
the architecture of the HiFi++ generator (Andreev et al. 2022) by combining it with a self-supervised pre-trained WavLM encoder (Chen et al. 2022b). Our final model delivers state-of-the-ard performance on real-world data, producing high-quality, studio-like speech at 48 kHz.
2 Mode Collapse and Speech Enhancement
The first question that we address is what is the practical purpose of speech enhancement model. The practical goal of a speech enhancement model is to restore the audio signal containing the speech characteristics of the original recording, including the voice, linguistic content, and prosody. Thus, loosely speaking, the purpose of the speech enhancement task for many applications is not "generative" in its essence, in the sense that the speech enhancement model should not generate new speech content but rather "refine" existing speech as if it was recorded in ideal conditions (studio-like quality). From the mathematical point of view, this means that the speech enhancement model should retrieve the most probable reconstruction of the clean speech y given the corrupted version x, i.e., y arg max, Petemn (v)
This formulation re-considers the probabilistic speech enhancement formulation, which widely used in the literature. In such formulation, the speech enhancement model is aimed to capture the entire conditional distribution Pelean (z). This formulation might be especially appealing in situations with high generation ambiguity, e.g., a low SNR scenario where clean speech content

---

could not be restored unambiguously. In this case, the speech enhancement model could be used to generate multiple reconstructions, the best of which is then selected by the end user. However, we note that this formulation might be redundant and not particularly relevant for many practical applications since the ambiguity in generation can be resolved by more straightforward means such as conditioning on linguistic content (Koizumi et al. (023c).
In practice, for many applications, more natural way of formalizing speech enhancement to treat it as a regression problem which aims at predicting the point of highest probability of the conditional distribution arg max, Petean (y)z). This formulation has the advantage of simplifying the task, since finding the highest mode of the distribution might be significantly easier than learning the entire distribution. Therefore. the speech enhancement models built for this formulation are likely to be more efficient after deployment since they solve a simpler task. We note that in the context of speech enhancement, the speed of inference is always of major concern in practice.
Given this formulation, we argue that the framework of generative adversarial networks (GANs) is more naturally suited for the speech enhancement problem than diffusion models. We show that GAN training naturally leads to the mode-seeking behaviour of the generator, which aligns with the formulation introduced above. Additionally, GANs enjoy one forward pass inference, which is in contrast to the iterative nature of diffusion models.
Let Py (y/2z) be family of waveform distributions produced by the generator (x))). Mao al. (2017) showed that training with Least Squares GAN (LS-GANN leads to the minimization of the Pearson X
divergence Xrearson [Pg] Po We propose that if (u(x)) approaches ō(y - go(x)) under some
parametrization, the minimization of this divergence leads to go(x) = arg max, Patean (y|ær). This means that if the generator deterministically predicts the clean waveform from the degraded signal, the LS-GAN loss encourages the generator to predict the point of maximum Petcan u/zz density. We note that although prior work by (Li & Farnia (023) demonstrated the mode-covering property for the optimization of Pearson x2 divergence, our result pertains to a deterministic generator setting, which is outside the scope of analysis provided by Li & Farnia (003)..
To prove this result, we consider the delta function as a limit of indicator density functions Pf (yl2z) go (x)) E [-1/E. 1 / and otherwise, where is the number of dimensions of y. Note that f (u]x))
<"/ /2"h. 1--- i.e., pf (y/zz) = --"/2" if
(z) dy 1 for any positive & and lime-++0o p (y)zz) 8(y go (z)). This approximation of the delta function by such a limit is practical due to the finite precision arithmetic used within computers; in other words, the delta function within computer arithmetic is actually an indicator function.
Proposition 1.
Let (yle) > 0 be a finite and Lipschitz continuous density function with unique then
global maximum and (5(()l)) = = /2" I
lim arg min X (pG) [Pelea + / = arg max Petean (y) ar) E-+ go(x)
y
Thus, LS-GAN training under ideal conditions should lead to the solution ge (z) arg max, Pokca (y/z) for the generator. In practice, however, success is highly dependent techni- calities, such as additional losses to stabilize training and architectures of neural networks. Below, we address these questions by revisiting the notion of perceptual loss for audio generation and assessing the effectiveness of neural architectures.
3 Perceptual Loss for Speech Generation
Adversarial training is known for its instability issues (Brock et al. 2018 It often leads to suboptimal solutions, mode collapse, and gradient explosions. For paired tasks, including speech enhancement, adversarial losses are often accompanied by additional regressive losses to stabilize training and guide the generator towards useful solutions (Kong et al. 2020 Su et al. 2020). In the context of GAN mode-sceking behaviour discussed above, regressive losses could be seen as merit to push the generator towards the "right" (most-probable) mode. Therefore, finding an appropriate regression loss guide adversarial training is of significant importance.
Historically, initial attempts to apply deep learning methods to speech enhancement were based on treating this problem as a predictive task (Defossez et al. 2020; Hao et al. 2021 Chen et al. 2022a
3

---

Isik et al., 2020). Following the principle of empirical risk minimization, the goal of predictive modelling is find a model with minimal average error over the training data. Given noisy waveform or spectrogram, these approaches attempt to predict the clean signal by minimizing point- wise distance in waveform and spectrum domains or jointly both domains, thus treating this problem as predictive task. However, given the severe degradations applied to the signal, there is an inherent uncertainty in the restoration the speech signal (i.e., given the degraded signal, the clean signal is not restored unambiguously), which often leads to oversmoothing (averaging) of the predicted speech. A similar phenomenon is widely known in computer vision (Ledig et al. 2017).
One promising idea to reduce the averaging effect is to choose an appropriate representation space for regression, which less "entangled" than waveform or spectrogram space. In simpler terms, the regression space should be designed so that averaged representation of sounds that are indistinguish- able to humans (such as the same phoneme spoken by the same speaker with the same prosody) still representation of this sound (see Appendix B.1)
We formulate two heuristic rules compare different regression spaces based on their structure:
Clustering rule: Representations of identical speech sounds should form one cluster that is separable from clusters formed by diffferent sounds.
SNR rule: Representations of speech sounds contaminated by different levels of additive noise should move away from the cluster of clean sounds monotonically with the increase in the noise level.
The clustering rule ensures that minimizing the distance between samples in the feature space causes the samples to correspond to the same sound. The SNR rule ensures that minimizing the distance between features does not contaminate the signal with noise, meaning noisy samples are placed distantly from clean samples.
Clustering rule
SNR rule
SNR 10
SNR
SNR 16
same sound
same sound
SNR SNR
same sound
12-
Figure 1: Illustration of heuristic rules for feature space structure. The Clustering rule (left) states that representations of the same speech sound should form clusters. The SNR rule (right) states that noise samples should deviate from the centre of the cluster as the amount of noise increases. Illustrations created using real samples are presented in Figure g Figure 7
In practice, we check these conditions by the following procedure:
1. We sample identical speech sounds with a multi-speaker VITS text-to-speech model (Kim let al., 2021). We define the group of waveforms corresponding to the same sound as 0.5-second waveform segments generated by the same speaker with the same text and the same phoneme duration (i.e., the stochastic duration predictor is used only once for each group). Thus, the waveform variation is produced due to sampling of latents from the prior distribution. We note that we do not explicitly fix the prosody; however, we observed that prosody variation was low by default and the sounds generated by this procedure were mostly perceptually indistinguishable from each other while corresponding to different waveform realizations. Overall, we define 354 groups of sounds with 80 speakers saying 177 different phrases, different speakers for each phrase; we sample 20 samples for each group, totalling 7080 waveform segments corresponding to 354 groups

---

2. Each waveform segment is then mapped to the feature space, and the resulting tensors are flattened obtain a vector for each sample. The vectors are clustered using K-means clustering (MacQueen et al. 1967). the number of clusters is set to the number of groups (354). After clustering, we compute the Rand index (Rand 1971) of the clustering produced by K-means with the clustering induced by initial groups of sounds splits. We treat the resulting number as a way to measure the adherence to the clustering rule.
3. Each waveform segment within group is randomly mixed with noise at SNR levels ranging from 10 to 20 dB. The segments are then mapped to the feature space to obtain vector for each sample. For each noisy sample, we compute the Euclidean distance between its features and the centre of the clean feature cluster. After that, we compute the negative Spearman's 's rank correlation between the SNR level and the distance from the sample the centre of the cluster. The negative correlations are averaged over all groups of sounds, and the resulting number treated as a quantitative measure of adherence the SNR rule.
Using these metrics, we assess the effectiveness of different feature spaces formed by several speech feature extractors, as well as conventional representations speech. Namely, we produce features by Wav2Vec 2.0 (Baevski et al. 2020), WavLM (Chen et al.) 2022b), the encoder of EnCodec (Détossez al.) 2023). and CDPAM (Manocha et al. 2022). As a conventional representation, we use waveform and spectrogram features. For Wav2 Vec 2.0 and WavLM, we consider the output of the last transformer layer and the output of the convolutional encoder as separate cases. We also train a HiFi-GAN generator (Kong et al.) (020) with each feature type used as representation for mean squared error loss computation on a neural vocoding task. To assess experimentally the suitability of the feature space to be used loss function for waveform generation, report MOS scores for samples generated by vocoders trained with each feature map. The results are presented Table] T
Table 1: Comparison of different features using Clustering rule, SNR rule, and MOS on neural vocoding.
Feature space
Rand score (t) (Clustering rule)
Negative correlation (t) (SNR rule)
MOS (t) (Vocoding)
Waveform
0.00 0.00
0.31 + 0.02
Failed
Spectrogram
0.00 + 0.00
0.08 + 0.03
1.78 + 0.08
Wav2 Vec 2.0
0.25 0.03
0.19 + 0.03
1.65 0.08
Wav2Vec 2.0-conv
0.94 0.01
0.78 0.02
2.23
WavLM
0.46 + 0.05
0.46 + 0.03
+.07
WavLM-conv
0.96 0.01
0.89 + 0.02
3.27 0.10
EnCodec
0.55 + 0.03
0.67 + 0.03
1.80 - 0.08
CDPAM
0.00 0.00
0.17 + 0.03
Failed
As can be seen, the features extracted by the convolutional encoder of the WavLM model present the best performance according to both our internal metrics and MOS score on the neural vocoding task. We have also tested the other layers of WavLM but did not observe significant improvements when using other layers (see Appendix [B.4). Empirically, we have found that if WavLM-conv feature loss is used together with spectrogram loss (L1-distance magnitudes of STFT), the performance on the vocoding task increases significantly, likely due the looseness of the WavLM representation (Appendix B.I. Without any ad adversarial training, the WavLMMccovv+LISTTE Equation (2) achieves an MOS score of 4.31 0.08, approaching the original adversarially trained HiFi GAN generator which achieves 4.69 = 0.05 (Appendix B.3
loss (LMOS-
4 FINALLY
4.1
Architecture
Our method is based on HiFi++ (Andreev et al. (022). The HiFi++ generator is a four-component neural network consisting of SpectraiUNet, Upsampler, WaveUNet, and SpectralMaskNet modules. SpectralUNet is responsible for initial preprocessing of audio in the spectral domain using two- dimensional convolutions. The Upsampler a HiFi-GAN generator-based module that increases the temporal resolution of the input tensor, mapping it to the waveform domain. WaveUNet performs
5

---

post-procees prsocssssssing in the waveform domain and improves the output of the Upsampler by incorporating phase information gleaned directly from the raw input waveform. Finally, SpectralMaskNet is applied to perform spectrum-based post-processing and, thus, remove any possible artefacts that remained after WaveUNet. Thus, the model alternates between time and frequency domains, allowing for effective audio restoration.
We introduce two modifications the HiFi++ generator's architecture. First, we modify the generator by incorporating WayLM- model output (last hidden state of the transformer) as an additional input to the Upsampler. Prior works (uugg et al. 2022 Byun al. 2023) have demonstrated the usefulness of Self- Supervisee Learning (SSL) features for speech enhancement tasks, and we validate this by observing significant performance gains from using SSL features. Second, we introduce the Upsample WaveUNet at the end of the generator. This module acts as a learnable upsampler of the signal sampling rate. For its architecture, we use the WaveUNet with an additional convolutional upsampling block in the decoder that upsamples the temporal resolution by 3 times. This allows the model to output 48 kHz signal while taking a kHz signal input.
WoulM
LMGS loss
(BSL retwork
Upsampler (ATI-GAN
Concel
WoveUMet
Speciall
Upcomppe Wieet
Fixal lose
Coper
Dhorrel tolors
SpectralUNel
PESO and
pIFt and nease
Spscrallther
Morittee Libel
Spectsaiitea for
Uoet Mitccaar
servid
Up
Figure 2: FINALLYY model architecture
4.2 Data and training
We use LibriTTS-R (Koizumi et al. 2023b), DAPS-clean (Mysore 2014) as the sources of clean speech data. LibriTTS-R is used at 16 kHz, while DAPS at 48 kHz. Noise samples were taken from the DNS dataset (Dubey et al. 2022). After mixing with noise, we apply several digital distortion effects (see Appendix D.2 for details).
We train the model in three stages. The first two stages concentrate on restoring the original speech content, and the final stage aims to enhance the aesthetic perception of the speech. The multi-stage approach is necessary due to the characteristics of the employed datasets: the LibriTTS-R dataset has a lot of samples but limited perceptual quality, whereas the DAPS dataset is high-qualit but contains a smaller number of samples. Consequently, we utilize the LibriTTS-R dataset for learning speech content restoration and the DAPS dataset for aesthetic stylization.
The loss functions that we use can be written as follows:
(f) =.y=p(xx,))
[100 (0(v) (690 (x)) + |[ST))())] (STFT(
(2)
2nd stage (16 kI12)
Cgen(0) ALMOS LIMOS (0) +AGAN LGAN-gen (0) + APM LFM(O) + AHH LHF(O), (0),
(3)
1st stage (16 kIlk)
3rd stage (48 kHz)
Caise (1.) LGAN == k.
(4)
Here, é denotes the WavLM-Conv feature mapping. (x) denotes the generator neural network with parameters 0, LGAANgenn(0 (0) denotes the LS-GAN generator loss (Mao et al. 2017). L()) denotes the combined generator loss, LGAN (yp:) denotes the LS-GAN discriminator Mao et al. 2017) loss for
the i-th discriminator with parameters sP. LrM denotes the feature matching loss (Kumar et al. 2019 Défossez et al. 2023). LeE denotes the human feedback loss, and A= denotes the corresponding loss

---

weights. Following Dérossez et al. (2023), we employ 5 discriminators with STFT window lengths of [2048, 1024, 512, 256, 128 for the 2nd stage and 5 discriminators with STFT window lengths of [4096, 2048, 1024, 512, 256] for the 3rd stage, which allows for the capture of spectral information at different resolutions.
Firstly, we train the FINALLY architecture without Upsample WaveUNet (we refer to this truncated architecture as FINALLY-16) at 16 kHz sampling rate on the LibriTTS-R dataset. We train the model with the proposed LMOS regression loss to provide the generator with better initialization before adversarial training.
Stage 1:
Stage 2: Second, we start adversarial training of FINALLY-16 with MS-STFT discriminators (Défos sez et al. 2023) and the LMOS loss. The learning of the generator undergoes linear warm-up to give the discriminators time to learn meaningful representations. At this stage, we focus the generator on producing reliable reconstruction of línguistic content by assigning larger values for LMOS and feature matching losses (AGAN gen = 0.4, APM = 20, ALMOS = 20).
Stage 3: Lastly, we attach Upsample WaveUNet to FINALLY-16 and start adversarial training of the FINALLY model to produce 48 kHz output. We subsample 48 kHz waveforms of the DAPS dataset and apply distortions to form the 16 kHz input. The discriminators are initialized randomly, and the learning rate for the generator undergoes a warm-up similar to the second stage. This stage is focused on producing the final output: therefore, we focus the generator perceptual quality by increasing the relative weight of GAN loss (AGAN-gen = 5, APM = 15, ALMOS = 0.5) and introducing additional human feedback loss Lur, which is based on UTMOS and PESQ metrics. The UTMOS
loss (Saeki et al. 2022) is based on a neural network model that simulates subjective MOS metric results. On the other hand, the PESQ loss delivers differentiable version of the PESQ metric (Rix iet al.] (0011),, as presented by Kim et al. (2019): Martin-Dona et al. (2018). Both UTMOS and PESQ help enhance speech aesthetic quality, as they incorporate insights from human preference studies. These metrics are differentiable with respect to their inputs, making them suitable for use as
loss functions (multiplied by negative constants) LHF = -20 LUTMOS 2 CPESQ AHF- 1.
5 Related Work
Self-supervised features for speech enhancement Several works have employed self-supervised
(SSL) features (Baevski et al. 2020 Chen al. 2022b Hsu et al. 202T) for training of speech
enhancement models as intermediate representation (Wang et al. 2024
Koizumi et al. 2023c).
auxiliary space for loss computation (Sato al. 2023 Hsieh et al. 2020 Close et al. 2023a b) or
input features (Hung et al. 2022 Byun et al. 2023). Sato et al. (2023): y; Close et al. [2023a b): Hsieh
al. (2020) proposed to use features of self-supervised models as an auxiliary space for regression loss computation. In our work, we similarly study the effect of SSL features on speech enhancement models; however, our study systematically compares different feature backbones and develops criteria for probing the structure of different feature spaces. In particular, we show that features of the WavLM's convolutional encoder are the most effective for the loss function, while recent work (Sato) al. 2023) has used the outputs of transformer layers. CCose et al. (2023 proposed similar loss function for speech enhancement based on convolutional features of HuBERT (Hsu et al,, 2021), our work can be considered extension of this work show that additional spectrogram loss greatly benefits the quality and provide experimental evidence for the advantages of LMOS--Lss usage for adversarial training.
2020 2021 consider a GaN-based GAN-based approaches The HiFi GAN works (Su et al.,
approach to speech enhancement, which is similar to ours. Importantly, these works base their generator architectures on feed-forward WaveNet (Rethage et al. 2018). a fully-convolutionall neural network that operates at full input resolution, leading slow training and inference times. We show that our model is able to achieve superior quality compared to this model while being much more efficient.
Koizumi et al. (023cc proposes to use w2v-BERT features (Chung et al., 2021 as intermediate representations for speech restoration. The features extracted from the noisy waveform are processed by a feature cleanser which is conditioned on the text representation extracted from transcripts via PnG-BERT, and speaker embedding extracted from the waveform. The feature cleanser is trained
htaps//ggiiuubccom///aoobbbbbobr
7

---

to minimize a regression loss between w2v-BERT features extracted from the clean signal and the predicted ones. the second stage, the WaveFit vocoder is trained to synthesize a waveform based on the predicted features (Koizumi et al. 2023a An important difference with our work is that the method uses text transcript, while our model does not. Despite this difference, we show that our model delivers better perceptual quality as reported by human listeners (Appendix C
Diffusion-basee approaches The recent success of diffusion based generative models has led to their use in wide array of applications. including speech enhancement. Numerous studies (Lemercier
ct al. 2023 Welker et al. 2022 Richter et al.) [203]] Lay et al. (223)) have applied diffusion models in various configurations to generate high-fin high-fidelity, clear speech from noisy input. For instance,
Welker et al. 2022);; ; Richter et al. (2023) introduced a novel stochastic diffusion process to design
generative model for speech enhancement the complex STFT domain for speech denoising and dereverberation. In the UNIVERSE study (Serrà et al. 2022), the authors propose a diffusion model for universal speech enhancement. They create a paired dataset using 55 different distortions and train a conditional diffusion model on it. Although their model performs well in terms of quality, it requires up to 50 diffusion steps to produce the final audio. The authors demonstrate that the number of steps can be reduced; however, the effect of this reduction on perceptual quality remains somewhat unclear. In our experiments, we show that our model can achieve similar results with just a single forward pass.
Results
6.1 Evaluation
We evaluate our models using the following sources of data:
VoxCeleb Data: 50 audio clips selected from VoxCeleb1 (Nagrani et al. 2017) cover the Speech Transmission Index (STI) range of 0.75-0.99 uniformly and balanced across male and female speakers.
UNIVERSE Data: 100 audio clips randomly generated by the UNIVERSE (Serrà et al. 2022) authors from clean utterances sampled from VCTK and Harvard sentences, together with noikes/background from DEMAND and FSDnoisy188 18k. The data contains various artificially simulated distortions including band limiting, reverberation, codec, and transmission artefacts. Please refer to (Serrà et al. 2022) for further details. The validation data of UNIVERSE is artificially simulated from clean speech recordings using the same pipeline that the authors utilized for training. Therefore, we must note that the comparison is conducted in manner advantageous to UNIVERSE, since our data simulation pipeline is different.
VCTK-DEMAND we use validation samples from popular Valentini denoising bench- mark (Valentini- Dalentini-Botinhooee Botinhao et al. 2017). This dataset used for a broad comparison with a wide range of speech enhancement models. The test set (824 utterances) includes artificially simulated noisy samples from speakers with 4 SNR (17.5, 12.5, 7.5, and 2.5 dB).
We utilize DNSMOS (Reddy et al. 2022). UTMOS (Sacki et al. 2022). and WV-MOS (Andreev jet al., p as non-intrusive metrics to objectively assess the samples generated by our speech enhancement model on the VoxCeleb dataset. The non-intrusive nature of these metrics is essential since the dataset comprises recordings from real-life scenarios, lacking ground- -trth samples.
In addition, we compute the Phoneme Error Rate (PhER) and Word Error Rate (WER) by comparing ground truths with the generated samples (see Appendix B for details) for both the UNIVERSE and VCTK-DEMAND datasets. For subjective quality assessment, we conduct 5-point Mean Opinion Score (MOS) tests. All audio clips are normalized ensure volume differences do not influence the raters' evaluations. The raters are required to be English speakers using appropriate listening equipment (more details in Appendix (F).
The Real-Time Factor (RTF) is determined by measuring the processing time (in seconds) for a 10-second audio segment a V V100 GPU and then dividing this time by 10. All confidence intervals are calculated using bootstrapping method.
We also evaluate our model on the additional metrics such as PESQ (Rix et al. 2001a), STOI (Tall et al. 2011), and SI-SDR (ooux et al. 2018). These metrics are included to ensure consistent comparison with previous works.
dataset Valentini- Botinnha et al. 2017) with
8

---

6.2 Comparison with existing approaches
We consider BBED Lay al, 2023). STORM (Lemeriier al. 2023 and UNIVERSE (Serà et al. 2022) diffusion models, along with Voicefixer and DEMUCS regression models, as our baselines. In addition, we consider our closest competitor, HiFi GAFi-G as GAN-basee baseline. The data for comparison with HiFi-GAN-2 and UNIVERSE were taken from their demo pages, since the authors did not release any code. We conduct comparisons with BBED, STORM, Voicefixer, DEMUCS, and HiFi-GAN-2 on real-world VoxCelebl samples and the comparison with UNIVERSE on the simulated data, provided by the authors of this work. The results are presented Table 2 We also compare these models on VCTK-DEMAND dataset, results can be found in Table 3 We complement
this table by two additional models: MetricGAN+ (Fu et 2021 and DB-AIAT
Yu et al. 2021
Table 2: Comparison with prior work on Voxceleb and UNIVERSE validation data.
VoxCeleb (HiFi-GAN-2 validation set, real data)
Model
MOS (1)
UTMOS (1)
WV-MOS (1)
DNSMOS (t)
RTF (L)
Input
3,46 - 0.07
2,76 0.13
2.90 0.16
2.72 0.11
VoiceFixer
3,41 + 0.07
2.60
2.79 = 0.09
3,08 + 0.06
0.02
DEMUCS
3.79 0.07
+
3.72
3.27+
0.08
STORM
3.75 0.06
3.29
3.54
3.17
1.05
BBED
3.97 0.06
0.10
3,47
3.23 0.04
0.43
HiFi-GAN-2
4.47 0.05
3.67 0.09
3.96 0.06
3.32 0.03
0.50
Ours
4.63 + 0.04
4.05 - 0.07
3,98 - 0.06
3.31 = 0.04
0.03
UNIVERSE validation set (simulated data)
Model
MOS (t)
UTMOS (t)
WV-MOS (t)
DNSMOS (t)
PhER (C)
RTF (1)
Input
2.87 + 0.05
2.27 + 0.28
1.72 + 0.61
2.25 + 0.19
0.31 + 0.05
Ground Truth
4.39 +: 0.05
0.06
4.28 + 0.06
3.33 0.04
0
UNIVERSE
4.10 + 0.07
3.89
3.85 + 0.12
3,23+
0.20 + 0.04
0.5
Ours (16 kHz)
3.99 0.07
4.21
+
3.25 0.05
0.14 0.03
0.03
Ours
4.23 0.07
4.21 0.10
4.43 0.08
3.25 0.05
0.14 0.03
0.03
Table 3: Comparison with the baselines on VCTK-DEMAN
Model
MOS (1)
UTMOS (f)
WV-MOS (1)
DNSMOS (1)
PESQ (1)
STOU (1)
SL-DDR
WER (L)
Input
3.18 0.07
3.08 :: 0.14
2.99 à: 0.24
2.53 J: 0.10
1.98 1 0.17
0.92 0.01
8.4
0.09 0.03
MenicGAN+
0.06
3.62
0.09
3.80 =
2.95 2..05
3.11
0.90
8.6
0.10 0.04
DEMUCS
3.96++0.06
3.95 + 0.05
4.37 + 0.06
3.14 0.004
2.04
0.95 0.01
18.5= 0.6
0.07 # 0.03
2,90
17.9
0.03
HiFI-GAN-2
4.13
3.99 0.05
4.26 = 0.05
0.02
3.14
0.95 0.01
18.6=0.6
0.07 0.03
DR-ATAT
4.22-4.0.5
0.05
4.38 1 0.06
3.18 0.004
3.26 + 0.12
0.96 - 0.01
19.3
0.07 0.03
Ours (16 kllz)
4.41 0.04
4,32
0.02
4.87
1 0.05
3,22 1. 0.04
2.94 2.94 0.10
0.92 0.001
4.6
0.07 0.03
Ours (48 kH2)
4.66 0.04
0.02 4.32
4.87 1 0.05
3.22 1: 0.04
0.11
0.00
0.3
0,07 0.03
GT (16 kHz)
4.26 0.05
4.07
0.04
4.52 1: 0.04
3.16 1: 0.04
GT (48 kltz)
4.56 1.561.0..01
0.03
4.07
0.04
4.52 1: 0.04
3.16 1: 0.04
Importantly, our study confirms the observation from Serrà et al. (2022) that speech enhancers based on generative models significantly outperform regression based approaches. Our model performs comparably in terms of perceptual MOS quality to all the considered baselines, while being more than five times as efficient as the closest competitors, HiFi-GAN-2 and UNIVERSE. We have also found that our model is less prone to hallucinating linguistic content than UNIVERSE, delivering lower Phoneme Error Rate (PhER) value.
Lastly, we would like to comment on the results in Table 3 Our model outperforms baselines in subjective evaluation and no-reference metrics, e.g. UTMOS, but underperforms in terms of PESQ (Rix et al.) 2001a), STOI (Taal et al.) 2011) and SI-SDR (Roux et al. 20188. Notably, there are numerous works consistently reporting low correlation of reference-basee based metrics with human perceptual judgment (Manocha et al. 2022 Manjunath] | Andreev et al. 2022). In particular, the study (Manocha et al. (022) reports that no-reference metrics (including DNSMOS, reported in our work) correlate significantly better with human perception and therefore have higher relevance for objective comparison between methods. Furthermore, our study, we report the MOS score, which directly reflects human judgments of restoration quality.

---

6.3
Ablation study
To validate the improvements proposed in this work, we conduct an ablation study assessing the effectiveness of the design choices made.
Firstly, we compare the LMOS loss against two other regression losses in the context of training a small speech enhancement model (10 times smaller than the final model; please see the Appendix [B.2 for details). The first regression loss is the Mel-Spectrogram loss, which was proposed by Kong et al. (2020). As the second alternative, we employ the Reconstruction loss (RecLoss) proposed by Défossez et al. (2023). It consists a combination LI and L2 distances between seelspeetso computed with different resolutions. For these experiments, conducted a grid search for the weights of each reconstruction loss (details are in Appendix E.2) and report results for the best option
in Table 4
Table 4: Ablation study (VoxCeleb real data).
Loss
MOS (t)
UTMOS (t)
WV-MOS (t)
DNSMOS (1)
input
3.46 = 0.07
2.77 + 0.14
2.88 + 0.16
2.72 = 0.11
w/o reg. loss
3.71 0.08
3.05 + 0.09
2.70 + 0.11
3.18 + 0.06
w/ L1Spec
4.05
3.45 0.08
3.28 0.04
2nd stage
w/ RecLoss
+
3.46 :.46:00 0.07
+.006
3,28 0.04
LMOS
4.20
3.48
3.58
3.26 0.04
WavL.M enc.
4.21 0.06
3.65
0.08
3.75 + 0.05
3.26 0.04
+ SScaling
4.30 = 0.05
3.83
0.07
4.00 + 0.06
3.21 0.05
3rd st.
+ 3rd stage
4.59 + 0.05
3.78 + 0.07
3.99 + 0.06
3.29 + 0.04
HF Loss
4.63 + 0.04
4.05 + 0.07
3.98 + 0.06
3.31 + 0.04
The other proposed improvements bring incremental gains in perceptual MOS quality. Firstly, we add the features of the WavLM encoder as an additional input to HiFi++. Next, we scale the architecture by increasing the number of channels within the model. After that, we concatenate the Upsample WaveUNet with the FINALLY-16 architecture and implement the 3rd stage of training on the DAPS- clean dataset (48 kHz). Finally, we use additional human feedback losses (HF Loss) during the 3rd stage to further improve perceptual quality.
Since Table44ooes not clearly demonstrate the advantages of using the WavLM (Chen et al. 2022b) en- coder in terms of MOS, we provide additional ablation study on more challenging UNIVERSE (Serra et al. 2022) validation dataset. The results of these additional experiments, presented in Table 5 clearly demonstrate the importance of WavLM encoder.
Table Ablation of WavLM encoder on UNIVERSE validation data.
MOS (t)
UTMOS (t)
WV-MOS (1)
DNSMOS (f)
PhER (4)
w/o WavLM
3.49 + 0.08
3.33 + 0.18
3.80 = 0.15
3.15 + 0.09
0.27 + 0.04
w/ WavLM
3.75 0.07
3.56 0.20
3.99 + 0.08
3.07 + 0.16
0.21 + 0.04
7 Conclusion
In conclusion, we theoretically demonstrate that LS-GAN training encourages the selection of the point of maximum density within the conditional clean speech distribution, aligning naturally with the objectives of speech enhancement. Our empirical investigation identifies WavLM as an effective backbone for perceptual loss, supporting adversarial training. By integrating WavLM- perceptual loss into the MS-STFT adversarial training pipeline and enhancing the HiFi++ architecture with WavLM encoder, we develop a novel speech enhancement model, FINALLY, which achieves state-of-the-at performance, producing clear and high-quality speech at 48 kHz.
10

---

References
Andreev, P., Alanov, A., Ivanov, O., and Vetrov, D. Hifi++: unified framework for bandwidth extension and speech enhancement, arXiv preprint A.Xiv:2003.10886 2022.
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449-12460, 2020.
Brock, A., Donahue, J., and Simonyan, K. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809. 1106, 2018.
Byun, J., Ji, Y., Chung, S.- S. Choe, S., and Choi, M.-S. An empirical study speech restoration guided by self-supervised speech representation. In ICASSP 2023-2023 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Chen, J., Wang, Z., Tuo, D., Wu, Z., Kang, S., and Meng, H. Fullsubnet+: Channel attention fullsubnet with complex spectrograms for speech enhancement. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 7857-7861. IEEE, 2022a.
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. Wavim: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics Signal Processing, (166::505-1515 2022b.
Chung. Y.-A,, Zhang, Y., Han, W. Chiu, C.-C., Qin, Pang, R., and Wu, Y. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 244 250. IEEE, 2021.
Close, G., Hain, T., and Goetze, S. The effect of spoken language on speech enhancement using self-supervised speech representation loss functions. In 2023 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 1-5. IEEE, 2023a.
Close, G., Ravenscroft, W., Hain, T., and Goetze, S. Perceive and predict: self-supervised speech representation based loss functions for speech enhancement. In ICASSP 2023-2023 IEFE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023b.
Defossez, A., Synnaeve, G., and Adi, Y. Real time speech enhancement in the waveform domain. arXiv preprint arXiv:2006. 2020.
Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. Transac- tions Machine Learning Research, 2023.
Dubey, H., Gopal, V., Cutler, R., Aazami, A., Matusevych, S., Braun, S., Eskimez, S. E., Thakker, M.. Yoshioka, T., Gamper, H., al. Icassp 2022 deep noise suppression challenge. In ICASSP 1022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 9271 0271-9275. IEEE, 2022.
Ephraim, Y. and Malah, D. Speech enhancement using minimum-mean square error short-time spectral amplitude estimator. IEEE Transactions on acoustics, speech, and signal processing, 32 (6):1109-1121, 1984.
Fu, S. Yu, C., Hsieh, T., Plantinga, P., Ravanelli, M., Lu, X., and Tsao, Y. Metricgan+: An improved version of metricgan for speech enhancement. CoRR, abs/2104.03538, 2021. URL https://arxiv wwweexivvvorgaaee org/abs at//aabb221444 03538
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Hao, X., Su, X., Horaud, R., and Li, X. Fullsubnet: full-band and sub-band fusion model for real- time single-channel speech enhancement. In ICASSP 202/-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6633-6637. IEEE, 2021.
11

---

Ho. J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-653 6851, 2020.
Hsieh, T.-A., Yu, C., Fu, S.-W., Lu, X., and Tsao, Improving perceptual quality by phone- fortified perceptual loss using wasserstein distance for speech enhancement. arXiv preprint arXiv:2010. 15774, 2020.
Hsich, T.-A., Yu, C., Fu, S.-W., Lu, X., and Tsao, Y. Improving perceptual quality by phone-fortified perceptual loss using wasserstein distance for speech enhancement, 2021.
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM Transactions Audio, Speech, and Language Processing. 29:3451-344 2021.
Hung, K.-H., Fu, S.-w., Tseng, H.-H., Chiang, H.-T., Tsao, Y., and Lin, C.-W. Boosting self- supervised embeddings for speech enhancement, arXiv preprint arXiv:2204.03339 2022.
Hwang, J., Hira, M., Chen, C., Zhang, X., Ni, Z., Sun, G., Ma, P., Huang, R., Pratap. V., Zhang, Y., Kumar, A., Yu, C.-Y., Zhu, C., Liu, C., Kahn, J.. Ravanelli, M.. Sun, P., Watanabe, S., Shi, Y.. Tao, Y., Scheibler, R., Cornell, S., Kim, S., and Petridis, S. Torchaudio 2.1: Advancing speech recognition, supervuid learning, and audio processing components for pytorch, 2023.
Isik, U., Giri, R., Phansalkar, N., Valin, J.-M., Helwani, K., and Krishnaswamy, Poconet: Better speech enhancement with vequency-positionn embeddings, semi-supervised conversational data, and biased loss. arXiv preprint arXiv:2008. 040000000440 2020.
Ito, K. and Johnson, L. The lj speech dataset. 2017.
Datasset
Kim, J., EI-Khamy, M., and Lee, End-to-end multi-task denoising for joint sdr and pesq optimiza- tion. arXiv preprint arviv: 10t.:0090000016660 2019.
Kim, J., Kong, J., and Son, J. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference Machine Learning, pp. 5530-5540. PMLR, 2021.
Koizumi, Y., Yatabe, K., Zen, H., and Bacchiani, M. Wavefit: An iterative and non aunttrrrrreessveee neural vocoder based on fixed-point iteration. In 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 884-891. IEEE, 2023a.
Koizumi, Y., Zen, H., Karita, S., Ding, Y., Yatabe, K., Morioka, N., Bacchiani, M., Zhang, Y., Han, W., and Bapna, A. Libritts-r: restored multi-speaker text-to-speech corpus. arXiv preprint arXiv:2305. 18802, 2023b.
Koizumi, Y., Zen, H., Karita, S., Ding, Y., Yatabe, K., Morioka, N., Zhang, Y., Han, W., Bapna, A., and Bacchiani, M. Müipher: robust speech restoration model integrating self-supervised speech and text representations. In 2023 IEEE Workshop Applications of Signal Processing Audio and Acoustics (WASPAA), pp. 1-5. IEEE, 2023c.
Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022 2020.
Kumar, K., Kumar, R., De Boissiere, T., Gestin, I.., Teoh, W. 7., Sotelo, J., De Brebisson, A., Bengio, Y., and Courville, A. C. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
Lay, B., Welker, S., Richter, J., and Gerkmann, T. Reducing the prior mismatch of stochastic differential equations for diffusion- based speech enhancement. arXiv preprint arXiv:2302.14745. 2023.
Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. Photo-realistic single image super-resolution using generative adversarial network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 4681-4690, 2017.
12

---

Lee, T., Yasunaga, M., Meng, C., Mai, Y., Park, J. S., Gupta, A., Zhang, Y.. Narayanan, D., Teufel, H., Bellagente, M., et al. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems, 36, 2024.
Lemercier, J.-M., Richter, J., Welker, S., and Gerkmann, Storm: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 2023.
Li, C. T. and Farnia, F. Mode-seeking divergences: theory and applications to gans. In International Conference on Artificial Intelligence and Statistics, pp. 8321-8350. PMLR, 2023.
Liu, H., Liu, X., Kong, Q., Tian, Q. Zhao, Y., Wang, D., Huang, C., and Wang, Y. Voicefixer: A unified framework for high-fidelity speech restoration. arXiv preprint arXiv:2204.05844 2022.
Loshchilov. I. and Hutter, F. Fixing weight decay regularization in adam. CoRR, abs/1711.05 2017. URLppttp://axxii http:/rrggaabb/111..555111 1111/11111...010
MacQueen, al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pp. 281-297. Oakland, CA, USA, 1967.
Manjunath, T. Limitations of perceptual evaluation of speech quality on voip systems. In 2009 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, pp. 1-6, 2009. doi: 10.1109/1SBMM 11L.1///BMSB.000..5133333.. 2009.513379..
Manocha, P. Jin, Z.. Zhang, R., and Finkelstein, Cdpam: Contrastive learning for perceptual audio similarity. In ICASSP 202/-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 196-200. IEEE, 2021.
Manocha, P., Jin, Z., and Finkelstein, A. Audio similarity is unreliable as proxy for audio quality, 2022. URL http::tttt////tt
org/ abs abs/2206. 13411
Mao, X., Li, Q. Xie, H., Lau, R. Y., Wang. Z. and Paul Smolley, S. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2794-2802, 2017.
Martin-Donas, M., Gomez, A. M., Gonzalez, J. A., and Peinado, A. M. A deep learning loss function based on the perceptual evaluation of the speech quality. IEEE Signal processing letters, 25(11):1680-166 2018.
Mysore, G. J. Can automatically transform speech recorded common consumer devices in real-world environments into professional production quality speech? dataset, insights, and challenges. IEEE Signal Processing Letters, 2806) 2014.
Nagrani, A., Chung, J. S., and Zisserman, A. Voxceleb: a large-scale speaker identification dataset. arXiv preprint arviv: 00661. 2017.
Pascual, S., Bonafonte, A., and Serra, J. Segan: Speech enhancement generative adversarial network. arXiv preprint arXiv: 000.770..04..,. 2017.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text- image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2):3, 2022.
Rand, W. M. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):846-85555 -500 1971.
Reddy, C. K., Gopal, V., and Cutler, R. Dnsmos p. 835: A non-intrusive perceptual objective speech quality metrie to evaluate noise suppressors. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 886-890. IEFF, 2022.
Rethage, D., Pons, J., and Serra, X. wavenet for speech denoising. 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5069-5077 IEEE, 2018.
13

---

Richter, J. Welker, S., Lemercier, J.-M., Lay, B., and Gerkmann, T. Speech enhancement and dereverberation with diffusion-based generative models. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2351-2366 2023. doi: 10. 10.11099TASLLP33233222224
Rix, A., Beerends, J., Hollier, M., and Hekstra, A. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE Interna- tional Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.0ICH37 volume 2, pp. 749-752 vol.2, 2001a. doi: 10.110//CAASPPP2000999900
Rix, A. W., Beerends, J. G., Hollier, M. P., and Hekstra, A. P. Perceptual evaluation of speech quality (pesq)-a a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE international conference on acousties, speech, and signal processing. Proceedings (Cat. No. 01CH37221), volume 2, pp. 749-752. IEEE, 2001b.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 00684-10695, 2022.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. CoRR, abs/1505, http/1555500597,.0 2015. /tLLippp///rrri..org/aaa/////
org/abs/1505 1505/
Roux, J. L.,, Wisdom, S., Erdogan, H., and Hershey, J. R. SDR half- baked or well done? CoRR,
abs/1811.02500 2018. URL http::/arxxii http://awwv...r/bbb////1/1...../ 02508
Sacki, T., Xin, D., Nakata, W., Koriyama, T., Takamichi, S., and Saruwatari, H. Utmos: Utokyo-
sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02/55, 2022.
Salimans, T. and Kingma, D. P. Weight normalization: A simple reparameteriization to accelerate training of deep neural networks. CoRR, abs/1602.00868, 2016. URL http::////xxi..oo/ org/bbs..bb.... /bs/
1602.07868
Sato, H., Masumura, R., Ochiai, T., Delcroix, M., Moriya, T., Ashihara, T., Shinayama, K., Mizuno, S., Ihori, M., Tanaka, T., et al. Downstream task agnostic speech enhancement with self-supervised representation loss. arXiv preprint 1435:33305..4723, 2023.
Serrà, J., Pascual, S., Pons, J., Araz, R. O., and Scaini, D. Universal speech enhancement with score-based diffusion. arXiv preprint 2XXv:2206.03065 2022.
Song. Y.. Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv: 1.14566, 2020.
Stoller, D., Ewert, S.. and Dixon. S. Wave-u-net: A multi-scale neural network for end-to-end audio source separation. CoRR, abbs/880600311 2018. URL
Su, J., Jin, Z., and Finkelstein, A. Hifi-gan: High-fidelity denoising and dereverberation based on speech deep features in adversarial networks. 2020.
Su, J., Jin, Z.,, and Finkelstein, A. Hifi-gan-2: adversarial networks conditioned on acoustic features. In 202/ IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 166-170. IEEE, 2021.
Studio-quality speech enhancement via generative
Taal, C. H., Hendriks, R. C., Heusdens, R., and Jensen, J. An algorithm for intelligibility prediction of time-frequency weighted noisy speech. IEFE Transactions on Audio, Speech, and Language Processing, 19(7):2125 12--2136, 2011. doi: 10. 10.1009/AAS..2111 1.21 14881.
Valentini- Botinhao, Valentini-Botinhao, C. et al. Noisy speech database for training speech enhancement algorithms and tts models. 2017.
Wang, Z.. Zhu, X., Zhang, Z., Lv. Y., Jiang, N., Zhao, G., and Xie, L. Selm: Speech enhancement using discrete tokens and language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11561-11565. IEEE, 2024.
Welker, S., Richter, J., and Gerkmann, T. Speech enhancement with score-based generative models in the complex stft domain. arXiv preprint arXiv:2203.17004 2022.
14

---

Yu, G., Li, A., Wang, Y., Guo, Y., Wang, H., and Zheng, C. Dual-branch attention-in -n-ttetntion
transformer for single-channel speech enhancement. CoRR, abs/2110.06467, 2021. URL https 7/arxiv. //arxiv.org/abartt /rtp.or///ab///11006666/
15

---

A Proof
Proposition 1. Let Peleoa (yle > 0 be a finite and Lipschitz continuous density function with unique global maximum and (5(2z) 5" 12" yy- (1) then
lim arg min XPearso (r) [Pelean + p5) arg max Peleun (y)22) + go(x)
Proof.
XPearsoo (pé (Peccan + +15)/2)= = (Fg(vl()) (Polean (1)(a) + (2)
dy
(Polean (1/2 + (
2 (1 Petean y (2) /2)p (v(=))² dy
1 Petean (2)22) 2))
(y)ax)
8- 90)0(t))
1 (etean (2) (z)))
dy
2 Polenn (3) (z)
RM -0-u-90000 11/6./1/1/
5"h
(1 2" Pelean (1) //42) 2n Pelean (1/ (x)//"
dy
2
8
2 Petemn (y b)yy.
R B=M00-90000E--1/55//1
Since Petean (y/xx) is Lipschitz continuous, there exist L, > 0 and C > such that for any & >> C (x()) E 1-1/ 1/E]", then Pokan (ylæx) E (Petean (00(=))ar) - L./5, Patean (ge + L/E).
Therefore, the divergence could be lower-bounded by
1 (1
3" (Pelean (go (x))xx) + L/E))²
2n11
5 (Pelean (x)(x)) L/E).
2
1
2
(Petean (go (x)[x) L/C)
g"
and upper-bounded by
1
(1
(Pelean (z))(x) - L/E))²
2n-1 (Petean (g0(x))>) L/E).
2
1
+ (Delean (ge (z) L/E)
g"
After taking Taylor expansion up to the linear term of 1 1/5", we get that
2
Pelenn (go (x)(=) o(1/E") S X ( (Petean + p2)/2),
and
2n+1
Pearsoo (p) (Petean + +p$)/2) <
2
&" Polenn (g0(2))z)+
Thus,
2
Pecasson (15)l (Petean + p2)/2)
E" Pelemn Petem (x) o(15").
As is approaching +oo, the main term depending on the ge(x) is minimized then ge (æ) arg Peteau (y|z) as global maximum is unique.
2n+1
gm Potcan (go (x)(æ) which is
16

---

From the probabilistic point of view, minimization of the point-wise distance leads to averaging effect. For example, optimization the mean squared error between waveforms delivers the expectation of the waveform over the conditional distribution of clean speech given its degraded B Additional results for perceptual losses
B.1
More on motivation
version go (z) (yy=passs)aa) []] The key thing is that the expectation over the distribution is not guaranteed to lie in the regions of high density of this distribution (see Figure and Figure
Entengled
Disenlangied
v())
Same sound
By Feteratris)
By By- Prteem (Py)
Figure 3: Ground truth waveform and waveform resynthesized by HiFi GAN vocoder. While wave- forms significantly differ, they correspond to the same sound, creating ambiguity in generation.
Figure 4: Regression in an entangled feature space might cause the expectation to lie outside the re- gions of high density, while regression in a dis- entangled space facilitates the expectation to lic within the regions of high probability density.
Mathematically speaking, let ((x)) be the mapping to this space, and assume there exists reverse mapping (z) such that á o (o(x)) purposes, there no need to explicitly know the reverse mapping é -=((2) as long as ((z) is differen- tiable. The regression in the space produced by the mapping leads to averaging in this space; thus, after minimizing the MSE loss
r. We would like to note that in practice, for regression
Ex (e..-maandoolettellllll (6g0 (x)())))
one would obtain the solution
go (x) 0- (By-panna
Therefore, desirable property for the d(x) mapping to be used the regression space that
o (wl)) = arg max paxem (g)(z),
ie., we would like averaging in é-space to provide a representation corresponding to the most likely solution (the most probable reconstruction as discussed in the previous section). In practice, this property is difficult to verify. Moreover, finding such mapping is likely to be task which is not easier than the original problem. Therefore, based on this intuition, we propose some heuristic rules for assessing the structure of regression spaces produced by different mappings.
B.2 Influence of losses on final audio
To illustrate the influence of LMOS loss, as well as the importance of adversarial training, we examine the impact of different training setup using vocoding and discuss the results using spectrograms in Figure(5] We start with training the vocoding model using only convolutional features from WavLM. This training turns out to be suboptimal, as the model produces noticeable artefacts, visible in the top spectrogram in Figure S Next, we complement the WavLM feature loss with the STFT loss, which is L. distance between spectrograms of reference and predicted audios. This method yields better quality, however some artefacts still remain. Notice, that such training particularly struggles reconstruct high frequencies and capture the harmonic content. Finally, train the
17

---

1009
110
120
120
130
Figure 5: Comparison of training schemes on spectrograms. Going from the top to the bottom, the first spectrogram is obtained from the model trained only on WavLM (Chen et al.) 2022b) features, the second one is produced by the model trained on both WavLM features and STFT LI loss. The third spectrogram obtained by training with both mentioned losses and adversarial loss. The last spectrogram is computed with ground truth audio.
18

---

model using adversarial loss. For this experiment we use original HiFi GAN (Kogg et al. 2020) setup. Adversarial training helps to alleviate artefacts and produces better quality, as compared the regression training used in both former experiments. For these reasons, we rely both on our newly developed reconstruction loss and adversarial training for creating our speech enhancement model.
B.3
Comparison with prior work
additionally compare the LMOS against other speech perceptual losses from the literature: (Hsieh)
al. 2020 Close al. 2023a Defossez et al. 2020 Kong et al. 2020). The comparison is outlined
in Table The experiments are conducted on neural vocoding task using LJSpeech dataset (Ito &
Johnson 2017). The HiFi-GAN generator V1 was trained for 1M iterations with batch size 16 for
each loss function.
Table 6: MOS on neural vocoding for different loss functions.
Method
MOS (t)
PFPL (Hsieh al. 2021)
2.40 0.08
SSSR loss with HUBERT features CCloe et al.
2023a
2.58 + 0.08
MS-STFT LI waveform (Defossez et al. 2020)
3.16 0.08
LMOS (ours)
4.21 + 0.07
adv. MPD-MSD (Kong et al. 2020
4.65 0.04
Ground Truth
4.66 + 0.04
B.4
WavLM layers
To find suitable layer of WavLM (Chen et al. (202bb model for our loss function, we take activations of each layer and measure Rand index and correlation with SNR. We seek to find such layer, that would produce high score for both metrics. Intuitively, this suggests that the layer's activations create space where audio with varying acoustics is effectively separated, and their hidden representations are responsive to noise.
9.9
begele Comeation lard Coo
-0.
-03
02
coty Oayer
Ioneeeetoo Sayers
Frav an
Wiai. Layer
Figure 6: Comparison of WavLM features using Rand index and negative correlation of SNR
As we see in Figure 6 the most suitable features come from the convolutional encoder or first transformer layer. For convenience, compute loss using features from convolutional encoder (we call the these outputs WavLM-Conv features).
B.5 WayLM features visualization
To provide visual evidence for the successful choice of WavLM features, well as to illustrate clustering rule and SNR rule, we show the first two principal components of WavLM-Con features and depict corresponding clusters (Figure 7 and noisy features (Figure [8. Figure Jillustrate that audio samples with the same lexical content form well-defined clusters, whereas those with different content are properly disentangled. Figure & shows that increasing SNR in audio moves it away for the centre of the cluster. These figures provide empirical evidence for the heuristics discussed in Section 3
19

---

ledyy
The socee
Theyres
Figure 7: Clustering rule visualized on Wavlm-Cove PCA features. The phrases corresponding to each cluster are visualized.
Figure 8: SNR rule visualized for Wavlm-Conv PCA features for a particular cluster.
C
Comparison with MIIPHER
We compare the enhancement quality of LibriTTS test__otseee other samples as relcased by the Müipher authors (Koizumi al. 2023b). The results outlined in Table 7 For the information WER refer to Appendix E.1 computations.
Table 7: MOS Scores for comparison with Miipher (LibriTTS, test other).
Method
MOS (t)
UTMOS (t)
DNSMOS (t)
WV-MOS (t)
WER (L)
Input
3.59 + 0.07
3,41 + 0.13
2.88 = 0.12
3.69 + 0.11
0.23 + 0.04
Miipher
4.18 0.06
3.95 + 0.13
2.99 0.12
4.15 +
0.22 + 0.05
Ours (16 kHz)
4.24 + 0.05
4.18 + 0.08
3.15 + 0.09
4,28 + 0.10
0.18 + 0.04
Ours (48 kHz)
4.54 0.05
4.18 + 0.08
3.15 0.09
4.28 0.10
0.18 0.04
20

---

D Implementation details
D.1 Reproducibility statement
Our model is based on HiFi++ architecture (Andreev et al. 2022), for which code is available in https://githut thub:://ggiubb..... wtt.:/ooomaaaaaaa http://aauaaaaaaasiiiiiiia..
We used several open-source datasets for training. More specifically, we used DAPS LibriTTS-R and Deep Noise Suppression Challenge (DNS) (only noises). We rely official implementations of MS-STFT discriminators, which can be found in htttt::// http//githbbbb com/ facebookccbeeeooooo esear ch/ encodec For LMOS loss and HiFi++ conditioning, we used WavLM large model from Hugging Face, which can be found in https //huggingface. www.wcooooottaaa///////ooo large
D.2 Augmentations
Before mixing with noise, convolve the speech signal with randomly chosen microphone impulse response from the Microphone Impulse Response projec apply other digital distortions. With a probability of 0.8, we also convolve the signal with room impulse response randomly chosen from those provided in the DNS dataset. We additionally apply audio effects from the torchaudio library (Hwang et al. 2023), trying simulate digital distortions. Parameter values chosen randomly; only one codec is applied.
Table 8: Applied augmentations.
Augmentation
Prob.
Param. name
Interval of values
acrusher
0.25
bits
[1.9]
crystalizer
0.4
intensity
[1.4]
flanger
0.15
depth
[1,8]
vibrato
0.15
frequency
[5,8]
codec ogg
encoder
vorbis, opus
0.45
codec mp3
bit rate
[4000,16000]
D.3 Model architecture
In this section detail parameters for our speech enhancement models. We use HiFi++ (Andreev) ct al. 2022) as backbone. This architecture consists of four parts: Spectral UNet, HiFi Upsampler, WaveUNet and SpsetalMaskNet. In addition that, we use WavLM-large feature encoder with transformer features, which takes a waveform as an input and outputs the last hidden state of the transformer, which stacked with the SpectralUNet output and fed into the HiFi Upsampler. In the next paragraphs, we thoroughly discuss the architecture of each component. Architecture summary is presented in Table
Residual blocks To better understand Spectral UNet, WaveUNet and SpectralMaskNet we first discuss the key building block. Each residual block is composed of convolution, followed by LeakyReLU, We use weight norm (Salimans & Kingmal 2016) instead of BatchNorm. we found it to be more efficient in our preliminary experiments. We use kernel size for 2D UNets and kernel size 5 for 1D UNet. Each residual block has additive skip connection. In our architecture, we use stacked composition of residual blocks. The number of residual blocks stacked together one layer is called depth.
SpectralUNet architecture SpectralUNet processes a mel-spectrogram input with dimensions [B, 80, Tl and outputs hidden state with dimensions [B, 512, Tj. The input mel-spectrogram is combined with positional encoding. The architecture based on a 2D UNet (Ronneberger
2015) with five layers, each having respective channels of [16, 32, 64, 128, 256) and depth of 4. Additional convolutions are applied after these layers transform 2D data from shape [B, 16. W. T] to [B, I, W, TJ, and then to [B, 512, TJ. This output is concatenated along the channel dimension with
http://micirp. http:///mliipp.bloggpoo... blogspot.com
21

---

the final transformer hidden state of WavLM (Chen et al. 2022b), which has been interpolated using the nearest-nan neighbour method to match the output length of SpectralUNet in the time dimension. The concatenated result then passes through a Residual block with kernel size 3 and with a width of 1536 (1024 from the final transformer hidden state and 512 from SpectralUNet output), then passes through 1d Convolution with kernel size 1 and LeakyReLUU to obtain size 512 in the channel dimension, and is subsequently fed into the HiFi Upsampler.
HiFF Upsampler architecture For HiFi Upsampler we use HiFi generator architecture from (Kong al. 2020). In our implementation we use 4 layer model with upsample rates [8, 8, 2, 2/, kernel size 116, 16, 4, 4], hidden sizes [512, 256, 128, 64). For each layer we use residual blocks with kernels [, 7, 11] and dilations [(1, 3, 5), (1,3, 5), (,).
WaveUNet architecture We implement WaveUNet following (Stoller et al. 2018). We use 4-layer WaveUNet with channels [128, 128, 256, 512), each layer has depth 4. Our WaveUNet takes the concatenation of input waveform and HiFi Upsampler as input. In this way we provide residual connection for Spectral UNet and HiFi Upsampler.
SpsktralMaskNet architecture The final Stage of our pipeline is SpectralMaskNet.. It applies channel-wise Short Time Fourier Transform (STFT) to the input, decomposes it into phase and amplitude, and then processes amplitude with 2D UNet. Finally, it multiplies processed amplitude and phase and applies inverse STFT obtain the final audio. For processing amplitude 2D UNet with channels (64, 128, 256, 512] backbone, each layer has depth 1. The output of SpectralMaskNet the final output of our 16kHz model.
WNveUNet Upsampler In order to enable the model to output samples in 48kHz we also add WaveUNet Upsampler, that upsamples the final audio. For that task we use WaveUNet with layers and channels [128, 128, 128, 128, 256], each layer has depth 3. The outputs of each layer are downsampled with the scale 4. After the output of the final upsample layer, we have an additional head that has 512 features and is used for directly upscaling the WaveUNet output into 48kHz.
Architecture summary More details about the architecture can found in (Andreev et 2022). We present the summary of parameters for Spectral UNet, WaveUNet and SasktralMaskNNtt in T
Table 9: Summary of parameters.
Kernel Size (constant for each layer)
Layer Depth (constant for each layer)
Submodel
Channel Sizes
Spectral UNet
[16, 32, 64, 128, 256]
3 [3, 7, 11] 5
HiFi Upsampler
[512, 256, 128, 64)
3
WaveUNet
[128, 128, 256, 512]
4
Spectral MaskNet
(64, 128, 256, 512)
3
1
WNveUNet Upsampler [128, 128, 128, 128, 256] + 512
5
3
2023 for adversarial training. We use 5 MS-STFT discriminators with n ffts [4096, 2048 1024, 512, 256). Jengtss [1024, 512, 256, 128, 64), win_lengths [4096, 2048, 1024, 512, 256) in 48kHz setting, for 16kHz all above-mentioned parameters are divided by 2. Our implementation closely follows (Défossez et al. 2023
Discriminators We employ MS-STFT discriminators Défossez et al.
Training details The main model is trained using 8 Nvidia P40 GPUs with effective batch size 32 and AdamW (Loshchilov & Hutter (017)) our main optimizer. For pretraining we use learning rate 0.0002, betas (0.8, 0.99) and learning rate exponential decay of 0.996 for each 200 iterations, the pretraining lasts 100,000 iterations. For the second stage we use learning rate 0.0002 with betas (0.8, 0.99) and learning rate decay 0.995, whereas for discriminators we use learning rate 0.0002 with betas (0.5, 0.999) and learning rate decay 0.995, the discriminators perform 2 optimization iterations for
22

---

every 1 generator's 's optimization iteration, the training lasts 30,000 generator's iterations. We also use a linear warm-up for 2000 iterations for the generator. We use the same training parameters for the third stage during generator's 40,000 iterations. The first and the second stage are trained in 16kHz using LibriTTS-R dataset with noises from DNS and augmentations, discussed in Appendix [D.2 Whereas the third stage is trained in 48kHz using DAPS dataset with noises from DNS as well. Datasets are discussed in Appendix D.I
Comparison of the datasets and resources used to train FINALLY and baseline models. To better contextualize our results, we provide a detailed comparison of the model sizes and training data of the baseline models.
Table 10: Comparison of resources and data used for training.
Model
Training Data Scale (clean data)
Model Size (parameters)
RTF V100 GPU
VoiceFixer
44 hours (VCTK)
112
0.02
DEMUCS
500 hours (DNS)
61 M
0.08
STORM
200 hours (WSJO and VCTK)
28 M
1.05
BRED
140 hours (WSIO)
65 M
0.43
HIFI-GAN-2
hours (DAPS)
34 M
0.50
Universe
1500 hours (private data)
189
0.50
FINALLY (ours)
200 hours (LibriTTS-R and DAPS) 454 M (including 358 of WavLM)
0.03
Although our model has more parameters compared to the baseline models, the majority of these parameters are dedicated to handling low-resolution features. For instance, the Transformer in WavLM processes representations of the waveform that are downsampled by a factor of 320, operating at 50 Hz. On the other hand, models such as HiFi-GAN-- primarily work the full waveform resolution, similar to WaveNet. This design enables our model to be more efficient in terms of computational resources, resulting in significantly lower Real-Time Factor (RTF).
D.4 Limitations
The primary area for further developing in the proposed model lies in improving perceptual quality, particularly in instances when the speech is severely degraded. Although our model is capable of capturing the content of such signals, even in situations that may be impossible for humans, some artefacts remain. These artefacts could possibly be attributed to the high uncertainty of the initial signal, such as the speaker's voice, for instance.
Another crucial aspect to consider the streaming mode. While the proposed model fast, doesn't suit low latency scenarios, which are widely applicable in telecommunications The big obstacle to such improvement could be the WavLM (Chen et al. (2022b) model that utilizes context for identifying the content of speech. This model has the boundaries of the required context and, consequently, the minimum latency achievable.
E
Evaluation details
E.1 Metric computation
To compare models in their ability to correctly reconstruct the speech, we use Phoneme Error Rate (PhER) and Word Error Rate (WER). Given the reference audio and audio, produced by the speech enhancement model, metries are computed in the following way. First, Automatic Speech Recognition (ASR) model is applied to both audios, yielding phoneme representations. Then, the editing distance computed over predicted phonemes (or words for WER) for the reference and the predicted audio. The distance is normalized to the length of the reference phoneme representation (or the length of the sentence for WER) and averaged over the validation set. We rely on https:: for the ASR models and metric computations. For PhER we use wuw.uunnnnnoooa/eee---e--------a [2-arctic-phoneme model, for WER we use Nonusssoonnavawwweeeee///sssssss These
co/
Wav2 Vec2-Vec2based (Baevski al. 2020) models for ASR.
23

---

E.2
Ablation details
For the ablation study, we utilized smaller version of our original model to expedite evaluations, with the reduced model containing approximately 22 million parameters. The dataset used for this ablation was provided by (Nagrani et al. 2017). The ablation process was carried out in several stages. Initially, we compared the performance of our LMOS loss with 1.1 loss (Kong et al.) (020). MS-STFT loss (Defossez et al. 2020). and RecLoss (Défossez al. (023) during the pretraining phase. Each loss function was used independently to train the model in a regressive manner for 500,000 steps with a batch size 12, utilizing two Nvidia P40 GPUs. Our findings indicated that LMOS consistently outperformed the other losses. Detailed results presented in Appendix E.2 It's important to note that MOS evaluations were not conducted during the first stage, as the enhanced audio typically displayed artefacts following this stage of training.
Table 11: Ablation for regression losses.
Metric
UTMOS (1)
WV-MOS (t)
DNSMOS (t)
MS-STFT
2.54
2.77 + 0.10
3.04 = 0.05
RecLoss
2.53 353:0.11 0.10
2.77 0.10
3.04 + 0.05
LISpec
did not converge
did not converge 3.57 + 0.05
did not converge
LMOS
3.43 + 0.09
3.14 + 0.04
Next, we ablated the use of mentioned losses in purely adversarial setup without pretraining. The same small model was trained from scratch using one of the discussed reconstruction losses with adversarial LS-GAN loss (Mao et al. (017)) and feature matching loss (Kong al. 2020). We used MS-STFT discriminators (Défossez al. 2023) for adversarial training. We fixed coefficients for the feature matching loss at and for the LS LS-GAN loss at 15, and then grid searched the coefficient for cach reconstruction loss. We experimented with the possible coefficient to be 3, 10, 20, 50 and found that is all cases the best coefficient is 3. We found that LMOS loss outperforms other reconstruction losses in this setup as well.
In the first stage we experimented with the addition of SSL features to boost model quality. For these experiments we took our best model trained with LMOS loss. As our ablation indicates, transformer features of WavLM significantly increase the model quality.
Finally, we train our big model, using the ablated architecture. We report the results in 'Scaling' row in Table 4 We also show how the introduction of the WaveUNet Upsampler and Human Feedback (HF) losses influence the final quality of our model.
In conclusion, found that the best model is the one, trained with WavLM transformer features and LMOS reconstruction loss. Moreover, we found, that using HF losses also significantly increases the final quality.
Comparison with HiFi++. Since our model is based on HiFi++ architecture (Andreev et al., 2022). explicitly show how the improvements introduced increase the quality of enhanced audio compared to HiFi++ baseline using VoxCelex dataset (Nagrani et al. 2017). The results are presented in Table 12
Table 12: Comparison with HiFi++ on VoxCeleb data.
Model
UTMOS (t)
WV-MOS (t)
DNSMOS (t)
Input
2.72 + 0.11
2.90 + 0.16
2.72 + 0.11
HiFi++
2.76 + 0.13
2.68 0.14
2.98 0.07
FINALLY (ours)
4.05 + 0.07
0.06
2.3100004
F
Subjective evaluation
We measure mean opinion score (MOS) of the model using a crowdsourcing adaptation of the standard absolute category rating procedure. Our MOS computing procedure is as follows.

---

Usten the tollwling speech dip.
Usen to the sokawfigsspechhhd
0100
you qualitty the followingg speech sample!
How do you rote the overall quality lowingg spocch sample?
The quality of thresampii
quality sittrsserpiiss
tollweingg sprecciidyy
0500
overall qualiit the following speech
How the overall quality of speech sample?
sample?
The Good
H5) extdtat
Fair
Phoo
Eootert
(2))
Red
seneert play
Rwel
The didanouri
Figure 9; The assessor's interface.
carofully
assess quality.
Please liste the audios headphones environment.
your computer Check that ran heve and understand the frlliwiing combinations of threw dights (for example 5. spriee with reise in the background (Ink1, ink?, link)), If you are not able hear these digits, your setup must be not good anough for this assignment.
you
sample very (link)
The assignment will rejected, verification shoos thet you broke the above
your anviron not quist anough your setup not good enough (sea above))
led answar control quest correctly
you were paying aflent on ring the ansignment
Figure 10: The rules for the assessor.
1. Select subset of 40 random samples from the test set (once per problem, i.e. for bandwidth extension or speech enhancement).
2. Select set of models to be evaluated; inference their predictions on the selected subset.
3. Randomly mix the predictions and split them into the pages of size 20 almost uniformly.
20 Almost uniformly means that on each page there are at least L models_ samples from each model.
4. Insert additional 4 trapping samples into random locations on each page: 2 samples from ground truth, and 2 samples of a noise without any speech.
5. Upload the pages to the crowdsourcing platform, set the number assessors for each page least 30.
Assessors are asked to work headphones in a quiet environment; they must listen to the audio until the end before assess it.
6. Filter out the results where the ground truth samples assessed with anything except (good) and (excellent), or the samples without voice are assessed with anything except (bad).
7. Compute 95% confidence intervals via bootstrapping.
Since the models are distributed uniformly among the pages, assessor's ss biases affect all models in the same way, so the relative order of the models remains. On the other hand, the assessor will have access to variety of the models on one page and thus can scale his ratings better. The other side is that the models' ratings depend on each other in this setting, because assessors tend to estimate the sample quality relatively to the average sample of the page. This means that when some models perform poorly in comparison, the higher scores given to the better models. 4 trapping samples
25

---

page is also a reasonable choice, because one cannot just random guess the correct answers for these questions.
The instructions for the assessors and the screenshots the evaluation interface are provided on
Figure Figure resppcttivel
The total amount of money spent on the surveys while preparing the paper is somewhere between $1000 and $1500 (it is hard to estimate exactly because there were lot of exploration tests during creating the model and writing the paper). According to the crowdsourcing system statistics, the average hourly wage varies between tasks from $2.5 to $4, which exceeds by a large margin the minimal wage in the countries where the test was conducted.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix D.4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:[Yes)
27

---

Justification:
Please see the Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please the Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [No]
Justification: We do not open the code for training of the model due to our organization policy, however, we use only publicly available data and most of the source codes available as stated in Appendix D. Therefore, we expect that the work could be reproduced with moderate efforts.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
ctaps.cc/publlce hwwpbliicgggideeeeeeeeeeeeeeeeee guwdee/dde/eee http://oodouuuoissooooooooooe
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see the Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report the confidence intervals computed by bootstrapping.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: please see the Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please the Section 5,
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Please see the Appendix
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "WPPuGH5ltA", "reviewer": "Reviewer_dAZE", "summary": "This paper describes a new formulation of GAN-based speech enhancement. It includes an analysis of the convexity in different feature spaces of the distribution of TTS utterances generated from the same inputs, concluding that WavLM's convolutional encoders provide the most convex such space. This representation is then incorporated into the input of HiFi++ and also a separate, unrelated upsampling stage is added at the end.\n\nOn the VoxCeleb real data validation set and the validation data of UNIVERSE, the proposed approach outperforms other strong baselines in both quality (MOS from a subjective listening test and other objective metrics) as well as in realtime factor. For example, on VoxCeleb, the proposed system is rated at 4.63 MOS compared to the second best system HiFi-Gan-2 at 4.47 while the proposed system is ~15x faster. For systems with comparable RTFs, the difference in MOS is 4.63 vs 3.79 for DEMUCS.", "questions": "N/A", "limitations": "Limitations of the approach are discussed in the appendix, section D.4. The limitation of this approach being non-streaming is mentioned there and is important to highlight.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Significance:\n* The argument for the relevance of GANs to speech enhancement is thoughtful, interesting, and convincing. It brings clarity on a point that I did not previously appreciate.\n* Analysis of the different feature spaces in terms of their convexity is another valuable and interesting contribution. It provides clearly actionable findings that are shown in the experiments to make a meaningful difference to system performance. This analysis can be applied more broadly to compare different SSL representations for various tasks.\n\nSystem performance:\n* Improving both the performance and the efficiency of a speech enhancement system is quite valuable and these differences appear to be large compared with strong baselines.\n* Listening to the provided audio examples in the supplementary material shows impressive performance, especially in comparison to the halucinations of UNIVERSE in low-SNR instances, although it is not clear how these particular examples were selected.\n\nClarity:\n* The paper is well written and easy to follow. The figures are helpful in understanding it. The description of the loss and different training stages is particularly clear and helpful.\n* The ablations are thorough and informative and show clear benefits to each of the stages of the model/training.", "weaknesses": "One point that could be added to the discussion of previous work is that of Maiti and Mandel (2019) and related work, which introduced the idea of speech enhancement by synthesis of a clean utterance that contains the same content as the original utterance.\n\nThe evaluation was conducted on a crowdsourcing platform without IRB review. This should be reviewed by ethics reviewers.\n\nS Maiti and M Mandel (2019). Parametric resynthesis with neural vocoders. Proc. IEEE WASPAA.\n\nMinor comments:\n* Line 187 states, \"we report MOS score\" but it does not state where this MOS score comes from. Please describe the experiment/measurement that generated the MOS scores. Presumably it was the human listening test described in the appendix, but this is not clear at this point in the manuscript.\n* Equation (2): please define phi\n* Line 299 calls losses based on UTMOS and PESQ \"human feedback\" losses, but since these are algorithms predicting human feedback, I don't think their outputs can be called human feedback itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe are very grateful for the high assessment of our work and your valuable suggestions. Below, we address your concerns.\n\n**W1. Work by Maiti and Mandel (2019)**\n\nThank you very much for pointing out this work. We will add a discussion of it to the related work section in the camera-ready version.\n\n**W2. Minor comments.**\n\n- “Presumably it was the human listening test described in the appendix, but this is not clear at this point in the manuscript.” – You are correct. We will add a reference to the relevant appendix section.\n- “Equation (2): please define phi.” – Phi denotes the WavLM-Conv feature mapping. We will add this clarification in the camera-ready version.\n- “Since these are algorithms predicting human feedback, I don't think their outputs can be called human feedback itself.” – We can call these losses “predicted human feedback losses” instead of “human feedback losses.” We believe this clarification will indeed improve clarity, and therefore we will use this naming in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e2vfEbk2Au", "reviewer": "Reviewer_gE54", "summary": "This paper proposes a universal speech enhancement model for real-world recording environments utilizing GAN, referred to as FINALLY. The authors theoretically analyze that using LS-GAN loss leads to finding the point of maximum density within the conditional clean speech distribution. To stabilize the adversarial training process, WavLM-based perceptual loss is integrated into the MS-STFT pipeline.", "questions": "- I'm curious about why DNSMOS and other metrics trend differently in Table 3.\n- Could you clarify why the order of the metrics in Tables 2 and 3 is presented differently?", "limitations": "- Artifacts mentioned in the inference results can degrade perceptual quality.\n- The paper has a clear contribution, but the presentation and organization are lacking.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper provides a meaningful analysis of LS-GAN loss in the context of speech enhancement. \n- The adoption of the WavLM neural network shows performance improvements.", "weaknesses": "- Although the paper focuses on real-world scenarios, it still needs results with objective metrics such as PESQ and STOI for a solid evaluation.\n- The three-stage training process of FINALLY is complex, and the performance gains over HiFi-GAN2 do not justify the increased cost.\n- Reducing the model size by 90% for ablation studies is not convincing, as it can significantly affect the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your time and consideration. We would like to address your concerns about the paper.\n\n**W1. It still needs objective metrics such as PESQ and STOI for a solid evaluation**\n\nOur paper does not include similarity-based metrics such as PESQ and STOI for two reasons. First, since PESQ and STOI require ground truth reference audio, we are unable to compute these metrics for real data such as VoxCeleb and LibriTTS, as there is no ground truth reference for such data. Second, there have been numerous works consistently reporting low correlation of reference-based metrics with human perceptual judgment [1, 2, 3]. In particular, study [1] reports that no-reference metrics (including DNSMOS, reported in our work) correlate significantly better with human perception and therefore have higher relevance for objective comparison between methods. Furthermore, in our study, we report the MOS score, which directly reflects human judgments of restoration quality. Therefore, we believe that our evaluation could be considered solid.\n\nHowever, we agree that outlining conventional metrics could facilitate comparison with prior work. Therefore, we have decided to measure these metrics on the popular denoising benchmark VCTK-DEMAND\n\nTable 1. Comparison with baselines on VCTK-DEMAND dataset with conventional metrics.\n| Model                | MOS          | UTMOS        | WV-MOS       | DNSMOS       | PESQ         | STOI         | SI-SDR       | WER         |\n|----------------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|\n| input                | 3.18 ± 0.07  | 2.62 ± 0.16  | 2.99 ± 0.24  | 2.53 ± 0.10  | 1.98 ± 0.17  | 0.92 ± 0.01  | 8.4 ± 1.2    | 0.09 ± 0.03  |\n| MetricGAN+           | 3.75 ± 0.06  | 3.62 ± 0.09  | 3.89 ± 0.10  | 2.95 ± 0.05  | 3.14 ± 0.10  | 0.93 ± 0.01  | 8.6 ± 0.7    | 0.10 ± 0.04  |\n| DEMUCS               | 3.95 ± 0.06  | 3.95 ± 0.05  | 4.37 ± 0.06  | 3.14 ± 0.04  | 3.04 ± 0.12  | 0.95 ± 0.01  | 18.5 ± 0.6   | **0.07 ± 0.03**  |\n| HiFi++               | 4.08 ± 0.05  | 3.89 ± 0.06  | 4.36 ± 0.06  | 3.10 ± 0.04  | 2.90 ± 0.12  | 0.95 ± 0.01  | 17.9 ± 0.6   | 0.08 ± 0.03  |\n| HiFi-GAN-2           | 4.13 ± 0.05  | 3.99 ± 0.05  | 4.26 ± 0.05  | 3.12 ± 0.05  | 3.14 ± 0.12  | 0.95 ± 0.01  | 18.6 ± 0.6   | **0.07 ± 0.03**  |\n| DB-AIAT              | 4.22 ± 0.05  | 4.02 ± 0.05  | 4.38 ± 0.06  | 3.18 ± 0.04  | **3.26 ± 0.12**  | **0.96 ± 0.01**  | **19.3 ± 0.8**   | **0.07 ± 0.03**  |\n| FINALLY (16 kHz)     | 4.41 ± 0.04  | **4.32 ± 0.02** | **4.87 ± 0.05**  | **3.22 ± 0.04**  | 2.94 ± 0.10  | 0.92 ± 0.01  | 4.6 ± 0.3    | **0.07 ± 0.03**  |\n| FINALLY (48 kHz)     | **4.66 ± 0.04**  | **4.32 ± 0.02**  | **4.87 ± 0.05**  | **3.22 ± 0.04**  | 2.94 ± 0.10  | 0.92 ± 0.01  | 4.6 ± 0.3    | **0.07 ± 0.03**  |\n| Ground Truth (16 kHz)| 4.26 ± 0.05  | 4.07 ± 0.04  | 4.52 ± 0.04  | 3.16 ± 0.04  | -            | -            | -            | -            |\n| Ground Truth (48 kHz) | 4.56 ± 0.03  | 4.07 ± 0.04  | 4.52 ± 0.04  | 3.16 ± 0.04  | -            | -            | -            | -            |\n\n\nAs one can see, our model clearly outperforms all baselines in terms of the MOS metric and reference-free objective metrics, while reference-based metrics correlate poorly with human judgments. Note that our model has slightly higher MOS than the ground truth due to the fact that our training data is of higher quality than VCTK ground truth samples.\n\n**W2. The three-stage training process of FINALLY is complex, and the performance gains over HiFi-GAN-2 do not justify the increased cost**\n\nWe agree that our training process is slightly more complex than that of HiFi-GAN-2. However, we would like to point out that our training pipeline, although intricate, is not considerably more complex compared to HiFi-GAN-2. HiFi-GAN-2 (similarly to our model) has three stages: 1) acoustic feature prediction network, 2) WaveNet training, and 3) adversarial training. Furthermore, our final model is more than 10 times faster (0.03 RTF compared to 0.5 RTF for HiFi-GAN-2). Therefore, we believe that the complexity of the training process is well justified by the dramatic increase in the efficiency of the resulting model.\n\n**W3. Reducing the model size by 90% for ablation studies is not convincing, as it can significantly affect the results.**\n\nWe follow a well-established practice in deep learning literature of conducting ablation studies on a smaller scale in order to reduce the costs of training, as many leading papers in the field have done (e.g., [4]). While this procedure may influence the results, practical considerations in a resource-intensive field such as ours remain significant. Therefore, we use a smaller model for some parts of the ablation study.\n\n**Q1. Why DNSMOS and other metrics trend differently in Table 3.**\n\nDue to the imperfections of different objective metrics, they can trend differently as some of them pay more attention to certain artifacts than others. Please consider taking into account relevant papers [1, 2, 3] in order to understand the issues with objective metrics for speech quality in more detail.\n\n**Q2. Order of metrics in Tables 2 and 3.**\n\nThank you very much for noticing this. We will rearrange the columns to have a consistent order in these tables to improve the clarity of presentation in the camera-ready version.\n\n\n[1] Manocha, Pranay, et al.. \"Audio Similarity is Unreliable as a Proxy for Audio Quality.\" \n\n[2] Manjunath, T. \"Limitations of perceptual evaluation of speech quality on VoIP systems.\" \n\n[3] Andreev, Pavel, et al. \"HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement.\"\n\n[4] Karras, Tero, et al. \"Analyzing and improving the training dynamics of diffusion models.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k5qNrqq15O", "reviewer": "Reviewer_sXBh", "summary": "The paper proposes a GAN based method for universal speech enhancement (SE) demonstrating competitive experimental performance.\n\nTo justify the use of adversarial learning for SE, the authors provide a theoretical insight regarding the effectiveness of the mode-covering property of LSGAN for the SE task under ideal conditions. However, in practice several additional losses needs to be employed for stable training and better performance.\n\nThe authors also propose a perceptual loss, LMOS, which combines L1 loss between STFT magnitudes and L2 loss between WavLM convolution encoder features. The choice of using WavLM-conv features for the perceptual loss is driven by two heuristic rules regarding the features, 1) identical (content) speech sounds should be clustered together and 2) adding noise should increase distance from clean cluster centers proportional to SNR. Based on these rules, several models were evaluated and finally the WavLM-conv was chosen.\nMoreover a Human Feedback (HF) loss is also utilized, which is provided by means of differentiable PESQ and UTMOS predictions.\n\nFor the model architecture, the authors extend the HiFi++ model architecture, incorporating WavLM features along with SpectralUNet features as input for the Upsampler. Additionally, an upsampling WaveUNet is added which increases the output sampling rate upto 48khz.", "questions": "- From the ablation study, the LMOS loss, model scaling, Upsampler (to 48khz) and HF losses show clear performance benefits, however the advantage of incorporating the WavLM-enc is not clear, as it does not lead to significant performance gain. Is the WavLM-enc necessary?\n- Since the goal is universal speech enhancement, it would be interesting to see the performance in more focussed evaluation such as bandwidth extension or reverberation individually.", "limitations": "Limitations have been adequately addressed.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall the paper is well written.\n- Strong speech enhancement performance.\n- Interesting theoretical insight showing the relevance of mode-covering property of LSGAN for speech enhancement task.\n- The proposed criteria for choosing the SSL feature for perceptual loss is very interesting and the results of the comparison with several SSL models is a very good contribution.\n- Architectural changes to the HiFi++ model are significant as it enables the model to output 48khz speech and incorporate SSL features.", "weaknesses": "**Main weakness is in the experimental evaluations**\n- Table 2 (main experimental result):\n  - Since the model architecture is based on HiFi++, a direct comparison is probably necessary, though the ablation study covers it somewhat, but comparison with a full scale baseline HiFi++ in the main experiment would be more convincing.\n  - There is no description of the model size/training data scale for the baseline methods, which makes it difficult to contextualize the results.\n  - A self contained description of the evaluation dataset is missing.\n\n- Table 3 (ablation study):\n  - The ablation study is missing the results for the 1st stage (pretraining setup, without adversarial loss), while it is mentioned in Appendix E2, but the results are not in the table.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your high assessment of our work. We truly appreciate your valuable comments. Below, we address your concerns.\n\n**W1. Comparison with HiFi++ & Q1. Advantage of incorporating the WavLM Encoder**\n\nThe original HiFi++ model was proposed for speech denoising and bandwidth extension applications separately. Therefore, a significant practical difference between our work and HiFi++ is that our model was trained to support a wide range of degradations emerging in practice, and thus our model is able to generalize to real-world data. Additionally, our model is trained with novel LMOS and HF losses, the effectiveness of which is validated by ablation studies. Therefore, our training framework is substantially different from that of HiFi++, and the importance of this difference is validated by practical observations.\n\nHowever, you are absolutely correct that the generator architecture of our model is mostly based on HiFi++. The main differences in this regard are the introduction of Upsample WaveUNet and the WavLM encoder. While the importance of Upsample WaveUNet is clearly validated by ablation studies, the effect of the WavLM encoder appears to be somewhat marginal in terms of MOS score (although we must pinpoint that objective metrics are considerably higher for the case with WavLM). To validate the importance of the WavLM encoder, we have conducted an additional ablation test on the UNIVERSE validation set, which contains more challenging cases than the Voxceleb data. The results are provided in the table below.\n\nTable 1. Ablation of WavLM encoder on UNIVERSE validation.\n\n|                   | MOS  | UTMOS | WV-MOS | DNSMOS | PhER |\n|-------------------|------|-------|--------|--------|------|\n| w/o WavLM    | 3.49 ± 0.08 | 3.33 ± 0.18 | 3.80 ± 0.15 | 3.15 ± 0.09 | 0.27 ± 0.04 |\n| w/ WavLM     | 3.75 ± 0.07 | 3.56 ± 0.20 | 3.99 ± 0.16 | 3.07 ± 0.08 | 0.21 ± 0.04 |\n\n\nThus, we conclude that the introduction of the WavLM encoder is very important to achieve high quality on more challenging data. We will provide these additional results in the camera-ready paper.\n\n**W2. Description of the Model Size/Training Data Scale for the Baseline Methods**\n\nWe agree that for better contextualization of our result, detailed information on the model size and training data of the baseline models is needed. Below, we provide a table, which will be included in the camera-ready version:\n\nTable 2. Comparison of resources with baselines.\n| Model          | Training Data Scale (clean data)                                                     | Model Size (parameters)                 | RTF on V100 GPU |\n|----------------|--------------------------------------------------------------------------|-----------------------------|-----------------|\n| VoiceFixer     | 44 hours  (VCTK)                                    | 112 M             | 0.02            |\n| DEMUCS         | 500 hours (DNS)                                    | 61 M              | 0.08            |\n| STORM          | 200 hours  (WSJ0 and VCTK) | 28 M              | 1.05            |\n| BBED           | 140 hours  (WSJ0)                                   | 65 M              | 0.43            |\n| HIFI-GAN-2     | 5 hours (DAPS)                                        | 34 M              | 0.50            |\n| Universe       | 1500 hours (private data)                               | 189 M             | 0.5             |\n| FINALLY (ours) | 200 hours (LibriTTS-R and DAPS)                           | 454 M  (including 358 M of WavLM) | 0.03            |\n\n\nWe note that, while our model has a larger number of parameters than the baselines, most of these parameters are used to process low-resolution features (e.g., the Transformer of WavLM operates on 320-times downsampled representations of the waveform, i.e., at 50 Hz). In contrast, models like HiFi-GAN-2 mostly operate at full waveform resolution (due to WaveNet). This allows our model to be more compute-efficient and thus have a much lower Real-Time Factor (RTF).\n\n**W3. Self-contained Description of Evaluation Dataset**\n\nSince we took the evaluation data from prior work, we did not describe it in much detail. However, we agree that our paper would benefit from such a description. We will include it in the camera-ready paper. Please find the evaluation dataset details below.\n\n**VoxCeleb Data:** 50 audio clips selected from VoxCeleb1 [1] to cover the Speech Transmission Index (STI) range of 0.75-0.99 uniformly and balanced across male and female speakers.\n\n**UNIVERSE Data:** 100 audio clips randomly generated by the UNIVERSE [2] authors from clean utterances sampled from VCTK and Harvard sentences, together with noises/backgrounds from DEMAND and FSDnoisy18k. The data contains various artificially simulated distortions including band limiting, reverberation, codec, and transmission artifacts. Please refer to [2] for further details.\n\n**W4. The Results for the 1st Stage (Pretraining Setup)**\n\nThank you very much for noticing this issue. Indeed, the table with these results is missing in the Appendix. We provide it below and will include it in the Appendix of the camera-ready paper.\n\nTable 3. Results on Voxceleb data after pretraining with different regression losses.\n| Loss       | UTMOS       | WV-MOS      | DNSMOS      |\n|------------|-------------|-------------|-------------|\n| MS-STFT    | 2.54 ± 0.10 | 2.77 ± 0.10 | 3.04 ± 0.05 |\n| RecLoss    | 2.53 ± 0.10 | 2.77 ± 0.10 | 3.04 ± 0.05 |\n| LMOS       | 3.44 ± 0.08 | 3.25 ± 0.03 | 3.57 ± 0.06 |\n| L1_Spec    | failed to converge | -           | -           |\n\n**Q2. More Focused Evaluation**\n\nWe agree that additional results concerning individual degradations would be helpful. We will include qualitative examples on the project web page upon paper publication.\n\n[1] Nagrani, Arsha et al.. \"Voxceleb: a large-scale speaker identification dataset.\" \n\n[2] Serrà, Joan, et al. \"Universal speech enhancement with score-based diffusion.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yV12jRzEwR", "reviewer": "Reviewer_mzEg", "summary": "The authors propose FINALLY, a speech enhancement algorithm based on GANs and the Wav LM encoder. They evaluate different feature extractors and then show qualitative and quantitative results on speech enhancement.", "questions": "Can you provide qualitative comparisons against the baselines like Hifi? The reviewers would appreciate hearing those as well as your generated outputs that you have already provided.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "A major strength is the fidelity of the generated outputs. The results are impressive and are demonstrated across a variety of SNR noise levels, noise types, and accents. I was very impressed by the samples, and the fact that it can be generated in a single pass. The generative nature of the model allows for a greater degree of restoration from low quality signals, compared to discriminative/masking based approaches (e.g. Demucs, Conv-Tasnet, TF-GridNet) that can't restore the speech to this studio like quality. \n\nI also liked the rigorous comparison to choose WavLM as the feature extractor in the network. There was proper consideration given to why you chose that encoder. Although a better choice would have been to  compare WavLM to using other methods in the ablation study, e.g. WavLM does better on the clustering and SNR rule, but does that actually mean it does better as the network component?", "weaknesses": "The main weakness is that the approach is very close to HiFi with only slight modifications. The other contributions stated in the paper are not as major as the authors claim. The first section about sampling from the conditional distribution vs the max seems like it misses the point a bit. The reason people use diffusion models, GANs etc for for conditional sampling is because those models produce high quality unconditional outputs and can often be used without modification Not necessary because there is a desire to sample from the posterior distribution to allow multiple generations for example. Therefore I don't think the analysis in section 2 is such a big contribution.\n\nA second major weakness is the experiment section. For speech enhancement, there are some standard datasets and metrics people use to compare results. These include metrics like PSNR, PESQ, STOI, and datasets like VCTK, WHAM, and Librispeech. This would allow comparison against a greater number of recent methods (e.g. TF GridNet, Demucs) which are not included", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nFirstly, we would like to thank you for your invaluable work. Below, we address your concerns about the paper.\n\n**W1. Diffusion Models and GANs for Conditional Sampling**\n\nWe generally agree that generative models, such as GANs and diffusion models, are not necessarily used to allow multiple generations from posterior distributions but rather due to their high-quality results. However, we think that this consideration does not downplay our contribution. We provide a theoretical interpretation for the case when samples are generated using an LS-GAN generator. Theoretical interpretations are important for in-depth analysis and might have a significant impact on how the field evolves in the future. From the practical side, we argue that learning the whole posterior distribution might not be necessary for the speech enhancement problem and therefore diffusion models might be solving an unnecessarily complex task. Our analysis reveals that GANs provide a natural remedy to regress for the most probable speech reconstruction directly, thus speech enhancement GANs solve a simpler task with fewer potential resources. Therefore, we believe that Section 2 holds a significant contribution to the field.\n\n**W2. Standard Datasets and Metrics for Speech Enhancement**\n\nWe would like to clarify the reasons behind our choice of datasets and metrics.\n\nWe chose the VoxCeleb and UNIVERSE validation set because these data include several degradations at the same time, and the strongest baselines for universal speech enhancement, which are HiFi-GAN-2 and UNIVERSE, release results on this data. Other methods from the literature usually consider only one degradation (e.g., only noise or reverberation) or are significantly inferior to HiFi-GAN-2 and UNIVERSE. For instance, the popular speech enhancement VCTK-DEMAND dataset considers only additive noise as the distortion, and methods achieving good results on this dataset tend to poorly generalize to real data. Thus, comparison on this data is not particularly important from a practical point of view.\n\nOur paper does not include similarity-based metrics such as PESQ and STOI for two reasons. First, since PESQ and STOI require ground truth reference audio, we are unable to compute them for real data such as VoxCeleb and LibriTTS, as there is no ground truth reference for such data. Second, there have been a number of works consistently reporting low correlation of reference-based metrics with human perceptual judgment [1, 2, 3]. In particular, the [1] study reports that no-reference metrics (including DNSMOS, which we reported in our work) correlate significantly better with human perception and therefore have higher relevance for objective comparison between methods. Furthermore, in our study, we report the MOS score, which directly reflects human judgments of restoration quality.\n\nHowever, we agree that outlining conventional metrics such as PESQ and STOI on a popular VCTK-DEMAND benchmark could facilitate comparison with prior work. One can find this comparison in the table below. Note that we will add these results in the appendix of the camera-ready paper.\n\n\nTable 1. Comparison with baselines on VCTK-DEMAND dataset.\n| Model                | MOS          | UTMOS        | WV-MOS       | DNSMOS       | PESQ         | STOI         | SI-SDR       | WER         |\n|----------------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|\n| input                | 3.18 ± 0.07  | 2.62 ± 0.16  | 2.99 ± 0.24  | 2.53 ± 0.10  | 1.98 ± 0.17  | 0.92 ± 0.01  | 8.4 ± 1.2    | 0.09 ± 0.03  |\n| DEMUCS               | 3.95 ± 0.06  | 3.95 ± 0.05  | 4.37 ± 0.06  | 3.14 ± 0.04  | 3.04 ± 0.12  | 0.95 ± 0.01  | 18.5 ± 0.6   | **0.07 ± 0.03**  |\n| HiFi++               | 4.08 ± 0.05  | 3.89 ± 0.06  | 4.36 ± 0.06  | 3.10 ± 0.04  | 2.90 ± 0.12  | 0.95 ± 0.01  | 17.9 ± 0.6   | 0.08 ± 0.03  |\n| HiFi-GAN-2           | 4.13 ± 0.05  | 3.99 ± 0.05  | 4.26 ± 0.05  | 3.12 ± 0.05  | 3.14 ± 0.12  | 0.95 ± 0.01  | 18.6 ± 0.6   | **0.07 ± 0.03**  |\n| DB-AIAT              | 4.22 ± 0.05  | 4.02 ± 0.05  | 4.38 ± 0.06  | 3.18 ± 0.04  | **3.26 ± 0.12**  | **0.96 ± 0.01**  | **19.3 ± 0.8**   | **0.07 ± 0.03**  |\n| FINALLY (16 kHz)     | 4.41 ± 0.04  | **4.32 ± 0.02** | **4.87 ± 0.05**  | **3.22 ± 0.04**  | 2.94 ± 0.10  | 0.92 ± 0.01  | 4.6 ± 0.3    | **0.07 ± 0.03**  |\n| FINALLY (48 kHz)     | **4.66 ± 0.04**  | **4.32 ± 0.02**  | **4.87 ± 0.05**  | **3.22 ± 0.04**  | 2.94 ± 0.10  | 0.92 ± 0.01  | 4.6 ± 0.3    | **0.07 ± 0.03**  |\n| Ground Truth (16 kHz)| 4.26 ± 0.05  | 4.07 ± 0.04  | 4.52 ± 0.04  | 3.16 ± 0.04  | -            | -            | -            | -            |\n| Ground Truth (48 kHz) | 4.56 ± 0.03  | 4.07 ± 0.04  | 4.52 ± 0.04  | 3.16 ± 0.04  | -            | -            | -            | -            |\n\n\nAs one can see, our model clearly outperforms all baselines in terms of the MOS metric and reference-free objective metrics, while reference-based metrics correlate poorly with human judgments. Note that our model has a slightly higher MOS than the ground truth likely due to the fact that our training data is of higher quality than the VCTK ground truth samples. Additionally, we would like to point out that one can find a comparison on real data with the mentioned DEMUCS model in Table 2 of the paper.\n\n**Q1. Qualitative Comparisons Against the Baselines**\n\nWe agree that the work could benefit from more qualitative results. We will provide more qualitative comparison results on the paper's web page upon acceptance, as we cannot modify supplementary materials during the rebuttal period according to NeurIPS rules.\n\n[1] Manocha, Pranay et al. \"Audio Similarity is Unreliable as a Proxy for Audio Quality.\" \n\n[2] Manjunath, T. \"Limitations of perceptual evaluation of speech quality on VoIP systems.\" \n\n[3] Andreev, Pavel et al.. (2022). \"HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We would like to express our sincere gratitude for your thoughtful comments and suggestions. Your appreciation of our insights into GAN training and the analysis of SSL models’ feature spaces is truly encouraging. We have worked hard to address the questions and concerns you raised during the review process. Below, we have summarized our responses to the key concerns:

**Use of Conventional Metrics and Datasets**

In response to the reviewers' concerns about not using PESQ, STOI, and other classic metrics, we point out their relatively weak correlation with perceptual quality. We cite several papers where this issue has been discussed in detail. Moreover, we elaborate on the difficulty of applying such metrics to real-world data, which frequently lacks ground truth samples. Therefore, we reason that subjective metrics, like MOS, provide a more accurate and fair assessment of the method’s performance. Nevertheless, we concur that the paper could benefit from benchmarking our method against other baselines using classic metrics and well-established datasets. Therefore, we have included a table comparing our method with other baselines on the VCTK-DEMAND dataset.

**Complexity of the Presented Method**

We agree that the level of complexity of our method in comparison to other methods is a significant concern. We address this in two ways. First, we point out that comparable SOTA algorithms (e.g., HIFI-GAN-2) also involve complex training procedures, consisting of numerous stages that must be trained consecutively. Therefore, we believe that while our method is complex, it is nevertheless on par with other existing algorithms. Second, we provide clear evidence that our algorithm is considerably faster (~x15 times, as noted by reviewers as well), while delivering comparable or better perceptual quality. We think that the benefits of increased inference speed and sound quality outweigh the possible training complexity issues.

**Comparison with HiFi++ and Importance of WavLM Encoder**

The generator architecture of our model is largely based on HiFi++, with the main differences being the introduction of Upsample WaveUNet and the WavLM encoder. While the importance of Upsample WaveUNet is clearly validated by our ablation study, the effect of the WavLM encoder appears to be somewhat marginal in terms of MOS score (although we must pinpoint that objective metrics are considerably higher for w/ WavLM case). To validate the importance of the WavLM encoder, we conducted additional ablation tests on the UNIVERSE validation set, which contains more challenging cases than the VoxCeleb data. The results clearly indicate the importance of the WavLM encoder in achieving high quality on the challenging UNIVERSE validation data.

**Theoretical Analysis of Conditional Generation**

We agree that the key advantage of modern generative models is their ability to generate high-fidelity objects. However, it is important to note that the choice of a generative model necessarily involves trade-offs. Diffusion models can generate realistic and diverse objects but do so slowly, whereas GANs are capable of generating realistic objects quickly in one forward pass, though they may lack diversity. In our paper, we argue that the lack of diversity is not an issue for speech enhancement and provide an analysis of why GANs are likely to sample the desirable mode. Our main argument is that sacrificing diversity is not problematic for speech enhancement, but sacrificing inference speed is. Thus, GANs might be better suited than diffusion models for the speech enhancement problem.

Once again, we would like to thank all the reviewers for their efforts and time.



Sincerely,

Authors

Meta-review: The paper proposed a GAN based method for universal speech enhancement and demonstrated impressive results across differen SNRs. As agreed among the reviewers, the theoretical discussion presented in the paper of the relevance of mode-covering property of LSGAN for speech enhancement is very insightful. The detailed ablations and the addtional results addressing reviewers' concerns such as more standard SE metrics and comparisons with HiFi provide solid evidence. One weakness as pointed out by the reviewer is the similarity to Hifi, the complexity of the training procedure and the non-streaming aspect of the approach. Given these, I hence recommend an accept.
Decision: Accept (poster)