GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling
Authors: Jixun Yao
Hexin Liu
Chen Chen
Yuchen Hu
EngSiong Chng
Lei Xie
Venue: ICLR 2025 Poster (2025)
Keywords: speech enhancement
language model
semantic information
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A generative speech enhancement framework tailored for language model-based speech enhancement.

Abstract: Semantic information refers to the meaning conveyed through words, phrases, and contextual relationships within a given linguistic structure. Humans can leverage semantic information, such as familiar linguistic patterns and contextual cues, to reconstruct incomplete or masked speech signals in noisy environments. However, existing speech enhancement (SE) approaches often overlook the rich semantic information embedded in speech, which is crucial for improving intelligibility, speaker consistency, and overall quality of enhanced speech signals. To enrich the SE model with semantic information, we employ language models as an efficient semantic learner and propose a comprehensive framework tailored for language model-based speech enhancement, called GenSE. Specifically, we approach SE as a conditional language modeling task rather than a continuous signal regression problem defined in existing works. This is achieved by tokenizing speech signals into semantic tokens using a pre-trained self-supervised model and into acoustic tokens using a custom-designed single-quantizer neural codec model. To improve the stability of language model predictions, we propose a hierarchical modeling method that decouples the generation of clean semantic tokens and clean acoustic tokens into two distinct stages. Moreover, we introduce a token chain prompting mechanism during the acoustic token generation stage to ensure timbre consistency throughout the speech enhancement process. Experimental results on benchmark datasets demonstrate that our proposed approach outperforms state-of-the-art SE systems in terms of speech quality and generalization capability. Codes and demos are publicly available at https://anonymous.4open.science/w/gen-se-7F52/.

Published as a conference paper at ICLR 2025
GENSE: GENERATIVE SPEECH ENHANCEMENT VIA LANGUAGE MODELS USING HIERARCHICAL MODEL-
ING
Jixun Yao', Hexin Liu", Chen Chen", Yuchen Hu², EngSiong Chng², Lei, Xie 1. Northwestem Polytechnical University 2. Nanyang Technological University
ABSTRACT
Semantic information refers to the meaning conveyed through words, phrases, and contextual relationships within given linguistic structure. Humans can leverage semantic information, such familiar linguistic patterns and contextual cues, reconstruct incomplete or masked speech signals in noisy environments. However, existing speech enhancement (SE) approaches often overlook the rich semantic information embedded in speech, which is crucial for improving intelligibility, speaker consistency, and overall quality of enhanced speech signals. To enrich the SE model with semantic information, employ language models semantic learner and propose a comprehensive framework tailored for language model-based speech enhancement, called GenSE. Specifically, approach SE a conditional language modeling task rather than a continuous signal regression problem defined in existing works. This is achieved by tokenizing speech signals into semantic tokens using pre-trained self-supervised model and into acoustic tokens using custom-designed single-quantizer neural codec model. To improve the stability of language model predictions, we propose hierarchical modeling method that decouples the generation of clean semantic tokens and clean acoustic tokens into two distinct stages. Moreover, we introduce token chain prompting mechanism during the acoustic token generation stage to ensure timbre consis- tency throughout the speech enhancement process. Experimental results on bench- mark datasets demonstrate that our proposed approach outperforms state-of off---f-thhee art SE systems in terms of speech quality and generalization capability. Codes and demos are publicly available at wufhtpp:: : //aox/naxxnn .tw.ggittii github. . i lo/gen-- se
efficient
INTRODUCTION
Speech enhancement (SE) aims to improve the intelligibility and quality of speech, particularly in situations where degradation is caused by various noises. Most existing SE approaches (Richter et al. 2023 Lemercier et al. 2023 Tai et al. 2024 Yang et al.) (024b) focus on learning a deter- ministic mapping from noisy clean speech modeling the distribution of clean speech. However, these approaches often ignore the rich semantic information embedded in speech, which can be cru- cial for improving intelligibility, speaker consistency, and overall signal quality. By neglecting se- mantic information, conventional approaches may experience performance degradation in complex noise conditions or when applied to unseen domains (Kato Baese- 2024| Wang et al. 2024c). Integrating semantic information into the SE process can better understand speech attributes, leading to more robust enhancement even in challenging acoustic environments.
Language models (LMs) have demonstrated their superiority in learning and representing seman- tic information across various natural language processing (NLP) tasks (Lehmann et al., 2023 Jin et al. (202)). Employing LMs in the enhancement process in generative manner can harness their powerful semantic modeling capabilities, enabling better speech reconstruction beyond simple noise removal. LM-based speech generation approaches first tokenize speech into discrete tokens before training models in a next-token prediction manner. There are two commonly used speech tokens: semantic and acoustic tokens. Acoustic tokens function at fine granularity, capturing detailed infor-
"Corresponding authors

---

Published as a conference paper at ICLR 2025
mation about the audio waveform and enabling high-quality reconstruction (Borsos al. 2023a). In
contrast, semantic tokens primarily encode higher-level information, such as phonetics, syntax, and semantics, which is essential for capturing language structure. In the conventional speech-to-specte generation pipeline, a continuous speech signal is first tokenized into semantic tokens, followed by LMs generating acoustic tokens based these semantic tokens other instructive information. Finally, acoustic tokens are reconstructed into the speech signal. Semantic tokens are crucial in this process, as they carry essential information about various speech attributes. In the SE task, the input speech is contaminated by noise, which complicates the modeling of both acoustic and semantic aspects. Noise distorts the speech signal, making it difficult to model semantic tokens accurately, which in turn affects the generation of acoustic tokens. Consequently, directly applying generative paradigms designed for clean speech to SE tasks can lead predictable performance degradation.
On the other hand, acoustic tokens offer the advantage of reconstruction paradigm, which plays a crucial role in bridging continuous speech and laken-based language models. Current acous- tic codec models (Délossez et al. 2022. Zeghidour et al., 2021: Zhang et al. (023) employ a ries quantizers organized in a residual manner to tokenize speech signals. While these models have demonstrated impressive performance, the multiple quantizers result in long acoustic token sequences, significantly increasing the computational expenses and difficulty for LMs. Meanwhile, when multiple quantizers employed, downstream LMs require additional design efforts to handle the increased complexity of predicting tokens extracted from multiple quantizers. These efforts may include incorporating both autoregressive and non-autoregressive structures (Wang et al. 2023a), parallel generation (Borsos et al. 2023b), or delay prediction (Copet et al. 2024).
In this study, we propose a novel generative framework, GenSE, that leverages LM and discrete speech tokens for speech enhancement. We design single quantizer codec model with a reorgani- zation process to reduce the number of tokens the LM needs to predict, resulting in simpler and more efficient prediction process. The reorganization process selects the most frequently used to- kens and combines them into a new quantization space, resulting significantly sacrificing performance. Additionally, we introduce hierarchical modeling method that separates the denoising and generation stages. Specifically, a noise-to-semann (N2S) module transforms the noisy speech signal into clean semantic tokens, while semantic-to-speech (S2S) module generates clean acoustic tokens. By decoupling denoising from generation, the N2S mod- ule focuses solely on transforming noisy semantic tokens into clean ones, ensuring that noise does not affect the subsequent generation process. This reduces the prediction difficulty for the LM, im- proving stability and performance in generating high-quality clean speech. We also explore various prompt types to leverage knowledge from language models in our approach, and we propose a to- ken chain prompting mechanism to concatenate noisy and clean semantic tokens along with noisy acoustic tokens to form a comprehensive prompt for the S2S module. This prompting mechanism helps capture more speaker characteristics from the noisy speech, ensuring timbre consistency. The main contributions of this study can be summarized as follows:
larger codebook size without
We present GenSE, novel generative framework designed for LM-based speech enhance- ment. It employs series of decoder-only LMs to predict discrete acoustic tokens based on semantic information, significantly improving both enhancement quality and generalization performance, even in challenging noise conditions.
We propose a simple neural codec model, SimCodec, which incorporates a reorganization process and contains only a single quantizer to reduce the number of acoustic tokens in the temporal dimension. SimCodec achieves remarkable reconstruction quality at a lower bit rate, optimizing both efficiency and performance for downstream LM prediction.
We introduce a hierarchical modeling method to decouple the denoising process from the generation process and improve the stability of LM prediction. A token chain prompting mechanism employed to ensure timbre consistency throughout the enhancement process.
RELATED WORK
2.1
GENERATIVE SPEECH ENHANCEMENT
Generative speech enhancement learns the distribution of clean speech as a prior for enhancement, rather than directly mapping noisy speech to clean speech. Most of generative models have been
2

---

Published as a conference paper at ICLR 2025
applied in SE, including generative adversarial networks (GANs) (Fu et al. 2019 Liu et al. 2021
variational autoencoders (VAEs) (Fang et al. 202T; Bie et al. 2022), J. flow based models
(Nugraha Yang al.
et al. 2020), and diffusion probabilistic models (Lemercier al. 2023 Tai et al.
2024
2024b Scheibler et al. 2024). A series of studies (Le et al. 2024 Yang et al.) 2023b Wang et al,
2024a) have leveraged large-scale datasets develop audio generation models capable of handling multiple tasks, such as speech synthesis (Zhang et al. (224)) voice conversion (Yao et al., 2024b a) and speech recognition (Liu al. 2024b). While these models demonstrate versatility, their perfor- mance may be constrained compared to systems specifically optimized and designed for particular task. Meanwhile, with the growing popularity of discrete representations extracted from neural audio codecs, some works have begun to use discrete tokens for speech enhancement. Genhancer (Yang) jet al. 2024a) argues that discrete codec tokens offer an efficient latent domain, which can replace the conventional time or time-frequency domain of signals, allowing for more complex modeling of speech. Another work, called SELM (Wang et al., 2024c), a pioneer in exploring using LMs for speech enhancement. It also employs discrete semantic tokens as the speech representation and incorporates LMs to generate clean semantic tokens conditioned on noisy input. However, these approaches still fall short in terms of speech quality and similarity compared to clean speech.
2.2 SPEECH TOKENIZATION
There are two widely used speech tokens in speech language models: semantic tokens and acoustic tokens. Semantic tokens are typically derived from representations produced by self-supervised speech models like HuBERT (Hsu al. 202T) or WavLM (Chen et al.) [022), and they primarily capture the phonetic and semantic content of speech signals. While semantic tokens are initially designed as training targets for self-supervised models, recent efforts have used them to directly represent high-level semantic information (Borsos et al.) (2023a). Acoustic tokens, on the other hand, are extracted from neural audio codec models, which tokenize high-rate audio signals into finite set of tokens (Défossez et al. 2022). AudioLM (Borsos et al. (023a) pioneered the use of LMS for audio generation by employing both semantic and acoustic tokens and building several LMs across different stages. VALL-E (Wang et al. (2023a) further extended the AudioLM framework and applied to text-to-speech (TTS), achieving impressive zero-shot voice cloning. Inspired by VALL- E. several speech-to-spedt voice conversion (Wang et al. (2023b), have replaced phonemes with semantic tokens and introduced powerful LMs to achieve remarkable results.
generation tasks, such speech translation (Dong al., 2023) and
2.3 NEURAL AUDIO CODEC
Previous neural audio codec models have been widely used in audio communication to compress audio into discrete representations. Recently, these codec models have gained interest in speech generation tasks, bridging continuous speech and token-based language models. A typical codec model consists of an encoder, a quantization module, and a decoder, with the quantization module playing a crucial role in tokenizing continuous speech features into discrete tokens. Several works explore different architectures to pursue better reconstruction quality ( (Wu et al. 2023) Kumar et al., 2024 Ai et al., 2024), others enhance the compression rate (Yang al. [202aa] et al. 22024a b) or optimize codec space (Ren et al., (2024) Ju et al. 2024 Zhang et 20.3 Du et al. 2024 Liu et al.) 2024a). Most approaches employ residual vector quantization (RVQ), which typically uses eight quantizers, each with a 1024-codebook size, to provide sufficient discrete space for quantization. However, the numerous acoustic tokens generated by these models create significant complexity and challenges in the LM prediction. Although some works (Li al. 2024 Ji et al. 2024b) explore the use of a single quantizer, their reconstruction quality often falls short or relies on an additional reference encoder to disentangle global representations, which are unsuitable for speech enhance- ment.
3 GENSE
3.1 OVERALL ARCHITECTURE
In this section, we introduce GenSE, a cutting-edee system designed to enhance various types of degraded signals, providing better speech quality, speaker similarity, and generalization ability. As
3

---

Published as a conference paper at ICLR 2025
shown in Figure 1 GenSE is composed of three key components: 1) SimCodec, neural speech codec that extracts acoustic tokens from speech through quantizer and reconstructs the speech sig- nal from these tokens via decoder; 2) A pre-trained self-supervised model, XLSR (Conneau al. 2020), used for extracting high-level semantic information from speech; 3) Two decoder-only LMs that generate clean speech signals conditioned noisy input. GenSE employs hierarchical mod- cling framework with two-stage process: N2S transformation front-end, which converts noisy speech into clean semantic tokens, and an S2S generation back-end, which synthesizes clean speech using both semantic tokens and noisy acoustic tokens while preserving the speaker's personalized characteristics. We introduce the SimCodec in Section 3.2 and the hierarchical modeling framework in Section 3.3
Noley Semantic Token
Noiey Acoustic Token
SimCodec Decoder
Clean Semantic Token
Clean Acoustic Token
Clean Speech
1
M
O
N2S Language Model
S2S Language Model
Token Chain Prompting
Semantic Extractor
SimCodec Encoder
Noisy Speech
Figure 1: The hierarchical modeling framework of language model in GenSE
3.2
REORGANIZATION PROCESS FOR SIMCODEC
3.2. SIMCODEC OVERVIEW
We propose an efficient neural codec model, SimCodee, to compress speech signals into discrete tokens by single quantizer, thereby reducing the number of tokens the LM needs to predict. Sim- Codec is based on the VQVAE architecture (Van Den Oord et al. 2017) and follows the same pattern as EnCodec (Défossez et al. 2022]. Specifically, SimCodec consists of three modules: 1) fully convolutional encoder that compresses the input speech signal into a frame-level latent represen- tation; 2) quantizer that discretizes the frame-level latent features into discrete tokens using the codebook; and 3) a decoder that mirrors the encoder's structure to reconstruct the speech waveform from discrete tokens.
The quantized discrete token in SimCodec acts as the acoustic token, which serves as the generation target for the S2S module. Unlike most existing codec models that employ multiple quantizers, SimCodec utilizes a single quantizer with larger codebook size of 8192. While training model with such a large codebook size often results in low codebook usage and slow convergence, we address these challenges through a specially designed two-stage training process with a codebook reorganization process. This approach improves both codebook usage and reconstruction quality, enabling SimCodec to achieve high performance with fewer tokens.
3.2.2 EXPLORATION OF CODEBOOK SIZE AND QUANTIZE SPACE
acoustic token extractor for speech language models, with focus on achieving higher reconstruction quality and audio fidelity using fewer tokens. In neural codec models, the quality is primarily determined by the bit rate (Défossez et al. 2022), which is influenced by factors such as the codebook size, token count per second, and the number of quantizers (Mentzer et al. 2023b). Initially, we aim to simplify the model by reducing the number of quantizers from the conventional eight to a single quantizer, while keeping other factors constant. However, this approach resulted in suboptimal performance, with the training process being difficult to converge.
SimCodec is designed
4

---

Published as a conference paper at ICLR 2025
We then consider expanding the codebook size to accommodate the vast space of discrete speech, while still using only a single quantizer. This means that when reducing the number of quantizers from eight to one, try to expand the codebook size by eight times, though it doesn't It result in the same bit rate. As result, we progressively increase the codebook size from 2 to 213 and evaluate the codebook usage of quantized embeddings and the reconstruction quality.
As shown in Figure 2 we use the Perceptual Evaluation of Speech Quality (PESQ) metric to evaluate the reconstruction quality of Sim- Codec. We observe clear trend: as the code- book size increases, so does the reconstruction quality. This aligns with expectations from compression perspective, where larger code- book size provides more bits to represent in- formation, typically resulting in better recon- struction performance. Specifically, we find that conventional codec codebook size of 2 may not fully utilize the potential of the speech space. When we increase the codebook size from 210 to 211 the PESQ score improves, in- dicating better reconstruction quality, However, further increasing the codebook size to 2 results in a significant drop in codebook usage from nearly 100% around 50%. This means that half of the quantized embeddings are left unused in the quantizer, which leads to a degradation in reconstruction quality. This diminishing return at larger codebook sizes in a single quantizer led us explore a more refined solution.
4,0
100
Codebook Usage
90
3.5
80
3.0
70
2.5
Codebook Usage
60
2.0
2
2
2
2
2
2
Figure 2: The relationship between reconstruction quality and codebook usage across different code- book sizes.
Staget:Training with group quantizer
Reorganize Process
Vector
Encoder
Quantizer
Decoder
Vector
Quantizer
Selected
Goncat
Copy
Copy
Reorganize
Encoder
Decoder
New Vector
Stage2:Training with single quantizer
Final quantized embedding
The detailed architecture and training process of SimCodec, with the reorganization pro- cess of the group quantizer is highlighted the red dashed block.
Figure 3:
3.2.3 CODEBOOK REORGANIZATION
To improve the codebook usage and reconstruction quality with a larger codebook size, we design a two-stage training process for SimCodec, as illustrated in Figure 3 In the first stage, we employ two quantizers with smaller codebook sizes, organized in a group quantization scheme rather than the traditional residual quantization method (Défossez et al. (022). In conventional residual quan- first quantizer, leading to an unequal
tization, the most important information is concentrated distribution of informativeness across subsequent quantizers. This imbalance makes it difficult to merge the two quantizers equitably to create a new quantization space.
After the first stage, we sort the quantized embeddings in each quantizer by codebook usage and select the top N and K quantized embeddings from each quantizer, respectively. We then pairwise concatenate these selected quantized embeddings and reorganize them into codebook size fit the new codebook dimension, while copying the parameters from the first stage for continued
codebook with K. the second stage, we initialize the encoder and decoder parameters
5

---

Published as a conference paper at ICLR 2025
training. At this point, the codec model is nearly fully trained, except for the projection layer adapted to the new codebook. As a result, the training in this stage converges quickly due to the quantized space remains almost identical to the first stage. Additionally, since the quantization process is tied to the dimensional space rather than the temporal domain, the concatenated embeddings will maintain high codebook usage within similar quantized space. SimCodec is trained in end-to- end GAN-based manner, optimizing a reconstruction loss over frequency domains and perceptual loss using discriminators different resolutions, details can be found in Appendix A.2
3.3 HIERARCHICAL MODELING FRAMEWORK
To improve the stability of LM prediction, we propose hierarchical modeling framework with two main components: a N2S transformation front-end, which transforms noisy speech into clean semantic tokens, and S2S back-end, which synthesizes clean speech while preserving the person- alized characteristics of the original speaker. By explicitly decoupling the denoising process from the generation process, noise affecting semantic token extraction is addressed early on, preventing it from influencing speech generation. Figure provides an illustrative overview of GenSE with the hierarchical modeling framework.
3.3. NOISY-TO-SEMANTIC TRANSFORMATION
The N2S module first employs powerful pre-trained self-supervised model to extract semantic tokens from the speech signal. For this purpose, we employ XLSR which is a large-scale multi- lingual pre-trained model for speech representation. XLSR first encodes the speech signal using stack of convolutional and transformer layers to produce continuous representations at every 20-ms frame and then applies k-means clustering to discretize these representations into set of cluster indices = {$1 $2, } T is the number frames and ST E [K], where K is the number of cluster centroids and we set K 1024.
By compressing speech signals into discrete semantic tokens through the self-supervised model, semantic irrlevvnt information from continuous speech representations is eliminated, enabling more effective training in an NLP paradigm. Mcanwhile, the self-supervised model is also noise-robust some extent (Chen et al. 2022). The N2S module leverages language model learn denoising transformations through a next-token prediction approach. The training objective is as follows:
( II (sn $1...., Sn-1, $1....))
where å and & represent clean semantic tokens and noisy semantic tokens, respectively. n is denote the frame length, while is concatenated with the and trained in an autoregressive manner.
3.3.2 SEMANTIC-TO-SPELRE
GENERATION
The S2S module is responsible for generating clean acoustic tokens and reconstructing clean speech signals using the SimCodec decoder. Instead of directly converting clean semantic tokens into acous- tic tokens, we introduce token chain prompting mechanism in the generation process. This mecha- nism leverages both the noisy semantic token and the predicted clean semantic token as intermediate prompts, which concatenated with the noisy acoustic token to generate the clean acoustic token. The acoustic token in the token chain prompting mechanism provides characteristics of the original speaker, helping ensure consistency of speaker timbre in the enhancement process.
Benefit from the SimCodec, the extracted acoustic token is a single sequence in the temporal dimen- sion, which becomes straightforward to directly connect semantic and acoustic tokens and compute the joint probabilities conditioned on the prompt token:
pΦz) II p (ān ......., an 31...... $1..... @1.....)
where à and à represent clean acoustic tokens and noisy acoustic tokens, respectively. Once the S2S module generates the clean acoustic tokens, the decoder component of SimCodec reconstructs

---

Published as a conference paper at ICLR 2025
them back into the enhanced speech signals. The hierarchical modeling method allows each stage to specialize in specific tasks tasks-denoising in the first stage and speech generation in the second stage. As a result, the prediction difficulty for the LM is reduced, leading improved stability and performance in denoising the degraded speech and generating high-quality speech.
Comparison results between our proposed GenSE and various baseline systems. "Noisy" refers to the original degraded signals and higher scores indicate better performance.
Table 1:
Without Reverb
With Reverb
DNSMOS
DNSMOS 1
SECS
vQ.
SECS
vQ.
SIG
BAK
OVL
SIG
BAK
OVL
Noisy
0.566
1.76
1.50
1.39
0.527
FullSubNet
3.05_-0.0%
3.51,000%
2.93...0%
0.62
0.623
2. 2.74
+.01+0.0%
2.27
0.63
0.615
Inter-Subnet
3.17+3..0g
3..3%
2.98_..77%
0.62
0.616
2.85
3.09 42.99
2.36
0.61
0.594
CDiffuSE
3.35-10.5%
3.5240-096
1.84
0.58
0.624
2.82,22.9%
3.31 3.314
2.42.6..9
0.57
0.611
SGMSE
3.47 -1..5%
3,41 2.8%
3.11..
0.58
0.651
3.07
3.02 40.0%
2.49. 99.19
0.56
0.632
StoRM
3.54 10.00
3.69; 45.156
3.15.
0.61
0.687
3.02.-10.2%
45.8%
2,56
0.62
0.646
SELM
3.47 -15.5%
3.81 45.855
3.21 .j.0%%
0.62
0.673
3.04 10.09
3.51
4.0.683
2.67 417.655
0.61
0.659
DOSE
3.58 17.3%
3,98, 15.4%%
3.31
0.61
0.692
3.20.
3.79
20.999
3,13 4a7.00.
0.63
0.668
GenSE
3.68 19.75%
4.18, 10.4%
3.43
0.67
0.717
3.49
-27.44
3.73
3.19,
0.65
0.671
4 EXPERIMENTS AND RESULTS
4.1 EXPERIMENTAL SETUP
Following previous works (Wang al. 2024c Tai al.] ) the clean speech data con- 2019), VoiceBank (Veaux) al. 2013), and the deep noise suppression (DNS) challenge datasets RRedy et al.] 2021 I). The noise datasets used are WHAM! (Wichern et al. 2019) and DEMAND (Thiemann et al.) 2013). Room impulse responses (RIRs) from openSLR26 and openSLR28 (Ko et al.) 2017) are randomly selected to simulate reverberation. All training data are generated on the fly, with an 80% prob- ability of adding noise at signal-to-noiss ratio (SNR) ranging from dB to 20 dB, and a 50% probability of convolving the speech with RIRs. We use the testset from publicly available DNS Challenge (Reddy et al. 202I) to compare GenSE with existing state-of-the-an baseline systems. Following Tai et al. (2024). we use the CHiME-4 dataset (Du et al. 2016 as additional test set to evaluate the generalization ability of the models. All samples are at a 16kHz sample rate.
Dataset:
sists of subsets from LibriLight (Kahn et al. 2020). LibriTTS (Zen et al.
Baseline systems: We compare our GenSE with recent remarkable and state-of- of-the-ar art speech en- hancement models containing: FullSubNet (Hao et al. 2021), Inter-Subnet (Chen et al. 2023), CDif- fuSE (Lu et al. 2022), SGMSE (Welker al. 2022). ), StoRM (Lemercier et al. 2023) SELM (Wang) et al. 2024c). and DOSE (aii al. 2024). Model configuration and details of baseline systems can
be found in Appendix A.3 and Appendix A.4 respectively.
Evaluation metrics: Both objective and subjective metrics are used to evaluate the performance between our proposed GenSE and baseline systems.
For objective metrics, we employ DNSMOS, speaker embedding cosine similarity (SECS), VQScore (Fu et al. 2024) and word error rate (WER) the evaluation metrics. Specif- ically: 1) We use the official DNSMOS (Reddy et al., 2022) model from the DNS chal- lenge and VQscore toolkit evaluate the quality of the enhanced signals. 2) We use the LavLM-TDCHL speaker verification mode to measure speaker similarity between en- hanced and clean signals. 3) We use pre-trained ASR mode to transeribe the generated speech and compare with ground-truth transcription. Meanwhile, we employ widely used metrics such as PESQ, short-time objective intelligibility (STOI), and Mel-Ceptral Distor- tion (MCD) for codec evaluation. Additionally, we introduce UTMOS (Sacki et al. 2022
https: htttes///gtthb..coom////ss//////
veri perification

---

Published as a conference paper at ICLR 2025
automatic Mean Opinion Score (MOS) prediction system to evaluate the reconstructed quality of codec from a perceptual perspective.
For subjective metrics, we use the naturalness mean opinion score (NMOS) and the simi- larity mean opinion score (SMOS) evaluate the naturalness and speaker similarity of the enhanced signals, respectively. Participants are invited to listen to the enhanced signals and provide their subjective perception scores 5-point scale: '5' for excellent, '4' for good, '3' for fair, '2' for poor, and 'I" for bad.
4.2 EXPERIMENTAL RESULTS ON SPEECH ENHANCEMENT
Objective Results. We compare our proposed GenSE with various baseline systems, and the objec- tive metric results are summarized in Table We observe the following: 1) The signals enhanced by GenSE show a significant improvement over the noisy input, achieving an OVL score of 3.43 with- out reverb and 3.19 with reverb. This indicates that GenSE effectively enhances degraded speech to a much higher quality. 2) GenSE outperforms baseline systems by substantial margin in evaluation metries, confirming its superiority in speech enhancement tasks. 3) GenSE achieves an SECS score of 0.67 without reverb and 0.65 with reverb, outperforming all baseline systems. This highlights its ability to preserve speaker identity while enhancing speech quality. These results demonstrate the superiority of GenSE in addressing various forms of degraded speech signals.
Generative models typically aim to fit the distribution of the training samples, which makes them more robust against domain shifts in the input data. We investigate this property of our proposed GenSE and compare the generalization abilities of GenSE to other baseline systems using the CHiME-4 dataset (Du et al.) (016). The CHiME-4 dataset includes real-recorded noises from four distinct real-world environments: street, pedestrian areas, cafeteria, and bus. These environments significantly differ from the training data domain, providing a comprehensive test for evaluating the generalization performance of the models.
Table 2: Comparison results of enhancement performance on the CHiME-4 dataset.
DNSMOS
SECS 1
VQ
WER +
SIG
BAK
OVL
Noisy
1.71
1.45
1.36
0.502
37.9
FullSubNet
2.42+0.0%
2.71+0.0%
2.01
0.54
0.632
34.1
Inter-Subnet
2,45
2.69-1.895 -18%%
2.08 +3.5%
0.53
0.627
29.6 29.6
CDiffuSE
2.61 47.8%%
2.72+0..%%
2.38 488.4%
0.56
0.622
40.7 4..4%
SGMSE
2.77 14.59%
2.73 4..4%
2.41 + 19.99
0.57
0.639
37.2 1.8%
StoRM
3.21 4.3..4%
3.08 1...7%
2.57 +27..9%
0.55
0.645
35. 5.8%
SELM
3.09 4.0.7%
3.26
226.220.9% 20.
2.62
2.62+33.355 30.
0.58
0.641
29. 22.4%
DOSE
3.17 3.17 +30.99
2.88 +6.3%
2.61 +20.99
0.61
0.654
31. $1.2. 17.7%
GenSE
3,22 +33.1%
3,47
4.28.009 0%%
2.89
+-4..8%
0.62
0.650
28.4 25.1%
As shown in Table 2. when there is a domain shift from the DNS test set to the CHiME-4 test set, deterministic-based approaches like FullSubNet and Inter-Subnet experience significant drop in performance compared to generative-bar based approaches. The difference in noise characteristics be- tween the training data and the CHiME-4 test set leads drastic decline in DNSMOS scores of the enhanced signal, in some cases nearly reaching the levels of the noisy signals. Our proposed GenSH achieves the best performance in both quality and speaker similarity compared with baseline systems, demonstrating more excellent generalization capability against domain shifts in noise char- acteristics. Additionally, GenSE achieves the lowest WER results, highlighting its ability to enhance speech intelligibilitt and preserve linguistic information, even in challenging noise environments. While the metrics on the CHiME-4 test set are lower than those on the DNS test set, GenSE shows less degradation compared other baseline systems, proving its robustness to variations in noise characteristics.
Subjective Results.
We visualize the subjective evaluation results of speech naturalness and speaker similarity using violin plots, as shown in Figure Participants rated GenSE significantly higher
8

---

Published as a conference paper at ICLR 2025
than all baseline systems in both NMOS and SMOS. The median values for speech naturalness and speaker similarity of signal enhanced by GenSE surpass those the baseline systems. Additionally, the overall distribution of subjective scores for GenSE outperforms the baselines, further highlight- ing its superiority in naturalness, quality, and speaker similarity of the enhanced speech. These subjective findings further corroborate the objective results presented in Table and Table
5
MO5
2
SMOS
NMOS
FullSubNet Inter- CDiffuSE
SGMSE
StoRM
SELM
DOES
GenSE
Figure 4: Violin plots for speech naturalness and speaker similarity, comparing the signals enhanced by baseline systems and GenSE. The thin inner line of each violin plot represents the full range of MOS values, while the thick inner line marks the interquartile range (between the first and third quartiles). The white point in each plot indicates the median value.
4.3 EXPERIMENTAL RESULTS ON SIMCODEC
In this subsection, we compare the proposed SimCodec in terms reconstruction quality with strong baselines such as EnCodec (Détossez et al. 2022). HiFi-Codec (Yang et al., 2023a). Speech Tok- enizer (Zhang et al. 2023), Vocos (Siuzdak) 2023). and Descript-Audio-Cotle (DAC) (Kumar al.] 2024). Table 3 presents the comparison results across different bit rates. Our proposed SimCodec significantly outperforms DAC under similar bandwidth settings (1.73 in PESQ, 0.17 in STOI, 1.09 in MCD, and 1.31 in UTMOS, respectively) and achieves comparable performance with other base- lines that operate at higher bit rates. Additionally. SimCodec delivers high-quality reconstruction while using the fewest tokens, significantly reducing the complexity and difficulty of downstream tasks compared to other codec models. We also compare the performance GenSE using different codec models for speech enhancement, with results provided in Appendix A.5) Moreover, inves- tigate the performance replacing our proposed reorganization strategy with different quantization strategies, results shown Appendix A.6 These experiments further confirm the necessity and effectiveness of SimCodec within the entire speech enhancement framework.
Table 3: Comparison results of reconstruction quality between our proposed SimCodec and base- line codec models. "Nq" represents the number of the quantizer in the codec model and "token/s" represents the token count per second after quantization.
Model
Bandwidth
Nq
token/s
PESQ
STOI f
MCD
UTMOS 1
DAC
9.0 kbps
9
900
3.77
0.971
2.67
4.05
SpeechTokenizer
6.0 kbps
600
3.25
0.957
3.06
3.57
Encodec
6.0 kbps
8
600
2.92
0.946
3.51
3,41
Vocos
6.0 kbps
8
600
3.37
0.961
3.22
3.48
DAC
4.0 kbps
4
400
2.61
0.917
4.03
3.09
HiFi-Codec
4.0 kbps
400
3.17
0.959
3.66
3.48
SpeechTokeninn
3.0 kbps
300
2.71
0.922
4.07
3.16
Encodec
3.0 kbps
4
300
1.93
0.884
4.78
2.30
Vocos
1.5 kbps
2
150
1.59
0.812
4.74
2.55
DAC
1.0 kbps
100
1.32
0.784
4.91
2,06
SimCodec
0.65 kbps
50
2.45
0.903
3.99
3.04
SimCodec
1.3 kbps
100
3.05
0.954
3.82
3.37

---

Published as a conference paper at ICLR 2025
4.4 ABLATION AND ANALYZE
We conduct an ablation study to evaluate the performance of removing the token chain prompting mechanism and hierarchical modeling from our GenSE framework, highlighting the contribution of each component the system's 's overall performance. Specifically: 1) We only use the clean seman- tic token to generate the clean acoustic token in the S2S module rather than concatenating the clean semantic token with the noisy semantic token and noisy acoustic token as intermediate prompts. 2) We further remove the hierarchical modeling, employing a single LM with the same model parame- ters the combined LMs in N2S and S2S modules to directly predict the clean acoustic token from the noisy semantic token.
Results of ablation study on GenSE framework, where "w/o" denotes the absence of specific method or component.
Table 4:
DNSMOS
SECS
VQ
SIG
BAK
OVL
GenSB
3.57.40.0%
0.96
3.31 3.31 40.0%
0.66
0.649
w/o token chain prompting
3.54-0.850
4.01 +1.3%
3.28 p.9%
0.43
0.636
w/o hierarchical modeling
3.38 -5.8%
3.75 9.65%
3.17-4.4%
0.41
0.621
As shown in Table 4 removing the token chain prompting mechanism leads to significantly lower SECS results compared to the original GenSE. This indicates a degradation in speaker similarity, with the S2S module failing to maintain speaker consistency after enhancement, This highlights the crucial role of the token chain prompting mechanism in capturing more acoustic characteristics of the original speaker. Furthermore, a notable decline in DNSMOS is observed when switching from hierarchical modeling to a single LM. This confirms the effectiveness of separating denoising and generation into different stages. The hierarchical modeling method not only improves the stability of the generation process but also allows the system focus more effectively on both semantic and acoustic aspects, resulting in superior performance when enhancing degraded signals. A case comparison of the hierarchical modeling ablation presented in Appendix A.7
5 CONCLUSION
In this study, we propose GenSE, an LM-based generative SE system with two key components: 1) A novel neural speech codec, SimCodec, with reorganizing process to improve the codebook usage under larger codebook size using a single quantizer. The proposed codec delivers high-quality re- construction with fewer tokens, significantly reducing the complexity and difficulty of downstream tasks. 2) A hierarchical modeling framework designed to enhance degraded signals through a two- stage generative process. In our framework, N2S front-end transforms noisy speech into clean semantic tokens, and an S2S back-end synthesizes clean speech from the generated clean semantic tokens. We also design token chain prompting mechanism within the hierarchical modeling frame- work to maintain speaker consistency. Experimental results demonstrate that GenSE outperforms state-of-the-art speech enhancement systems in both subjective and objective evaluations, demon- strating its effectiveness speech enhancement. Ablation studies and codec comparison results further confirm the effectiveness of each component in our proposed GenSE framework. Limita- tions and future works are discussed in Appendix A.8 A.8
ACKNOWLEDGMENT
This work was supported by Tencent and Tencent-NTU Joint Research Laboratory (CENTURY), Nanyang Technological University, Singapore.
REFERENCES
Yang Ai, Xiao-Hang Jiang, Ye-Xin Lu, Hui-Peng Du, and Zhen-Hua Ling. Apcodec: A neural audio codec with parallel amplitude and phase spectrum encoding and decoding. arXiv preprint arXiv:2402. 10533, 2024.
10

---

Published as a conference paper at ICLR 2025
Xiaoyu Bie, Simon Leglaive, Xavier hancement using dynamical variational autoencoders. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:2993-30077 2022.
and Laurent Girin. Unsupervised speech
Zalán Borsos, Raphačl Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier. Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 21:2253-25333 2023a.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.09636 2023b.
Jun Chen, Wei Rao, Zilin Wang, Jiuxin Lin, Zhiyong Wu, Yannan Wang, Shidong Shang, and Helen Meng. Inter-subnet: Speech enhancement with subband interaction. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):5505- 1518, 2022.
Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint SEGI:2000611972 2020.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and controllable music generation. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi High fidelity neural audio compression. arXiv preprint arXiv:2210.13433. 2022.
Qianqian Dong, Zhiying Huang, Qiao Tian, Chen Xu, Tom Ko, Yunlong Zhao, Siyuan Feng, Tang Li, Kexin Wang, Xuxin Cheng, et al. Polyvoice: Language models for speech to speech transla- tion. arXiv preprini 2300:22306.02882.. 2023.
Jun Du, Yan-Hui Tu, Lei Sun, Feng Ma, Hai-Kun Wang, Jia Pan, Cong Liu, Jing-Dow Chen, and Chin-Hui Lee. The ustc-iflytek system for chime-4 challenge. Proc. CHiME, 4(1):36-38, 2016.
Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng, Funcodec: fundamental, reproducible and integrable open-source toolkit for neural speech codec. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 591-595, 2024.
Huajian Fang, Guillaume Carbajal, Stefan Wermter, and Timo Gerkmann. Variational autoencoder for speech enhancement with a noise-aware encoder. In ICASSP 2021-2021 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 676-680, 2021.
Szu-Wei Fu, Chien-Feng Liao, Yu Tsao, and Shou-De Lin. Metricgan: Generative adversarial net- works based black-hox metric scores optimization for speech enhancement. In International Conference on Machine Learning, pp. 2031-2041. PmLR, 2019.
Szu-Wei Fu, Kuo-Hsuan Hung, Yu Tsao, and Yu-Chiang Frank Wang. Self-supervised speech qual- ity estimation and enhancement using only clean speech. arXiv preprint arXiv:2402.1632. 2024.
Xiang Hao, Xiangdong Su, Radu Horaud, and Xiaofei Li. Fullsubnet: A full-band and sub-band fusion model for real-time single-channel speech enhancement. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6633-6637, 2021.
11

---

Published as a conference paper at ICLR 2025
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions audio, speech, and language processing, 29:3451-3460, 2021.
Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, and Yossi Adi. Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech enhancement. arXiv preprint arXiv:2212. 1137,, 2022.
Shengpeng Ji, Minghui Fang. Ziyue Jiang, Rongjie Huang, Jialung Zuo, Shulei Wang, and Zhou Zhao. Language-codec: Reducing the gaps between discrete codec representation and speech language models. arXiv preprint arXiv:2402. /2208, 2024a.
Shengpeng Ji, Ziyue Jiang, Xize Cheng, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Ruiqi Li, Ziang Zhang, Xiaoda Yang, et al. Wavtokenizer: efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint arXiv:2408. 16532, 2024b.
Bowen Jin, Hansi Zeng. Guoyin Wang, Xiusi Chen, Tianxin Wei, Ruirui Li, Zhengyang Wang, Zheng Li, Yang Li, Hanqing Lu, et al. Language models semantic indexers. arXiv preprint arXiv:2310.07815, 2023.
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng. Kaitao Song, Siliang Tang, et al. Naturalspeech Zero-shot speech synthesis with factor- ized codec and diffusion models. arXiv preprini arXiv:2403. S4K:::2003.30000 03100, 2024.
Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri- light: A benchmark for with limited or no supervision. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673. IEEE, 2020.
Misaki Kato and Melissa M Baese-Bert The effects of acoustic and semantic enhancements on perception of native and non-native speech. Language and Speech, 67(1):40-71, 2024.
Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael Seltzer, and Sanjeev Khudanpur. A study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE interna- tional conference on acousties, speech and signal processing (ICASSP), pp. 5220-5224, 2017.
Rithesh Kumar, Kundan Kumar, Vicki Anand, Yoshua Bengio, and Aaron Courville. Nu-gan: High resolution neural upsampling with gan. arXiv preprint arXiv:2010.11362,, 2020.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36, 2024.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36, 2024.
Jens Lehmann, Preetam Gattogi, Dhananjay Bhandiwad, Sébastien Ferré, and Sahar Vahdati. Lan- guage models as controlled natural language semantic parsers for knowledge graph question an- swering. In ECA/ 2023, pp. 1348-1356 2023.
Jean-Marie Lemercier, Julius Richter, Simon Welker, and Timo Gerkmann. Storm: A diffusion- based stochastic regeneration model for speech enhancement and dereverberation. JEEF/ACM Transactions Audio, Speech, and Language Processing, 2023.
Hanzhao Li, Liumeng Xue, Haohan Guo, Xinfa Zhu, Yuanjun Lv, Lei Xie, Yunlin Chen, Hao Yin, and Zhifei Li. Single-codec: Single-codebook speech codec towards high-performance speech generation. arXiv preprint arXiv:2406. 00742, 2024.
Haoyu Li and Junichi Yamagishi. Noise tokens: Learning neural noise templates for environment- aware speech enhancement. arXiv preprint axXiv:200404001,, 2020.
12

---

Published as a conference paper at ICLR 2025
Shijia Liao, Yuxuan Wang, Tianyu Li, Yifan Cheng, Ruoyi Zhang, Rongzhi Zhou, and Yijin Xing. Fish-speech: Leveraging large language models for advanced multilingual text-to exxt-to-speec synthe- sis. arXiv preprint arXiv:2411.01133 2024.
Haohe Liu, Qiugiang Kong, Qiao Tian, Yan Zhao, DeLiang Wang. Chuanzeng Huang, and Yux- uan Wang. Voicefixer: Toward general speech restoration with neural vocoder. arXiv preprint WrXiv:2109.1373: 2021.
Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang. and Mark D Plumbley. Se- manticodec: An ultra low bitrate semantic audio codec for general sound. arXiv preprint 00f1:22005000333, 2024a.
Hexin Liu, Xiangyu Zhang, Leibny Paola Garcia, Andy WH Khong, Eng Siong Chng, and Shinji Watanabe. Aligning speech to languages to enhance code-switching speech recognition. arXiv preprint arXiv:2403.05887 2024b.
Yen-Ju Lu, Zhong-Qun -iu Wang. Shinji Watanabe, Alexander Richard, Cheng Yu, and Yu Tsao. Con- ditional diffusion probabilistic model for speech enhancement. In ICASSP 2022-2022 IEEE In- ternational Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 7402-7406, 2022.
Soumi Maiti and Michael Mandel. Speaker independence of neural vocoders and their effect parametric resynthesis speech enhancement. In ICASSP 2020-2020 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pp. 206-210, 2020.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made simple. arXiv preprint arXiv:2309. 15505, 2023a.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made simple. arXiv preprint arXiv:2309. 15505, 2023b.
Aditya Aric Nugraha, Kouhei Sekiguchi, and Kazuyoshi Yoshii. A flow-based deep latent variable model for speech spectrogram modeling and enhancement. IEEE/ACM Transactions Audio, Speech, and Language Processing, 28:1 18:1104 104-1117, 2020.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispecch: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 52065520110 2015.
Chandan KA Reddy, Harishchandra Dubey, Kazuhito Koishida, Arun Nair, Vishak Gopal, Ross Cutler, Sebastian Braun, Hannes Gamper, Robert Aichner, and Sriram Srinivasan. Interspeech 2021 deep noise suppression challenge. arXiv preprint ar wxXi:2101.009000 2021.
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2022- 2022 IEEE Inter- national Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 886 886-890, 2022.
Yong Ren, Tao Wang, Jiangyan Yi, Le Xu, Jianhua Tao, Chu Yuan Zhang, and Junzuo Zhou. Fewer-token neural speech codec with time-invariant codes. In ICASSP 2024-2024 IEEE In- lernational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12737-12741 IEEE, 2024.
Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhancement and dereverberation with diffusion-based generative models. IEEF/ACM Transac- fions on Audio, Speech, and Language Processing, 31 31:2351-2364, 2023.
Takaaki Sacki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hi- roshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.021152 2022.
Robin Scheibler, Yusuke Fujita, Yuma Shirahata, and Tatsuya Komatsu. Universal score-based speech enhancement with high content preservation. arXiv preprint arXiv:2406. /2/94, 2024.
13

---

Published as a conference paper at ICLR 2025
Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint 1xXiv:33066008144 2023.
Wenxin Tai, Yue Lei, Fan Zhou, Goce Trajcevski, and Ting Zhong. Dose: Diffusion dropout with adaptive prior for speech enhancement. Advances in Neural Information Processing Systems, 36, 2024.
Joachim Thiieman, Nobutaka Ito, and Emmanuel Vincent. The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings. In Proceedings of Meetings on Acoustics, volume 19. AIP Publishing, 2013.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Christophe Veaux, Junichi Yamagishi, and Simon King. The voice bank corpus: Design, collection and data analysis of large regional accent speech database. In 2013 international conference oriental COCOSDA held jointly with 2013 conference on Asian spoken language research and evaluation (O-COCOODAVCASLRE)) pp. 1-4, 2013.
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arX1v:2301.02111, 2023a.
Xiaofei Wang. Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang, Shujie Liu, Jinyu Li, and Takuya Yoshioka. Speechx: Neural codec language model as versatile speech transformer. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 2024a.
Zhichao Wang, Yuanzhe Chen, Lei Xie, Qiao Tian, and Yuping Wang. Lm-ve: Zero-shot voice conversion via speech generation based on language models. IEEE Signal Processing Letters, 2023b.
Zhichao Wang, Yuanzhe Chen, Xinsheng Wang, Zhuo Chen, Lei Xie, Yuping Wang, and Yuxuan Wang. Streamvoice: Streamable context-aware language modeling for real-time zero-shot voice conversion. arXiv preprint 2rXiv:2401.11053 2024b.
Ziqian Wang, Xinfa Zhu, Zihan Zhang, YuanJun Lv, Ning Jiang, Guoqing Zhao, and Lei Xie. Selm: Speech enhancement using discrete tokens and language models. Proc. ICASSP, pp. 11561- 11565, 2024c.
Simon Welker, Julius Richter, and Timo Gerkmann. Speech enhancement with score-based genera- tive models in the complex stft domain, arXiv preprint arXiv:2203. 17004, 2022.
Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. Wham!: Extending speech separation to noisy environments. arXiv preprint arXiv: 1907.01 160, 2019.
Yi-Chiao Wu, Israel D Gebru, Dejan Marković, and Alexander Richard. Audiodec: An open-source streaming high-fidelity neural audio codec. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023.
Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi-codec: residulesidual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765. 2023a.
Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704, 2023b.
Haici Yang, Jiaqi Su, Minje Kim, and Zeyu Jin. Genhancer: High-fidelity speech enhancement via generative modeling on discrete codec tokens. 2024a.
Yiyuan Yang. Niki Trigoni, and Andrew Markham. Pre-training feature guided diffusion model for speech enhancement. arXiv preprint 240.v:2406.07646 2024h.
14

---

Published as a conference paper at ICLR 2025
Jixun Yao, Yuguang Yan, Yu Pan, Ziqian Ning, Jiaohao Ye, Hongbin Zhou, and Lei Xie. Stableve: Style controllable zero-shot voice conversion with conditional flow matching. arXiv preprint arXiv:2412.04724, 2024a.
Jixun Yao, Yuguang Yang, Yi Lei, Ziqian Ning, Yanni Hu, Yu Pan, Jingjing Yin, Hongbin Zhou, Heng Lu, and Lei Xie. Promptve: Flexible stylistic voice conversion in latent space driven by natural language prompts. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10571-10575, 2024b.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 50:495-507, 2021.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02885. 2019.
Xiangyu Zhang, Daijiao Liu, Hexin Liu, Qiquan Zhang, Hanyu Meng, Leibny Paola Garcia, Eng Siong Chng, and Lina Yao. Speaking in wavelet domain: A simple and efficient approach to speed up speech diffusion model. arXiv preprint arXiv:2402.1064 10642, 2024.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308. 16692, 2023.
Chuanxia Zheng and Andrea Vedaldi. Online clustered codebook. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22798-22807 2023.
A
APPENDIX
In the supplemental material:
A.I
We discuss several common questions for the design of GenSE.
A.2
We provide training objectives of SimCodec.
A.3
We provide the model configurations of SimCodec and LM.
A.4
We describe the details of baseline systems employed in our experiments.
A.S
We ablate different codec models in our framework and compare the performance speech enhancement.
A.6
We compare the performance of using different quantization strategies the Sim-
Codec.
A.7
We present visual case of the enhanced sample.
A We discuss the limitations and future works.
A.1 ADDITIONAL DISCUSSIONS THE DESIGN OF GENSE FRAMEWORK
Question 1: Can GenSE employ different SSL models to extract semantic tokens?
GenSE general framework that can be compatible with various SSL models. We conduct addi- tional experiments to evaluate the enhancement performance when we replace the XLSR in GenSE with the widely used WavLM (Chen et al. 2022), as shown in Table [) We can find that GenSE achieves similar results when using these two different SSL models.
On the other hand, we chose XLSR as the semantic extractor due to the key advantage its multilin- gual speech representation capabilities. While other SSL models also offer robustness to noise and high-quality semantic representations, XLSR stands out because it is pre-trained on a large-scale, multilingual dataset, enabling it to learn robust speech features across different languages. This multilingual generalization ability is highly valuable for SE tasks, which frequently involve diverse languages and varying acoustic environments.
15

---

Published as a conference paper at ICLR 2025
Table 5: Comparison results of employing XLSR and WavLM as semantic extractors in GenSE.
DNSMOS 1
SECS t
VQ f
SIG
BAK
OVL
GenSE with XLSR
3.57
3,96
3.31
0.66
0.694
GenSE with WavLM
3,48
4.05
3.25
0.65
0.712
Question 2: Why the conventional PESQ metric not involved in the evaluation?
Several speech enhancement papers (Kumar al. 2020: Liu al. 2021 Maiti & Mandel 2020
&i Yamagishi, 2020) have demonstrated that PESQ may not accurately reflect the true quality of speech, particularly for speech generated by generative models. The key issue lies in the fact that PESQ is an intrusive metric, meaning it evaluates quality by directly comparing the enhanced speech its clean counterpart. However, generative models aim model the distribution of real data, and as a result, the generated speech might sound realistic despite having more considerable distance from the clean version. Therefore, intrusive metrics such as PESQ and STOI may not provide precise evaluation of signal quality for generative models.
Lower PESQ results do necessarily indicate that the generated speech has lower quality, as noted by studies such as (Reddy et al.) 2022) and (Hsu et al. 2022). To address this limitation, we also conducted an ABX test assess the perceptual quality of the enhanced speech compared to clean speech, as shown in Figure 5 This evaluation provides a more subjective measure of quality, capturing how close the generated speech sounds to clean speech.
Question 3: How does the performance of GenSE compare with clean samples from subjective perceived aspect?
We conduct ABX test to compare the performance between GenSE and baseline systems, aiming to evaluate human perception and measure how distinguishable the enhanced speech from clean samples. As shown in Figure [ GenSE is significantly closer to clean speech samples compared to the baseline systems. Participants in the ABX test consistently preferred the speech enhanced by GenSE over the other baseline systems, indicating that GenSE effectively preserves key characteris- ties of the original speech, such as naturalness, intelligibility, and speaker consistency. This suggests that the subjective perceptual quality of enhanced signals from GenSE is much higher and closer to the clean samples, demonstrating its effectiveness in handling noisy signals.
FullSubNet
21.2
Inter-Subnet
61.7
14.6
23.7
CDiffuSE
28.3
SGMSE
54.5
20.2
25.3
StoRM
54.3
18.3
27.4
SELM
42.1
21.1
36.8
DOSE
GenSE
38.4
21.8
39.8
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Clean
No Preference
Enhanced
Figure 5: ABX results between GenSE and baseline systems. "No Preference" means that partici- pants perceive the enhanced signal to be as realistic as the clean signal.
Question 4: Can the N2S module autoregressive manner?
16

---

Published as a conference paper at ICLR 2025
We believe that the generation performance of the N2S module will benefit from autoregressive modeling due to the sequential nature of semantic token generation. The input noisy speech is processed in step-by-step manner to ensure that the semantic representation remains consistent and aligned with the temporal structure of the speech signal, making autoregressive modeling necessary. To further investigate this, conducted experiments comparing the performance of the N2$ module using autoregressive and non-autoregreesivve modeling.
Table 6: Comparison results employing auto-regressive (AR) modeling and non-auto-regrresive (NAR) modeling in N2S module.
DNSMOS
SECS 1
VQ 1
Latency
SIG
BAK
OVL
N2S with AR
3.57
3.96
3.31
0.66
0.694
N2S with NAR
3.35
3.79
3.03
0.63
0.662
-45.7%
As shown in Table non-autoregressive methods achieve significantly faster inference speeds due to their inherent nature of parallel processing. However, all metrics are degraded compared with autoregressive modeling methods. Non-autoreereesive models often struggle with tasks that require capturing complex dependencies maintaining sequence consistency, such transforming noisy tokens into semantically meaningful clean tokens. In this case, the autoregressive nature of the N2S module ensures more precise and stable transformation of noisy semantic tokens into clean ones, which is crucial for the downstream S2S module's 's generation process.
A.2 TRAINING OBJECTIVES OF SIMCODEC
Suppose X is the input speech signal, the total training objectives contain conventional reconstruc- tion loss, commitment loss for the quantizer, and several discriminator losses with the correspond- ing feature matching loss. The reconstruction loss is a linear combination between the LI and L2 losses over the mel-spectrogram using several frequency scales:
K
Cree
Σ (IISTFT&(X) - (T)T&(X)]] + +|STTFT(()) - STFT& k=
(3)
where X is the reconstructed signal and K represents the hop size index. Meanwhile, we can caleu- late the commitment loss based on the following formula:
Leom = || (2))l22
(4)
where and represent the hidden output of the encoder and quantizer, respectively. The adversarial losses for the generator G and the discriminator D are defined as follows:
Lais xxxx) [D(X) 1)² (D(X))P]
(5)
Egen Ex (D(X)-1)) 1)²
Furthermore, we introduce feature matching loss to measure the difference in intermediate features of the discriminator between a ground truth sample and generated sample. This loss serves as an additional objective to train the generator. Specifically, every intermediate feature layer of the discriminator is extracted, and the LI distance between corresponding features from the ground truth sample and the conditionally generated sample is computed in each feature space:
T Σ / D((x) D((X) D -
Pim = E (x,x)
(6)
N.
where T denotes the number of layers in the discriminator; D; and N; denote the features and the number of features in the i-th layer of the discriminator, respectively. The final objectives are the weighted sum:
Lustal Xi Toree Aa + Tegen L-m,
17

---

Published as a conference paper at ICLR 2025
where we set X1 45 and A2 0.1.
We conduct additional experiments to compare our SimCodec with a contemporaneous work, Wav- Tokenizer (Ji et al., 2024b), which also utilizes single quantizer. For fair comparison, produced WayTokenizer using its official code and trained on the same dataset, i.c. LibriSpeech. As shown in Table, our proposed SimCodec (0.65 kbps) outperforms WayTokenizzr (0.5 kbps) large margin with only 10 additional tokens per second and achieves similar performance compared WayTokenizer (0.9 kbps). We believe these results further demonstrate the effectiveness of our SimCodec. On the other hand, our proposed codec supports tokenizing speech with a larger codebook size (8 (922 compared WayTokenizer (4096) while using a single tokenizer.
Table 7: Comparison results between proposed SimCodec and WayTokenizer.
Model
Bandwidth
Nq
token/s
PESQ
STOI
MCD
UTMOS
Wav Tokenizer
0.5kbps
40
1.92
0.857
4.72
2.77
Wav Tokenizer
0.9kbps
75
2.58
0.911
4.14
3.15
SimCodec
0.65 kbps
50
2.45
0.903
3.99
3.04
SimCodec
1.3 kbps
100
3.05
0.954
3,82
3.37
We also visualize the performance between SimCodec and other codec models, as shown in Figure
4.0
600 token's
3.5
400 token's
100 token's
600 token's
3.0
300 token's
PESQ
2.5
90 token's
300 token's
600 token's
token/s
400 token's
DAC
Encodec
2.0
token's
HiFiCodec
300 token's
Vocos
SpeeechTokenizer
1.5
WavTokenizer
100 token's
SimCodec (Ours)
1.0 0
2
3
4
5
6
7
8
9
10
Kbps
Figure 6: Comparison between SimCodec and state-of-the-arr acoustic codec model. The vertical axis PESQ metric measures the reconstructed quality, and the horizontal axis Kbps represents audio compression levels. The size of the circles represents the token numbers per second.
A.3 MODEL CONFIGURATION
We employ LibriSpeech (Panayotov et al., 2015) for training the SimCodec model, which contains 960 hours of 16kHz speech data. The training is conducted using 2 A100 GPUs with a batch size of 128. The SimCodec model is trained for 50k steps in the first stage and 10k steps in the second stage. We employ the AdamW optimizer with a learning rate of le-4 to optimize the codec model. For language model training, steps. We employ the Adam W optimizer with a learning rate le-4 and 5k warmup steps, following the inverse square root learning schedule to adjust the learning rate dynamically during training.
use 8 A100 GPUs with batch size of 256, training for million
18

---

Published as a conference paper at ICLR 2025
A.4 BASELINE SYSTEMS
We compare the performance of our proposed approach with several baseline systems, which we describe in detail below:
FullSubNet (Hao et al. 2021): ): A full- full-band and sub-band fusion model for single-channel real-time speech enhancement, in which the input full-band and sub-band noisy spectral feature, and output full-band and sub-band speech target.
Inter-Subnet (Chen et al. 2023); A single-channel lightweight speech enhancement frame- work employs the subband interaction with the global spectral information such as cross-band dependencies and global spectral patterns.
way to complement the subband model
CDiffuSE (Lu et al.
2022): : A speech enhancement algorithm that incorporates character-
istics of the observed noisy speech signal into the diffusion and reverse processes.
SGMSE (Welker et al. 2022): : A diffusion speech enhancement framework that does not start the reverse process from pure Gaussian noise but from mixture of noisy speech and Gaussian noise. This procedure enables using only 30 diffusion steps to generate high- quality clean speech estimates.
StoRM (Lemercier et al. 2023): A stochastic regeneration approach where an estimate given by predictive model is provided as guide for further diffusion, which uses the predictive model to remove the vocalizing and breathing artifacts while producing very high-quality samples thanks the diffusion model, even in adverse conditions.
SELM (Wang et al. 2024c): A novel speech enhancement paradigm that integrates discrete tokens and leverages language models, comprises three stages: encoding, modeling, and decoding.
DOSE TTai et al., that employs two efficient condition-augmentalie techniques.
2024); A model-agnostic diffusion-based speech enhancement method
A.5 ABLATION FOR DIFFERENT CODEC MODEL IN SE
To evaluate the necessity of SimCodee in our proposed framework, replace SimCodec with the current DAC model. We compare the performance of GenSE when employing different codec mod- els with varying numbers of quantizers. For models with more than quantizer, explore two prediction strategies: 1) autoregressive and non-autoregressive hybrid prediction as used in (Wang al. 2023a), and 2) parallel prediction as employed in (Borsos al., 2023b) for acoustic token generation.
Table 8: Comparison results of GenSE employ different codec models.
DNSMOS 1
Codec Model
N
Type
SECS
VQ f
SIG
BAK
OVL
SimCodec
1
Directly
3.57 0.00%
3,96 10.0%
3.31 10.0%
0.66
0.694
DAC
1
Directly
2.39
2.37-40.1%%
2.18
34.1%
0.35
0.547
DAC
4
AR+NAR
14.0%%
3.46
12.6%
2.77
16.3%
0.61
0.656
DAC
4
Parallel
3.21-10.199
2,98
10.0%
0.63
0.672
X we observe the following: 1) Replacing SimCodec with a single-quantizer DAC codec results in significant decline in both DNSMOS and speaker similarity, highlighting the superiority of SimCodec achieving high-quality speech enhancement only using single quan- tizer. 2) While increasing the number of quantizers improves enhancement performance when using either autoregressive and non-autoregressive hybrid prediction or parallel prediction, the results still do not surpass the performance achieved by employing SimCodec.
As shown in Table
Furthermore, we compare the performance between GenSE with current bandwidth and lower band- width and we investigate trade-offs between performance gains and computational costs by employ- ing a 50Hz SimCodec (0.65kbps) to replace the current 100Hz (1.3kbps) version for acoustic token extraction. This adjustment reduces the number of acoustic tokens required for prediction by half,
19

---

Published as a conference paper at ICLR 2025
and the number of prefix acoustic tokens needed is also halved, thereby improving computational efficiency.
Table 9: Comparison results between GenSE with different bandwidth.
Model
SIG
BAK
OVL
SECS
VQ
RTF
GeenSE(1.3kbps)
3.57
3.96
3.31
0.66
0.694
100%
GenSE(0.65kbps)
3.34
3.56
3.18
0.63
0.648
-24.7%
As shown in Table 9 we observe a performance degradation in GenSE with lower bandwidth, par- ticularly in DNSMOS metrics (but still outperforms most bascline systems). However, we also com- puted the real-time factor (RTF) and found that the lower bandwidth version of GenSE achieves over 20% speedup compared to the current version. This improvement is attributed to the significantly reduced number of tokens required for prediction, which enhances prediction efficiency.
Our findings demonstrate that SimCodec offers significant advantages in reducing the number of tokens required for LM prediction while maintaining competitive level of reconstruction quality compared to higher bit-rate codecs. This reduction in token count simplifies the LM prediction task, contributing more efficient and stable speech enhancement process. Moreover, SimCodec's reorganization process and larger codebook size help to further improve performance in terms of both reconstruction quality and compression efficiency.
A.6 COMPARISON RESULTS FOR USING DIFFERENT QUANTIZATION STRATEGIES IN SIMCODEC.
We investigate the performance of using different quantization strategies and the comparison results are shown in Table. Our proposed reorganization strategy outperforms the clustering vector quan- tization (CVQ) strategy (Zheng & Vedaldi 2023) in PESQ. STOI, and MCD metrics, with only a slight degradation in UTMOS. We also observe that finite scalar quantization (FSQ) (Mentzer al.) 2023a) demonstrates lower reconstruction quality. We attribute this to several factors: the smaller latent dimension of the vector in FSQ, the high variance of gradients during training it approxi- mates hard quantization, and the smooth but less accurate approximation in the early training stages. These challenges particularly pronounced when employing a single quantizer, potentially limit- ing FSQ's s effectiveness in achieving high-quality reconstruction. An effective group FSQ strategy employed in (Liao et al. 2024), but needs several quantizers.
Table 10: Comparison results of using different quantization strategies.
Model
PESQ
STOI
MCD
UTMOS
SinCoole-reorgaanzzztio
3.05
0.954
3.82
3.37
SimCodec-CVQ
2.97
0.945
3.95
3.39
SimCodec-FSS
2.51
0.913
4.53
2.94
A.7 VISUALIZE OF THE ENHANCED SPECTROGRAM FROM SEVERELY DEGRADED SAMPLES
We present a case study on severely noisy signal, as shown in Figure 7 The spectrogram produced by GenSE shows clear harmonic structures and speech formants, demonstrating its ability to effec- tively suppress noise and retain speech details. In contrast, the variant models without hierarchical modeling exhibit more distortion and residual noise, which indicates inferior noise reduction and signal reconstruction compared to GenSE. This visual comparison further highlights the effective- ness of hierarchical modeling under challenging noise conditions.
A.8 LIMITATION AND FUTURE WORKS
Despite GenSE has made great progress, it still suffers from the following issues.
Data coverage. Although we employ several datasets to train our models, the performance of the GenSE framework could benefit from larger datasets and more model parameters. Expanding the
20

---

Published as a conference paper at ICLR 2025
Noisy signal
GenSE w/o Hierarchical Modeling
GenSE w/ Hierarchical Modeling
Figure 7: Visualization of the enhanced spectrograms generated by different models for a severely noisy signal.
training data would provide greater coverage of diverse noise types, speaker characteristics, and acoustic environments, thereby further improving the model's generalization to unseen conditions. Additionally, increasing the model's parameters would enable it to capture more complex patterns and nuances in the speech data, further enhancing its ability to produce high-quality, intelligible speech under wider range of conditions. In the future, will scale GenSE to more generalized and larger-scale benchmarks to enhance the generation ability.
Inference efficiency. Although GenSE outperforms state-of thetherrrt baselines in both enhancement quality and generalization ability, its inference speed remains a bottleneck due to the autoregressive generation process. This slows down the system, limiting its real-time applicability in practical sce- narios. To address this, we believe that the architecture can remain unchanged to support real-time applications by modifying the token prediction pattern during training. Specifically, we propose alternating token prediction in the order of [$1, 01, $2, 02,...,
..., Sn, a.) to replace the current sequen- tial prediction pattern [$1, $2. a1 a2 .... a,,l This approach has been demonstrated effectively in streaming voice conversion (Wang et al. 2024b) and real-time spoken language modeling") sug- gesting its potential to achieve real-time performance within our framework. Furthermore, model quantization and acceleration techniques are necessary for GenSE to be effectively deployed in real-world applications. future work, we plan to explore efficient strategies to improve inference speed, such model pruning, or lightweight architectures, enabling faster and more efficient speech enhancement.
md
21


Reviews:
[{"review_id": "PdY7uWdRdJ", "reviewer": "Reviewer_bQqp", "summary": "- This paper introduces a language model-based generative speech enhancement system, termed GenSE.\n- The system comprises two primary components: a decoder-only model that enhances noisy tokens into clean tokens, and a neural speech codec, SimCodec, which reconstructs waveforms from the enhanced clean tokens.", "questions": "- The lack of novelty mentioned in the weaknesses section diminishes the overall contribution of this paper. Without a substantially innovative approach, I am inclined to recommend rejection.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written and easy to follow.\n- The proposed approach demonstrates the effectiveness of the decoder-only architecture for conventional signal processing tasks, such as speech enhancement (SE).", "weaknesses": "- The proposed approach lacks significant novelty, which is the primary reason for my decision to reject the paper. However, please correct me if I am mistaken, as I am open to revisiting my assessment.\n\n- Concerning speech enhancement (SE) using language models (or the decoder-only architecture), similar approaches have already been introduced in:\n\n[1] Wang, X., Thakker, M., Chen, Z., Kanda, N., Eskimez, S. E., Chen, S., ... & Yoshioka, T. (2024). Speechx: Neural codec language model as a versatile speech transformer. IEEE/ACM Transactions on Audio, Speech, and Language Processing.  \n[2] Yang, D., Tian, J., Tan, X., Huang, R., Liu, S., Chang, X., ... & Meng, H. (2023). Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704.\n\nNeither of these references are cited.\n\n- Similarly, with regard to the neural speech codec, an analogous method was proposed in:\n\n[3] Li, H., Xue, L., Guo, H., Zhu, X., Lv, Y., Xie, L., ... & Li, Z. (2024). Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation. arXiv preprint arXiv:2406.07422.\n\nThis work is also not referenced. Given these omissions, I judge the paper as lacking sufficient originality for acceptance. I believe all referenced works were available prior to the ICLR submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MdKwxV3MQn", "reviewer": "Reviewer_KVEG", "summary": "This paper presents GenSE, a novel generative framework for speech enhancement that leverages language models (LMs) and discrete speech tokens. GenSE employs a single-quantizer neural codec model called SimCodec to extract acoustic tokens from speech, reducing the complexity compared to previous multi-quantizer codecs. It also introduces a hierarchical modeling approach that separates the denoising and generation stages, with a noise-to-semantic (N2S) module transforming noisy speech into clean semantic tokens, and a semantic-to-speech (S2S) module generating clean acoustic tokens.", "questions": "Could you provide a comparison between SimCodec and Vocos (Siuzdak, 2023) and WavTokenizer(Ji, 2024)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed generative framework leverages language models and discrete speech tokens to outperform state-of-the-art speech enhancement systems in terms of speech quality and generalization capability.\n2. The paper introduces a hierarchical modeling approach that separates the denoising and generation stages, improving the stability and performance of the LM-based generation process.\n3. The paper is clearly written and easy to follow.", "weaknesses": "The ablation studies are relatively insufficient. For example, it would be helpful to provide detailed analysis on what information are contained in noisy/clean semantic tokens and noisy/clean acoustic tokens, respectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aPUjLVelsp", "reviewer": "Reviewer_tRzd", "summary": "The paper introduces GenSE, a generative speech enhancement (SE) framework that integrates language models (LM) to leverage semantic information for enhancing speech signals. Unlike traditional SE methods that focus on signal mapping, GenSE treats SE as a conditional language modeling task. By tokenizing speech into semantic and acoustic tokens using a novel codec (SimCodec) and employing a hierarchical approach, GenSE aims to maintain speaker consistency and improve speech quality under noisy conditions. Experiments demonstrate GenSE’s significant improvements over state-of-the-art SE systems in both quality and robustness to noise.", "questions": "1. Could the authors provide more insight into how SimCodec might perform under different network conditions, especially with low latency or limited bandwidth?\n2. How does the system handle speaker identity in cases of domain shifts, such as across different languages, accents, and ages, and would an alternative to XLSR affect GenSE’s generalization capability?\n3. For practical implementation, are there considerations for reducing the computational overhead of the hierarchical modeling method, perhaps through model pruning or compression techniques?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. GenSE offers a unique perspective by reframing SE as a language modeling task, using semantic information to enhance robustness. This represents a notable departure from conventional deterministic mapping in SE.\n2. The hierarchical modeling method, separating semantic and acoustic token generation, improves both quality and intelligibility of enhanced speech, as evidenced by superior metrics across DNSMOS and SECS.\n3. The authors present a detailed breakdown of the methodology and technical architecture, providing clear diagrams and tables that make complex processes accessible.\n4. By addressing the limitations of traditional SE approaches in handling complex noise environments, GenSE has the potential to impact real-world applications in noisy and challenging acoustic settings.", "weaknesses": "1. The hierarchical design and multiple components in GenSE, while effective, may pose a challenge in real-time applications. Simplifying or optimizing these processes further could improve usability.\n2. Although SimCodec effectively reduces token count, further exploration into balancing token complexity and quality in low-bandwidth scenarios could enhance GenSE’s adaptability.\n3. The two-stage quantizer reorganization might benefit from more empirical comparisons with other single-quantizer methods such as WavTokenizer, as these details are relatively underexplored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yBZQPueWy9", "reviewer": "Reviewer_fv2J", "summary": "This paper proposes a novel approach to speech enhancement (SE) called GenSE, which integrates semantic information into the enhancement process using language models (LMs). Traditional SE methods often ignore the semantic context, focusing solely on mapping noisy to clean speech, which can lead to performance issues in challenging environments. GenSE redefines SE as a conditional language modeling task by leveraging LMs to predict discrete acoustic tokens based on semantic information. It also separates the denoising and generation stages, improving prediction stability and incorporating a token chain prompting mechanism to maintain timbre consistency. The proposed SimCodec Model achieves remarkable reconstruction quality at a lower bit rate. Experimental results show that GenSE outperforms existing SE systems, demonstrating improved intelligibility and robustness in noisy conditions.", "questions": "My questions are included in the weaknesses part.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed hierarchical modeling method that separates the denoising and generation stages is effective.\n2. The proposed SimCodec reduces the number of tokens in the generation process, which would benefit all speech generation tasks.\n3. The experimental results and demo audios are promising.", "weaknesses": "**The main issues with this paper lie in the design of SimCodec and the lack of some experimental details:**   \n*SimCodec*:\n1. The issue of low codebook usage with a large codebook size has been identified in the field of computer vision for a long time, and there are already many solutions available [1, 2]. Although this work proposes the codebook reorganization strategy to solve this issue, there are no ablation comparisons between this strategy and baselines like CVQ [2] and FSQ [3]. These comparisons are important for validating the effectiveness of the reorganization strategy proposed in this paper.  \n2. The codebook reorganization strategy employs two quantizers at the first stage and concat the two quantizers at the second stage. This process is slightly similar to the GRVQ technique of Hifi-Codec [4] and the multichannel quantization of MoVQ [5]. I think the comparative experimental results of these two techniques should be added to Table 3. And the authors should discuss how their approach differs from or improves upon GRVQ and MoVQ.  \n3. I think Figure 6 looks extremely similar to Figure 1 in WavTokenizer [8], even the colors of the baselines are the same. However, this paper does not compare with WavTokenizer. An explanation why WavTokenizer was not included in the comparison and how their work differs from or builds upon WavTokenizer is required, or 2) Include WavTokenizer as a relevant baseline.  \n\n*Some experimental details*:  \n1. Real-time generation is crucial for speech enhancement models, but the experiments of this paper do not mention the real-time factor (RTF) of the GenSE model. While Table 4 demonstrates that token chain prompting and hierarchical modeling are highly effective, it also does not indicate how much delay these methods introduce.\n2. In Section 3.3.2, the prefix token of GenSE at the S2S stage contains noisy acoustic tokens, clean semantic tokens, and noisy semantic tokens, which significantly increase the sequence length in training and inference. This paper lacks a specific analysis of the trade-offs between performance gains and computational costs of the introduced prefix sequence.  \n3. Mapping from semantic to acoustic using a flow-matching model has proven to be highly effective in many previous studies [6, 7].  The authors could explain why they chose their current approach instead of a flow-matching model for the S2S module, discussing potential advantages and disadvantages. Alternatively, they might consider implementing a flow-matching model as an additional baseline in their experiments to compare its performance with their current method.  \n\n**Minor questions that would not influence the scores:**\n1. Do you use greedy decoding for decoder LM? Will beam search improve the performance of the model?  \n\n**Minor clarity issues**:\n1. In Section 3.2.3, Line 264, ``we reinitialize the encoder and decoder parameters to fit the new codebook dimension, while copying the parameters from the first stage``, the use of \"reinitialize\" in the first half of the sentence introduces clarity issues;\n2. In Section 3.3.1, Line 293, ``Meanwhile, the self-supervised model is also noise-robust to some extent.`` Some citations can be added here to demonstrate that this phenomenon actually exists.\n\n**Minor typos**:  \n1. In Section 1, Line 052, the quotes of ``textless NLP``;\n2. In Figure 6, `Our` -> `Ours`.\n\n**Conclusion**:  \nThe SimCodec and hierarchical modeling method proposed in this paper are not particularly novel, as there have been related studies in fields such as Computer Vision and Speech Generation. However, the experimental results are still quite impressive. If the authors could address my concerns, I would increase the score.\n\n[1] Yu, Jiahui, et al. \"Vector-quantized image modeling with improved vqgan.\" arXiv preprint arXiv:2110.04627 (2021).  \n[2] Zheng, Chuanxia, and Andrea Vedaldi. \"Online clustered codebook.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.  \n[3] Mentzer, Fabian, et al. \"Finite scalar quantization: Vq-vae made simple.\" arXiv preprint arXiv:2309.15505 (2023).  \n[4] Yang, Dongchao, et al. \"Hifi-codec: Group-residual vector quantization for high fidelity audio codec.\" arXiv preprint arXiv:2305.02765 (2023).  \n[5] Zheng, Chuanxia, et al. \"Movq: Modulating quantized vectors for high-fidelity image generation.\" Advances in Neural Information Processing Systems 35 (2022): 23412-23425.  \n[6] Du, Zhihao, et al. \"Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens.\" arXiv preprint arXiv:2407.05407 (2024).  \n[7] Anastassiou, Philip, et al. \"Seed-TTS: A Family of High-Quality Versatile Speech Generation Models.\" arXiv preprint arXiv:2406.02430 (2024).  \n[8] Ji, Shengpeng, et al. \"Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling.\" arXiv preprint arXiv:2408.16532 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)