GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling

Authors: Jixun Yao, Hexin Liu, Chen Chen, Yuchen Hu, EngSiong Chng, Lei Xie
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: speech enhancement, language model, semantic information

Abstract:
Semantic information refers to the meaning conveyed through words, phrases, and contextual relationships within a given linguistic structure. Humans can leverage semantic information, such as familiar linguistic patterns and contextual cues, to reconstruct incomplete or masked speech signals in noisy environments. However, existing speech enhancement (SE) approaches often overlook the rich semantic information embedded in speech, which is crucial for improving intelligibility, speaker consistency, and overall quality of enhanced speech signals. To enrich the SE model with semantic information, we employ language models as an efficient semantic learner and propose a comprehensive framework tailored for language model-based speech enhancement, called GenSE. Specifically, we approach SE as a conditional language modeling task rather than a continuous signal regression problem defined in existing works. This is achieved by tokenizing speech signals into semantic tokens using a pre-trained self-supervised model and into acoustic tokens using a custom-designed single-quantizer neural codec model. To improve the stability of language model predictions, we propose a hierarchical modeling method that decouples the generation of clean semantic tokens and clean acoustic tokens into two distinct stages. Moreover, we introduce a token chain prompting mechanism during the acoustic token generation stage to ensure timbre consistency throughout the speech enhancement process. Experimental results on benchmark datasets demonstrate that our proposed approach outperforms state-of-the-art SE systems in terms of speech quality and generalization capability. Codes and demos are publicly available at https://anonymous.4open.science/w/gen-se-7F52/.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730535949697

thread_type:
official_review

<thread_object>
creator: reviewer_fv2j

<content>
summary:
This paper proposes a novel approach to speech enhancement (SE) called GenSE, which integrates semantic information into the enhancement process using language models (LMs). Traditional SE methods often ignore the semantic context, focusing solely on mapping noisy to clean speech, which can lead to performance issues in challenging environments. GenSE redefines SE as a conditional language modeling task by leveraging LMs to predict discrete acoustic tokens based on semantic information. It also separates the denoising and generation stages, improving prediction stability and incorporating a token chain prompting mechanism to maintain timbre consistency. The proposed SimCodec Model achieves remarkable reconstruction quality at a lower bit rate. Experimental results show that GenSE outperforms existing SE systems, demonstrating improved intelligibility and robustness in noisy conditions.

strengths:
1. The proposed hierarchical modeling method that separates the denoising and generation stages is effective.
2. The proposed SimCodec reduces the number of tokens in the generation process, which would benefit all speech generation tasks.
3. The experimental results and demo audios are promising.

weaknesses:
**The main issues with this paper lie in the design of SimCodec and the lack of some experimental details:**   
*SimCodec*:
1. The issue of low codebook usage with a large codebook size has been identified in the field of computer vision for a long time, and there are already many solutions available [1, 2]. Although this work proposes the codebook reorganization strategy to solve this issue, there are no ablation comparisons between this strategy and baselines like CVQ [2] and FSQ [3]. These comparisons are important for validating the effectiveness of the reorganization strategy proposed in this paper.  
2. The codebook reorganization strategy employs two quantizers at the first stage and concat the two quantizers at the second stage. This process is slightly similar to the GRVQ technique of Hifi-Codec [4] and the multichannel quantization of MoVQ [5]. I think the comparative experimental results of these two techniques should be added to Table 3. And the authors should discuss how their approach differs from or improves upon GRVQ and MoVQ.  
3. I think Figure 6 looks extremely similar to Figure 1 in WavTokenizer [8], even the colors of the baselines are the same. However, this paper does not compare with WavTokenizer. An explanation why WavTokenizer was not included in the comparison and how their work differs from or builds upon WavTokenizer is required, or 2) Include WavTokenizer as a relevant baseline.  

*Some experimental details*:  
1. Real-time generation is crucial for speech enhancement models, but the experiments of this paper do not mention the real-time factor (RTF) of the GenSE model. While Table 4 demonstrates that token chain prompting and hierarchical modeling are highly effective, it also does not indicate how much delay these methods introduce.
2. In Section 3.3.2, the prefix token of GenSE at the S2S stage contains noisy acoustic tokens, clean semantic tokens, and noisy semantic tokens, which significantly increase the sequence length in training and inference. This paper lacks a specific analysis of the trade-offs between performance gains and computational costs of the introduced prefix sequence.  
3. Mapping from semantic to acoustic using a flow-matching model has proven to be highly effective in many previous studies [6, 7].  The authors could explain why they chose their current approach instead of a flow-matching model for the S2S module, discussing potential advantages and disadvantages. Alternatively, they might consider implementing a flow-matching model as an additional baseline in their experiments to compare its performance with their current method.  

**Minor questions that would not influence the scores:**
1. Do you use greedy decoding for decoder LM? Will beam search improve the performance of the model?  

**Minor clarity issues**:
1. In Section 3.2.3, Line 264, ``we reinitialize the encoder and decoder parameters to fit the new codebook dimension, while copying the parameters from the first stage``, the use of "reinitialize" in the first half of the sentence introduces clarity issues;
2. In Section 3.3.1, Line 293, ``Meanwhile, the self-supervised model is also noise-robust to some extent.`` Some citations can be added here to demonstrate that this phenomenon actually exists.

**Minor typos**:  
1. In Section 1, Line 052, the quotes of ``textless NLP``;
2. In Figure 6, `Our` -> `Ours`.

**Conclusion**:  
The SimCodec and hierarchical modeling method proposed in this paper are not particularly novel, as there have been related studies in fields such as Computer Vision and Speech Generation. However, the experimental results are still quite impressive. If the authors could address my concerns, I would increase the score.

[1] Yu, Jiahui, et al. "Vector-quantized image modeling with improved vqgan." arXiv preprint arXiv:2110.04627 (2021).  
[2] Zheng, Chuanxia, and Andrea Vedaldi. "Online clustered codebook." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.  
[3] Mentzer, Fabian, et al. "Finite scalar quantization: Vq-vae made simple." arXiv preprint arXiv:2309.15505 (2023).  
[4] Yang, Dongchao, et al. "Hifi-codec: Group-residual vector quantization for high fidelity audio codec." arXiv preprint arXiv:2305.02765 (2023).  
[5] Zheng, Chuanxia, et al. "Movq: Modulating quantized vectors for high-fidelity image generation." Advances in Neural Information Processing Systems 35 (2022): 23412-23425.  
[6] Du, Zhihao, et al. "Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens." arXiv preprint arXiv:2407.05407 (2024).  
[7] Anastassiou, Philip, et al. "Seed-TTS: A Family of High-Quality Versatile Speech Generation Models." arXiv preprint arXiv:2406.02430 (2024).  
[8] Ji, Shengpeng, et al. "Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling." arXiv preprint arXiv:2408.16532 (2024).

questions:
My questions are included in the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691130846

thread_type:
official_review

<thread_object>
creator: reviewer_trzd

<content>
summary:
The paper introduces GenSE, a generative speech enhancement (SE) framework that integrates language models (LM) to leverage semantic information for enhancing speech signals. Unlike traditional SE methods that focus on signal mapping, GenSE treats SE as a conditional language modeling task. By tokenizing speech into semantic and acoustic tokens using a novel codec (SimCodec) and employing a hierarchical approach, GenSE aims to maintain speaker consistency and improve speech quality under noisy conditions. Experiments demonstrate GenSE’s significant improvements over state-of-the-art SE systems in both quality and robustness to noise.

strengths:
1. GenSE offers a unique perspective by reframing SE as a language modeling task, using semantic information to enhance robustness. This represents a notable departure from conventional deterministic mapping in SE.
2. The hierarchical modeling method, separating semantic and acoustic token generation, improves both quality and intelligibility of enhanced speech, as evidenced by superior metrics across DNSMOS and SECS.
3. The authors present a detailed breakdown of the methodology and technical architecture, providing clear diagrams and tables that make complex processes accessible.
4. By addressing the limitations of traditional SE approaches in handling complex noise environments, GenSE has the potential to impact real-world applications in noisy and challenging acoustic settings.

weaknesses:
1. The hierarchical design and multiple components in GenSE, while effective, may pose a challenge in real-time applications. Simplifying or optimizing these processes further could improve usability.
2. Although SimCodec effectively reduces token count, further exploration into balancing token complexity and quality in low-bandwidth scenarios could enhance GenSE’s adaptability.
3. The two-stage quantizer reorganization might benefit from more empirical comparisons with other single-quantizer methods such as WavTokenizer, as these details are relatively underexplored.

questions:
1. Could the authors provide more insight into how SimCodec might perform under different network conditions, especially with low latency or limited bandwidth?
2. How does the system handle speaker identity in cases of domain shifts, such as across different languages, accents, and ages, and would an alternative to XLSR affect GenSE’s generalization capability?
3. For practical implementation, are there considerations for reducing the computational overhead of the hierarchical modeling method, perhaps through model pruning or compression techniques?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709353924

thread_type:
official_review

<thread_object>
creator: reviewer_kveg

<content>
summary:
This paper presents GenSE, a novel generative framework for speech enhancement that leverages language models (LMs) and discrete speech tokens. GenSE employs a single-quantizer neural codec model called SimCodec to extract acoustic tokens from speech, reducing the complexity compared to previous multi-quantizer codecs. It also introduces a hierarchical modeling approach that separates the denoising and generation stages, with a noise-to-semantic (N2S) module transforming noisy speech into clean semantic tokens, and a semantic-to-speech (S2S) module generating clean acoustic tokens.

strengths:
1. The proposed generative framework leverages language models and discrete speech tokens to outperform state-of-the-art speech enhancement systems in terms of speech quality and generalization capability.
2. The paper introduces a hierarchical modeling approach that separates the denoising and generation stages, improving the stability and performance of the LM-based generation process.
3. The paper is clearly written and easy to follow.

weaknesses:
The ablation studies are relatively insufficient. For example, it would be helpful to provide detailed analysis on what information are contained in noisy/clean semantic tokens and noisy/clean acoustic tokens, respectively.

questions:
Could you provide a comparison between SimCodec and Vocos (Siuzdak, 2023) and WavTokenizer(Ji, 2024)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730801121934

thread_type:
official_review

<thread_object>
creator: reviewer_bqqp

<content>
summary:
- This paper introduces a language model-based generative speech enhancement system, termed GenSE.
- The system comprises two primary components: a decoder-only model that enhances noisy tokens into clean tokens, and a neural speech codec, SimCodec, which reconstructs waveforms from the enhanced clean tokens.

strengths:
- The paper is clearly written and easy to follow.
- The proposed approach demonstrates the effectiveness of the decoder-only architecture for conventional signal processing tasks, such as speech enhancement (SE).

weaknesses:
- The proposed approach lacks significant novelty, which is the primary reason for my decision to reject the paper. However, please correct me if I am mistaken, as I am open to revisiting my assessment.

- Concerning speech enhancement (SE) using language models (or the decoder-only architecture), similar approaches have already been introduced in:

[1] Wang, X., Thakker, M., Chen, Z., Kanda, N., Eskimez, S. E., Chen, S., ... & Yoshioka, T. (2024). Speechx: Neural codec language model as a versatile speech transformer. IEEE/ACM Transactions on Audio, Speech, and Language Processing.  
[2] Yang, D., Tian, J., Tan, X., Huang, R., Liu, S., Chang, X., ... & Meng, H. (2023). Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704.

Neither of these references are cited.

- Similarly, with regard to the neural speech codec, an analogous method was proposed in:

[3] Li, H., Xue, L., Guo, H., Zhu, X., Lv, Y., Xie, L., ... & Li, Z. (2024). Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation. arXiv preprint arXiv:2406.07422.

This work is also not referenced. Given these omissions, I judge the paper as lacking sufficient originality for acceptance. I believe all referenced works were available prior to the ICLR submission.

questions:
- The lack of novelty mentioned in the weaknesses section diminishes the overall contribution of this paper. Without a substantially innovative approach, I am inclined to recommend rejection.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732582532980

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the rebuttal revision

comment:
We thank all the reviewers for their efforts and constructive suggestions for our paper. We summarize the main revision of the manuscript according to the comments and suggestions of reviewers:
- In Section 1, we include definitions and differences between semantic tokens and acoustic tokens.
- In Section 2.1, we add an introduction of generative audio language models, highlighting their objectives and methodologies, and emphasize the differences between our framework and these models.
- In Section 2.3, we add the discussion of the single quantizer codec model.
- In Section 4.4, we add comparison results of Vocos as additional baseline in Table 3.
- In Appendix 2, we present a detailed performance comparison between our proposed SimCodec and WavTokenizer. While WavTokenizer is a contemporaneous work (a submission to ICLR 2025), we added these results to improve the soundness of our proposed SimCodec following reviewers' valuable suggestions.
- In Appendix 5, we investigate the trade-offs between performance gains and computational costs of GenSE under different bandwidths.
- In Appendix 6, we compare the performance of using different quantization strategies in the SimCodec.
- In Appendix 8, we provide a detailed discussion of potential strategies to enhance the practicability of our framework for real-world applications.

These revisions have addressed the reviewers' concerns while strengthening the paper's contributions and evaluation rigor. We believe these changes have markedly improved the manuscript's quality and clarity. We greatly appreciate the reviewers' great efforts and valuable comments, which have significantly improved the soundness of our manuscript.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper is clearly written and easy to follow, with detailed breakdowns and clear diagrams. It introduces a hierarchical modeling method that separates denoising and generation stages, improving stability and performance, and reframes speech enhancement (SE) as a language modeling task. The proposed generative framework outperforms state-of-the-art SE systems in terms of speech quality and generalization, with promising experimental results and demo audios.