DC-Spin: A Speaker-invariant Speech Tokenizer For Spoken Language Models

Authors: Heng-Jui Chang, Hongyu Gong, Changhan Wang, James R. Glass, Yu-An Chung
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: speech tokenizer, self-supervised learning, spoken language model, speech language model, speech resynthesis, audio codec

Abstract:
Spoken language models (SLMs) have gained increasing attention with advancements in text-based, decoder-only language models. SLMs process text and speech, enabling simultaneous speech understanding and generation. This paper presents Double-Codebook Speaker-invariant Clustering (DC-Spin), which aims to improve speech tokenization by bridging audio signals and SLM tokens. DC-Spin extracts speaker-invariant tokens rich in phonetic information and resilient to input variations, enhancing zero-shot SLM tasks and speech resynthesis. We propose a chunk-wise approach to enable streamable DC-Spin without retraining and degradation. Comparisons of tokenization methods (self-supervised and neural audio codecs), model scalability, and downstream task proxies show that tokens easily modeled by an n-gram LM or aligned with phonemes offer strong performance, providing insights for designing speech tokenizers for SLMs.

Decision: reject
Reviewer scores: 5, 8, 1, 5

Reviews and discussion:
<thread>
creation date:
1730024164507

thread_type:
official_review

<thread_object>
creator: reviewer_wrzi

<content>
summary:
This paper proposed a new method for speech tokenization named SpinHuBERT, this tokenization is based on pre-training a HuBert speech encoder with the spin objective, then with an ASR target, and finally fine-tuning with a DC-Spin objective. The evaluation shows the high performance of the method across a wide range of tasks, such as spoken language modeling, speech resynthesis, robustness under distortions and inference efficiency. Furthermore, the authors evaluate different proxy tasks for spoken language modeling which identifies decent candidates.

strengths:
1. The proposed method is novel, and since speech tokanization is widely used improvement are significant. 
2. The paper is clearly written which enables to easily re-implement and use the proposed method. 
3. The proposed method shows a strong performance over the other baseline methods.

weaknesses:
1. Proposed method out-performs the baseline on most tasks, but the margins are small while the proposed method is significantly more complex.
2. Evaluation on directly synthesizing speech is not performed. It is unclear if in a real setting the proposed method is able to generate coherent and intelligible speech. This evaluation should be added to show the methods performance in the task SpeechLMs are used for.
3. Throughout the paper, UTMOS is used instead of real human raters. While UTMOS is a good proxy, it does not fully replace human raters. At least some of the evaluations should be done using MOS with human raters. 
4. Table 3 shows evaluation of other method with much larger models. While a 150m parameter model is decent, SpeechLMs are regularly scaled to larger sizes. The paper is lacking an evaluation of a full-sized SpeechLM model. Such a model would give the reader a better option to appreciate the performance of the proposed method.
5. While the tokens are claimed to be speaker invariant, this is not evaluated in the experimental setup. I would expect the authors to include an experiment which verifies this claim. 
6.

questions:
1. In table 8 and 4 ASR-WER is very high. What ASR model was used? What is the reason for this?
2.  Why is using the proxy tasks better than directly using TSC/ sWUGGY / sBLIMP. They don't seem easier to compute and TSC/ sWUGGY / sBLIMP are already proxies for the ability to generate intelligible audio. 
3. Why use HuBert as the base encoder and not some other encoder? what are the properties of HuBert that make it best for this task?
4. Why fine-tune and not train from scratch with the SPIN objective?
5. In figure 1, we see that different tasks have the best performance with different code-book sizes. For example the 1000 size codebook has the best performance on TSC (but is not used later in the paper). How should the final model be picked?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730274278700

thread_type:
official_review

<thread_object>
creator: reviewer_seng

<content>
summary:
This paper introduces Double-Codebook Speaker-invariant Clustering (DC-Spin) as a method to enhance speech tokenization by making the speech tokens to be speaker-invariant. 
By decreasing the proportion of the speaker information in the speech tokens, DC-Spin aims to improve zero-shot SLM tasks and speech resynthesis. Through comparisons of tokenization techniques and model scalability, the study highlights the effectiveness of tokens easily modeled by n-gram language models or aligned with phonemes, offering valuable guidelines for designing speech tokenizers for SLMs.

strengths:
This paper proposes a method called DC-Spin to improve speech tokenization.
The speech tokens generated by the proposed method is speaker-invariant and can preserve more content information compared to open-source tokenizers.

weaknesses:
The motivation of this paper does not sound convincing.
A common way of connecting a speech encoder to a LM is directly through speech representation, which is of continuous values. 
The discretization from representation to discrete tokens will only degrade the quality of the speech input. Check out the papers below:
https://arxiv.org/pdf/2402.08846
https://arxiv.org/abs/2309.00169
https://www.isca-archive.org/interspeech_2024/yang24f_interspeech.pdf
The discretized tokens will eventually be converted to continuous vectors through the embedding table of the LM. 
Therefore, there is no need to discretized the continuous embedding for the input of LM.

From the LM output perspective, yes, there is a need for speech tokenization if we want the LM to generate speech.
However, the TTS community has already gone far beyond word accuracy. We are talking about voice clone, high naturalness.
Thus, removing the speaker information may even hinder these speech tokens to be used.
Even if we neglect the voice clone issue, the ASR-WER of the speech resynthesis is far below standard (18.x in the paper). That means the system can't be used. Checkout the numbers in the following papers:
https://arxiv.org/abs/2309.00169
https://arxiv.org/abs/2406.02430
https://arxiv.org/abs/2301.02111

questions:
The Spin mentioned in the paper perturb the speech utterance by randomly scaling the F0 and formant frequencies.
As I know, speaker variation is more than different F0 and formants. How do you handle speaking style, which is also a kind of speaker-variant?

In the most right hand side of Figure 2, it is not clear to me how the codebook is formed and how the discrete speech tokens are generated? Is K-means clustering still be used?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730641790599

thread_type:
official_review

<thread_object>
creator: reviewer_eraa

<content>
summary:
This paper presents the development of robust and efficient speech tokenizers for spoken language modeling and speech resynthesis. The authors introduce two innovative methods, SpinHuBERT and DC-Spin, designed to meet four essential criteria for ideal tokenizers: capturing phonetic information, preserving acoustic details, robustness to perturbations, and enabling efficient inference. Experimental results demonstrate that a 150M parameter Transformer decoder, trained on downstream tasks, performs optimally with DC-Spin speech tokens. Furthermore, DC-Spin delivers superior reconstruction quality at extremely low bitrates compared to popular codecs. The authors also identify proxy metrics — shown to correlate highly with downstream task performance — and advocate for these metrics as a basis for evaluating future speech tokenizers.

strengths:
* **Originality:** The paper’s contribution of SpinHuBERT and DC-Spin tokenizers demonstrates originality.
* **Quality:** The evaluation is comprehensive, evaluating the proposed tokenizers across multiple downstream tasks and metrics to show their effectiveness.
* **Clarity:** The presentation is clear.
* **Significance:**  Disentangling factors within discrete speech tokens, as explored in DC-Spin, addresses a growing area of interest (e.g., FACodec). DC-Spin seems to stand out as one of the first speech tokenizers to separate speaker information from codebooks.

weaknesses:
**Comparison in Speech Resynthesis:** While the results show that DC-Spin performs well at low bitrates (<1.5 kbps), it is worth noting that current state-of-the-art speech synthesis systems (e.g., NaturalSpeech2, VoiceCraft, MaskGCT) use codecs with higher bitrates to achieve high-quality speech reconstruction. A comparison of DC-Spin against existing tokenizers at these higher bitrates would provide valuable insight into its performance for speech resynthesis under more typical conditions.

questions:
1. **Claim:** “The first codebook (primary) extracts discrete units for downstream applications. The second codebook (auxiliary) is a large codebook that enhances the encoder’s capability to capture fine-grained phonetic units.” Might there be an ablation study on the independent use of each codebook to support this finding? (i) Evaluating downstream task performance using only the primary codebook vs. only the auxiliary codebook, (ii) analyzing the phonetic content captured by each codebook independently, and (iii) examining how varying the size of each codebook impacts the overall system performance are useful experiments to help validate the claim on the role of codebooks.
2. **Robustness Testing:** Could the authors provide further details on the types and degrees of perturbations applied in testing the robustness of DC-Spin? Specifically, can you provide quantitative details on the noise levels, time stretch factors, reverberation parameters, and pitch shift ranges, and comment on how well they compare to real-world conditions?
3. Can DC-Spin can be adapted to operate at higher bitrates comparable to systems like NaturalSpeech2 or VoiceCraft? If so, how does its performance compares in terms of speech quality metrics (e.g. MUSHRA scores) at those higher bitrates?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687808555

thread_type:
official_review

<thread_object>
creator: reviewer_q3wd

<content>
summary:
The paper presents a novel approach to speech tokenization for spoken language models (SLMs), focusing on speaker invariance and phonetic richness through the Double-Codebook Speaker-invariant Clustering (DC-Spin) technique. DC-Spin utilizes a dual codebook design to separate phonetic and speaker-invariant features, enhancing SLM performance in tasks like automatic speech recognition (ASR) and speech synthesis. Additionally, the paper proposes SpinHuBERT, a pre-trained model that leverages DC-Spin for robust and accurate tokenization. The contributions of DC-Spin are evaluated against existing tokenizers, demonstrating improved robustness, phonetic alignment, and real-time processing efficiency, setting a new standard in resource-constrained and scalable speech tokenization.

strengths:
1. Introduced the Double-Codebook Spin to capture the fine-grained phonetic units better, and gave the detail process about how to select the codebook size.
2. The experimental setup is thorough, covering both zero-shot and supervised tasks, with a clear evaluation on robustness and inference efficiency. 
3. This work analyzed multiple proxy tasks on speech tokenizers to reveal their relationship with the performance of the spoken language model.

weaknesses:
1. Although this work states three contributions, they were incremental somehow. For example, the DC-Spin doesn't show convincing advantages against Spin method in Table 2, especially considering the much larger codebook size of DC-Spin. As for the chunk-wise streaming simulation, it has been used in many similar works such as ASR and TTS. And the authors claimed it in the contribution part but they just mentioned it in a short paragraph in the experiment section.
2. The compared neural audio codec methods were a little bit dated. Encodec and SpeechTokenizer methods were proposed in 2022 and 2023. Many new and powerful works have been proposed since then. For example, descript audio codec (DAC) and hifi-codec should be included in the comparison. Besides, WavTokenizer is another work focusing on low bitrate, making it suitable for comparison.
3. The motivation of designing the DC-Spin is not clear. The authors claimed the auxiliary codebook can capture fine-grained phonetic units but didn't provide explanations. Moreover, it both primary and auxiliary codebooks pay attention to the phonetic units, the acoustic detail would be ignored in DC-Spin to some extent, which is harmful for the generative task.

questions:
1. I am a little bit confused about the bitrate of Kmeans and DC-Spin reported in Table 4 and Table 19. For example, DC-Spin_{500,4096} has the primary codebook with 500 size and the auxiliary codebook with 4096 size. But the final bitrate is the same as the Kmeans_{500} which only has a 500 size codebook. Could you explain it in detail?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose a method for speech tokenization that is speaker invariant. The paper received four reviews, with three generally negative and one positive. While the paper did receive one vote for acceptance, that vote had low confidence, while the three votes for rejection were much more confident in their evaluation. The reviewers were not too motivated to engage with the paper generally, but one reviewer did note that concerns remained after the rebuttal. In general, it was felt that the evaluation of the propose method should be improved before acceptance to a top-tier conference.