DC-Spin: A Speaker-invariant Speech Tokenizer For Spoken Language Models

## Abstract

Abstract Spoken language models (SLMs) have gained increasing attention with advancements in text-based, decoder-only language models.
SLMs process text and speech, enabling simultaneous speech understanding and generation.
This paper presents Double-Codebook Speaker-invariant Clustering (DC-Spin), which aims to improve speech tokenization by bridging audio signals and SLM tokens.
DC-Spin extracts speaker-invariant tokens rich in phonetic information and resilient to input variations, enhancing zero-shot SLM tasks and speech resynthesis.
We propose a chunk-wise approach to enable streamable DC-Spin without retraining and degradation.
Comparisons of tokenization methods (self-supervised and neural audio codecs), model scalability, and downstream task proxies show that tokens easily modeled by an n-gram LM or aligned with phonemes offer strong performance, providing insights for designing speech tokenizers for SLMs.

## 1 Introduction

Spoken language models (SLMs) and related applications have gained more interest with the advancements of large language models (LLM) and audio tokenization techniques (Wu et al., ).
These speech LMs resemble causal LMs in natural language processing, but SLMs take speech and, optionally, text as input and generate speech or text.
Hence, these LMs can perform tasks like speech continuation (Lakhotia et al., ), automatic speech recognition (ASR) (Rubenstein et al., ; Maiti et al., ), text-to-speech synthesis (TTS) (Wang et al., ), and the more complicated spoken language understanding (SLU) problems (Gong et al., ; Chu et al., ; Nguyen et al., ).
SLM has two main research directions: 1) LM architecture and training and 2) speech tokenization techniques, the latter of which is the focus of this paper.

Since directly taking raw audio waveform as input to an SLM is infeasible, tokenizing speech into text-like discrete units has become an essential component of recent SLMs.
We define four key qualifications for a good speech tokenizer inspired by prior studies.
First, the tokens should contain strong phonetic or semantic information so that the SLM can use the content of speech to perform ASR and SLU (Lakhotia et al., ).
Second, the tokens should retain acoustic details for being resynthesized into speech for generative tasks like TTS and speech-to-speech translation (Lee et al., ; Zhang et al., ; Wang et al., ).
Third, the tokenizer should be robust to perturbations like additive noise, reverberation, and speaker change because the perturbations are irrelevant to how an SLM understands human speech and language (Gat et al., ; Messica & Adi, ).
Fourth, the tokenizer should be lightweight and fast, supporting real-time interaction between users and SLMs.
Hence, this paper tries to answer the following question: how to build and evaluate a good speech tokenizer for spoken language models that satisfies these key qualifications?

We simplify the setup of this paper by training a unit-based speech LM (uLM) (Lakhotia et al., ) and a Hifi-GAN unit-to-speech synthesizer (Kong et al., ; Polyak et al., ).
This setup is commonly used in SLM studies and applications (Maiti et al., ; Messica & Adi, ; Hassid et al., ), which is an ideal proxy for more advanced SLMs.
uLMs are decoder-only transformer LMs (Vaswani et al., ) and trained with the next-token prediction objective on speech tokens.
uLMs can perform zero-shot tasks by estimating the probability of utterances, including detecting real spoken words and determining correct syntactic structures (Nguyen et al., ), and can be fine-tuned for ASR.
Moreover, we train Hifi-GANs to convert tokens to audio and quantify the intelligibility of the resynthesized speech to simulate speech generation with SLMs.
With uLM and resynthesis, we can examine speech tokenizers on the first two required qualities.
Next, we follow Gat et al. () to quantify the robustness by comparing the extracted tokens between clean and perturbed speech.
Finally, we measure the inference speed of offline and streaming tokenization.

After defining the goals and evaluation pipelines, we propose Double-Codebook Spin (DC-Spin) by extending speaker-invariant clustering (Spin) with an auxiliary codebook to extract better speech units, where Spin is a self-supervised fine-tuning method for capturing phonetic units via online clustering and speaker-invariant swapped prediction (Chang et al., ).
To further boost robustness and token quality, we propose pre-training the Hidden-unit BERT (HuBERT) self-supervised speech encoder with Spin codeword units as a better initialization for DC-Spin (Hsu et al., ), denoted as SpinHuBERT.
The contributions of this paper are listed as follows:

- 1.
The proposed speech tokenizer produces high-quality speaker-invariant speech tokens, achieving state-of-the-art spoken language modeling and speech resynthesis compared to open-source tokenizers on multiple benchmarks with limited resources.
- 2.
We propose a simple chunk-wise method to repurpose offline speech tokenizers into streaming mode with a negligible performance drop.
- 3.
We analyze multiple proxy tasks to understand the relation between speech tokenizer and SLM performance.
We find that phoneme and character-normalized mutual information and the proposed n-gram predictability are good proxies for downstream tasks.

## 2 Related Work

Spoken Language Models (SLM)
SLMs or speech language models usually refer to decoder-only LMs that input or output speech and sometimes text.
The two main approaches to integrating speech into LMs are adaptor and token-based.(^1^11Terms “token” and “unit” are used interchangeably in this paper, indicating discrete speech units.)
Because of the recent advancements in LLMs, researchers connect speech encoders and text-based LMs through adaptors (Chu et al., ; Ma et al., ; Tang et al., ), allowing speech understanding and ASR but requiring a more sophisticated design for speech generation (Dubey et al., ).
In contrast, a more common approach, which is the main focus of this paper, is to tokenize speech to serve as both input and output of SLMs (Lakhotia et al., ; Hassid et al., ; Maiti et al., ).
Under this setup, SLMs treat audio waveforms as text-like tokens, allowing SLMs to process speech and text jointly (Nguyen et al., ) and to generate speech by synthesizing tokens into audio (Polyak et al., ).

In Lakhotia et al. (), SLMs are trained with unlabeled speech tokens to discover the spoken content, which can be evaluated with zero-shot tasks in Nguyen et al. ().
This concept allows an SLM to be fine-tuned with paired speech-text data for ASR, TTS, and SLU (Maiti et al., ).
Advanced techniques like interleaving speech and text tokens (Nguyen et al., ), initializing with text-based LMs (Hassid et al., ), and integrating multiple token types (Borsos et al., ) are developed to improve performance.
Because SLMs simultaneously understand and generate speech, and the speech tokens are the only media between the models and audio signals, speech tokenizer design has become a crucial part of SLM research.

Self-supervised Learning (SSL)
SSL is introduced to leverage large unlabeled audio datasets to pre-train speech encoders, mitigating the need for extensive human labeling (Mohamed et al., ).
SSL models are trained to predict pseudo labels given a partial speech utterance.
Pseudo targets could be Mel spectrograms (Chung et al., ; Liu et al., ), vector-quantized features (Baevski et al., ; Hsu et al., ; Chiu et al., ), or an exponential average of the model itself (Baevski et al., ; Liu et al., ).
Pre-trained SSL models offer good initialization for speech processing tasks (Yang et al., ).
Moreover, evidence has shown that speech SSL models excel at extracting phonetic representations (Pasad et al., ; Chang et al., ; Choi et al., ), so quantizing SSL hidden layer embeddings with K-means clustering is widely adopted to tokenize speech (Lakhotia et al., ; Hassid et al., ; Maiti et al., ).
Gat et al. () and Messica & Adi () further fine-tune SSL encoders for robust speech tokenizers.

Neural Audio Codec
Neural network-based codecs compress audio into compact units and reconstruct high-fidelity signals from the units (Zeghidour et al., ; Défossez et al., ; Wu et al., ).
These models resemble autoencoders and comprise an encoder, a quantization module, and a decoder.
A commonly used technique for the quantization module is residual vector quantization (RVQ) (Zeghidour et al., ).
RVQ has multiple codebooks, each quantizing the residual features computed from the previous codebook, making the first few codebooks preserve more critical information for reconstructing audio waveforms.
Zhang et al. () proposes SpeechTokenizer by enforcing the first codebook to capture phonetic units by distilling knowledge from a pre-trained SSL teacher, but the teacher bounds the performance.
One of the benefits of neural codecs is that the model itself has an audio resynthesis module, i.e., the decoder.
Still, SSL-based tokenizers can resynthesize speech with a separate vocoder.

Besides the open-source tokenizers, closed models like USM (Rubenstein et al., ) are claimed to be powerful for SLMs, but these tokenizers are difficult to reproduce or compare because the details remain unrevealed.
In contrast, this paper aims to offer insights into designing tokenizers and shares all details for future studies.
Additionally, some works categorize speech tokens into semantic and acoustic tokens for understanding and generative tasks, respectively (Zhang et al., ; Borsos et al., ).
However, we will demonstrate that a single type of speech token is sufficient to perform well on both tasks.

## 3 Method

### 3.1 Background

Speaker-invariant Clustering (Spin)
Spin is a self-supervised fine-tuning approach inspired by Caron et al. () and captures speaker-invariant content in speech signals through online clustering and swapped prediction (Chang et al., ).
During training, each utterance is perturbed to sound like a different speaker but with the same content by randomly scaling the F0 and formant frequencies.
Both utterances are fed to a pre-trained SSL encoder, and the frame-level output of each utterance is transformed into a sequence of probability distributions with a learnable codebook.
The distributions are smoothed to enforce full codebook usage and serve as the learning target.
Finally, the model performs swapped prediction by minimizing the cross-entropy loss between the original codeword distribution and the smoothed targets from the perturbed output and vice versa.

Spin efficiently improves SSL encoders in content-related problems like ASR and phoneme recognition (PR).
Robust-Spin (R-Spin) extends Spin for robust speech recognition but requires more complicated training stages and implementation (Chang & Glass, ).
Although Chang & Glass () have shown that discrete units produced by Spin codebooks are closely aligned with phonemes and characters, the applications of these tokens remain undiscovered.

Hidden-unit BERT (HuBERT)
HuBERT is an SSL pre-training method for speech representation learning (Hsu et al., ).
Like BERT in NLP (Devlin et al., ), HuBERT is pre-trained with a mask prediction objective for multiple iterations with pseudo labels derived by K-means clustered continuous audio representations.
First, the labels are K-means cluster IDs of Mel-frequency cepstral coefficients (MFCCs).
Then, the second iteration model predicts K-means clusters from the first model’s hidden embeddings.
Besides serving as pre-training labels, K-means units are useful in SLM and speech-to-speech translation (Lakhotia et al., ; Lee et al., ).
This paper adopts HuBERT as the initialization of the proposed speech tokenizers (Section 3.2) and further improves HuBERT by introducing better learning targets (Section 3.3).

### 3.2 Spin as Speech Tokenizer

This section proposes tokenizing speech with Spin codebook along with methods to improve the quality of Spin discrete units.
Because Spin codebooks capture phonetic information and have a unique speaker-invariant property, the tokens extracted from Spin satisfy the first qualification in Section 1.
These properties are especially useful for speech generation because the vocoder can condition on different speakers, allowing more flexible speech synthesis.
Compared with K-means, Spin’s codebook is optimized with gradient descent, proven highly scalable (LeCun et al., ).
In contrast, K-means clustering requires extracting and storing hidden features, leading to high memory consumption and special implementation when scaling (Zanon Boito et al., ).
Furthermore, K-means tokens contain speaker and unrelated information, leading to suboptimal SLM performance (Yeh & Tang, ).
Motivated by the above reasons, this paper explores the possibilities of tokenizing speech with Spin for SLMs.

Figure: Figure 1: HuBERT + Spin tokenizers on zero-shot SLM (see Section 4.2).
Refer to caption: x1.png

First, we fine-tune HuBERT Base with different Spin codebook sizes and use the codeword IDs as discrete units to perform zero-shot spoken LM tasks, where the experimental setup can be found in Section 4.1.
As shown in Figure 1, ideal codebook sizes are between 200 and 500.
Note that the codebook size should be large enough for speech resynthesis since low bitrate degrades resynthesis quality (Appendix D).
Moreover, Chang et al. () found larger Spin codebooks capture better phonetic representations.
The contradictory properties motivate us to develop methods to obtain a small but high-quality codebook.

Double-Codebook Spin (DC-Spin)
DC-Spin extends Spin to two learnable codebooks optimized with the same objective.
The first codebook (primary) extracts discrete units for downstream applications.
The second codebook (auxiliary) is a large codebook that enhances the encoder’s capability to capture fine-grained phonetic units.
Because both codebooks share the same encoder, the auxiliary codebook is expected to indirectly help the primary codebook encode high-quality units.

Supervised Fine-tuning (SFT)
Inspired by the speech encoders in multimodal LLMs (Rubenstein et al., ; Gemini Team, ; Dubey et al., ), we include supervised fine-tuning to boost the token quality.
Specifically, we consider CTC-based (Graves et al., ) ASR and PR as the supervised tasks because 1) the data for these objectives are relatively easy to collect compared to frame-wise labels and 2) both tasks force the model to neglect redundant information and extract the content in speech.
CTC fine-tuning can be applied before or during DC-Spin fine-tuning, but we found the former leads to better results (Appendix B.3).

Figure: Figure 2: The proposed multi-stage training for the DC-Spin (Section 3.2). Stage (I) pre-trains a speech encoder with pseudo labels from K-means or Spin units, where the latter is the proposed SpinHuBERT (Section 3.3). The optional stage (II) fine-tunes the encoder with CTC-based ASR or phoneme recognition (PR). In stage (III), the encoder is fine-tuned with DC-Spin to obtain the codebook for extracting discrete speech tokens.
Refer to caption: x2.png

### 3.3 HuBERT Pre-training with Better Targets

Table: Table 1: Zero-shot SLM accuracy for DC-Spin with SSL encoders sharing similar architectures. See Section 4.2 for task descriptions.

Spin can be applied to any pre-trained speech encoder, but the fine-tuned performance depends on the encoder’s quality.
In Table 1, HuBERT and data2vec are superior to other methods, even though all models are fine-tuned with the same DC-Spin objective.
HuBERT is slightly inferior to data2vec on two tasks, but data2vec is more unstable because the learning target is an exponential moving average of itself (Baevski et al., ).
In contrast, HuBERT has fixed learning targets, which can be replaced with better pseudo labels.
The above findings have led us to propose SpinHuBERT by training HuBERT models with labels Spin units to better initialize DC-Spin.
Because of the speaker-invariant nature of Spin, Chang et al. () and (Chang & Glass, ) have shown that discrete units derived from Spin codebooks are closer to phonetic units than HuBERT K-means units.
Following this observation, SpinHuBERT is expected to extract better phonetic representations.

Summarizing the proposed DC-Spin and SpinHuBERT, the training pipeline is shown in Figure 2.
In stage (I), we pre-train a SpinHuBERT encoder with pseudo labels generated with Spin.
The optional stage (II) fine-tunes the encoder with CTC-based ASR or PR.
Stage (III) fine-tunes the encoder with the proposed DC-Spin objective to obtain the discrete speech tokens for downstream applications.

## 4 Experiment

### 4.1 Setup

Baseline Tokenizers
We adopt EnCodec 24kHz (Défossez et al., ) and SpeechTokenizer (Zhang et al., ) as the neural audio codec baselines.(^2^22[https://github.com/ZhangXInFD/SpeechTokenizer](https://github.com/ZhangXInFD/SpeechTokenizer))
For SSL-based methods, we consider K-means clustering, augmentation invariant discrete representation (Gat et al., ), and Noise Aware Speech Tokenization (NAST) (Messica & Adi, ), where the second and third methods are designed specifically for SLM by training with perturbation-invariant objectives.(^3^33[https://github.com/ShovalMessica/NAST](https://github.com/ShovalMessica/NAST))
An SSL-based tokenizer using K-means clustering with $K$ units is denoted as “K-means$K$.”

Self-supervised Pre-training
The HuBERT models are trained for 400k steps with 124k hours of unlabeled English speech.
Following the Large and X-Large models in Hsu et al. (), our 3rd-iteration HuBERT (it3) learns to predict 500-unit K-means clusters of the 9th layer of HuBERT Base.
SpinHuBERT learns from a Spin model with a codebook size of 4096.
Unless specified otherwise, SSL models operate at a 50Hz framerate.
Details can be found in Appendix A.2.

Supervised Fine-tuning
Under the SFT setup, we fine-tune pre-trained SSL models with ASR and PR before applying DC-Spin using two labeled datasets: LibriSpeech and English Labeled 3k.
The latter extends LibriSpeech with an additional 2k hours of speech.
The fine-tuned encoders are denoted by appending “ASR$n$k” and “PR$n$k” to the encoder’s name, where $n=$ 1 or 3, indicating the two dataset sizes.
See Appendix A.3 for more information.

Spin & DC-Spin Fine-tuning
We follow Chang et al. () and reimplement Spin in fairseq (Ott et al., ).
We fine-tune SSL models with unlabeled data from LibriSpeech on a single NVIDIA 32GB V100 GPU (see Appendix A.4).
“Spin$K$” denotes Spin with a codebook size of $K$.
DC-Spin with primary and auxiliary codebook sizes of $K_{1}$ and $K_{2}$ is denoted as “DC-Spin$K_{1}$,$K_{2}$.”

Spoken Language Models
We adopt unit-based LM as the SLM for a fair comparison with prior works (Lakhotia et al., ).
Each SLM is a 150M-parameter transformer decoder (Vaswani et al., ) that performs next-token prediction on discrete speech units.
The training data are obtained by extracting units from the 6k hours clean subset of Libri-Light (Kahn et al., ).
After training, SLM estimates the log probability of speech utterances normalized by length for zero-shot SLM tasks.
Furthermore, we fine-tune SLMs with the same training objective but with labeled data from LibriSpeech to perform ASR.
See Appendix A.5 for more details.

Speech Resynthesis
We use the Expresso dataset (Nguyen et al., ) to train and evaluate unit-to-speech Hifi-GAN vocoders (Kong et al., ; Polyak et al., ).
The input includes a sequence of tokenized speech units, a speaker ID, and a style ID.
After training, we resynthesize all utterances in the dev and test sets with the original speaker and style IDs.

In our experiments, the speech tokens are deduplicated for SLM, i.e., merging repeated consecutive tokens, so the SLM outputs are also deduplicated, requiring the vocoder to include a duration prediction module in real-world applications.(^4^44E.g., a token sequence 45 103 103 34 5 5 5 after deduplication would be 45 103 34 5.)
However, to avoid further uncertainties, we simplify the vocoder setup to take speech token sequences with the correct length as input (Chang et al., ).
We keep the SLM and vocoder simple to reduce the effects of downstream model design and amplify the impact of tokenizers since this paper aims to understand how to design speech tokenizers and how they affect SLM performance.
The applications can be extended by introducing more advanced modeling strategies, but we leave this part for future studies.

### 4.2 Zero-shot Spoken Language Modeling

Table: Table 2: Zero-shot SLM evaluation for unsupervised speech tokenizers based on HuBERT Base and the LibriSpeech dataset. All SLMs share the same architecture (150M parameters).

Table: Table 3: Unconstrained resources zero-shot spoken language modeling results. We use the first RVQ codebook of audio codecs to extract speech tokens.

This section discusses the impact of tokenizers on SLM by adopting the following tasks.
tSC
We use the “Topic” Spoken StoryCloze to evaluate an SLM’s ability to capture continuation coherence and fine-grained textual nuances (Hassid et al., ).
Each sample comprises two similar spoken stories with different endings.
The SLM must find the utterance with a consistent ending.
sWUGGY
We adopt the sWUGGY spot-the-word task from ZeroSpeech (Nguyen et al., ).(^5^55[https://github.com/zerospeech/benchmarks](https://github.com/zerospeech/benchmarks))
Each sample has two spoken words with similar pronunciations, with one of the words absent from the English vocabulary.
The “all” subset combines the “in-vocab” subset and out-of-vocabulary words that do not appear in the LibriSpeech training set.
sBLIMP
The sBLIMP acceptability metric is also adopted from ZeroSpeech.
Each sample comprises two similar utterances, but one is ungrammatical.
The above tasks require an SLM to compute a pseudo probability for each audio recording in a sample and compare the probabilities to determine which is more likely to be the correct answer.
The results are reported in accuracy.

Table 2 shows the results of unsupervised speech tokenization techniques based on HuBERT Base and LibriSpeech for a fair comparison.
DC-Spin demonstrates superior performance compared with previous methods.
We observe consistent improvement of DC-Spin over Spin across different unit sizes, but the gap is narrowed when the codebook size is 500.
Among all tasks, DC-Spin improves sWUGGY most significantly because this problem is closely related to how well speech tokens represent pronunciation, which is directly related to phonetic information.
The results strongly indicate the effectiveness of DC-Spin.

To compare the proposed methods with state-of-the-art SLMs, we report results with unconstrained resources in Table 3.
The proposed SpinHuBERT with DC-Spin offers the best performance on sWUGGY and sBLIMP, even using a relatively small SLM and training data size.
For tSC, DC-Spin performs similarly with 1.3B-parameter TWIST (Hassid et al., ), but the gap increases between DC-Spin and larger SLMs, showing that this task might correlate more with LM scaling, especially when comparing to the cascaded topline.
Furthermore, DC-Spin is improved using either ASR or PR SFT with similar performance gains, indicating that either task is suitable for assisting DC-Spin.
As for the baselines, the Whisper Small encoder (87M parameters) with K-means offers low accuracy even though the encoder was trained with 680k hours of speech.
EnCodec tokens result in the worst performance because no explicit constraints are imposed on the encoder or quantizer to extract phonetic or semantic representations.
SpeechTokenizer performs similarly to HuBERT with K-means, corroborating the hypothesis mentioned in Section 2 that the HuBERT teacher bounds this model.
Hence, building speech tokenizers from speech SSL models offers better representations for SLM.
Overall, the results suggest that speech tokenizers greatly impact SLMs, and the proposed SpinHuBERT and DC-Spin achieve state-of-the-art SLMs on several tasks with limited resources.

### 4.3 Speech Resynthesis

This section focuses on speech generation with SLMs by resynthesizing speech from discrete units and evaluating with the following metrics.

ASR-WER
This metric uses an ASR model to transcribe the resynthesized speech and computes the word error rate (WER) to quantify the intelligibility of the audio.(^6^66[https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_960h_pl.pt](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_960h_pl.pt))
UTMOS
Following prior works (Mousavi et al., ; Chang et al., ), we adopt UTMOS, a neural network-based mean opinion score (MOS) prediction, to assess the quality of the resynthesized speech because this metric highly correlates with human-rated MOS (Saeki et al., ).
Although other metrics exist to evaluate vocoders, we focus on whether the speech tokens preserve sufficient information to synthesize intelligible and human-like speech using the same vocoder.

Table: Table 4: Speech resynthesis ASR-WER and UTMOS on Expresso dev and test sets.

As shown in Table 4, HuBERT with DC-Spin reduces more than 10% relative WER compared with K-means, but the K-means and DC-Spin are similar in SpinHuBERT, showing that training HuBERT with Spin units helps representations for resynthesis.
SFT with 1k hours of data has little impact on the resynthesis results, although SFT has removed some acoustic details.
Moreover, SFT with more data (1k vs. 3k hours) lowers ASR-WER, which might be caused by increased robustness.
Compared with codec-based approaches, DC-Spin tokens can be synthesized to produce high-intelligibility and quality speech at a relatively low bitrate because the acoustic details are encoded across several RVQ codebooks in codecs.
We notice that UTMOS among SSL-based methods are similar, possibly indicating that the resynthesis quality is less relevant to the tokens than the vocoder.
To summarize, this section demonstrates the effectiveness of SSL-based tokenizers on speech resynthesis, corroborating with the findings in Shi et al. ().

### 4.4 Robustness

Table: Table 5: Unit edit distance using 500 units with four types of audio distortions. DC-Spin^⋆ is based on SpinHuBERT-PR3k.

This section focuses on the robustness of speech tokenizers via unit edit distance (UED) (Gat et al., ).(^7^77[https://github.com/ShovalMessica/NAST/tree/main/augmentations](https://github.com/ShovalMessica/NAST/tree/main/augmentations))
This metric computes the unit error rate of speech tokens between clean and distorted audio inputs, so lower values imply superior robustness.

In Table 5, Spin and DC-Spin surpass Gat et al. () under most distortions even though this baseline tokenizer is explicitly trained with a denoising objective while our methods only have a speaker-invariant constraint.
One surprising finding is that Spin and DC-Spin are less robust on pitch shift than other distortions, probably because the distortion always shifts the pitch by a major third, making the speakers with higher pitches sound unreal.
In contrast, the speaker perturbation approach in Spin training keeps the speech more natural (Choi et al., ).
Moreover, the overall best-proposed SpinHuBERT-PR3k + DC-Spin tokenizer (the last row) reduces the UED values further.
Overall, the proposed tokenizers demonstrate robustness even in unseen scenarios.

### 4.5 Inference Efficiency

Table: Table 6: Offline speech tokenizer inference efficiency. Only the first RVQ codebook in the audio codec models is included in the parameter calculation.

This section inspects the inference efficiency, the last qualification for good speech tokenizers.
The following metrics are averaged over three runs on LibriSpeech dev-clean and dev-other using a single V100 GPU.
Latency is the average time required to tokenize an utterance.
Real Time Factor (RTF) is the ratio between latency and utterance duration, so a lower value implies faster inference.

Offline Inference
We first compute the offline inference efficiency by tokenizing entire utterances.
As shown in Table 7, audio codecs are significantly slower than SSL models with a similar size because the RNNs in the former cannot be parallelized in contrast to the self-attention in the latter.
Next, NAST models are slow because the architecture is a Conformer encoder stacked on top of a HuBERT model (Gulati et al., ).(^8^88NAST50 and NAST100 have the same model architecture and size.)
HuBERT Base with DC-Spin and K-means have the same inference speed since the encoder and the quantization operation are similar.
In addition, large HuBERT models (300M+ parameters) are slow, which is less ideal for real-time speech tokenization.

Chunk-wise Streaming
To optimize user experience with SLMs, we repurpose speech tokenizers by chunk-wise token extraction to simulate streaming tokenization.
Initially, a tokenizer extracts the first chunk of speech with a duration of $T_{\text{chunk}}$ seconds.
And each time, the chunk expands by $T_{\text{shift}}$ seconds to tokenize the incoming audio.
Hence, the context is constantly expanding to improve tokenization accuracy.
As shown in Table 7, the proposed DC-Spin has less performance degradation than K-means and maintains downstream performance like tSC.
The results demonstrate the feasibility of repurposing to streaming mode without re-training.
Combining the offline and streaming experiments, DC-Spin satisfies the fourth qualification of being a good speech tokenizer.
Appendix G has a more detailed explanation of the chunk-wise approach and additional results.

### 4.6 Effects of SSL Pre-training

Table: Table 8: SSL pre-trained encoders comparison. Unless specified otherwise, discrete units are K-means clustered hidden features with 500 centroids. We report the sWUGGY in-vocab subset, the LibriSpeech test-other for SLM ASR, and the test set for resynthesis.

To understand the effects of SSL pre-training on tokenizing speech, we train HuBERT models with different sizes and objectives, quantize hidden representations with K-means for speech tokenization, and report the results in Table 8.
SLM ASR is the result of pre-trained SLM fine-tuned with ASR transcription (see Appendix A.5).

First, HuBERT second iteration (it2) models perform similarly on several SLM tasks, but HuBERT Large exhibits significantly worse accuracy on sWUGGY, the cause of which remains unknown even though we trained the SLM twice to verify.
The results suggest scaling model size helps SLM-ASR and resynthesis but is not always helpful and also decreases inference efficiency (Table 7).
Second, we pre-train HuBERT it3 models with different framerates and sizes.
Compared to different framerates, 25Hz offers the best overall SLM results, but resynthesis intelligibility is degraded because the lowered framerate increases reconstruction difficulty.
Like HuBERT it2, we found improvement for all metrics when scaling the model size (6 vs. 12 vs. 18 layers).
Third, SpinHuBERT surpasses HuBERT it3 on all tasks, indicating that enhancing pseudo labels for pre-training has a greater impact on performance than scaling the model.
SpinHuBERT even narrows the performance gap between 50 and 25Hz models.
Comparing K-means with DC-Spin (gray fonts), the performance gain from applying DC-Spin is more significant than all other effects.
Thus, results suggest we should focus more on tokenization techniques than scaling SSL encoders.

### 4.7 Finding Proxy Tasks for Spoken Language Modeling

Figure: Figure 3: Pearson correlation coefficients between proxy and downstream tasks.
Refer to caption: x3.png

This section inspects the correlation between tasks to find proper proxies for the actual SLM tasks.
See Appendix A.7 for more details.
Bitrate
We compute the bitrate of deduplicated tokens by considering the distribution of tokens via entropy.
N-gram Predictability
We propose training a 4-gram LM with deduplicated tokens on LibriSpeech and reporting the average perplexity.
This metric measures the difficulty of modeling speech tokens.
Phonetic ABX
ABX error rate quantifies how well a tokenizer can distinguish phonemes (Schatz, ; Nguyen et al., ).
Phone Normalized Mutual Information (PNMI)
Proposed by Hsu et al. (), PNMI computes the mutual information between the speech tokens and phoneme alignments.
Thereby, higher values imply better alignment with the underlying phoneme distribution.
Character Normalized Mutual Information (CNMI)
Similar to PNMI, CNMI compares tokens with character alignments (Chang & Glass, ).
We use UnitY2 to compute alignments (Seamless Communication et al., ).(^9^99[https://github.com/facebookresearch/seamless_communication/blob/main/docs/m4t/unity2_aligner_README.md](https://github.com/facebookresearch/seamless_communication/blob/main/docs/m4t/unity2_aligner_README.md))

Using 33 tokenizers with 50Hz framerate and 500 units, we compute the Pearson correlation coefficients between proxy and downstream metrics in Figure 3.
We make the values negative before calculating the coefficients for lower-better metrics (bitrate, 4-gram, ABX, ASR, and resynthesis).
According to Figure 3, bitrate positively correlates with tSC and sBLIMP, implying short and compact tokens are more suitable for capturing the long context of speech.
Next, low 4-gram perplexity correlates with SLM tasks, so repeating patterns in tokens improves SLM.
The high correlation between PNMI, ABX, and sWUGGY verifies that sWUGGY relies on well-aligned phonetic units (Section 4.2).
Similarly, CNMI quantifies the textual alignment quality, making this task more related to sBLIMP and ASR.
Nevertheless, the ABX error rate negatively correlates with tSC and sBLIMP, implying this metric might fail to serve as a proxy.
Furthermore, speech resynthesis highly correlates with phoneme alignment metrics (ABX and PNMI), suggesting this task relies on the phonetic representations captured by the tokens for synthesizing intelligible speech signals.
Overall, n-gram predictability, PNMI, and CNMI are ideal proxies for developing speech tokenizers.
More results can be found in Appendix H.

## 5 Conclusion

This paper studies building and evaluating effective and robust speech tokenizers for spoken language modeling and speech resynthesis.
We propose SpinHuBERT and DC-Spin, which demonstrate strong capabilities on several tasks compared with open-source speech tokenizers.
Our methods satisfy the four qualifications for an ideal tokenizer: captures phonetic information, preserves acoustic details for resynthesis, is robust to perturbations, and fast inference.
Furthermore, we found n-gram predictability, PNMI, and CNMI metrics highly correlate with downstream performance, making these tasks ideal proxies.
The findings and proxy tasks offer guidelines for future tokenizer and spoken language model development.

Limitations and Future Works
This paper focuses on the effectiveness of speech tokenizers, so the evaluation tasks are on a smaller scale.
Although the proposed tokenizers achieve state-of-the-art zero-shot metrics with small SLMs, it is worth investigating their gains on multimodal LLMs.
Our models are trained and evaluated on English speech, so extending to multilingual and general audio is left for future studies.
TTS and speech-to-speech translation are also potential applications.

## Ethics Statement

The speech tokenizers in this paper are trained with a limited set of audio data from several English corpora, which is inherently biased toward specific accents and dialects and might be less robust to unseen acoustic domains.
Because inaccurate tokens might lead to misinterpretation in spoken language models, the proposed tokenizers must be carefully examined when they are used for speech processing applications.

## Reproducibility Statement

The experiments of this paper utilize publicly available datasets and code for better reproducibility.
First, we use public datasets for model training and evaluation as described in Section 4.1 and Appendix A.
Second, the baseline speech tokenizers and SSL speech encoders are open models that can be accessed easily, as listed in Appendix A.1.
Third, the training code of Spin and DC-Spin is first adopted from the official code in Chang et al. () and reimplemented in the open-source fairseq library (Ott et al., ).
We also demonstrate that our implementation matches the original performance in Appendix A.4.
Fourth, we follow the original implementation for the evaluation tasks to ensure a fair comparison with prior works.
For reference, we provide the source of the code and data we use in footnotes throughout the paper.

## Appendix A Implementation Details

### A.1 Baselines

Table: Table 9: Layers for K-means clustering and the corresponding token quality in PNMI and CNMI with $K=$ 500.

K-means
Following Hsu et al. (), we train the K-means models with 100 hours of speech from LibriSpeech (Panayotov et al., ).
Table 9 lists the K-means clustering setup for the encoders we use in this paper, especially in Table 8.

EnCodec
We use the EnCodec 24kHz model trained on speech and general audio (Défossez et al., ).(^10^1010[https://huggingface.co/facebook/encodec_24khz](https://huggingface.co/facebook/encodec_24khz))
This model consists of an encoder, a residual vector quantizer (RVQ), and a decoder.
We use the codeword IDs extracted from the first codebook for SLM-related tasks.
Note that the speech tokens have a framerate of 75Hz.
Because this model takes audio input at 24kHz, we upsample audio to 24kHz before feeding it into the encoder and downsample the decoder output to 16kHz.

SpeechTokenizer
We adopt the official checkpoint trained on LibriSpeech (Zhang et al., ).(^11^1111[https://github.com/ZhangXInFD/SpeechTokenizer](https://github.com/ZhangXInFD/SpeechTokenizer))
The architecture is similar to EnCodec, but the framerate of speech tokens is 50Hz.

Noise Aware Speech Tokenization (NAST)
We follow the official implementation and checkpoints for NAST (Messica & Adi, ).(^12^1212[https://github.com/ShovalMessica/NAST](https://github.com/ShovalMessica/NAST)).
However, we found the inference function applies Gumbel noise before computing the probability distribution over the codewords (Jang et al., ), leading to random output tokens and degrading the performance.
Hence, we skip the Gumbel Softmax and residual information computation for accurate token prediction and faster inference.

### A.2 Self-supervised Pre-training

LibriSpeech
LibriSpeech is a labeled read English speech corpus commonly used for ASR and SSL pre-training (Panayotov et al., ).
The training set comprises 960 hours of speech.
The four evaluation subsets are used in this paper: dev-clean, dev-other, test-clean, test-other.
The Base speech SSL models from prior works are pre-trained with the 960 hours training set, including wav2vec 2.0 (Baevski et al., ), HuBERT (Hsu et al., ), data2vec (Baevski et al., ), and DinoSR (Liu et al., ).

English Unlabeled 124k
To improve robustness, we pre-train HuBERT models with a larger English corpus, covering more audio domains.
The 124k hours unlabeled speech corpus combines the English subsets Common Voice (Ardila et al., ), Fisher (Cieri et al., ), Multilingual LibriSpeech (Pratap et al., ), Voxlingua (Valk & Alumäe, ), VoxPopuli (Wang et al., ), LibriSpeech, and a subset originating from a
publicly available repository of crawled web data.
Different from prior works, we exclude Libri-Light (Kahn et al., ) because this corpus slightly degrades performance on domains other than LibriSpeech.

HuBERT it3 and SpinHuBERT models are pre-trained on the 124k hours dataset using 32 NVIDIA 80GB A100 GPUs with the same hyperparameters as the HuBERT Base iteration 2 (Hsu et al., ).(^13^1313[https://github.com/facebookresearch/fairseq/tree/main/examples/hubert](https://github.com/facebookresearch/fairseq/tree/main/examples/hubert))
The only difference is that we increase the batch size to 225 seconds of speech per GPU or, equivalently, two hours considering all 32 GPUs.
We list the differences for models operating at different framerates in Table 10.

**Table 10: HuBERT pre-training hyperparameters for models operating at different framerates.**
| Hyperparameters | 50Hz | 25Hz | 12.5Hz |
| --- | --- | --- | --- |
| CNN Extractor Layers | 7 | 8 | 9 |
| CNN Positional Encoding Kernel | 128 | 64 | 32 |
| Time Mask Length (frames) | 10 | 5 | 2 |

Additionally, the targets for SpinHuBERT Base@50Hz are derived from a Spin4096 model based on HuBERT Base and fine-tuned with LibriSpeech 960 hours dataset, which is the same setup as will be described in Appendix A.4.
The 25Hz targets are generated by another Spin4096 model, but this model downsamples the encoder representations by averaging every two consecutive frames before performing online clustering.

### A.3 ASR & PR Fine-tuning

English Labeled 3k
This 3k-hour labeled English speech corpus extends from LibriSpeech by including the transcribed English subsets in Common Voice (Ardila et al., ) and VoxPopuli (Wang et al., ), filtered with the same recipe in Seamless Communication et al. ().
We normalize the transcriptions to match LibriSpeech, i.e., removing all punctuation except for apostrophes and converting numbers to words.
E.g., converting “16th” to “SIXTEENTH.”
After normalization and removing ambiguous transcriptions, the corpus has 3100 hours of speech left.
All ASR experiments are character-based.
For phonemized transcription, we use the official LibriSpeech lexicon to convert English words into phonemes.(^14^1414[http://www.openslr.org/11/](http://www.openslr.org/11/))
We take the first pronunciation for words with multiple pronunciations for a deterministic behavior.
For out-of-vocabulary words, we use a neural network-based G2P model to obtain the phoneme transcriptions (Park & Kim, ).

The ASR and PR fine-tuning hyperparameters are shown in Table 11.
The ASR and PR models are trained on 8 NVIDIA 32GB V100 GPUs using CTC loss (Graves et al., ).
For the first 10k updates, the encoder is frozen, and the linear projector for CTC is fine-tuned.
Note that the hyperparameters are not tuned to optimal, so there still might be room for improvement.

**Table 11: Hyperparameters for ASR and PR fine-tuning.**
| Dataset | Training | Batch Size | Learning | Time Mask |
| --- | --- | --- | --- | --- |
| (hours) | Updates | (minutes) | Rate | Probability |
| 1 | 13k | 60.0 | 5e-5 | 0.075 |
| 10 | 25k | 26.7 | 2e-5 | 0.075 |
| 100 | 80k | 26.7 | 3e-5 | 0.075 |
| 960 | 150k | 26.7 | 3e-5 | 0.065 |
| 3100 | 150k | 26.7 | 3e-5 | 0.065 |

### A.4 Spin & DC-Spin

Following Chang et al. (), we reimplement Spin with fairseq for better scalability (Ott et al., ).(^15^1515[https://github.com/vectominist/spin](https://github.com/vectominist/spin))
All Spin models, including DC-Spin, are trained on a single NVIDIA 32GB V100 GPU for 20k updates.
We perturb each utterance in the LibriSpeech dataset with the same implementation in Chang et al. () before training to avoid on-the-fly data augmentation and reduce costs.
The learning rate linearly ramps up to 5e-5 in the first 4k updates and decreases to zero in the rest.
The batch size is 400 seconds of audio before speaker perturbation, equivalent to 20k frames for 50Hz models.
As shown in Table 12, we found that adding a small portion of masking improves performance slightly, so masking with a probability of 0.01 and a span of 5 frames is added to the input.
Predicting hard targets (one-hot) also offers better alignment with phonemes and characters.
Because we only fine-tune the Base models with 12 transformer encoder layers, all Spin and DC-Spin models freeze the first nine layers and fine-tune the last three layers with the learnable codebook(s).
The codeword embedding size is set to 256.

**Table 12: DC-Spin50,4096 with different training strategies. Soft target uses the probability distribution derived through the Sinkhorn-Knopp algorithm, while hard target converts the distribution to one-hot by taking argmax over all possible codewords.**
| Spin Swapped |  |  |  |
| --- | --- | --- | --- |
| Prediction Target | Masking | PNMI | CNMI |
| Hard | N/A | 0.485 | 0.350 |
| Soft | $p=$ 0.01 and length $=$ 5 | 0.482 | 0.349 |
| Hard | $p=$ 0.01 and length $=$ 5 | 0.490 | 0.355 |

Furthermore, to ensure that the reimplemented Spin in fairseq has a similar performance as in Chang et al. (), we report a comparison of Spin codebook quality between the original and our implementation in Table 13.
Cluster purity measures the purity of each phoneme’s associated token, and phone purity measures the average phoneme purity within one class of tokens (Hsu et al., ).
The small discrepancy in codebook quality metrics indicates our implementation successfully reproduces the results in Chang et al. ().

**Table 13: Codebook quality comparison between the original Spin implementation in Chang et al. () and ours. All models are based on HuBERT Base.**
|  | Cluster | Phone |  |
| --- | --- | --- | --- |
| Method | Purity | Purity | PNMI |
| Spin500 (original) | 0.085 | 0.693 | 0.707 |
| Spin500 (ours) | 0.082 | 0.687 | 0.702 |
| Spin1000 (original) | 0.047 | 0.732 | 0.747 |
| Spin1000 (ours) | 0.049 | 0.721 | 0.741 |
| Spin2000 (original) | 0.027 | 0.757 | 0.774 |
| Spin2000 (ours) | 0.026 | 0.759 | 0.777 |

### A.5 Spoken Language Model

Pre-training
Following prior works (Lakhotia et al., ; Gat et al., ; Messica & Adi, ), we pre-train unit-based SLMs with speech tokens extracted from the 6k hours clean subset of Libri-Light corpus (Kahn et al., ).(^16^1616[https://github.com/facebookresearch/fairseq/tree/main/examples/textless_nlp/gslm/ulm](https://github.com/facebookresearch/fairseq/tree/main/examples/textless_nlp/gslm/ulm))
We select 1% of the training data for validation, which covers all sequence lengths.
The unit-based LM has the transformer_lm_big architecture implemented in fairseq.
The LM is trained on 8 NVIDIA 32GB V100 GPUs with a gradient accumulation of 8 steps, a maximum of 8192 tokens per GPU, and 3072 tokens per utterance.
Utterances with lengths exceeding 3072 tokens are split into shorter sequences.
The learning rate linearly increases to 5e-4 in the first 4k steps and decays as in Vaswani et al. ().
We choose the checkpoint with the lowest validation perplexity for zero-shot evaluation and ASR fine-tuning.

ASR Fine-tuning
Similar to SLM pre-training, ASR fine-tuning has the same training setup except for the data preparation.
We extract tokens and concatenate transcription from the LibriSpeech 960h training corpus.
To construct the ASR data, a special token 
 is inserted after each tokenized utterance, followed by the corresponding character-based transcription.
E.g., an utterance for training would look like 69 10 ... 11 482 
 Z E U S | S E E S | ..., where “|” denotes whitespace.
The LM input embedding is extended by randomly initializing embeddings for English letters and the special 
 token.
The training batch size and computing resources are the same as SLM pre-training.
The learning rate linearly increases to 2e-4 in the first 2k steps and decays as in Vaswani et al. ().
After training, we decode with a beam size of 5 using the checkpoint at 10k updates.

### A.6 Speech Resynthesis

Expresso
Expresso is a high-quality expressive speech dataset covering 26 expressive styles (Nguyen et al., ).
This dataset is split into train, dev, and test sets.
We use the train set to train a Hifi-GAN vocoder.
During evaluation, we resynthesize speech in dev and test sets with the vocoder conditioned on the original speaker and style IDs.
Note that some expressive styles in this dataset differ from normal speech, e.g., whispering, leading to a lower UTMOS.

We follow the default training hyperparameters in Nguyen et al. () to train Hifi-GAN models on a single NVIDIA 32GB V100 GPU for 400k updates.(^17^1717[https://github.com/facebookresearch/speech-resynthesis/tree/main/examples/expresso](https://github.com/facebookresearch/speech-resynthesis/tree/main/examples/expresso))
We add extra upsample layers in the Hifi-GAN for lower framerate models like 25Hz and 12.5Hz.

### A.7 Proxy Task Metrics

Bitrate
We consider the distribution of the units extracted by the tokenizers during bitrate calculation.
Assuming a corpus of $T$ seconds of audio and $N$ tokens in total.
For each token ID $k=1,\dots,K$, the number of occurrences is denoted as $n(k)$.
Hence, the probability of occurrence of each token $k$ is $p(k)=n(k)/N$.
Then, we calculate the bitrate as follows

$$ bitrate $\displaystyle=\frac{N}{T}\mathbb{E}\left[-\log_{2}p(k)\right]$ $\displaystyle=\frac{N}{T}\sum_{k=1}^{K}\frac{n(k)}{N}\log_{2}\frac{N}{n(k)}$ $\displaystyle\leq\frac{N}{T}\log_{2}K.$ $$

The bitrates are calculated over the dev-clean and dev-other subsets of LibriSpeech.

N-gram Predictability
We implement a simple n-gram LM and estimate unseen n-grams by backing off to lower-order n-gram LMs.
First, all audio data in LibriSpeech are tokenized and deduplicated.
Second, for each utterance, we add 
 and 
 tokens at the front and end, respectively.
Then, we train n-gram LMs with orders from 1, $\dots$, $n$.
These LMs then estimate the log probability of the dev and test sets in LibriSpeech to get the perplexities.
Lower perplexities indicate that the tokens are easier to be predicted given a small context, which also implies similar token patterns appear frequently.

## Appendix B DC-Spin: Design and Analysis

### B.1 Effect of the Auxiliary Codebook

Figure: Figure 4: DC-Spin50,$\cdot$ with different auxiliary codebook sizes vs. zero-shot SLM tasks. Dashed lines indicate Spin50.
Refer to caption: x4.png

Here, we discuss the effect of the auxiliary codebook size in DC-Spin.
We plot the relation between the auxiliary codebook sizes and the zero-shot SLM results in Figure 4.
Comparing Spin and DC-Spin (dashed vs. solid lines), the proposed DC-Spin helps downstream tasks in most cases.
Moreover, the performance gain of larger auxiliary codebook sizes is more prominent in sWUGGY than in the other two tasks, corroborating with the findings in Chang et al. () and the discussions in Section 4.2.
Still, we observe that the overall performance drops when the codebook size is over 4096.
The results indicate the necessity of including a large auxiliary codebook for helping the primary Spin codebook for SLM applications.

### B.2 Quantization: K-means vs. Spin Codebook

Here, we discuss the difference between quantizing speech encoder representations with Spin codebook and K-means clustering.
Among K-means results in Table 14, Spin4096 offers the overall best performance because the large codebook used during self-supervised fine-tuning enhances the encoder in capturing phonetic units.
Still, the gap between Spin4096 and DC-Spin is narrowed when K-means has 500 centroids.
When comparing the two quantization techniques, K-means vs. codebook, we found that the codebook quantization method is slightly better because the codebooks are optimized jointly with the encoder.
The results demonstrate that Spin and DC-Spin codebooks are, in general, a better way of quantizing encoder embedding than K-means.
Another benefit of using Spin codebooks is to avoid the need for training a separate K-means model.

Table: Table 14: Zero-shot SLM results with different quantization approaches. All models are based on HuBERT Base. K-means clustering is performed on the transformer encoder output.

### B.3 Supervised DC-Spin Fine-tuning

Figure: Figure 5: DC-Spin with mel spectrogram reconstruction auxiliary objective. The reconstruction loss is $\mathcal{L}_{\text{Mel}}=\mathcal{L}_{\text{AA}}+\mathcal{L}_{\text{AB}}+ \mathcal{L}_{\text{BA}}+\mathcal{L}_{\text{BB}}$.
Refer to caption: x5.png

Method
Inspired by the speech encoders in multimodal LLM studies (Gemini Team, ; Dubey et al., ), we include supervised fine-tuning to boost tokenizer quality.
Different from Section 3.2, the SFT here is applied during DC-Spin fine-tuning so that the tokenizer is jointly optimized with several objectives, i.e., multitask learning.
We consider three types of SFT: Mel spectrogram reconstruction (Mel), CTC-based character recognition (CTC-ASR), and CTC-based phoneme recognition (CTC-PR).
Note that Mel reconstruction is unsupervised, but we consider these tasks together as SFT for a simpler presentation.
For Mel reconstruction, we adopt the speaker encoder and the decoder proposed in Chou & Lee ().
The speaker encoder transforms the Mel spectrogram of an utterance to a single speaker embedding.
The decoder then reconstructs the Mel spectrogram by taking the speech encoder’s output embedding and fuses the speaker embedding with adaptive instance normalization (Huang & Belongie, ).
Because the input for Spin training consists of pairs of utterances with the same content spoken by different speakers, we reconstruct each utterance into two Mel spectrograms using the original and the perturbed speaker embeddings, as illustrated in Figure 5.
This training objective is expected to help disentangle speaker and content representations.
We use a standard $L^{1}$ loss for this task.
For CTC-ASR and CTC-PR, a linear prediction is added to project hidden representations to logits over all possible output textual tokens.
Thus, the loss function of the supervised DC-Spin is

$$ $\mathcal{L}_{\text{DC-Spin-SFT}}=\mathcal{L}_{\text{DC-Spin}}+\lambda_{\text{ Mel}}\mathcal{L}_{\text{Mel}}+\lambda_{\text{CTC-ASR}}\mathcal{L}_{\text{CTC- ASR}}+\lambda_{\text{CTC-PR}}\mathcal{L}_{\text{CTC-PR}},$ $$

where $\lambda_{\text{Mel}}$, $\lambda_{\text{CTC-ASR}}$, and $\lambda_{\text{CTC-PR}}$ are hyperparameters.
With the SFT tasks, the learned codebooks are expected to align better with the underlying textual and phonetic distribution.

Setup
The training setup is almost identical to DC-Spin as discussed in Appendix A.4, but the peak learning rate here is 2e-5.
In the experiments, we always let

$$ $\lambda_{\text{Mel}}+\lambda_{\text{CTC-ASR}}+\lambda_{\text{CTC-PR}}=\text{5}$ $$

and assign the same value for each loss.
E.g., when Mel reconstruction and PR are applied, we have $\lambda_{\text{Mel}}=\lambda_{\text{CTC-PR}}=$ 2.5 and $\lambda_{\text{CTC-ASR}}=$ 0.
We compute 80-bin Mel spectrograms with torchaudio (Yang et al., ).

Table: Table 15: Supervised tokenizers on zero-shot SLM tasks. All tokenizers are DC-Spin models with auxiliary tasks indicated by the checkmarks. The top three results in each section are underlined.

Results
According to Table 15, fine-tuning DC-Spin jointly with supervised objectives offers limited improvement compared with applying ASR and PR before DC-Spin.
This phenomenon might be explained by the fact that ASR and PR require fine-tuning more hidden layers to perform well or have a greater impact on token quality.
However, fine-tuning too many layers with Spin leads to collapsed representations and requires more advanced techniques to mitigate this issue (Chang et al., ; Chang & Glass, ), making DC-Spin + SFT more difficult to find the optimal hyperparameters.
Thus, we exclude supervised DC-Spin fine-tuning from the main text and separate unsupervised and supervised fine-tuning into two stages.

### B.4 Codebook Quality

Table: Table 16: Codebook quality of speech tokenizers with 500 units. The ABX implementation differs from Gat et al. () and Messica & Adi (), so the scores are not comparable.

To quantify the codebook quality, we compute the ABX, PNMI, and CNMI values for several speech tokenizers in Table 16.
The ABX scores are averaged over LibriSpeech dev subsets used in ZeroSpeech 2021.
For HuBERT Base tokenizers, the trend of all three metrics on K-means, Spin, and DC-Spin indicate the effectiveness of the proposed DC-Spin tokenizer in capturing better phonetic representations.
However, when fine-tuned with ASR or PR, the ABX error rates increased while PNMI and CNMI were improved.
We suspect this phenomenon is caused by the fact that some fine-grained phonetic representations in SSL models become coarser because of the CTC-based supervised fine-tuning tasks.

### B.5 Codebook Visualization

Figure: (a) HuBERT + DC-Spin500,4096
Refer to caption: x6.png

Following prior works (Chang et al., ; Liu et al., ), we plot the $P(\text{phone}|\text{code})$ of the primary codebook in DC-Spin to visualize the assignment for each code to phonemes.
Figure 6 illustrates DC-Spin with different initialization models.
The distributions between different models are similar and follow the long-tail distribution of phoneme occurrences.
Still, SpinHuBERT + PR fine-tuning slightly assigns the codes to each phoneme more uniformly.

## Appendix C Additional Spoken Language Model Results

This section offers more complete results and ablation studies on SLM.

### C.1 SLM-ASR Initialization

As shown in Table 17, we compare decoder-only ASR fine-tuning with and without unsupervised SLM pre-training.
The lower WERs indicate that large-scale SLM pre-training benefits downstream fine-tuning.
Hence, all SLM-based ASR experiments in this paper are initialized with pre-trained SLMs.

**Table 17: Decoder-only ASR WERs with and without unit-based SLM pre-training. The discrete units are HuBERT Base Layer 9 K-means 500 units.**
| Method | dev-clean | dev-other | test-clean | test-other |
| --- | --- | --- | --- | --- |
| From Scratch | 8.5 | 18.7 | 8.9 | 18.8 |
| SLM Pre-training | 7.7 | 17.6 | 7.9 | 18.2 |

### C.2 SLM-based ASR

Table: Table 18: Decoder-only ASR WERs on LibriSpeech. All ASR models are initialized with pre-trained SLMs. The lowest WERs are boldfaced, and the second and third best values are underlined.

This section reports the complete SLM-based ASR results and 4-gram predictability in Table 18.
The findings are listed as follows:

- 1.
Audio codecs offer the worst ASR WERs, showing that the first codebook poorly captures the content of speech because the information is spread out to the other RVQ codebooks.
- 2.
Whisper encoder with K-means clustering performs worse than most SSL-based encoders.
Although the Whisper encoder is trained with 680k hours of labeled speech, the sequence-to-sequence ASR architecture does not explicitly constrain the encoder to align with the input signals or capture fine-grained phonetic units, making this model less ideal for tokenizing speech than SSL models.
- 3.
According to the HuBERT Large and X-Large results, scaling the encoders with more parameters improves ASR.
However, compared with DC-Spin in the last part of the table, scaling has less impact on the performance.
- 4.
Unsurprisingly, supervised fine-tuning with PR and ASR improves SLM-based ASR and 4-gram perplexity because the supervision directly relates to the downstream task.
- 5.
The 4-gram predictability (perplexity) correlates with ASR WER, indicating that this metric is a good proxy for SLM-based ASR, consistent with Figure 3.

Results in Table 18 demonstrate the proposed SpinHuBERT and DC-Spin tokenizer offer the best SLM-based ASR.

## Appendix D Additional Speech Resynthesis Results

This section reports complete results on speech resynthesis in Table 19.
Note that the first two RVQ codebooks are always used during EnCodec training, so the EnCodec results start from 1.5kbps bandwidth.
Because of this property, the tokens produced by the first codebook contain less useful information for downstream tasks, as shown in the zero-shot SLM experiments in Section 4.2.

In Table 19, EnCodec and SpeechTokenizer require multiple codebooks, i.e., high bitrates, to resynthesize the audio with low ASR-WER and high UTMOS.
For instance, when all codebooks are used (RVQ1:8), SpeechTokenizer performs similarly to our best model but requires a 9X bitrate.
The UTMOS of EnCodec is relatively low even though the intelligibility is high probably because this model is not specialized in synthesizing human speech.
Next, comparing DC-Spin with different codebook sizes (the third section of Table 19), a larger primary codebook offers superior resynthesis performance because the bandwidth increases.

Extending from Section 4.3, Table 19 reports additional results using randomly selected speaker and style IDs to simulate real-world applications.
Generally, we found slight degradation in random resynthesis intelligibility and similar UTMOS.
Since the Spin and DC-Spin tokenizers are only trained with a speaker-invariant objective, the style information is still preserved in the tokens, making resynthesizing to a different style more difficult.
One possible solution is to include more complex perturbations in the Spin fine-tuning process to force the tokenizer to neglect irrelevant information.

Table: Table 19: Complete speech resynthesis ASR-WER and UTMOS results on Expresso dev and test sets with different methods and bitrates. “Original” and “random” respectively denote resynthesizing speech with the original and random speaker and style IDs.

## Appendix E Self-supervised Pre-training

This section reports and discusses additional results of SSL pre-training, including ASR fine-tuning, SUPERB downstream evaluation, and layer-wise analysis.

### E.1 CTC-based Automatic Speech Recognition

Following prior studies, we fine-tune SSL models with limited labeled data for ASR with the setup described in Appendix A.3 (Baevski et al., ; Hsu et al., ; Chen et al., ).
The experiments here are conducted once without hyperparameter tuning, which might not reflect the true performance of SpinHuBERT.
As shown in Table 20, SpinHuBERT outperforms HuBERT it3 in all setups, showing that improving HuBERT pre-training targets helps capture better content information and offers a better initialization for ASR fine-tuning.
However, HuBERT it3 is slightly worse than HuBERT it2 (Hsu et al., ), which might be caused by the fact that HuBERT it3 is trained with a more diverse and noisy dataset without a denoising objective like WavLM (Chen et al., ), while the training and evaluation of HuBERT it2 are both on the clean LibriSpeech corpus.
Moreover, the HuBERT it3 and SpinHuBERT models are trained with 124k hours of speech but are optimized with only 400k steps, significantly fewer than that of WavLM Base+ (1M steps).
Although SpinHuBERT is slightly inferior to the prior state-of-the-art like multi-resolution HuBERT (Shi et al., ) and WavLM in some ASR cases, the main purpose of developing SpinHuBERT is to offer a better initialization for the proposed DC-Spin.

Table: Table 20: LibriSpeech CTC-based ASR results without LM.

### E.2 Speech Processing Universal Performance Benchmark

Table: Table 21: Results of SSL models on SUPERB. “ParaL.” indicates the paralinguistic task. Unless specified otherwise, all models are “Base” with approximately 95M parameters.

This section evaluates the SSL models in this paper on the Speech Processing Universal Performance Benchmark (SUPERB) (Yang et al., ; Tsai et al., ).
SUPERB is a benchmark that assesses the usefulness of hidden representations of pre-trained speech SSL encoders by applying these representations to a wide range of speech processing tasks.
During downstream task training, a speech encoder is frozen, and hidden layer features are extracted.
A learnable weighted-sum mechanism then aggregates each frame across all layers to a sequence.
The aggregated features are then fed to a lightweight prediction head optimized with a supervised objective like CTC.
We encourage the readers to refer to the original SUPERB papers for a more complete explanation of the tasks and evaluation metrics.
We follow the implementation in the S3PRL library.(^18^1818[https://github.com/s3prl/s3prl](https://github.com/s3prl/s3prl))
The results are shown in Table 21.

Comparing the HuBERT it3 models, we found increasing the number of layers improves almost all tasks (6 vs. 12 vs. 18 layers), showing the effect of model size scaling in downstream tasks.
When comparing 50Hz HuBERT it3 and SpinHuBERT models, the 25Hz models are usually better at content-related tasks like PR, ASR, keyword spotting (KS), and speech translation (ST), which might be a result of the shortened sequence of features.
The ASR results of HuBERT it3 and SpinHuBERT are worse than some prior methods, which is consistent with the findings in the previously discussed ASR experiments.
Similar to the results in Chang et al. (), the SpinHuBERT models trained with Spin units offer high-quality representations and improve content-related tasks like PR.
Although trained with speaker-invariant targets, SpinHuBERT performs similarly to prior methods on speaker-related tasks because the speaker information is preserved at the bottom layers, which will be discussed in the next paragraph.

Figure: (a) Phoneme Recognition (PR)
Refer to caption: x9.png

We visualize the weighted-sum mechanism of the SUPERB downstream models in Figure 7 to understand the importance of each hidden layer for different downstream tasks.
Following Chang et al. (), we normalize the weights by scaling with a factor of the averaged $L^{2}$ norm of each layer’s hidden representations.
We observed that top layers are more important for content and semantic-related tasks since the weights have higher values (layers eight to twelve), consistent with prior studies (Chang & Glass, ; Yang et al., ).
Moreover, emotion recognition (ER) relies on the top layers as well (Figure 7(g)), indicating that classifying speech emotion depends on the content representations.
In contrast to the previous tasks, speaker-related problems use the bottom layers, implying that speaker information is stored at those layers (Figures 7(h), 7(i), and 7(j)).
Comparing HuBERT and SpinHuBERT models, the proposed SpinHuBERT models encode speaker information at lower layers (Figures 7(h) and 7(i)), which is caused by the speaker-invariant pseudo labels for SSL pre-training, forcing the models to drop speaker information at early layers.
Hence, the results verify the findings in the previous paragraph.
Overall, this section comprehensively discusses the effectiveness of SpinHuBERT’s downstream applications and analyzes the importance of each layer.

### E.3 Layer-wise Phonetic ABX

Figure: (a) HuBERT iteration 2
Refer to caption: x19.png

This section discusses the phonetic representations in several speech SSL models by showing each layer’s phonetic ABX error rates.
The ABX scores are averaged over dev sets of the LibriSpeech and Fisher datasets (Cieri et al., ).(^19^1919[https://github.com/facebookresearch/libri-light/tree/main/eval](https://github.com/facebookresearch/libri-light/tree/main/eval))
Because the Fisher dataset is noisier than LibriSpeech, including this corpus helps simultaneously assess the robustness of these SSL models.

As shown in Figure 8(a), the behavior of the Large and X-Large HuBERT models are slightly different than the Base model.
The first difference is the lowest ABX layer of the Large and X-Large models, which is at the last, while the Base model is at the 10th because the former two models are trained with the 9th HuBERT Base layer K-means units.
Second, the ABX scores of some middle layers in Large and X-Large models are higher than other layers.
In Figure 8(b), we compare HuBERT models trained with three iterations.
SpinHuBERT achieves the lowest ABX error rate at the last layer compared with HuBERT models trained with K-means units.
We found the HuBERT it3 models with different sizes share a similar trend in ABX over the hidden layers.
Furthermore, we compare several SSL encoders with similar architectures and the number of parameters in Figure 8(c).
The SSL models all have low ABX scores near the last layer, but wav2vec 2.0 and data2vec have significantly higher values in the last two layers.
Moreover, because of the training objective, the Whisper encoder is worse than other SSL models at distinguishing phonemes, corroborating the findings in Appendix C.2.
To summarize this section, we found that HuBERT is a relatively superior method for capturing phonetic representations, and SpinHuBERT pushes the limit by improving the pre-training target.

## Appendix F Additional Robustness Results

Extending Table 5, this section reports the complete results of robustness experiments in Table 22.

Table: Table 22: Complete unit error distance (UED) robustness results. Unless specified otherwise, all tokenizers are based on HuBERT Base. See Section 4.4 for more information.

## Appendix G Repurposed Chunk-wise Streaming

### G.1 Method

This section describes the chunk-wise streaming token extraction approach.
As depicted in Figure 9, the tokenizer extracts units after the speaker completed speaking $T_{\text{chunk}}$ seconds of speech.
Then, after another $T_{\text{shift}}$ seconds, the tokenizer repeats the same operation with an increased context length ($T_{\text{chunk}}+T_{\text{shift}}$ seconds).
Assuming the tokenizer produces $L_{\text{chunk}}$ and $L_{\text{shift}}$ tokens given $T_{\text{chunk}}$ and $T_{\text{shift}}$ seconds of speech, respectively.
Let $L_{\text{overlap}}=\left(L_{\text{chunk}}-L_{\text{shift}}\right)/$ 2.
The last $L_{\text{overlap}}$ extracted tokens are neglected in each chunk because the lack of future frames degrades token quality.
Under this setup, the tokens extracted in the first few chunks might be less accurate than the offline extracted tokens but will gradually improve after an expanded context.

For instance, in Figure 9, $L_{\text{chunk}}=$ 7 and $L_{\text{shift}}$ = 3, making $L_{\text{overlap}}=$ 2.
The first chunk extracts seven tokens and neglects the last two, and the first five tokens are fed into the SLM.
When shifted by $T_{\text{shift}}$ seconds, the token sequence length increased by three.
We then neglect the last two tokens and take the three tokens before them as the SLM input.

We implement the chunk-wise streaming tokenization by repeatedly increasing the audio samples fed into the tokenizer.
Each time, the tokenizer (e.g., HuBERT) takes a longer input, and we select tokens according to the methods in the previous paragraph.
We then concatenate all selected tokens into one sequence, with approximately the same amount as in the offline extracted sequence.
Note that more advanced model architectures and streaming strategies can be implemented to improve streaming performance.
For example, reusing encoded hidden states and attention maps from the previous chunks might increase inference efficiency.
Nevertheless, we only consider this repurposing approach to achieving streaming tokenization to demonstrate that the proposed tokenizers are streamable without retraining.

Figure: Figure 9: Illustration of chunk-wise streaming token extraction with increasing context length.
Refer to caption: x22.png

Latency
The latency calculation is the average latency per chunk.
As shown in Table 7, the latency is lower than 20ms, which is about 5% of $T_{\text{shift}}=$ 0.4s.
Moreover, the latency of the last few chunks is about 60ms when a sequence length is 30 seconds long, still significantly lower than the $T_{\text{shift}}$, showing that the user experience mainly depends on $T_{\text{chunk}}$ and $T_{\text{shift}}$, not the tokenizer.
Hence, we focus on whether a tokenizer retains the performance when repurposed to streaming mode rather than the latency.

### G.2 Results

Figure: Figure 10: UED between tokenizing offline and chunk-wise streaming. All models are 500-unit tokenizers. Smaller $T_{\text{shift}}/T_{\text{chunk}}$ indicate higher overlap between chunks. Solid, dashed, and dotted lines depict $T_{\text{chunk}}=$ 1, 2, and 5 seconds, respectively.
Refer to caption: x23.png

Figure: (a) $T_{\text{chunk}}=$ 1 sec
Refer to caption: x24.png

Figure: (a) $T_{\text{chunk}}=$ 1 sec
Refer to caption: x27.png

As shown in Figure 10, the UED between offline and streaming tokenization can be significantly reduced by increasing $T_{\text{chunk}}$ (comparing between line styles: solid, dashed, and dotted).
The UED can also be improved by reducing $T_{\text{shift}}$ because the higher overlap between chunks ($L_{\text{overlap}}$) prevents using tokens that lack future context.
We found that DC-Spin without SFT has a lower UED compared with SFT, which is different from the robustness experiments in Appendix F.
A possible cause of this discrepancy is that fine-tuning with PR makes the speech encoder depend on a longer context because the average rate of phonemes is around 10Hz.
Still, SpinHuBERT with PR SFT outperforms the HuBERT K-means baseline.

In Figure 11, we show the tSC accuracy under several streaming settings.
Similar to the findings in Figure 10, higher $T_{\text{chunk}}$ and lower $T_{\text{shift}}$ results in better accuracy.
Unexpectedly, when $T_{\text{chunk}}$ is two or five seconds and $T_{\text{shift}}/T_{\text{chunk}}\leq$ 0.8, streaming sometimes outperforms offline tokenization (solid lines are higher than dashed).
We suspect the SLM’s robustness to token perturbation causes this phenomenon, but we leave this investigation for future studies.

As for resynthesis with streaming tokenization, results in Figure 12 show a similar trend as in UED and tSC experiments.
Unlike tSC, streaming tokenization always underperforms offline, indicating that speech resynthesis still requires accurate speech tokens.
Overall, experiments in this section demonstrate the possibility of repurposing offline tokenizers to streaming mode with a small performance drop.
The findings offer insights into designing speech tokenizers for real-world applications.

## Appendix H Correlation Between Metrics

Figure: Figure 13: Pearson correlation coefficients between evaluation metrics computed with tokenizers operate at a 50Hz framerate with 500 units. The upper right corner is the same as Figure 3.
Refer to caption: x30.png

This section reports more results on the correlation between proxy tasks and downstream performance for reference.
As shown in the upper left corner of Figure 13, some proxy tasks have high correlations, like ABX and PNMI, but correlate differently with downstream metrics.
This observation implies similar proxies should all be considered when predicting a tokenizer’s downstream performance.
In the lower right corner of Figure 13, most downstream tasks are highly correlated, showing that speech tokenizers usually improve all tasks simultaneously.
Furthermore, SLM-based ASR correlates with all other tasks with coefficients higher than 0.74.
Therefore, ASR with speech tokens can also serve as a hint to other downstream task performance.

We only compare tokenizers with 500 units operating at a 50Hz framerate because some metrics like ABX error rate are incomparable when the number of units or framerate differ.
Perhaps future studies can focus on revealing the relationship between unit size and framerate.

Additionally, Figures 14, 15, 16, and 17 visualize the correlation between these proxies and five major evaluation metrics.

Figure: (a)
Refer to caption: x31.png

Figure: (a)
Refer to caption: x36.png

Figure: (a)
Refer to caption: x41.png

Figure: (a)
Refer to caption: x46.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. I am a little bit confused about the bitrate of Kmeans and DC-Spin reported in Table 4 and Table 19. For example, DC-Spin_{500,4096} has the primary codebook with 500 size and the auxiliary codebook with 4096 size. But the final bitrate is the same as the Kmeans_{500} which only has a 500 size codebook. Could you explain it in detail?
soundness: 3
strengths: 1. Introduced the Double-Codebook Spin to capture the fine-grained phonetic units better, and gave the detail process about how to select the codebook size.
2. The experimental setup is thorough, covering both zero-shot and supervised tasks, with a clear evaluation on robustness and inference efficiency. 
3. This work analyzed multiple proxy tasks on speech tokenizers to reveal their relationship with the performance of the spoken language model.
summary: The paper presents a novel approach to speech tokenization for spoken language models (SLMs), focusing on speaker invariance and phonetic richness through the Double-Codebook Speaker-invariant Clustering (DC-Spin) technique. DC-Spin utilizes a dual codebook design to separate phonetic and speaker-invariant features, enhancing SLM performance in tasks like automatic speech recognition (ASR) and speech synthesis. Additionally, the paper proposes SpinHuBERT, a pre-trained model that leverages DC-Spin for robust and accurate tokenization. The contributions of DC-Spin are evaluated against existing tokenizers, demonstrating improved robustness, phonetic alignment, and real-time processing efficiency, setting a new standard in resource-constrained and scalable speech tokenization.
weaknesses: 1. Although this work states three contributions, they were incremental somehow. For example, the DC-Spin doesn't show convincing advantages against Spin method in Table 2, especially considering the much larger codebook size of DC-Spin. As for the chunk-wise streaming simulation, it has been used in many similar works such as ASR and TTS. And the authors claimed it in the contribution part but they just mentioned it in a short paragraph in the experiment section.
2. The compared neural audio codec methods were a little bit dated. Encodec and SpeechTokenizer methods were proposed in 2022 and 2023. Many new and powerful works have been proposed since then. For example, descript audio codec (DAC) and hifi-codec should be included in the comparison. Besides, WavTokenizer is another work focusing on low bitrate, making it suitable for comparison.
3. The motivation of designing the DC-Spin is not clear. The authors claimed the auxiliary codebook can capture fine-grained phonetic units but didn't provide explanations. Moreover, it both primary and auxiliary codebooks pay attention to the phonetic units, the acoustic detail would be ignored in DC-Spin to some extent, which is harmful for the generative task.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. **Claim:** “The first codebook (primary) extracts discrete units for downstream applications. The second codebook (auxiliary) is a large codebook that enhances the encoder’s capability to capture fine-grained phonetic units.” Might there be an ablation study on the independent use of each codebook to support this finding? (i) Evaluating downstream task performance using only the primary codebook vs. only the auxiliary codebook, (ii) analyzing the phonetic content captured by each codebook independently, and (iii) examining how varying the size of each codebook impacts the overall system performance are useful experiments to help validate the claim on the role of codebooks.
2. **Robustness Testing:** Could the authors provide further details on the types and degrees of perturbations applied in testing the robustness of DC-Spin? Specifically, can you provide quantitative details on the noise levels, time stretch factors, reverberation parameters, and pitch shift ranges, and comment on how well they compare to real-world conditions?
3. Can DC-Spin can be adapted to operate at higher bitrates comparable to systems like NaturalSpeech2 or VoiceCraft? If so, how does its performance compares in terms of speech quality metrics (e.g. MUSHRA scores) at those higher bitrates?
soundness: 3
strengths: * **Originality:** The paper’s contribution of SpinHuBERT and DC-Spin tokenizers demonstrates originality.
* **Quality:** The evaluation is comprehensive, evaluating the proposed tokenizers across multiple downstream tasks and metrics to show their effectiveness.
* **Clarity:** The presentation is clear.
* **Significance:**  Disentangling factors within discrete speech tokens, as explored in DC-Spin, addresses a growing area of interest (e.g., FACodec). DC-Spin seems to stand out as one of the first speech tokenizers to separate speaker information from codebooks.
summary: This paper presents the development of robust and efficient speech tokenizers for spoken language modeling and speech resynthesis. The authors introduce two innovative methods, SpinHuBERT and DC-Spin, designed to meet four essential criteria for ideal tokenizers: capturing phonetic information, preserving acoustic details, robustness to perturbations, and enabling efficient inference. Experimental results demonstrate that a 150M parameter Transformer decoder, trained on downstream tasks, performs optimally with DC-Spin speech tokens. Furthermore, DC-Spin delivers superior reconstruction quality at extremely low bitrates compared to popular codecs. The authors also identify proxy metrics — shown to correlate highly with downstream task performance — and advocate for these metrics as a basis for evaluating future speech tokenizers.
weaknesses: **Comparison in Speech Resynthesis:** While the results show that DC-Spin performs well at low bitrates (<1.5 kbps), it is worth noting that current state-of-the-art speech synthesis systems (e.g., NaturalSpeech2, VoiceCraft, MaskGCT) use codecs with higher bitrates to achieve high-quality speech reconstruction. A comparison of DC-Spin against existing tokenizers at these higher bitrates would provide valuable insight into its performance for speech resynthesis under more typical conditions.

---

confidence: 4
contribution: 1
initial_rating: 1
presentation: 2
questions: The Spin mentioned in the paper perturb the speech utterance by randomly scaling the F0 and formant frequencies.
As I know, speaker variation is more than different F0 and formants. How do you handle speaking style, which is also a kind of speaker-variant?

In the most right hand side of Figure 2, it is not clear to me how the codebook is formed and how the discrete speech tokens are generated? Is K-means clustering still be used?
soundness: 1
strengths: This paper proposes a method called DC-Spin to improve speech tokenization.
The speech tokens generated by the proposed method is speaker-invariant and can preserve more content information compared to open-source tokenizers.
summary: This paper introduces Double-Codebook Speaker-invariant Clustering (DC-Spin) as a method to enhance speech tokenization by making the speech tokens to be speaker-invariant. 
By decreasing the proportion of the speaker information in the speech tokens, DC-Spin aims to improve zero-shot SLM tasks and speech resynthesis. Through comparisons of tokenization techniques and model scalability, the study highlights the effectiveness of tokens easily modeled by n-gram language models or aligned with phonemes, offering valuable guidelines for designing speech tokenizers for SLMs.
weaknesses: The motivation of this paper does not sound convincing.
A common way of connecting a speech encoder to a LM is directly through speech representation, which is of continuous values. 
The discretization from representation to discrete tokens will only degrade the quality of the speech input. Check out the papers below:
https://arxiv.org/pdf/2402.08846
https://arxiv.org/abs/2309.00169
https://www.isca-archive.org/interspeech_2024/yang24f_interspeech.pdf
The discretized tokens will eventually be converted to continuous vectors through the embedding table of the LM. 
Therefore, there is no need to discretized the continuous embedding for the input of LM.

From the LM output perspective, yes, there is a need for speech tokenization if we want the LM to generate speech.
However, the TTS community has already gone far beyond word accuracy. We are talking about voice clone, high naturalness.
Thus, removing the speaker information may even hinder these speech tokens to be used.
Even if we neglect the voice clone issue, the ASR-WER of the speech resynthesis is far below standard (18.x in the paper). That means the system can't be used. Checkout the numbers in the following papers:
https://arxiv.org/abs/2309.00169
https://arxiv.org/abs/2406.02430
https://arxiv.org/abs/2301.02111

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. In table 8 and 4 ASR-WER is very high. What ASR model was used? What is the reason for this?
2.  Why is using the proxy tasks better than directly using TSC/ sWUGGY / sBLIMP. They don't seem easier to compute and TSC/ sWUGGY / sBLIMP are already proxies for the ability to generate intelligible audio. 
3. Why use HuBert as the base encoder and not some other encoder? what are the properties of HuBert that make it best for this task?
4. Why fine-tune and not train from scratch with the SPIN objective?
5. In figure 1, we see that different tasks have the best performance with different code-book sizes. For example the 1000 size codebook has the best performance on TSC (but is not used later in the paper). How should the final model be picked?
soundness: 3
strengths: 1. The proposed method is novel, and since speech tokanization is widely used improvement are significant. 
2. The paper is clearly written which enables to easily re-implement and use the proposed method. 
3. The proposed method shows a strong performance over the other baseline methods.
summary: This paper proposed a new method for speech tokenization named SpinHuBERT, this tokenization is based on pre-training a HuBert speech encoder with the spin objective, then with an ASR target, and finally fine-tuning with a DC-Spin objective. The evaluation shows the high performance of the method across a wide range of tasks, such as spoken language modeling, speech resynthesis, robustness under distortions and inference efficiency. Furthermore, the authors evaluate different proxy tasks for spoken language modeling which identifies decent candidates.
weaknesses: 1. Proposed method out-performs the baseline on most tasks, but the margins are small while the proposed method is significantly more complex.
2. Evaluation on directly synthesizing speech is not performed. It is unclear if in a real setting the proposed method is able to generate coherent and intelligible speech. This evaluation should be added to show the methods performance in the task SpeechLMs are used for.
3. Throughout the paper, UTMOS is used instead of real human raters. While UTMOS is a good proxy, it does not fully replace human raters. At least some of the evaluations should be done using MOS with human raters. 
4. Table 3 shows evaluation of other method with much larger models. While a 150m parameter model is decent, SpeechLMs are regularly scaled to larger sizes. The paper is lacking an evaluation of a full-sized SpeechLM model. Such a model would give the reader a better option to appreciate the performance of the proposed method.
5. While the tokens are claimed to be speaker invariant, this is not evaluated in the experimental setup. I would expect the authors to include an experiment which verifies this claim. 
6.

Meta-review: The authors propose a method for speech tokenization that is speaker invariant. The paper received four reviews, with three generally negative and one positive. While the paper did receive one vote for acceptance, that vote had low confidence, while the three votes for rejection were much more confident in their evaluation. The reviewers were not too motivated to engage with the paper generally, but one reviewer did note that concerns remained after the rebuttal. In general, it was felt that the evaluation of the propose method should be improved before acceptance to a top-tier conference.

Decision: Reject