StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs
Authors: Yuhan Song
Linhao Zhang
Chuhan Wu
Aiwei Liu
Wei Jia
Houfeng Wang
Zhou Xiao
Venue: ICLR 2026 Poster (2026)
Keywords: speech tokenizer
noise robustness
audio
multi-modality
speech language modeling
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This paper introduces StableToken, a novel speech tokenizer architecture with superior noise robustness and excellent performance.

Abstract: Prevalent semantic speech tokenizers, designed to capture linguistic content, are surprisingly fragile. We find they are not robust to meaning-irrelevant acoustic perturbations; even at high Signal-to-Noise Ratios (SNRs) where speech is perfectly intelligible, their output token sequences can change drastically, increasing the learning burden for downstream LLMs. This instability stems from two flaws: a brittle single-path quantization architecture and a distant training signal indifferent to intermediate token stability. To address this, we introduce StableToken, a tokenizer that achieves stability through a consensus-driven mechanism. Its multi-branch architecture processes audio in parallel, and these representations are merged via a powerful bit-wise voting mechanism to form a single, stable token sequence. StableToken sets a new state-of-the-art in token stability, drastically reducing Unit Edit Distance (UED) under diverse noise conditions. This foundational stability translates directly to downstream benefits, significantly improving the robustness of SpeechLLMs on a variety of tasks. Our code and model are publicly available at https://github.com/Tencent/StableToken.

Published as a conference paper at ICLR 2026
STABLETOKEN: A NOISE-ROBUST SEMANTIC SPEECH TOKENIZER FOR RESILIENT SPEECHLLMS
Yuhan Song
Linhao Zhang2+
Chuhan Wu?
Aiwei Liu?
Wei Jia?
Houfeng Wang't it
Xiao Zhou²
State Key Laboratory of Multimedia Information Processing.
School of Computer Science, Peking University
2 Basic Model Technology Center, WeChat AI, Tencent Inc.
(songyuhan, wanghf) (angghheeukk.e.o.. edu. 2hangl.inhac00@@ma1...cooo
ABSTRACT
Prevalent semantic speech tokenizers, designed to capture linguistic content, are surprisingly fragile. We find they not robust to meaning-irrelewee irrelevant acoustic perturbations; even at high Signal-to-Noise Ratios (SNRs) where speech is per- fectly intelligible, their output token sequences can change drastically, increasing the learning burden for downstream LLMs.. This instability stems from two flaws: brittle single-path quantization architecture and a distant training signal indif- ferent to intermediate token stability. To address this, we introduce StableToken, a tokenizer that achieves stability through a consensus-drice multi-branch architecture processes audio in parallel, and these representations are merged via a powerful bit-wise voting mechanism to form a single, stable token sequence. Stable'Token sets new state-of-the-ar art in token stability, drastically re- ducing Unit Edit Distance (UED) under diverse noise conditions. This foundational stability translates directly to downstream benefits, significantly improving the robustness of SpecchLLMs on a variety of tasks. Our code and model are publicly available httpps: thu. httph//w...ommeeeeeeeteeettteeee
mechanism. Its
INTRODUCTION
The application of Large Language Models (LLMs) to the speech domain has given rise a new class of powerful models: Speech Large Language Models (SpeechLLMs) (Hurst et al., 2024 Défossez/ jtt al. 2024 Zeng et al. 2024). These models rely on a discrete speech tokenizer to convert continuous audio into tokens sequences that the L.M can process. Among available methods, semantic tokenizers have been widely adopted, as their low-bitrate, semantically-alignes outputs are highly compatible
with LLM architectures (Défossez et al. 2024 Zeng et al. 2024 Ding et al. 2025
Wu et al. 2025).
The design of semantic speech tokenizers has evolved from early self-supervised learning (SSL)
202 Baevski et al. 2020) towards a more direct, supervised paradigm (Du et al. Zeng et al. 2024). This modern paradigm centers on optimizing a VQ-based quantizer ( Van Den Oord et al. 2017) with direct, end-to-ouee objective such automatic speech recognition (ASR)
methods (Hsu et al.
2024a b
This powerful combination has proven highly effective at producing semantically-rich and compact
discrete representations, leading to the widespread adoption of supervised semantic tokenizers as
the backbone of many modern SpeechLLMs (Zeng et al. 2024 Ding et al. 2025 Wu et al. 2025
Huang et 2025 Fang et al. 2025).
Despite their widespread adoption and apparent success, we find that these semantic tokenizers harbor critical vulnerability: profound lack of robustness. Contrary their core design principle of encoding semantics, even imperceptible acoustic noise can induce drastic shifts in their discrete outputs (Figure D]. The general issue of tokenizer instability is also supported by recent findings on earlier non-V --aeed SSI. tokenizers (Messica & Adi 2024). This instability creates a damaging downstream effect: small acoustic changes trigger large token jumps, which break the crucial
"quual contribution. Work conducted during Yuhan's internship at WeChat. Corresponning authors.

---

Published as a conference paper at ICLR 2026
Orginal miniee comtrbatooo
TMV
61tmmm 10am)
Tokenzer (osteedrm))
SMMR-25
(1]
2037
Stable Taken (propcoeed)
SMRR225
(6)
TM
14:37
Tokia audio
Tokia idanticall original
Taker diffarnnnt from
Figure 1: Illustration of StaeleToken: unlike traditional methods, StaeleToken yields consistent token sequences under minute perturbations with different Signal-to-Noiss Ratios (SNRs). Robustness is measured by Unit Edit Distance (UED, J) between token sequences of the original and noise- perturbed audio. StableToken achieves significantly lower UED, indicating enhanced token stability.
speech-text alignment and pose a significant modeling challenge for the LLM, forcing to learn
from an inconsistent or even chaotic input stream. This inherent fragility is severely amplified by
environmental noise, serving as a key cause for the performance degradation of SpeechLLMs in
real-world conditions (Ma et al.] 2025b Zhang al. [0255] Yang et al. 2022cc Jiang al.] (2025). We
argue that enhancing tokenizer robustness is therefore a direct and promising path toward building more resilient models.
We pinpoint the fragility of semantic tokenizers to two fundamental weaknesses. First, an architec- tural flaw: these tokenizers rely on single-patl quantization, design that lacks fault tolerance. A minor perturbation near a quantization boundary is inevitably magnified into a completely different output token. This architectural vulnerability then compounded by a distant supervisory signal: the standard ASR loss is indifferent to this intermediate token instability, as it only supervises the final transcribed text. This allows models to converge on solutions that are functionally correct but representationally fragile. The dual challenge of a brittle architecture and distant supervisory signal necessitates a new tokenization paradigm.
Addressing this dual challenge is non-trivial. For the brittle architecture, an offline ensemble of models seems intuitive. However, this approach untenable: (1) it prohibitively increases inference cost; (2) aggregating independently trained models is non-trivial, as their quantization boundaries are arbitrarily aligned; and (3) token-level majority vote is too coarse. To tackle the distant supervisory signal, one might introduce token-level consistency objective for clean and noisy input audios. Yet, this leads to unstable gradients when applied to discrete codes, making the model difficult to train. The failure of these straightforward approaches underscores the need for more integrated paradigm.
We propose StableToken, which integrates a co-designed architecture and training strategy to over- come the dual challenges of architectural fragility and distant supervision. Architecturally. it introduces the voting-LFQ module-a a multi-branch quantizer extended from the LFQ algorithm (Yu al. 2023), with negligible inference overhead. Its core mechanism is differentiable bit-level information,
majority vote. During training, this enables a more fine-grained fusion of leading more stable and robust representation learning. At inference, this same mechanism provides profound crror-correction, operating at the granular bit-level rather than the coarse token-level. This distinction is critical: not only does it ensure the final token remains correct when minority of branches err due to noise, but it can even recover the token when majority of branches fail at the teken- as long as the underlying bit-level errors remain sparse.
This architectural robustness is further solidified by tailored training strategy. We present the model with multiple "views" of an input-a clean version to majority of branches and perturbed version to random minority-to create stable reference. A consensus loss then leverages this reference to provide the explicit, intermediate supervision. The multi-branch architecture and multi- view training strategy are thus deeply intertwined: the architecture provides the necessary structure for the training signal, and the signal in turn unlocks the architecture's 's full potential.
We validate TokeleToken through comprehensive experiments. At the tokenizer level, it achieves a new of-te-of-the-an in noise robustness, slashing the Unit Edit Distance (UED) by over 60% relative (from 26.15 17% 10. 10%),, all while maintaining top-tier reconstruction fidelity. This
2

---

Published as a conference paper at ICLR 2026
Text Decder
Encoder Layer
Projection out
Speech Tokers
O
C
Special Stegle Thomgl Sidente)
Qummization
Average Pooling
Halaa
H paattrted
Projection
Projociun #2
Projection
Encoder Layer
Iopus
Trkking
Heme
H Hottcaal
Home
Hom
Home
Haro
Inguet mandor baanchee:
Addd Pentubations
Figure 2: The architecture of StaeleToken. Our model replaces the standard single-path quantizer with multi-branch Voting-L.FQ module. The zoomed-in in view shows n parallel branches generating independent binary representations. bit-wise majority vote then aggregates them into single token, During our Noise-Aware Consensus Training. randomly selected minority of branches receive perturbed inputs (H pentrred) while the majority receive clean inputs (Helem). consensus loss forces the perturbed branches to align with the consensus. The yellow paths are only used during training.
foundational superiority translates directly to downstream SpeechLLLMs. In speech understanding, the downstream models yield significant robustness gains that are especially pronounced under severe noise. The performance gap between StableToken and baselines widens dramatically as the noise level increases. Similarly, for speech generation, the enhanced token consistency simplifies the learning task, resulting in substantially superior synthesis quality of downstream models. These results confirm that improving tokenizer robustness directly and highly effective for building more resilient SpeechLLMM.
2
METHODS
2.1 OVERALL STRUCTURE
Our approach, StableToken,, is designed to overcome the fragility of prevailing VQ-based semantic tokenizers. These tokenizers often produce unstable token sequences the presence of subtle noise, vulnerability stemming from two core weaknesses: (I) a single-path architecture that lacks fault tolerance, and (2) distant supervisory signal that fail to enforce representational invariance.
SokeleToken adopts the architectural paradigm, established in works like (Du et al. 2024 Du et al., 2024b) of embedding semantic tokenizer within an end-to-en end ASR model. However, our approach fundamentally enhances this design by introducing two synergistic innovations to address its inherent instabilitie:: (1) the Voting-LFO Module, multi-branch quantizer that builds architectral robustness, and (2) Noise-Aware Consensus Training, training strategy that explicitly enforces invariance to acoustic perturbations.
2024at Zeng
2.2 THE VOTING-LFQ MODULE
The foundation of StableToken a novel quantizer architecture designed for intrinsic robustness. As shown in Figure the pretrained speech encoder first processes the input speech into sequence
3

---

Published as a conference paper at ICLR 2026
of hidden states. These states then downsampled via average pooling to produce compact representation, E IR,, for each time step.
While traditional quantizers map h to token in single, brittle step, Voting Look-up-Free Quantizer (Voting-LFQ) is founded on redundancy and consensus. begins by creating n inde- pendent "perspectives" of the input state h using parallel linear projection layers. Each branch i E [I. computes a projected vector p; E R
P = W,h+b. b;,
where W; E RdxD and b, E Rd are the unique learnable parameters for that branch. Each projected vector then binarized into B, (-1,+1}" using the non-differentiable sign function, ie.. B; = (p:). We use the Frriggh-TThruuee Estimator (STE) (Bengio et al. 2013) to enable end-to-end training.
During training, we aggregate these n binary vectors in bit-wise manner by averaging their values across branches for every dimension j E {1....,,0) d}, resulting in a real-valued score:
n (B.).
(Stina))
72
Unlike the rigid assignment of a single bit (+1 or 11), these averaged scores can take nuanced values representing the confidence consensus of all branches. This provides the model with richer feedback during optimization, helping it learn more robust and informative representations.
During inference, perform one additional step to apply the sign function to these aggregated scores to obtain the final consensus-basee based binary vector:
(Bina); = sign ((Smm))y)
By using an odd number of parallel branches (n). we enforce a strict majority rule via a bit-wise vote, creating exceptional robustness against noise. This approach not only corrects errors when a minority of branches fail, but can also recover the true token even if majority of branches are corrupted at the token-level. Recovery is possible long the underlying bit-level errors remain sparse. This resilience marks significant advantage over fragile single-path quantizers and, in parallel, allows for more expressive representations during training.
Finally, by mapping its 1 and +1 entries to and 1 respectively and treating the {0, 1] representation as binary number, the stabilized binary vector B is deterministically mapped to an integer index k E {0.... 2d 1}. This index serves as the final, robust speech token. It is worth noting that our Voting-LFQ structure introduces negligible additional parameters and computational overhead during inference. A detailed complexity analysis is provided in Appendix
B.6 B.6
2.3 NOISE-AWARE CONSENSUS TRAINING
The Voting-LFQ architecture enables our novel training paradigm, designed to explicitly instill representational invariance. Our goal is make the tokenizer robust to noise without degrading its performance on clean inputs.
The core mechanism works as follows: during each forward pass, for given input audio w, generate a perturbed audio sample w' = A(()), where A(-) is a stochastic augmentation function applied at the waveform level (e.g., adding Gaussian noise; further details in Appendix (B.3). Both w and w' are separately processed by the encoder to produce two corresponding hidden states, h and h'. we then randomly select minority subset of branches (where n/2) to receive the perturbed hidden state h', while the remaining n - l majority branches receive the clean hidden state h.
This setup allows the model perform self-stabilization. To enforce this, introduce the consensus loss (Cconsensus) which encourages all branches, whether they see clean or noisy input, to produce similar pre-quantization representations. We compute a dynamic, "online" target Pall by averaging the rre--uantization vectors p, from all n branches. The loss then penalizes the deviation of each branch from this global average:
Lconsensus 1
pp:
Pan 12 where
1 Pall
Σ n
P j=1
(4)
n
1=1
4

---

Published as a conference paper at ICLR 2026
By optimizing this objective, the clean-man majarrtyy branches act a stable anchor for the global average Pall- preventing it from being corrupted by the noisy inputs. Consequently, the noisy-minoris branches forced to learn representations that align with the clean consensus, effectively learning to ignore the perturbations. Optimizing on the continuous vectors pj provides a smoother and more effective gradient signal than working with the binarized b;.
2.4 FINAL TRAINING OBJECTIVE
The complete training objective for StableTooken combines the ASR task loss with our consensus loss and standard LFQ regularization terms. The primary task is optimized via Cross-Entropy Entropy loss (Casr) on the ground-truth transcripts. Following the LFQ framework (Yu al.) 2023). we also include commitment loss (Ccommiment) to encourage the hidden states to stay close to the quantized representations, and a codebook entropy loss (Leotehook) to promote uniform usage of the discrete codes. The final, composite loss function is weighted sum:
Lutal = LASR + A1 Leumensus + X comminmentt + A Lusdebbkok
(5)
where \1, X2, and λ are scalar hyperparameters that balance the influence of each component.
EXPERIMENTAL SETUP
Our experiments are structured to comprehensively validate StableToken from three perspectives. We first demonstrate its core superiority the tokenizer level, establishing new state-of-the-art in noise robustness without compromising reconstruction quality ($4.1). We then show that this fundamental stability translates directly into significant performance gains in diverse downstream SpeechLLM tasks, including ASR, Speech Emotion Recognition (SER), and Text-to-speech (TTS) ($4.2). Finally, we dissect the model through ablation studies, qualitative analysis, and case study to verify the contribution of each design component and provide insight into its inner workings (4.3)
Tokenizer Training. Our StableToken model is built upon an encoder-decoder architecture initial- ized from wwwsperre whisper-largee waat--aarre--e (Radford et al. 2023), with our Voting-LFQ module inserted into the encoder's mid-point. The tokenizer is pre-trained on a diverse 150k-hour speech corpus. Our tokenizer vocabulary size is set to 8192 (corresponding to d 13), and the frame rate is 25Hz. For our main experiments, the number of voters is set to N = 5. a choice justified by our analysis in Full details on training data hyperparameters are provided in Appendix B
Section
4.3
Baseline Models. We benchmark StableToken against comprehensive suite of SOTA models across three categories: SSL--based, distilled, and supervised tokenizers. For all baselines, we use their officially released models. The detailed list of baseline models can be found in Appendix D
Tokenizer-Leve Evaluation. We assess the tokenizer's intrinsic properties. Robustness mea- sured Unit Edit Distance (UED%, 4) (Messica & Adi 2024) the FLEURS (Conneau et al. (2025) benchmark under various synthetic perturbations and real-world noise conditions, which include challenging out-of-domain (OOD) noise. A detailed description of the noise profiles is in Appendix B Fidelity is measured by Word Error Rate (WER%, 4) and Mean Opinion Score (MOS, 1) on the
LibriSpeech [Panayotov al. 2015) and SEED (Anastassiou al. 2024) benchmarks.
For downstream tasks, we follow a controlled, isogenic setup to ensure fair comparison. Each tokenizer integrated into SpeechL.MM framework using a pre- trained Qwen2. 5B Yangg et 2022a)) backhone, which is then fine-tuned using prompt-based paradigm (Zeng et al. 2025). We evaluate on three tasks: (1) ASR: Assessed on noise-augmented LibriSpeech (Panayotov et al. 2015) and the CHiME-4 (Vincent al. 2017) benchmark using WER (%); (2) SER: Assessed on a noise-augmenter version the ESD (Zhou et al.) (022) test set using classification accuracy (%); (3) TTS: Assessed the SEED-TTS (Anastassiou al.) 2024) benchmark using both WER (%) and MOS. The aggregated training datasets, fine-tuning hyperparameters, and prompts for each task detailed in Appendix F
Downstream Task Evaluation.
5

---

Published as a conference paper at ICLR 2026
Table 1:
Noise robustness comparison across different semantic tokenizers. Results are reported in UED% (L) under synthetic perturbation (Gaussian, Pink, Brown, Bit Crush) and real noise conditions. is worth noting that a comparison most meaningful between tokenizers of the same type. For more comprehensive evaluation, we also include SSL and semantic distilled tokenizers as baselines.
Frame Rate
Codebook
Gauss. Noise
Pink Noise
Brown Naise
Bit Crush
Real Noise
Real (OOD)
Model
Size
Avg.
SSL Semantic Tokenizer
HoBERT-500 (Hsu al 2021
50Hz
500
26.42
20.38
18.82
18.02
18.48
19.18
20.22
NAST Messica & Adi 2024
50Hz
200
18.67
15.78
15.26
14.95
18.69
19.07
17.07
R-Spin Chang Glass
50Hz
2048
21.56
15.47
14.95
15.08
14.75
16.48
Semantic Distilled Tokenizer
50Hz
1024
37.39
28.05
28.06
21.38
22.33
23.09
26.72
Speech Tokenizer Zhang et
2023
50Hz
1024
55.69
59.84
35.29
33.16
S0Hlz
1024
72.74
72.72
75.91
54.01
48.43
48.63
62.07
50Hz
1024
53.54
43.85
40.17
36.95
27.82
28.78
38.52
X- X-Clee Ye ct
2025
50Hl7
1024
71.76
59.95
57.88
50.26
41.25
42.44
53.92
50Hz
1024
84,46
77.31
76.49
68 .7
59.89
62.28
71.48
Mimi Défossez
72.68
60.19
43.58
53.43
Supervised Semantic Tokenizer
Voic---Vice-Tokenn.. Zong al.
2025
12.5Hz
16384
42.44
32.12
30.22
25.53
27.67
28.62 28.62
31.10
8 Tokenizer Du al.
2024a
25Hz
4096
35.40
27.09
25.45
20.64
23.88
24.58
26.17
50Hz
4096
46.05
33.46
27.20
27.70
33.09
CosyVoice" Du al.
2024b
25Hz
6561
54.67
42,57
39.96
30.87
31.76
32.13
38.66
Token (Ours)
25Hz
8192
12,93
9.76
9.37
7.32
10.65
10.96
10.17
RESULTS
4.1
TOKENIZER-LEVER PERFORMANCE
4.1.1 SUPERIOR NOISE ROBUSTNESS
As shown in Table [ StableTokknn establishes new state-of-the-art in noise robustness. It achieves an average UED 10.17% %, a dramatic improvement over both the best supervised baseline (8 Tokenizer, 26.17% 17%) and the top-performing robust SSL-based model (R-Spin, 16.48%). Crucially, this strong performance holds even on out-of-distribution (OOD) real-world noise not seen during training. demonstrating the excellent generalization of our method. Furthermore, this outperformance achieved using significantly larger vocabulary than conventional tokenizers. This makes the result even significant, larger vocabulary creates a finer-grained decision space, making the task of maintaining coken-level invariance inherently more challenging. This substantial performance gap underscores the effectiveness of our co-designed architecture and training strategy.
4.1.2 EXCELLENT RECONSTRUCTION QUALITY
Table 2: Reconstruction results measured by WER (L) and MOS (f) on LibriSpeech (Panayotov et al. 2015 and SEED (Anastassiou et al. 2024) benchmarks.
WER
MOS f
Frame Rate
LS-
LS-
SEED
SEED
LS- clean
LS-
SEED
SEED
Model
#C
BPS
clean
other
other
zh
GLM-4 Voice-TTken. (Zeng el al. 2025)
12.51lz
4.04
9.33
3.54
3.23
4.07
3.99
4.16
4.10
8 Tokenizer (Du 2024a
25Hz
300
5.78
13.38
5.91
4.26
3.40
3.31
3.40
3.31
CosyVoice2 er 20240
25Hz
325
4.25
9.68
4.34
2.75
3.36
3.25
3.31
3.58
StableToken (Ours)
25Hz
325
3.84
7.99
3.44
2.62
4.09
3.83
4.01
4.18
To evaluate reconstruction quality, we follow the methodology of Du et al. (2024a b); Zeng et al. (004)) and train a flow matching model to synthesize audio from our speech tokens. The results, shown in Table 2 demonstrate that the leap in noise robustness does not compromise the tokenizer's fundamental quality. StableToken delivers sfate-of of-e-of-thhe-r the-art reconstruction performance, evidenced by its exceptional Word Error Rate (WER) and Mean Opinion Scores (MOS). These results validate

---

Published as a conference paper at ICLR 2026
StableTokkn as versatile tokenizer that excels in both resilience and fidelity. Details on the audio reconstruction setup are provided in Appendix B.5
4.2 DOWNSTREAM SPEECHLLM PERFORMANCE
The ultimate measure of tokenizer's utility is its impact on downstream tasks. We find that StableToken''s 's intrinsic robustness consistently translates to superior performance in ASR, SER, and TTS, especially in challenging, noisy conditions.
ASR WER vs. SNR: Synthetic Noise
ASR WER vs. SNR: Real Noise
ASR WER vs. SNR: Real (OOD) Noise
35
Casyleim??
Conylaino)
ASR WER (00
66-4 Sek--Vore-Tkeriie
SLM www.-eeeeeoeeebeee
Statkuiiiin Jouns)
Wour tears)
Stapkcheer buind
XSR WER
ASR WER
10
343
3,47
SNR (c
SKR (dh)
SNR (ch))
SER Accuracy vs. SNR: Synthetic Noise
SER Accuracy vs. SNR: Real Noise
SER Accuracy vs. SNR: Real (OOD) Noise
Accurccy 140
Accuraiyy
Caelenn B.M Uau
01gea
SMR
Orgin
SMR
Orphy 25
SMi
Figure 3: Performance of downstream SpeechLLMs under various noise conditions and SNR levels. (Top Row) ASR performance, measured in Word Error Rate (WER, 4). (Bottom Row) SER performance, measured in Accuracy (t). In both tasks, StableToken consistently demonstrates superior robustness, with the performance gap widening as noise severity increases.
4.2.1 ROBUST ASR PERFORMANCE
StableToken significantly contributes to robust downstream ASR model. Figure 3 shows that while all systems perform comparably on clean audio, the performance gap widens dramatically as noise increases. Under the most severe OOD real-world noise at OdB SNR, the model with StableToken achieves a WER of 20.34%, a relative reduction of over 30% compared to the baseline's 29.94%.
This robustness generalizes to complex acoustic scenes. On the CHiME-4 (Vincent et al. 2017) benchmark (Table the StableToken- based system achieves WERs of 35.909 50%% (test-real set) and 30.61% % (test-simulated set), marking relative reductions of approximately 30% over the next-best baseline. This confirms that token-level stability is a direct driver of downstream model resilience.
4.2.2 ROBUST SER PERFORMANCE
Figure 3 shows that the HtableToken-baseee model consistently achieves higher classification accuracy across all noise types and levels. While performance is similar across all tokenizers on clean audio, the Stable model's accuracy degrades much more slowly as noise increases, demonstrating greater robustness in isolating emotional cues from corrupted audio.
4.2.3 SUPERIOR TTS PERFORMANCE
As shown in Table 3 StableToken delivers superior TTS performance, with significantly lower WER on both subsets, as well as an improved MOS on SEED-TTS>H and competitive performance on SEED-TTSEN- The reduction in WER confirms that our tokenizer enables more intelligible speech synthesis that faithfully reproduces the intended text. Concurrently, the MOS score demonstrates high naturalness and auditory quality of the synthetic speech. Together, these results strongly support Stable an effective and information-rich representation for speech synthesis.
7

---

Published as a conference paper at ICLR 2026
Table Downstream SpeechLLMs performance comparison on ASR (CHiME-4) and TTS (SEED- TTS) benchmarks. In both tasks, integrating StableTokken into the downstream SpeechLLM leads to substantially improved performance, demonstrating its noise robustness and versatility.
ASR
TTS
Tokenizer
LLM-base
Dev Set
Test Set
SEED-TTSEN
SEED-TTSZH
Real
Sim.
Real
Sim.
WER + MOS
WER + MOS
CosyVoier
Qwen2.5-3B
38.66
40.82
54.63
47.71
7.80
3.52
8.73
3.47
CosyVoices Qwen2.5-3B
43.91
48.39
59.83
55.01
7.22
3.75
9.89
3.37
LLM---Voice
Qwen2.5-3B
36.92
36.38
51.08
43.09
6.19
4.19
5.26
3.85
StableTooken
Qwen2.5-3B
25.56
25.36
35.90
30.61
4.43
4.12
3.02
4.08
In summary, across understanding (ASR, SER) and generation (TTS), StableTToken consistently enables stronger, more reliable downstream performance. This dual advantage affirms its effectiveness powerful, versatile foundation for real-world speech systems.
4.3 ANALYSIS
4.3.1 COMPONENT ABLATION STUDY
Table Sequential ablation study of StableToken. We jointly evaluate tokenizer robustness (UED
J) and semantic preservation (ASR WER U). Results show that each component contributes to robustness, with the full model providing optimal stability and semantic fidelity. ASR measured on the validation set during tokenizer training.
Tokenizer Robustness (UED% 4)
ASR (WER% !)
Gauss. Noise
Brown Noise
Pink Noise
Bit Crush
Real
Real
LS-
LS-
Model Configuration
Noise
(OOD)
Clean
Other
StableToken (Full)
12.93
9.76
9.37
7.32
10.65
10.96
2.03
4.68
w/o Consensus Loss
24.80
19.06
17.81
14.03
16.97
17.43
2.03
4.88
w/o Noise-Aware Training
30.77
23.05
21.30
17.32
20.95
21.51
2.19
5.52
w/o Multi-Branch
34.53
25.44
24.58
19.83
23.68
24.47
2.39
5.85
Our sequential ablation study, presented in Table confirms that each component of StableToken is critical for its performance. First, removing the Consensus Loss causes the significant degradation in token robustness (e.g., UED Real OOD noise increases from 10.96% to 17.43%), which under- scores the importance of enforcing explicit agreement between branches. Subsequently, removing the Noise-Aware Training further harms performance, particularly the preservation of semantic content (WER on LibriSpeech-Otte baseline results the poorest performance overall. This highlights the Multi-Branch architecture' dual role: it is the structural enabler for our training strategy and acts as an effective ensemble at inference, mitigating quantization errors common in single-path designs (Ma al. 2025a).
increases from 4.88% to 5.52%).. Finally, reverting to a single-branch
4.3.2 ANALYSIS VOTER COUNT (N)
To determine the optimal number of voters that best balances performance and computational cost, we conduct preliminary training runs for each configuration. The results in Table 5 show clear trend: increasing N' from 5 yields substantial improvements both robustness and semantic preservation. However, further increase to N 7 offers only marginal gains that do not justify the added computational overhead. We therefore select N as the optimal configuration for all experiments. Analysis of parameters and FLOPss for different N can be found in Appendix B.6
8

---

Published as a conference paper at ICLR 2026
Table 5; Impact of the number of voters (N) on tokenizer robustness and semantic preservation.
Tokenizer Robustness
(UED% 4
ASR (WER % 4
Gauss. Noise
Brown Noise
Pink Noisc
Bit Crush
Real Noise
Real
LS- Clean
LS-
Number of Voters (N)
(OOD)
Other
N = 3
20.66
15.42
14.44
11.55
14.89
15.27
2.24
5,47
N = 5
18.68
13.87
13.11
10.50
14.06
14.49
2.22
5.38
N 7
18.10
13.35
12.51
9.84
13.79
14.11
2.36
5.52
Table Case study on error correction via bit-wise voting.
Token @ Pos. 68
Token @ Pos. 80
Token @ Pos. 105
Token @ Pos. 114
Output Source
(Vote Bit #4)
(Vote on #5, #7)
(Vote #3)
(Vote Bir #2,
5517
3485
2920
6939
Clean Reference
10001101
10011111111
,,.01101000
...0001 1011
5533
3485
2920
6939
Voter (Noisy)
1001 1101
10011101
101000
.,,0001 1011
5517
3517
2912
'9433"
Voter 2 (Noisy)
10001101
101111011
..001000000
1,,00011111
5517
3517
2920
6939
Voter 3 (Noisy)
10001101
1111111111
...01001000
0,00001 1011
5517""
3485
"9220"
"7003"
Voter 4 (Noisy)
10001101
100111011
1,101001000
001011011
5533
3357
2920
6939
Voter 5 (Noisy)
10011101
...00011101
1..01001000
...0001 1011
3485
6939
5517
2920
Final Voted Output
Bit #5: 3 vs 2 +
#2: vs 0
Bit #4: 3 vs
4 -
Bit #7: 4 1
Bit #6: vs. -
4.3.3 CASE STUDY
provids case study illustrating the error correction capability of the voting-LFQ module. For instance, at position 80, noise causes three voters to generate erroneous tokens. Specifically, Voters 2 and 3 flip bit #5, while Voter 5 flips bit #7. Despite most voters predicting incorrect tokens, the voting mechanism operating at the bit level allows for correct recovery. For bit #5, the correct value '0' wins by a 3-to-2 majority, and for bit #7, the correct value '1' wins by a 4-to-1 majority, successfully reconstructing the original token (3485). Similar corrections occur positions 68, 105 and 1 114. This case study highlights a key advantage of StableToken: its resilience does not depend on every branch being perfect, but on the collective ability override sparse bit-flip errors.
RELATED WORK
Semantic Speech Tokenizers The evolution of LLMs has driven the transition of spoken dialogue
models from traditional pipelines to end-to-end SpeechLLMs (Zhang & Wang. 2019
Zhang et al.
2020: Jacgmin et al., 2022 Lee al.) [202T] Fang et al. 2024, Défossez al., 2024 Wang al.
2024.. with semantic tokenizers becoming increasingly crucial. The design of semantic tokenizers
has evolved through several distinct paradigms. Early approaches utilized self-supervised learning
(SSL) to derive discrete units from unlabeled data (Hsu et al. 2021 Baevski et al. 2020) Chen et al.
2022 Chung et al. 2021 Conneau et al. 202 Chiu et al. 2022 Baevski al. 2019 Liu el al. 20233
Gat et al. 2023 Huang al. 2022 Lodagala et al. 2023 Chang et al. 2023 The vast majority of tokens produced by these methods are designed for discriminative tasks. is reported that discretized
SSL tokens primarily encode phonetic information, causing high Gross Pitch Error (GPE) when paired with a vocoder for audio generation, making them unsuitable for end-to-end SpeechLLMs
(Sicherman & Adi 2023 Polyak et al. 2021 Mousavi et al. 2024
Guo et al. 2025).

---

Published as a conference paper at ICLR 2026
A second category employs hybrid approach, enhancing acoustic tokenizer with semantic distillation to balance acoustic fidelity and semantic content (Zhang et al. 2023 Ye et al. [025] Détossez et al. 2024; Siahkoohi et al. 2022; Yang et al. 2024b). This design enables strong performance on both generative and discriminative tasks. However, their integration with downstream large language models (LLMSs) is hampered by several significant challenges. First, to preserve high fidelity, these methods tend to encode excessive acoustic details, which results in a high bits-per- second (BPS) rate. This high data rate generates longer token sequences, thereby increasing the computational load and impairing training efficiency. Furthermore, their reliance on Residual Vector Quantization (RVQ) produces hierarchical tokens that are inherently incompatible with the flat input structure expected by most LLMs. Collectively, the high data rate, the structural mismatch, and the overhead of processing superfluous acoustic information present substantial obstacles to their application in modern SpeechI.LMss
More recently, third and more direct paradigm has gained traction: fully supervised training. Given that the primary goal is to capture semantic and phonetic information, this approach directly uses an Automatic Speech Recognition (ASR) objective for supervision. The process involves quantizing the intermediate representations of powerful ASR encoder and optimizing the model with an ASR loss, ensuring the resulting discrete tokens directly represent linguistic units (Zeng et al.. 2025 Du et al. 2024a b). Subsequently, downstream vocoder is trained to convert these discrete tokens into mel-spectrograms for speech synthesis. This tokenizer design is foundational to the current state-of- the-art end-to-end SpeechLLMs, underscoring its effectiveness and growing adoption. Interestingly, research has revealed that while the ASR objective targets linguistic content, the resulting tokens retain sufficient extra-phonetic information, such as prosody. This is likely because the ASR encoder implicitly learns to model prosodic features as they serve as valuable auxiliary cues for achieving high transcription accuracy. This retained information allows an integrated LLM to generate highly expressive synthesis and convey complex emotions. Consequently, this design's gs ability to support expressive generation has made it foundational choice for state-of-the-art SpeechLLMMs (Zeng et al., 2024
Ding et al., 2025).
Noise Robustness Ensuring the stability of discrete speech tokens in the presence of noise is critical for the performance of modern Speech Language Models (SLMs). However, this issue has been
largely overlooked compared to the extensive research focused on improving the robustness of the
Automatic Speech Recognition (ASR) model itself (Wang et al. 2022 Tjandra al. 2023 Lickhoff
et al. 2023] Gong et al. 2023 Ahn et al., (225). ). Recently, two studies have begun to address this gap by investigating the noise robustness of traditional SSL-based speech tokenizers.
R-SPIN (Chang & Glass 2024) addresses this by learning speaker- and noise-invariant discrete units through data-efficient sulf-supervised framework. extends the speaker-invarian clustering of view of the input and an auxiliary loss that predicts "acoustic pieces," which are mhoneme--liigned pseudo-labels, to prevent model collapse and ensure the resulting discrete units represent pure linguistic content In contrast, NAST (Messica & Adi 2024) proposes an architecture designed explicitly for robust tokenization, consisting of predictor, a residual encoder, and a decoder. Its training is governed by combination of reconstruction loss, diversity loss to encourage codebook usage, and a crucial robustness loss that penalizes changes in the predicted token distribution between clean and nogse-adgmented versions of the same utterance, thereby directly optimizing for token-level stability. Liu et al. (2025) introduce dlice-consistency and perturbation-consiitenne constraints to mitigate discrete representation inconsistency, but their approach targets acoustic tokenizers (rather than semantic tokenizers), which prioritize audio detail reconstruction. Therefore, noise invariance is less meaningful in their context, making their work fundamentally different from ours.
Spin by using an additional
6 CONCLUSION
We introduce StableToken, a novel tokenizer designed to solve the critical instability of existing semantic tokenizers in noisy environments. By employing multi-branch architecture and consensus mechanism with bitwise voting, StableToken achieves state-of-the the-trrrt token stability. This stability directly translates to significant improvements in the robustness of downstream SpeechLLMs
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China (62576010). The corresponding authors are Linhao Zhang and Houfeng Wang.
REPRODUCIBILITY STATEMENT
To facilitate reproducibility of our work, we have provided detailed descriptions of the datasets, hyperparameters, and other experimental details used in our study in Section [ and Appendix B Our code and model checkpoint will be released publicly upon acceptance to further support reproducibility and foster future research.
REFERENCES
Adaeze Adigwe, Noé Tits, Kevin EI Haddad, Sarah Ostadabbas, and Thierry Dutoit. The emotional voices database: Towards controlling the emotion dimension in voice generation systems. arXiv preprint arXiv:1806.099 09514, 2018.
Hyebin Ahn, Kangwook Jang, and Hoirin Kim. Hubert-vic: Improving noise-robust automatic speech recognition speech foundation model via varianc--ivarranc--covraan regularization. arXiv preprint arXiv:2508. /2292, 2025.
Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint 2xXiv:2006.024300 2024.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: massively- multilingual speech corpus. arXiv preprint 1t.Xiv:112.0066611 2019.
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vee: Self-supervised learning of discrete speech representations. arXiv preprint 11Xiv:1910.05453, 2019.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for supervipee learning of speech representations. Advances in Neural Information Processing Systems, 13:12449-124444 12460, 2020.
Evelina Bakhturina, Vitaly Lavrukhin, Boris Ginsburg, and Yang Zhang. Hi-fi multi-speaker english ItS dataset. arXiv preprint arXiv:2104.01497, 2021.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 3432, 2013.
Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Aishell-1: An open-source mandarin speech corpus and speech recognition baseline. In 2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech I/O systems and assessment (O-COCOSDA), pp. 1-5. IEEE, 2017.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335-359, 2008
Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: coordes emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-390, 2014.
Heng-Jui Chang and James Glass. R-Spin: Efficient Speaker and Noise-invariat Representation
Learning with Acoustic Pieces. In Proceedings the 2024 Conference of the North American Chapter the the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 642-662. Association for Computational Linguistics, 2024.
11

---

Published as a conference paper at ICLR 2026
Heng-Jui Chang, Alexander H. Liu, and James Glass. Self-supervised fine-tuning for improved content representations by speaker-invartn clustering. In Proc. Interspeech 2023, pp. 2983-2987, 2023.
Guoguo Chen, Shuzhou Chai, Guanbo Wang. Jiayu Du, Wei- Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv: 106.06909, 2021.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujic Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6): 1518-1518, 2022.
Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random- projeccioonon quantizer for speech recognition. International Conference on Machine PMLR, 2022.
Learning, pp.
Yu-An Chung, Yu Zhang, Wei Han, Chung- Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for sulf--uue speech pre-training. In 202/ IEFE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 244-2500 250. IEEE, 2021.
Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross-lingual representation learning for speech recognition. 2021.
Alexis Conneau, Min Ma. Simran Khanuja, Yu Zhang, Vera Axelrod. Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Flcurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 798 805. IEEE, 2023.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00033 2024.
Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report. arXiv preprint arXiv:2504.1 18425, 2025.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang. Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint 2rXiv:2407.05... 2024a.
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.1011 2024b.
Kate Dupuis and M Kathleen Pichora-Fuller. Toronto emotional speech set (tess). 2010.
Patrick Eickhoff, Matthias Möller, Theresa Pekarek Rosin, Johannes Twiefel, and Stefan Wermter. Bring the Noise: Introducing Noise Robustness to Pretrained Automatic Speech Recognition. In Artificial Neural Networks and Machine Learning - ICANN 2023, pp. 381-392. Springer Nature Switzerland, 2023.
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. arXiv preprint 24Xv:24099066666 2024.
Qingkai Fang, Yan Zhou, Shoutao Guo, Shaolei Zhang, and Yang Feng. Llama-omni2: Lim- based real-time spoken chatbot with autoregressive streaming speech synthesis. arXiv preprint NXXi:2505500625. 2025.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: open dataset of Jubelel sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829-8833 852, 2021.
12

---

Published as a conference paper at ICLR 2026
Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón, Keith Achom,, Anjali Gopi, David Kanter, Maximilian Lam, Mark Mazumder, and Vijay Janapa Reddi. The people's 's speech: A large-seale diverse english speech recognition dataset for commercial usage. arXiv preprint ar arX1v:2111.0934,, 2021.
Itai Gat, Felix Kreuk, Tu-Anh Nguyen, Ann Lee, Jade Copet, Gabriel Synnaeve, Emmanuel Dupoux, and Yossi Adi. Augmentation invariant discrete representation for generative spoken language modeling. In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pp. 465-477, 2023.
Yuan Gong, Sameer Khurana, Leonid Karlinsky, and James Glass. Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers. In Proc. Interspeech 2023, pp. 2358-2362, 2023. doi: 10.24437lmlerspechh.2233 2013-1111.
Yiwei Guo, Zhihan Li, Hankun Wang. Bohan Li, Chongtian Shao, Hanglei Zhang, Chenpeng Du, Xie Chen, Shujie Liu, and Kai Yu. Recent advances in discrete speech tokens: A review. arXiv preprint arXiv:2502.064444 064.0, 2025.
Haorui He, Zengqiang Shang. Chaoren Wang. Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In 2024 IEEE Spoken Language Technology Workshop (SLT), pp. 885-890. IEEE, 2024.
Wei-Ning Hsu, Benjamin Bolte. Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. /EEE/ACM transactions on audio, speech, and language processing, 29:3451-3460, 2021.
Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, et al. Step-audio: Unified understanding and generation in intelligent speech interaction. arXiv preprint wwXiv25502.1146. 2025.
Wenyong Huang, Zhenhe Zhang, Yu Ting Yeung, Xin Jiang, and Qun Liu. Spiral: Self-supervised perturbation-invaaiant representation learning for speech pre-training. In ICLR, 2022.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Philip Jackson and SJUSSG Haq. Surrey audio-visual expressed emotion (savee) database. University of Surrey: Guildford, UK, 2014.
Léo Jacqmin, Lina M Rojas- hajas-Barahooa, and Benoit Favre. " "o you follow me?": A survey of recent approaches in dialogue state tracking. arXiv preprint arXiv:2207.14622 2022.
Jesin James, Li Tian, and Catherine Inez Watson. An open source emotional speech corpus for human robot interaction applications. In Interspeech, pp. 2768-2772, 2018
Chengze Jiang, Zhuangzhuang Wang, Minjing Dong, and Jie Gui, Survey of adversarial robustness in multimodal large language models. arXiv preprint arXiv:2503.13362, 2025.
Jacob Kahn, Morgane Riviere, Weiyi Zheng, Exgeny Kharitonov, Qiantong Xu, Fierre-Emmanue Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri-light: A benchmark for asr with limited or supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673. IEEE, 2020.
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 119-132, 2019.
Jungil Kong, Jachyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 13:17222-77033 2020.
13

---

Published as a conference paper at ICLR 2026
Thia-Hsuan Lee, Hao Cheng. and Mari Ostendorf. Dialogue state tracking with a language model using schema-driven prompting. In Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4937-4949 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2022 1/255///1/222.mmm///l. wwwnlmmmain.eeee URL
Moens, Xuanjing Huang, Lucia Specia, and
emnlp main. 4047
Keon Lee, Kyumin Park, and Daeyoung Kim. Dailytalk: Spoken dialogue dataset for conversational text-to-speech. In ICASSP 2023-2023 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Xinjian Li, Shinnosuke Takamichi, Takaaki Sacki, William Chen, Sayaka Shiota, and Shinji Watanabe. Yodas: Youtube-oriented dataset for audio and speech. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1 1-8. IEEE, 2023.
Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mingyu Xu, Kexin Wang, Ke Xu, Yu Hc, Ying Li, Jinming Zhao, et al. Mer 2023: Multi-label learning, modality robustness, and semi-super learning. In Proceedings of the 31st ACM international conference multimedia, pp. 9610-9611 2023.
Alexander H Liu, Heng-Jui Chang, Michael Auli, Wei-Ning Hsu, and Jim Glass. Dinosr: Self- distillation and online clustering for supervied speech representation learning. Advances in Neural Information Processing Systems, 045:58346-5366 2023.
Wenrui Liu, Zhifang Guo, Jin Xu, Yuanjun Lv, Yunfei Chu, Zemin Liu, and Junyang Lin. Analyzing and mitigating inconsistency in discrete speech tokens for neural codec language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2025.
1: Long Papers), pp.
Steven R Livingstone and Frank Russo. The ryerson audio-visual database emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5)ee0196391, 2018.
Vasista Sai Lodagala, Sreyan Ghosh, and Srinivasan Umesh. Ccc-wav2vec 2.0: Clustering aided cross contrastive self-supervised learning speech representations. 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 1-8. IEEE, 2023.
Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiao- juan Qi. Unitok: A unified tokenizer for visual generation and understanding. arXiv preprint arXiv:2502.20321, 2025a.
Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, and Xie Chen. Language model can listen while speaking. Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 24831-24823 2025b.
Shoval Messica and Yossi Adi Nast: Noise aware speech tokenization for speech language models. In Proc. Interspeech 2024, pp. 4179, 2024.
Poonch Mousavi, Luca Della Libera, Jarod Duret, Artem Ploujnikov, Cem Subakan, and Mirco Ravanelli. Dasb-discrete audio and speech benchmark. arXiv preprint 1eXiv:24006.4294 2024.
Tu Anh Nguyen, Wei-Ning Hsu, Antony d' Avirro, Bowen Shi, Itai Gat, Maryam Zazel-Zarani, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, et al. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. arXiv preprint arXiv:2308.05722, 2023.
Kari Ali Noriy, Xiaosong Yang, and Jian Jun Zhang. Emns/mmz/corpus: An emotive single- speaker dataset for narrative storytelling in games, television and graphic novels. arXiv preprint arXiv:2305. 13137. 2023.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 2206-5210. IEFE, 2015.
14

---

Published as a conference paper at ICLR 2026
Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015-1018, 2015.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprini arXiv:2104.00355, 2021.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv: 2018.
Vincel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411, 2020.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcl Leavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference machine learning, pp. 28492-28511, PMLR, 2023.
Ali Siahkoohi, Michael Chinen, Tom Denton, W Bastiaan Kleijn, and Jan Skoglund. Ultra-low-iirate speech coding with pretrained transformers. In Proc. Interspeech 2022, pp. 4421-4425, 2022.
Amitay Sicherman and Yossi Adi. Analysing discrete self supervised speech representation for spoken language modeling. In /CASSP, 2023.
Fengping Tian, Chenyang Lyu, Xuanfan Ni, Haoqin Sun, Qingjuan Li, Zhiqiang Qian, Haijun Li, Longyue Wang, Zhao Xu, Weihua Luo, et al. Marco-voice technical report. arXiv preprint arXiv:2508.000 020388, 2025.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. deHuBERT: Disentangling Noise in Self-supervised Model for Robust Speech Recognition. arXiv preprint arXiv:2302. 2023,
Aaron Van Den Oord, Oriol Vinyals. et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Christophe Veaux, Junichi Yamagishi, and Kirsten MacDonald. Cstr vetk corpus: English multi- speaker corpus for estr voice cloning toolkit. 2017.
Emmanuel Vincent, Shinji Watanabe, Aditya Arie Nugraha, Jon Barker, and Ricard Marxer. An analysis of environment, microphone and data simulation mismatches in robust speech recognition. Computer Speech & Language, 46:535-557, 2017.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.003900 2021.
Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking-face generation. In European conference computer vision, pp. 700-717 Springer, 2020.
Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen Ilm. arXiv 2024.
preprint arXiv:2411.000
Yiming Wang, Jinyu Li, Heming Wang, Yao Qian, Chengyi Wang, and Yu Wu. Wav2vec-Swin Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition. In ICASSP 2022 2022 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp.
IEEE, 2022.
Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, et al. Step-audio 2 technical report. arXiv preprint urXiv:2507.16632 2025.
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint a40i 16725, 2024.
15

---

Published as a conference paper at ICLR 2026
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint mIX1::2412.15115. 2024a.
Dongchao Yang, Haohan Guo, Yuanyuan Wang, Rongjie Huang, Xiang Li, Xu Tan, Xixin Wu, and Helen Meng. Uniaudio 1.5: Large language model-driven audio codec is a few-shot audio task learner. Advances in Neural Information Processing Systems, 37:56802-5666
2024b.
Wanqi Yang, Yanda Li, Meng Fang, Yunchao Wei, Tianyi Zhou, and Ling Chen. Who can with- stand chat-audio attacks? an evaluation benchmark for large language models. arXiv preprint arXiv:2411. 14842, 2024c.
Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, al. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 25697-25705, 2025.
Lijun Yu, José Lezama, Nitesh Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer key to visual generation. arXiv preprint 2rXiv:2310.05737, 2023.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv: 02882, 2019.
Aohan Zeng. Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02611 2024.
Aohan Zeng. Zhengxiao Du, Mingdao Liu, Lei Zhang, Yuxiao Dong, Jie Tang, et al. Scaling speech- text pre-training with synthetic interleaved data. In The Thirteenth International Conference on Learning Representations, 2025.
Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, et al. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition. In ICASSP 2022-2023 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 6182-6186. IEEE, 2022.
Jian Zhang, Linhao Zhang, Bokai Lei, Chuhan Wu, Wei Jia, and Xiao Zhou. Wildspeech--enchh Benchmarking audio lims in natural speech conversation. arXiv preprint arXiv:2506.22 2 1875, 2025.
JTFLM Zhang and Huibin Jia, Design of speech corpus for mandarin text to speech. In The blizzard challenge 2008 workshop, 2008.
Linhao Zhang and Houfeng Wang. Using bidirectional transformer-crf for spoken language under- standing. In CCF international conference on natural language processing and chinese computing, pp. 131- Springer, 2019.
Linhao Zhang, Dehong Ma, Xiaodong Zhang, Xiaohui Yan, and Houfeng Wang. Graph Istm with context-gated mechanism for spoken language understanding. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 9539-9546, 2020.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308. 16692,
Jinming Zhao, Tenggan Zhang, Jingwen Hu, Yuchen Liu, Qin Jin, Xinchao Wang, and Haizhou
Li. MBed: Multi-modal multi-scene multi-label emotional dialogue database. arXiv preprint arXiv:2205.10237 2022.
Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. Emotional voice conversion: Theory, databases and esd. Speech Communication, 137:1-18, 2022.
16

---

Published as a conference paper at ICLR 2026
A LARGE LANGUAGE MODEL (LLM) USAGE STATEMENT
In accordance with the conference policies on Large Language Model (LLM) usage, we hereby disclose the following: After completing the initial draft of this paper. we utilized an LLM to enhance grammar and polish the writing of this manuscript. No new research ideas, experimental designs, or scientific content were generated by the LLM. All scientific contributions, analyses, and conclusions presented in this work are solely those the authors. We take full responsibility for the content of this paper, including all sections that have been revised improved with LLM assistance. The LLM not an author and did not contribute to the research ideation or substantive scientific writing.
This statement is provided to ensure transparency and compliance with the conference's ' policies LLM usage.
B DETAILS OF STABLETOKEN
B. TRAINING DATASETS FOR STABLETOKEN
We train our StableTooken model on hundreds of thousands of hours of both open-source data and in-house data. All open-source datasets used in this work are listed in Table 7
Table 7: Summary of datasets used for training StakleToken
Dataset
Duration (#hours)
Task Language(s)
LibriSpeech Panayotov et al.
2015
960
ASR English
Multilingual ibriSpeech (Pratap et al.
2020
27,322
ASR English
The People's s Speech (Galvez al. 2021
5,568
ASR English
GigaSpeech (Chen et al. 2021
10,000 ASR English
Yodas (Li al. (2023)
29,155 ASR English
Hi-Fi TIS (Bakhturina et al. 2021
292 ASR English
VCTK /Veaux al.
2017)
44
ASR English
LibriTTSs (Zen et al.
2019)
586
ASR English
Soitt (Xie & Wu
2024
679
ASR English
AISHELL--1 (Bu et al. 2017)
150 ASR Chinese
WenetSpeech (Zhang et al. 2022
10,005 ASR Chinese
Common Voice (Ardila et 2019
2,133
ASR English, Chinese
Emilia (He et al. 2024)
96,750 ASR English, Chinese
B.2 TRAINING HYPERPARAMETERS FOR STABLETOKEN
Table summariies the main hyperparameters used throughout StableTooken training.
Table 8: Hyperparameters used for training StableToken
Hyperparameter
Value
optimizer type
Adam W
Ir sr_schedller
OneCycleLR
max Ir warmup_stte
1.5e-5
1000
weight_decay
0.01
grad_clip
1.0
consensus loss weight (A1)
0.25
commitment loss weight (A2)
0.25
codebook
mndehook_entropy_hhee eeteeenettyyooooe losss__igs____hhht (As)
1.0
17

---

Published as a conference paper at ICLR 2026
B.3 DETAILS OF STOCHASTIC PERTURBATIONS DURING TRAINING
We detail the construction and parameterization of the stochastic augmentation function A(-) as introduced in Section 2.3 For each sample, one perturbation type is randomly selected from the following five categories: Gaussian Noise, Pink Noise, Brown Noise, Bit Crush Distortion, and Real-world Noise. The intensity of the selected perturbation is then uniformly sampled from predefined range specific to each type, as summarized in Table For real-world noise, an additional random selection of noise audio clip is performed. The pool of noise clips consists of samples from the AudioCaps (Kim et al. 2019). FSD50k (Fonseca et al.) (02T), and ESC-50 50 (Piczak) 2015) datasets. Notably, the ESC-10 subset ESC-50 excluded from training and reserved exclusively for evaluation out-of-domain real-world noise.
Table 9: Perturbation types and their corresponding intensity ranges utilized during training.
Perturbation Type
Intensity Range
Gaussian Noise
16 S SNR I 30
Pink Noise
16 S SNR - 24
Brown Noise
12 < SNR < 24
Bit Crush Distortion
8 < Bit Depth S 14
Real-world Noise
12 < SNR S 24
B.4 DISCUSSION OF CONSENSUS OBJECTIVE Loss CHOICES
We choose the L2 loss (Mean Squared Error Loss) for the consensus objective in Eq. due to several considerations:
First, Le loss offers simple and direct way to minimize the differences among multiple branches. At the same time, its form is also naturally compatible with the existing commitment loss in quantization- based models, thus facilitating stable optimization and consistent gradient behavior across objectives. Furthermore, by averaging representations from all branches, the resulting target inherently incorpo- rates a form of confidence weighting, outlier branch results are diluted in the mean.
Second, since all branches originate from the same underlying input, with only noise perturbations in minority branches, the goal is not to increase inter-class separation in contrastive learning, but to enforce similarity across the noisy and clean versions. The L2 loss directly minimizes the Euclidean distance between the branches and their consensus, effectively encouraging robust invariance without introducing additional factors or requiring extra sampling. Morcover, cosine similarity is less sensitive to the number of bit flips in high-dimensional binary codes, especially when representations already close, such as the flip of a single bit, which corresponds to small changes angle and often results in diminished gradient signals and less effective correction of localized errors.
B.5 AUDIO RECONSTRUCTION DETAILS
Table 10: Summary of datasets used for training the flow matching model
Dataset
Language(s)
Librilight (Kahn et al. 2020)
English
WenetSpeech (Zhang et al. 2022
Chinese
Yodas2 (Li et al. 2023)
English, Chinese
Emilia He et 2024
English, Chinese
Following the framework of Cosy Voice (Du et al. 2024a) and CLM-4-Voice (Zeng et al. 2025). we train a flow matching model to reconstruct audio from speech tokens. The model takes as input the speech token representations and produces Mel spectrograms. Finally, a HiFi-GAN vocoder (Kong let al., (2200) converts the generated Mel spectrograms into the speech waveforms, The datasets for traning the flow matching model is listed in Table [axle|OOeexclui our in-house datasets, which comprise both English and Chinese speech data), and the training hyperparameters in Table 11
18

---

Published as a conference paper at ICLR 2026
Table 11: Hyperparameters used for training the flow matching model
Hyperparameter
Value
optimizer_ty type Ir schedulerr learning rate rate
Adam
WarmupLR
3.0e-4
warmup_ster steps
25000
grad_clipp batch_type
1.5
dynamic
max_ max_frames_in satss__ii__heell
10000
B.6 ANALYSIS ON COMPUTATIONAL EFFICIENCY
B.6.1 COMPUTATIONAL OVERHEAD OF DIFFERENT VOTER COUNTS
To further explore the computational overhead brought by increasing the number of voters (N), we present the model parameter counts and floating-poin operations (FLOPs) for models with N = 1, 3, 5, and 7 in Table 12 Both metrics are measured using the THOP library For FLOPs calculation, we an input audio with a duration of 30 seconds.
The results show that as N increases, the model parameters increase linearly, but the increment between adjacent N values is relatively small (about 0.033M parameters). The FLOPs for different N are also very close, indicating that enlarging cost. This suggests that increasing the number of voters achieves better performance and robustness with only minimal impact on model size and inference efficiency.
does not introduce significant extra computational
Table 12: Model parameters and inference FL.OPs for different voter counts N on 30-second input audio.
Number Voters (N)
#Parameters (M)
#FLOPs (G)
N
320.261
480.978
N 3
320.294 (t 0.010%)
481.003 (f 0.005%)
N
320.328 (1 (10211) 320.361 (1 0.031%)
481.028 (1 0.010%)
N
481.053 (1 0.016%)
B.6.2 EMPIRICAL. INFERENCE EFFICIENCY BENCHMARK
From an architectural standpoint, the overhead introduced by the multi-branch design is negligible. The core design of StableToken confines the multi-branch architecture to the lightweight quantization stage only, while the computationally expensive Transformer Encoder remains a singlc-path process. Since the parallel quantization branches can be executed concurrently with high efficiency on modern hardware, this design introduces almost no additional end-to-end latency compared to conventional single-path tokenizers.
To rigorously evaluate the practical inference efficiency of StableToken, we conducted a comprehen- sive benchmark test measuring latency, Real-Time Factor (RTF), throughput, and memory footprint. Voicc-ooooo-ieiieee (Zeng al. 2025).
We compared StableToken against strong bascline. under identical hardware conditions using both an NVIDIA H20 GPU and an AMD EPYC 9K84 96-Core Processor.
The tests covered various batch sizes and audio durations. The detailed results are summarized in Table B (GPU) and Table 14] (CPU).
Based on these results, we observe the following: (1) Latency, RTF, and Throughput: The empirical data aligns with our theoretical analysis. The end-to-end latency and RTF of StaencToken nearly identical to the baseline, showing slight advantage at larger batch sizes (e.g., 2.0% faster at a batch
19

---

Published as a conference paper at ICLR 2026
Table 13: Inference Efficiency and Resource Usage on single NVIDIA H20 GPU.
Batch Size
GLM-4-Voice StableToken (25 tokens/s)
Duration (s)
Metric
Difference
(12.5 tokens/s)
Latency (ms)
9.11
9.00
+1.22%
RTF
0,0091
0.0090
+1.22%
Throughput (s/s)
109.77
111.11
-1.21%
Memory (MB)
1330
1538
-13.5%
Latency (ms)
63.17
63.19
-0.03%
RTF
0.0021
0.0021
-0.03%
30
Throughput (s/s)
474.92
474.79
+0.03%
Memory (MB)
1693
1853
-8.6%
Latency (ms)
235.96
239.57
-1.5%
RTF
0.0015
0.0015
-1.5%
16
10
Throughput (s/s)
678.08
667.87
+1.5%
Memory (MB)
2173
2328
-6.7%
Latency (ms)
1622.86
1656.49
-2.0%
RTF
0.0008
0.0009
-2.0%
32
60
Throughput (s/s)
1183.10
1159.08
+2.1
Memory (MB)
13978
14230
-1.8%
Table 14: Inference Efficiency and Resource Usage on an AMD EPYC 9K84 96-Core Processor.
Batch Size
StableToker GLM-4-Voice
Duration (s)
Metric
Difference (%
(25 tokens/s)
(12.5 tokens/s)
Latency (ms)
104.27
116.62
-10.6% %
RTF
0.1043
0.1166
-1066% %
Throughput (s/s)
9.59
8.57
+11.9
Memory (MB)
2349
2727
-13.9% %
Latency (ms)
1790.86
1809.86
-1.0%
RTF
0,0597
0.0603
-1.0%
30
Throughput (s/s)
16.75
16.58
+1.0%
Memory (MB)
2357
2739
-13.9%
Latency (ms)
5287.85
5687.53
-7.0%
RTF
0.0330
0.0355
-7.0% %
10
Throughput (s/s)
30.26
28.13
+7.6%
Memory (MB)
2566
2982
-1400%
Latency (ms)
52021.89
55549.44
-6.4%
RTF
0.0271
0.0289
-6.4% %
32
60
Throughput (s/s)
36.91
34.56
+6.8%
Memory (MB)
2957
3117
-5.1% %
size of on GPU). Throughput matches or slightly exceeds the baseline across tested scenarios. (2) Memory Footprint: StableToken consistently consumes less memory than the baseline, with savings of up to 13.5% on GPU and 14.0% on CPU. We attribute this efficiency to the use of Lookup- Free Quantization (LFQ), which is inherently more resource-efficient than the Vector Quantization (VQ) employed by the baseline.
In summary, the benchmarks demonstrate that StableToken achieves improved token stability without compromising inference efficiency. Its latency and throughput are competitive with strong baselines, while offering lower memory footprint.
20

---

Published as a conference paper at ICLR 2026
B.7 ANALYSIS TOKEN DISTRIBUTION AND CROSS-LINGUAL EFFICIENCY
In this section, we investigate the fundamental properties of the token distribution generated by our multi-branch architecture. Specifically, we analyze the entropy, vocabulary efficiency, and language-specifcit of the learned representations. We conduct statistical analysis on the Chinese and English token distributions to verify whether the architecture effectively learns distinet phonetic representations without introducing significant bias or inefficiency.
Our analysis focuses two main aspects: (1) Evaluating the efficiency vocabulary space utilization for individual languages. (2) Assessing the degree of distributional overlap between Chinese and English to confirm the capture of language-specifie features.
B.7.1 VOCABULARY UTILIZATION AND DISTRIBUTION ENTROPY
We first analyze the vocabulary usage StableToken on the LibriSpeech-tes (English) and AISHELL (Chinese) datasets. As shown in Table 15 both languages utilize substantial portion of the available vocabulary. The high entropy values observed for both languages indicate that the token distributions rich and dispersed, rather than collapsing into a small subset of tokens. This suggests that the tokenizer maintains high representational capacity for both languages.
Table 15: Vocabulary Utilization Statistics on LibriSpeech and AISHELL
Metric
English
Chinese
Total Tokens
969.440 8,139 (99.35%)
906,250 7,565 (92.35%)
Unique Tokens Used
Token Entropy
12.43
11.68
B.7.2 CROSS-LINGUAL TOKEN SPECIFICITY
To assess the language specificity of the learned codebook, we examine the overlap of high-frequene tokens between languages. We identify the top-frequency tokens for one language and calculate their occurrence frequency in the other.
Tables 16 and 17 present these comparisons. The results indicate that tokens with high frequency in one language are extremely rare or non-existent in the other. This disjointed usage pattern demonstrates that the model has successfully learned specialized. language-specific phonetic units, allocating distinct sub-spaces of the codebook to different languages.
Table 16: Frequency Analysis of Top 5 English Tokens
Token ID
Freq. (English) Freq. (Chinese)
Rank in Chinese
1877
0.4803%
0.0119%
2455
3813
0.3693%
0.00 %
N/A
3809
0.3377%
0.00 %
N/A
3812
0.3090%
0.00 %
N/A
3808
0.2957%
0.0001%
7320
Table 17: Frequency Analysis of Top 5 Chinese Tokens
Token ID Freq. (Chinese)
Freq. (English)
Rank in English
3810
0.7531%
0,0042%
6276
3811
0.7530%
0.0002%
8045
7910
0.7392%
0.0027%
6850
3815
0.6933%
0.0001
8104
3234
0.6815%
0.00019 %
8085
21

---

Published as a conference paper at ICLR 2026
B.7.3 QUANTITATIVE DISTRIBUTIONAL DIVERGENCE
To formally quantify the difference between the token distributions the two languages, we calculate the Kullback-Leibler (KL) Divergence. As shown in Table 18 the significant non-zero KL Divergence values confirm that the model maintains distinct probability distributions for each language.
Table 18: KL Divergence between Language Token Distributions
Metric
Value
KI. Divergence (English
Chinese)
2.81
KL Divergence (Chinese
English) 2.05
Collectively, these analyses demonstrate that the multi-branch voting mechanism robustly allocates specific regions of the codebook different languages, capturing unique phonetic characteristics while maintaining efficient vocabulary utilization.
B.8 LONG AUDIO PROCESSING AND BOUNDARY STABILITY
In this section, we clarify the model's configuration regarding input duration and analyze the stability of tokenization at segment boundaries. This analysis is crucial for understanding the model's behavior in streaming scenarios or when processing long audio via chunking.
B.8.1 MODEL CONFIGURATION AND SEGMENT LENGTH
StakleToken inherits the architectural specifications and processing conventions of its backbone, wwssper-rrrge--e Consequently, the model is configured as follows: (1) Maximum Duration: The tokenizer processes a maximum input audio duration of 30 seconds per inference pass, consistent with the Whisper context window. (2) Training Constraint: During training, all audio segments constrained to this 30-second window. Audio files exceeding this duration in the training dataset were truncated to the first 30 seconds. (3) Inference Strategy: For audio exceeding 30 seconds, StableTToken employs standard chunking strategy. The input is segmented into 30-second (or smaller) chunks which are processed independently. The resulting token sequences are then concatenated form the final output.
B.8.2 ANALYSIS OF BOUNDARY STABILITY
A potential concern with chunk-based processing is whether token stability degrades at the boundaries of audio segments (i.e., the beginning and end a chunk), which could lead to defects when chunks are concatenated together.
To investigate this, we analyzed the distribution of tokenization inconsistencies across different temporal regions of the input audio under Gaussian noise perturbation. We segmented the test audio clips into three regions: (1) Start: The first 15% of the audio duration; (2) Middle: The central 70% of the audio duration; and (3) End: The final 15% of the audio duration. We calculated the Unit Edit
Distance (UED) independently for each region. The results are presented Table 19
Table 19: Distribution of UED across different temporal regions of audio clips under Gaussian noise.
Region
Inconsistencies
Ref. Tokens
UED % of Total Inconsistencies
Start (0-15%)
16,585
119,770
13.85%
16.29%
Middle (15-85%)
69,333
550,069
12.60%
68. 18.111%
End (85-100%)
15,884
116.016
13.69%
15.60%
Overall
101,802
785,855
12.95%
100.00%
The empirical results indicate that UED is consistent across all regions. The UED in the middle region (12.60%) is only marginally lower than that the start (13.85%) and end (13.69%). Furthermore, the
22

---

Published as a conference paper at ICLR 2026
proportion of total inconsistencies occurring in the start (16.29%) and end (15.60%) regions closely aligns with their respective 15% share the audio duration.
This uniformity demonstrates that there is no significant degradation in token stability at the segment boundaries. The consensus mechanism employed by StableToker maintains robustness consistently throughout the audio clip, supporting the effectiveness of the chunking strategy for long audio processing.
C ABLATION STUDY ON ARCHITECTURAL HYPERPARAMETERS
In this section, investigate the impact of two key architectural hyperparameters on model per- formance: the depth at which the quantizer is placed within the encoder, and the ratio of clean to perturbed branches during training. We compare our proposed configuration (L16+3:2 3:2, which means the quantizer is placed at Layer 16 with 3:2 clean-to-noist branch ratio) against three variants: (1) Shallower Quantization (L8 3:2): The quantizer is placed Layer 8; (2) Deeper Quantization (L24 + 3:2): The quantizer is placed at Layer 24; and (3) Reduced Noise Ratio (L16 + 4:1): The ratio of clean to perturbed branches is set 4:1.
the performance of each model checkpoint at 100k training steps across tokenizer robustness (UED), Automatic Speech Recognition (ASR), and Speech Emotion Recognition (SER) tasks.
Table 20: Ablation study on quantizer placement and perturbation ratio. UED is calculated under Gaussian Noise. LibriSpeech WER is reported for test-clean/ test-other. SER is evaluated on the ESD dataset.
Configuration
Robustness
ASR (WER %, L)
SER
(Layer Clean:Noisy)
(UED%, U)
LibriSpeech
WenetSpeech KeSpeech
(Acc%, 1)
L16 + 3:2 (Ours)
18.68
2.22 5.38
10.91
11.00
67.38
L8 + 3:2 22.05
/
11.84
11.24
66.90
L24 3:2
13.65
/
10.16
11.03
59.51
L16 4:1
20.94
2.40/ 5.91
11.00
11.57
62.14
C.1 EFFECT OF QUANTIZER PLACEMENT
The results demonstrate a clear trade-off associated with the quantizer's depth: (1) Deep Quantization (L224): Placing the quantizer the deep layer forces it operate highly abstract, semantic features. This yields the highest tokenizer robustness (lowest UED) and strong ASR performance on challenging datasets like WenetSpeech. However, at this depth, much of the fine-grained acoustic information (e.g., prosody and timbre) has been abstracted away, resulting in a significant degradation in SER performance (59.51% 31% vs. 67.38%). (2) Shallow Quantization (L8):: Operating on shallower features at Layer 8 retains acoustic details but lacks sufficient semantic abstraction. This leads to both poorer tokenizer robustness and degraded ASR performance compared the deeper configurations. (3) Balanced Configuration (L166): Our proposed placement at Layer 16 strikes an effective balance. captures sufficiently robust semantic content for gegh-performnnce ASR while retaining enough acoustic detail to excel in paralinguistic tasks like SER.
C.2 EFFECT OF PERTURBATION RATIO
Comparing the 3:2 and 4:1 clean-to-noisy ratios reveals the importance of the training signal difficulty: The 3:2 ratio results in significantly better tokenizer robustness compared to the 4:1 ratio (18.6 (18.68% vs. 20.94% UED). A higher proportion of perturbed branches creates a more challenging training objective for the consensus mechanism.
This enhanced robustness translates directly to downstream tasks. We hypothesize that the more diverse training signal forces the model to learn representations that truly invariant to perturbations rather than relying on the clean majority. This richer representation benefits both semantic (ASR) and paralinguistic (SER) performance.
23

---

Published as a conference paper at ICLR 2026
In conclusion, these ablation studies validate the design choices of the proposed model. The combina- tion of Layer 16 placement and 3:2 noise ratio optimizes the trade-off between semantic robustness and acoustic feature preservation.
D BASELINE MODELS
The baseline models considered in our study are as follows: (1) HuBERT-500 (Hsu et al. 2021) I.
a self-supervised speech representation model that leverages iterative offline clustering to produce pscudo-labels and employs masked prediction loss; use the official checkpoint with 500 cluster (2) NAST (Messica & Adi 2024), a noise-aware speech tokenization approach comprising a predictor, residual encoder, and decoder, in which the predictor representations of clean speech and augmented speech are explicitly aligned: (3) R-Spin (Chang & Glass) 2024). which enhances the robustness of
speech representations by learning discrete, speaker- and noise-invariant acoustic units through prediction-based training objective; (4) Sokezzoo
Tokenizer (Zhang et al. 2023), which introduces a
hierarchical encoder-decoder framework with residual vector quantization (RVQ) unify semantic
and acoustic tokens; (5) X-Codec Ye et al. 2025). which augments the RVQ backbone with semantic
features from pre-trained semantic encoder and applies a semantic reconstruction loss achieve
higher fidelity in audio generation; (6) Mimi (Défossez et al., 2024), neural audio codec using RVQ to convert audio into discrete tokens, where the first quantization level is distilled to capture semantic information; (7) Cosy Voice (8 Tokenizer) (Du et al. 2024a), which extracts supervised semantic tokens from a multilingual speech recognition encoder for LLM-based TTS, thereby improving content consistency and speaker similarity in voice cloning; (8) Cosy Voice2 (Du et al. 2024b),
which introduces finite-scalar quantization (FSQ) to improve the codebook utilization and updates the model architecture for streaming synthesis capabilities; (9) GLM-4-Voice (Zeng al. 2025), which fine-tunes a pre-trained ASR model by including a pooling layer and a vector quantization layer, producing discrete tokens that strongly preserve semantic information at low frame rates.
E NOISE PROFILES
In Lakenizer-level evaluation, augment the FLEURS (Conneau et al. 2023) benchmark with variety of synthetic perturbations, including Gaussian Noise, Pink Noise, Brown Noise and Bit Crush Distortion, well as real-world noise samples from the ESC-50 (Piczak 2015) dataset. Specifically, the ESC-10 (Piczak) (2013) subset used as out-of-domain (OOD) roal-world noise and is excluded from our Stable Token training pipeline, while the remaining 40 noise categories from ESC-50 incorporated into the training process and thus are considered in-domain real-world noise.
We carefully adjusted the noise level ensure that the added noise does not obscure the semantic content of the original audio and does not affect human perception of the speech. A summary of all perturbation types and their corresponding intensity is provided in Table 21
Table 21: Details of synthetic and real-world perturbations used for noise augmentation.
Perturbation Type
Intensity Value
Gaussian Noise
SNR 25
Pink Noise
SNR = 22
Brown Noise
SNR 16
Bit Crush Distortion
Bit Depth 10
Real- soal-world Noise
SNR = 16
OOD Real-world Noise
SNR = 16
http://////oooooooooooeeeeeeeeee /hubert
24

---

Published as a conference paper at ICLR 2026
F DETAILS OF DOWNSTREAM TASK EVALUATION
F.1 TRAINING DATASETS FOR SPEECHLLMS
In this section, we summarize the speech datasets employed for training SpeechLLMs in Table 22 covering various tasks including Automatic Speech Recognition (ASR), Speech Emotion Recognition (SER), Text-to-Speech (TTS), and Speech Next Token Prediction (SNTP).
Table 22: Summary of datasets used for training SpeechLLMs
Dataset
Task
Language(s)
LibriSpeech (Panayotov et al.
2015
ASR
English
Multi-Lingual Librispeech (Pratap et al.
2020)
ASR
English
TESS (Dupuis & Fichora-Fuller 2010
SER
English
SAVEE (Jackson & Haq. 2014)
SER
English
RAVDESS (Livingstone & Russo 2018
SER
English
MELD Poria et al. 2018)
SER
English
MEAD Wang et al. 20200)
SER
English
Jo--Corpus (James et al.
2018
SER
English
JEMOCAP (Busso al.
2008
SER
English
Expresso Nguyen et al. 2023
SER
English
EmoV-DB (ddıwe
2018
SER
English
EMNS (Norly et al. 2023
SER
English
Dailytalk (Lee al. 2025
SER
English
CREMA-D (Cao et al. 20014
SER
English
CASIA /Zhang Jia 2008
SER
Chinese
M3ED (Zhao et al. 2022)
SER
Chinese
MER2023 (Lian el al.
2023
SER
Chinese
CSEMOTIONS Tian et al.
2025
SER
Chinese
ESD (Zhou et al.
(022))
SER
English, Chinese
Hi-Fi TTS (Bakhturina et al. 2021
TTS
English
VCTK (Veaux al. 2017
TTS
English
LibriTTS Zen et al.
2019
TTS
English
GigaSpeech (Chen et al.
2021
SNTP
English
VoxPopuli Wang et al. 2021
SNTP
English
MaticData
TTS
Chinese
AISHELL-I Bu al. 2017
TTS Chinese
WenetSpeech Zhang et al.
2022)
SNTP Chinese
F.2 TRAINING HYPERPARRMETERS FOR SPEECHL.LMS
Table 2 summarizes the main hyperparameters used throughout downstream SpeechLLM training. Unless otherwise specified, these settings are uniformly adopted for all tasks and models.
F.3 PROMPTS USED IN DOWNSTREAM TASKS
This appendix contains the complete lists of textual prompts used for Automatic Speech Recognition (ASR). Speech Emotion Recognition (SER), and Text-to-Speech (TTS) tasks. For both fine-tuning and inference, a prompt was randomly selected from the corresponding set for each sample.
https://uww.opens11>
25

---

Published as a conference paper at ICLR 2026
Table 23: Hyperparameters used for training all downstream SpeechLLMss
Hyperparameter
Value
optimizer type
Adam
Ir scheduler
Cosine
learning_rad rate min Ir decay _ ratio
4.0c-4
4.0e-5
0.75
weight_ grad_clip
0.1
1.0
F.3.1
ASR TASK PROMPTS
Please transcribe the following audio content into text.
Please convert the following recording into text.
Please transcribe this audio recording into text.
Transcribe the following audio content into text.
Convert the following recording into text.
This audio recording needs to be transcribed into text. Recognize and convert the following speech content into text. Turn the following audio file into text. Transcribe this recording into text.
Transcribe the following audio file into text. Convert this speech recording into text.
Recognize the following audio content and convert it into text. Transcribe the following recording into text.
F.3.2 SER TASK PROMPTS
What is the emotion of this text?
Analyze the sentiment of the following sentence.
Identify the feeling expressed in this audio.
IS the tone of this message positive Detect the emotion in the user's feedback. what emotion is being conveyed here?
negative?
Classify the emotion of this statement. Tell me the emotional state of the speaker. Analyze the emotional content of this speech.
F.3.3 TTS TASK PROMPTS
Please synthesize the following text into speech. Convert the following text to speech.
Transform the following text into speech.
This text needs to synthesized into speech.
Synthesize the following text into speech. Turn the following text into speech. Generate speech from the following text. Convert the text below into speech. Create speech from the following text.
Produce speech from the following text. Render the following text speech.
26

---

Published as a conference paper at ICLR 2026
G FULL RECONSTRUCTION RESULTS
The comprehensive results for the tokenizer-level reconstruction quality evaluation are provided in Table 24 Note that SSL-based semantic tokenizers are included this comparison, as there no publicly available decoders for reconstructing audio from their generated tokens.
Table 24: WER ()) and MOS (f) on LibriSpeech (Panayotov et al. 2015) and SEED (Anastassiou al. 2024). StableToken combines strong noise robustness with competitive reconstruction quality.
It is worth noting that comparison is most meaningful between tokenizers of the same type.
WER
MOS
Frame Rate
LS-
LS-
SEED
SEED 2h
LS- clean
LS- other
SEED
SEED
Model
#C
BPS
clean
other
zh
Semantic Distilled tokenizer
50Hz
500
4.77
16.06
10.37
74.98
2.51
2.49
2.51
2.44
SpeeciTok Zhhang al. 2023
S0Hz
1500
4.03
10.72
4.93
7.81
3.00
2.89
2.89
3.06
50Hz
4000
3.21
6.58
2.77
2.25
3.32
3.10
3.22
3.44
50Hz
500
3,98
9.02
4.72
5.96
3.17
3.04
3.05
3.18
Yo X-Codec
al.
2025
50Hz
1500
3.16
6.11
2.74
2.24
3.43
3.17
3.19
3.38
50Hz
4000
3.09
5.49
2.25
1.74
3.47
3.19
3.19
3.33
Mimi DDfossez ul
2024
12.5Hz
1100
4.65
9.84
3.86
2.81
3.26
3.06
3.15
3.19
Supervised Semantic tokenizer
GLM-4-Voicc-Tkoen Zeng et al. 2025
12.5Hz
175
4.04
9.33
3.54
3.23
4.07
3,99
4.16
4.10
8 Tokenizer (Du et al.
2024a
25Hz.
300
5.78
13.38
5.91
4.26
3.40
3.31
3.40
3.31
Du et al. 2024b
251kz
325
4.25
9.68
4.34
2.75
3,36
3,25
3.31
3,58
StableToken (Ours)
25Hz
325
3.84
7.99
3.44
2.62
4.09
3.83
4.01
4.18
27


Reviews:
[{"review_id": "wFBdREYBfF", "reviewer": "Reviewer_vfbD", "summary": "This paper proposes StableToken, a noise-robust semantic speech tokenizer aimed at improving the stability of SpeechLLMs under noisy conditions. The design integrates a multi-branch quantizer with bit-wise voting to mitigate token assignment instability, and a Noise-Aware Consensus Training (NACT) objective to enforce consistency between clean and noisy inputs.", "questions": "1. Do StableTokens still preserve paralinguistic cues such as speaker identity, prosody, and emotion, or does the stability objective reduce these attributes that are important for tasks like TTS or emotion recognition?\n2. Could the multi-branch voting mechanism introduce inefficiencies or biases in multilingual settings, where phonetic distributions differ widely across languages?\n3. Beyond consensus training, could robustness be further improved by combining StableToken with joint pretraining that explicitly incorporates noise augmentation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is novel in combining multi-branch quantization with voting and noise-consensus training.\n2. StableToken outperforms state-of-the-art baselines across tokenizer-level and downstream tasks in noisy conditions.\n3. Comprehensive analysis includes ablations on branch count, noise levels, and downstream performance.", "weaknesses": "1. The method emphasizes semantic token consistency but does not address whether acoustic nuances (e.g., prosody, speaker traits) are preserved. This may limit usefulness in tasks like TTS or voice cloning, where fine-grained acoustic details are crucial.\n2. Most experiments rely on additive or SNR-controlled noise (e.g., 0 dB, 5 dB). Real-world noise conditions such as overlapping speech, environmental interference, or channel distortion are underexplored, leaving practical robustness uncertain.\n3. Multi-branch quantization may generate redundant tokens. While voting improves stability, it could reduce the information density of tokens and thus impact efficiency in downstream models.\n4. The paper does not study how multi-branch tokenization affects token distribution, entropy, or scaling laws. Shifts in token distribution could potentially influence pretraining dynamics in large-scale SpeechLLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fsjod7ON7k", "reviewer": "Reviewer_Shzy", "summary": "The paper targets the brittleness of semantic speech tokenizers under meaning-irrelevant acoustic noise, arguing that instability in discrete token sequences burdens downstream SpeechLLMs. It proposes StableToken, which replaces single-path quantization with a multi-branch Voting-LFQ module that aggregates per-branch binary codes via bit-wise majority voting, coupled with Noise-Aware Consensus Training that aligns clean/noisy branch representations using a consensus loss. StableToken reduces token instability, while matching or improving reconstruction (WER/MOS) and translating to better downstream ASR/SER/TTS robustness. Ablations confirm each component’s contribution.", "questions": "1. Latency/RTF and throughput. Please report RTF, tokens/sec, and memory usage for StableToken vs. the strongest baselines on a common GPU and on CPU. Include per-branch FLOPs and end-to-end wall-clock with batch size effects.\n2. Segment length & streaming. What is the maximum audio duration processed without truncation? Was training/inference chunked to 30s due to Whisper Large-v3 initialization? How does overlapping/chunk stitching affect stability?\n3. Training audio window. Was the tokenizer trained strictly with $\\leq$ 30s segments, or longer via chunking? Does stability degrade across chunk boundaries?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. A simple but elegant architectural redundancy (multi-branch) with bit-level voting, paired with a clean consensus loss that sidesteps discrete-code gradient issues.\n\n2. Strong tokenizer-level robustness (UED 10.17%), maintained reconstruction (WER/MOS), and clear downstream benefits on ASR/SER/TTS, including OOD noise. Solid ablations and voter-count analysis.\n\n3. Architecture/training described crisply. The evaluation protocol is broad (synthetic and real noise, multiple tasks).\n\n4. Stability is a real deployment pain point. Showing robustness gains that carry to ASR/SER/TTS is valuable for the community.", "weaknesses": "# 1. Latency/RTF/throughput and memory footprint\nThe paper claims negligible overhead, but lacks wall-clock numbers (GPU/CPU), batch size effects, and memory usage. This matters given multiple branches and a large backbone.\n\n# 2. Streaming/segment length\nThe encoder is initialized from Whisper Large-v3 (commonly used with 30s segments). The paper doesn’t clarify the maximum context during training/inference, the chunking strategy, or whether the tokenizer itself inherits a 30s limit.\n\n# 3. Fair comparison controls\nRobustness comparisons span models with different frame rates and codebook sizes (e.g., 12.5/25/50 Hz & vocab 4k–16k). Please provide a control where bitrate/frame rate are aligned across tokenizers or report robustness at matched bits-per-second to rule out confounds. Or perhaps using some normalization methods to make the UEDs comparable.\n\n# 4. Ablations\nYou analyze voter count. I would also suggest evaluating the effect of varying the proportion of perturbed branches ($k$), and the placement of the quantizer within the encoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UDYscsIlWa", "reviewer": "Reviewer_p2V9", "summary": "The paper introduces StableToken, a noise-robust discrete speech tokenizer designed to improve stability and representation consistency under diverse acoustic conditions. The method incorporates Noise-Adaptive Token Alignment (NATA) and Multi-Granular Contextual Quantization (MGCQ) to achieve invariant discrete token representations, enabling better downstream generalization for speech generation and understanding models. Comprehensive evaluations are conducted across noisy and clean conditions, demonstrating improved token stability, representation purity, and semantic retention compared to previous models such as HuBERT, SpeechTokenizer, and AudioTokenizer.", "questions": "- Can the authors quantify how much granularity adaptation (e.g., variable codebook resolution) contributes to stability gains vs. standard augmentation?\n- Is the granularity selection deterministic or learned during inference?\n- How is token correspondence established across variable frame rates or time shifts?\n- Could the method generalize to music signals where spectral structures differ substantially?\n- How does StableToken perform under limited training data or unseen noise conditions?\n- Can the model generalize to domains not covered by augmentations?\n- What is the runtime cost compared to standard quantizers like FSQ or RVQ?\n- How scalable is MGCQ to high frame rates or large codebooks? Is there any observed trade-off between speed and robustness?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The combination of NATA and MGCQ introduces a fine-grained, adaptive quantization strategy that balances stability and expressiveness. The paper effectively frames quantizer instability as a consistency problem between clean and noisy latent spaces.\n- The evaluation was conducted under diverse noise conditions (additive noise, reverb, SNR degradation), including both analysis level (token consistency, MI) and task-level (ASR, LM perplexity, reconstruction) metrics. The improvements are demonstrated in real speech-LM setups, not fixed probing or static benchmarks.", "weaknesses": "- The token stability is part of the major claim but insufficiently formalized. The paper describes it as “the percentage of tokens invariant across clean–noisy pairs,” but does not clarify the matching threshold or how it scales with temporal misalignment.\n- The model’s success may partially stem from extensive noise augmentation rather than architectural novelty. There is insufficient ablation isolating the contribution of NATA and MGCQ from data-scale effects.\n- Although StableToken integrates seamlessly, the multi-quantizer architecture and contextual alignment loss may introduce overhead.\n- Experiments focus heavily on additive noise and mild reverberation, but real-world degradation often involves non-linear distortions (e.g., clipping, compression, far-field effects). The robustness of the method in unseen noise cases would be a good extension elaboration of the robustness of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CbMePgKiog", "reviewer": "Reviewer_Khaj", "summary": "The paper proposes a new Tokenizer (StableToken) to encode speech while being less affected by noises. The proposed architecture is based on a majority voting system with a Lookup-Free-Quantizer, training with perturbed and clean views so the model learns to predict the same tokens with noisy and clean signal.\n\nThe model is compared to a range of SOTA tokenizers, which it outperforms on all evaluated tasks.\nThe model's robustness is measured on automatic speech recognition and speech emotion recognition tasks for a range of noises and SNRs, and show a reduced impact from the raising SNRs compared to other SOTA approaches.", "questions": "Did you try to apply this tokenizer on a larger set of downstream tasks related more closely to the semantic aspects of the speech (like machine translation, summarization or question answering) or to phonetic and accoustic aspects of the speech (like speaker characterization or accent identification) ?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper shows a clear structure and detailed experiments to prove the main contributions: the production of a SOTA tokenizer (see Table 1) with good performances on reconstruction quality (see Table 2) and downstream tasks under noise (see Table 3/Figure 3).\n\nThe ablation study comes to complete the full justification of the architectural choices, while the voter count analysis and the case study on error correction clarify further the intuitions behind the work.", "weaknesses": "The main weakness of the article is its framing. The claim is to produce a robust tokenizer for speech LLMs, where the only experiments performed for speech LLMs are for the downstream tasks of SER and ASR. \nThose tasks could have been performed with a range of different models, so no experiment really highlights why this tokenizer is specifically fit for speech LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)