StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient
  SpeechLLMs
Authors: name: Yuhan Song; name: Linhao Zhang; name: Chuhan Wu; name: Aiwei Liu; name: Wei Jia; name: Houfeng Wang; name: Xiao Zhou
Published: 2025-09-26 11:32:51

Abstract: Prevalent semantic speech tokenizers, designed to capture linguistic content,
are surprisingly fragile. We find they are not robust to meaning-irrelevant
acoustic perturbations; even at high Signal-to-Noise Ratios (SNRs) where speech
is perfectly intelligible, their output token sequences can change drastically,
increasing the learning burden for downstream LLMs. This instability stems from
two flaws: a brittle single-path quantization architecture and a distant
training signal indifferent to intermediate token stability. To address this,
we introduce StableToken, a tokenizer that achieves stability through a
consensus-driven mechanism. Its multi-branch architecture processes audio in
parallel, and these representations are merged via a powerful bit-wise voting
mechanism to form a single, stable token sequence. StableToken sets a new
state-of-the-art in token stability, drastically reducing Unit Edit Distance
(UED) under diverse noise conditions. This foundational stability translates
directly to downstream benefits, significantly improving the robustness of
SpeechLLMs on a variety of tasks.