Scaling Speech Tokenizers with Diffusion Autoencoders
Authors: Yuancheng Wang
Zhenyu Tang
Yun Wang
Arthur Hinsvark
Yingru Liu
Yinghao Aaron Li
Kainan Peng
Junyi Ao
Mingbo Ma
Mike Seltzer
Qing He
Xubo Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Speech Tokenizer
Diffusion Autoencoder
Codec
ASR
Speech Language Model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Speech tokenizers are foundational to speech language models, yet existing approaches face two major challenges: (1) balancing trade-offs between encoding semantics for understanding and acoustics for reconstruction, and (2) achieving low bit rates and low token rates. We propose Speech Diffusion Tokenizer (SiTok), a diffusion autoencoder that jointly learns semantic-rich representations through supervised learning and enables high-fidelity audio reconstruction with diffusion. We scale SiTok to 1.6B parameters and train it on 2 million hours of speech. Experiments show that SiTok outperforms strong baselines on understanding, reconstruction and generation tasks, at an extremely low token rate of 12.5 Hz and a bit-rate of 200 bits-per-second.

Published as a conference paper at ICLR 2026
SCALING SPEECH TOKENIZERS WITH DIFFUSION AU- TOENCODERS
Yuancheng Wang' 2 Zhenyu Tang', Yun Wang', Arthur Hinsvark', Yingru Liu', Yinghao Li', Kainan Peng', Junyi , Mingho Ma', Mike Seltzer', Qing He', Xubo Liub
1 Meta Superintelligence Labs
2 The Chinese University of Hong Kong, Shenzhen
ABSTRACT
Speech tokenizers are foundational to speech language models, yet existing ap- proaches face two major challenges: (1) balancing trade-offs between encoding semantics for understanding and acoustics for reconstruction, and (2) achieving low bit rates and low token rates. We propose Speech Diffusion Tokenizer (SiTok), a diffusion autoencoder that jointly learns semantic-rich representations through supervised learning and enables high-fidelity audio reconstruction with diffusion.
We scale SiTok to 1.6B parameters and train it on 2 million hours of speech. Experiments show that SiTok outperforms strong baselines on understanding, re- construction and generation tasks, at an extremely low token rate of 12.5 Hz and a bit-rate of 200 sessppe--seond.
INTRODUCTION
Speech tokenizers Parker et al. 2024, Guo et al. 2025 Mousavi et al. 2025) define the inter-
face between continuous speech signals and discrete speech language models (Borsos et al. 2023 Nguyen et al. 2025 Défossez et al. 2024 Grattatiori et al. 2024 Zeng et al. 2202bb a Fang et al. 2024 Wang al., 2024; Huang et al. [025 Ding et al. [025]] Xu et al. (025), directly shaping how speech is perceived, modeled, and generated. As speech language models continue to scale and unify understanding and generation, the quality and structure of their discrete speech representations
become a critical bottleneck.
effective speech tokenizer is generally expected balance three competing objectives: (1) ex- treme compression for scalable language modeling, (2) high-fidelity reconstruction for natural speech generation, and (3) semantic-rich representations for downstream speech understanding. However, existing speech tokenizers struggle to satisfy these objectives jointly, particularly in the low token-rate regime, where each discrete token is required to encode substantial information.
Most prior approaches address this tension through heuristic compromises rather than principled solutions. (1) To mitigate reconstruction degradation at low bitrates, many methods rely on residual vector quantization (RVQ) (Zeghidour et al. 2021 Défossez et al. 2022 Kumar et al. 2023) or
increased frame rates (Xin et al.. 2024 Ji et al. [2024] Ju et al. 2024; Ye et al. 2025b] Wang et al.,
2025a), which directly inflate token budgets and undermine efficiency for language modeling. (2) Conversely, tokenizers optimized primarily for acoustic fidelity often neglect linguistic structure, re- sulting in representations that are poorly suited for speech understanding.
(3) Some approaches rely
on multi-stage training pipelines (Guo et al. [2024] Anastassiou et al., 2024 Du et al. [0224a] Zeng jet al., 2024a] Zhang al. 2025c] Wang et al., (025c). where representation learning is decoupled from waveform reconstruction, requiring separate second-stage ooken-to-wavefong mapping and thus preventing end-to-end joint optimization.
In this work, aim explore speech tokenizer paradigm that simultancously achieves extreme compression, high-quality reconstruction, and effective representations for speech language model- ing. Crucially, we observe that under traditional acoustic reconstruction objectives, simply scaling training data or model size yields diminishing returns at low token rates. This limitation suggests
Work done during intersship Meta. Email: yuanchengwangel i ink.

---

Published as a conference paper at ICLR 2026
structural bottleneck imposed by vector quantization: when trained solely with deterministic construction losses, aggressive compression forces the discrete latent space to collapse uncertainty, often prioritizing low-level signal details over linguistically meaningful structure. As a result, such tokenizers yield suboptimal semantic representations for downstream speech understanding tasks such automatic speech recognition (ASR) (Zhang et al., 2023 Défossez al. (024). To over- come this bottleneck, a speech tokenizer requires a generative framework that can explicitly model the uncertainty induced by aggressive quantization, rather than enforcing deterministic inverse mapping. Diffusion models (Ho et al., 2020) Song et al., 2021] Lipman et al. 2023) offer a natu- ral solution, as they learn to reverse a stochastic corruption process and have demonstrated strong
generative capability and scalability across many domains (Rombach et al.) 2022 Liu et al. 2023
Shen et al. 2024 Le et al. 2023 Polyak et al. (024). Some prior works (Anastassiou et al. 2024
Guo et al.
2024 Du et al. 2024a b
Zhang et al.) 2025c have explored diffusion-based speech
tokenization, but typically adopt a two-stage design: first quantizing speech self-supervised repre-
sentations (Baevski et al. 2020 Chung ct al. 2021 Hsu et al. 2021] Chen et al., 2022 Chiu al.
2022]. and then training separate diffusion model for waveform or mel-spectrogram synthesis. Such decoupled training prevents the quantizer from being optimized for reconstruction fidelity and forces the diffusion decoder to adapt to suboptimal discrete codes. In contrast, we jointly opti- mize quantization and reconstruction within diffusion autoencoder (Rey et al., 2019; Preechakul et al. 2022). ensuring that discrete codes are both highly compressive and explicitly aligned with the generative distribution of speech.
Beyond reconstruction, we introduce semantic regularization to explicitly shape the discrete latent space. Speech tokenizers trained solely with reconstruction losses tend to emphasize acoustic fi- delity while lacking alignment with linguistic information, which is detrimental for speech language modeling in both understanding and generation tasks. Prior works (Zhang et al. 2023 Défossez et al. 2024; Li et al. 2025a, Della Libera et al., 2025) attempt to alleviate this issue through se- mantic distillation, aligning latent representations with self-supervised speech features via MSE or cosine similarity losses. However, such indirect alignment does not explicitly enforce linguistic consistency, In this work, we directly impose semantic supervision on the quantized latent space by introducing an auxiliary Connectionist Temporal Classification (CTC) decoder and optimizing it with CTC loss (Graves et al.] 2006), encouraging discrete tokens to preserve semantic-rich and linguistically meaningful structure. We provide more comprehensive review of related work low-bitrate speech tokenizers and ciffusiio-bbsed speech tokenization in the Appendix B
Since diffusion models require iterative sampling during inference, decoding efficiency becomes key challenge. We further investigate shortcut fine-tuning (Frans et al., 2024) and additional ac- celeration techniques that substantially reduce the number of diffusion steps (e.g., or 4) while maintaining high reconstruction quality.
In summary, we propose the Speech Diffusion Tokenizer (SiTok), scaling it 1.6B parameters and training it on 2 million hours of speech data. SiTok achieves strong performance under an extreme compression setting of 12.5 Hz token rate and 0.2 kbps. We comprehensively evaluate SiTok on both speech reconstruction and diverse understanding tasks, including emotion recognition, keyword spotting, speaker verification, and automatic speech recognition. In addition, we demonstrate that the learned discrete representations can be effectively used for speech generation, enabling high- quality synthesis under the same low token-rate setting. Extensive ablation studies on codebook size, codebook dimension, and residual vector quantization (RVQ) further provide insights into the design of scalable diffusion- speech tokenizers. Reconstruction and speech generation samples shown in https: http://ttookddeeee qi github.oo/
2
METHOD
In this section, we introduce SiTok. We first present the speech tokenization architecture based on a diffusion au- toencoder. We then describe our key design, semantic reg- ularization. Finally, we explain how decoding can be accel- erated through shortcut fine-tuning, along with additional techniques that further improve reconstruction quality. Fig- ure provides an overview of our model.
Forward Diffision
Encoder
VQ
DiT Decoder
Lete
CTC Decuder
Lim
Figure 1: Overview of SiTok.
2

---

Published as a conference paper at ICLR 2026
2.1 OVERVIEW
Speech tokenizers are generally based on autoencoders: the raw speech representation is first mapped into latent features by an encoder, then quantizer encodes the latent features into quence of discrete tokens, and finally a decoder reconstructs the raw speech representation from these discrete tokens. Some speech tokenizers directly use raw waveform signals modeling tar- gets and rely on adversarial training to improve perceptual quality. However, we argue that this paradigm unfavorable for scaling because: (1) directly processing the waveform sequence inef- ficient due to its excessive length, which necessitates substantial up- and down- Nown-sampling and often forces prior work to cut waveforms into only few seconds; and (2) adversarial training requires complicated loss designs and additional discriminator optimization, which tends to be unstable.
In this work, we propose novel framework: (1) we utilize mel-spectrogrrass as both the input and the reconstruction target, leveraging vocoder to directly synthesize the corresponding waveform; and (2) we replace adversarial training with a diffusion autoencoder, which facilitates more stable and scalable training. We posit that this diffusion-basse compression and reconstruction. By learning to reverse the diffusion process, the model is trained to effectively capture the underlying data distribution, enabling more robust recovery of the original signal from its quantized representation. Formally, given an input mel-spectrogram æ, the training process is as follows:
framework achieve superior
1. Downsampling:
The temporal resolution of æ is reduced for computational efficiency. In this work, we set the default frame rate 12.5 Hz. Encoding: The encoder Ey maps the downsampled spectrogram x to a sequence of latent fea- tures:
z == (xr)
3. Quantization: Each feature vector in the latent sequence is mapped its closest entry in a discrete codebook E = {er, eg..... ek) } producing a sequence of discrete indices This vector quantization step can be denoted as:
q vQ(2;
4. Diffusion Modeling: The decoder Dø is trained to reconstruct æ conditioned on the quantized representation. The discrete indices q are first mapped back to their corresponding codebook embeddings Zq. Using flow-matching objective, the decoder D& learns to predict a velocity field that transforms a noisy sample back the original data. The process is defined as:
x₁ tæ + (1 t) where NN I) and ~ U(0, 1).
The decoder's predicted velocity us is optimized to match the true velocity (x E):
uo t, 2,) = Do( 1, 2q) + e.
2.2 SEMANTIC REGULARIZZTION
In our preliminary study, we found that relying solely on reconstruction, whether based on a dif- fusion loss regression loss, results in poor intelligibilityy (much higher WER) and degraded performance on downstream understanding tasks. Motivated by prior works, we introduce an aux- iliary loss to directly supervise the latent space after vector quantization. Unlike approaches that employ representation alignment (Yu et al. 2025) or semantic distillation (Zhang et al.) 2023] Défossez al. 2024) match the latent representations with features from self-supervised mod- els, we directly predict the textual content through an additional lightweight decoder Dew. trained with a CTC loss. Some previous works like Beichuan- Tokenizer (Gong et al. 2025) also incorporate ASR- based supervision to enrich semantic repre- sentations, but they still rely on an additional ASR model as semantic encoder to extract latent features. In contrast, SiTok learns representations directly from raw speech.
tokenizer (Li et al., 2025b) and XY-
The total loss Lisal combines three components: the diffusion reconstruction loss (Lrec, the seman- tic CTC loss (Lere), and the vector quantization loss (Lrg). Given the ground-truth text transcript y, the objective is:
Liotal Er,,r. [|| (2 2g)) (x c)ll +Aare CTC om (Zy). +
Lrg
Reconstruction Loxs
CTC Loss
VQ Loss
3

---

Published as a conference paper at ICLR 2026
where zq is the sequence of quantized embeddings, Daw is the auxiliary CTC decoder, and Xew is balancing hyperparameter. We find that dew crucial for performance.
2.3 EFFICIENT DECODING
Traditional diffusion models often require multiple inference steps, which can make decoding com- putationally inefficient. To address this, we explore two strategies to significantly accelerate diffu- sion decoding: Shortcut Fine-tuning and Leght-weight Diffusion Head.
Shorteut Fine-tuning We explore efficient few-step decoding with fine-tuning the decoder using the shortcut model objective proposed by Fyrrns al. (2024). During the tinettning stage, we freeze the encoder and VQ modules. The fine-tuning process then updates only the decoder weights. The key idea behind shortcut models to train network that conditioned not only on the time step t but also a desired step size d. This allows the model to learn direct mapping from noisy input to a significantly denoised output a single forward pass, effectively "jumping" over many intermediate steps of a standard iterative diffusion process.
The fine-tuning employs a combined loss function that jointly optimizes flow-matching objective and self-consiseency objective. The total loss is formulated as:
Cs L.. ..d [1,0( t, - (x1 - x0)1 + | t, 2d) - sarget 1|
Flow-Matching Loss
Seif-Consistency Loss
where 86 is the shortcut model (our decoder) being trained. The target for the self-consistency loss, Stargets is generated by the model itself using two consecutive smaller steps, with gradients detached:
(fou(eent))) 7 = (0 t d. d.d)
sarget stopgrad
and Br4d W. + 8q (ZEr 1, ddd.
The first term grounds the model's behavior for infinitesimal step sizes = 0), ensuring it matches the true data velocity. The second term enforces that single large step of size 2d yields the same result as two sequential steps size d. This self-consistency training enables the decoder to accu- rately perform large, discrete jumps along the denoising trajectory, significantly reducing inference steps while maintaining high reconstruction quality.
Light-weight Diffusion Head We also explore Light-weight diffusion head that reduces the cost of iterative denoising by splitting the decoder into a main body (run once) and a small head reused across diffusion steps. This design lowers per-step computation; see Appendix C. for details.
2.4 RECONSTRUCTION REFINEMENT
To further enhance the quality of the reconstruction speech, we employ two distinct refinement strategies. The first is a decoder finetuning, where the encoder and VQ modules are frozen, and only the decoder is trained further. This step specializes the decoder for high-fidelity synthesis from the fixed discrete representations. The second the introduction of token classifier-free guidance (Token CFG). To enable this, we train the decoder to be conditionally dependent on the discrete tokens by randomly dropping all input tokens with a 10% probability, which forces the decoder to also learn unconditional generation objective. During inference, this allows us to steer the decoding process by combining predictions from both conditional (with tokens) and unconditional (with dropped tokens) passes, leading to more robust and accurate reconstruction. The efficacy of both optional refinement techniques is empirically validated in our results (Table
3 EXPERIMENTS AND RESULTS
3.1 IMPLEMENTATION DETAILS
Data and Preprocessing We use 2 million of in-house speech data to train models. The dataset covers multiple languages, with English accounting for the vast majority. We do not apply additional preprocessing to the speech data, such as splitting into shorter segments; instead, we train
4

---

Published as a conference paper at ICLR 2026
directly on the original utterance lengths paired with their transcripts. We use Hz, 128-bin mel- spectrograms as both the input and reconstruction targets of our tokenizer, while first stacking every four consccutive frames to reduce the frame rate to 12.5 Hz for more efficient training. For waveform synthesis, we employ a Vocos-hased (Siuzdak 2024) vocoder to convert the mel spectrograms back to audio waveforms at 24K Hz.
Model Our model is constructed using standard Llama-style Transformer blocks (Touvron et al, 2023 Grattafiori et al. 2024). The encoder and the auxiliary CTC decoder are composed of causal Llama decoder layers, with 16 and layers, respectively. Unless otherwise specified, we set the hidden size to 1536, the intermediate size to 4096, and the number of attention heads to 16. For the VQ module, we adopt default configuration of 32 dimensions with a codebook of 65 65,536 entries, updated using an exponential moving average (EMA) (van den Oord et al. 2017). The diffusion decoder is implemented by modifying the causal Llama decoder layers into a non-causal form with 16 layers. We incorporate the diffusion step embedding by replacing RMSNorm (Zhang & Sennrich 2019) with an Adaptive RMSNorm variant. Additional architectural details are provided in Appendix A while ablation studies on the codebook dimension, codebook size, and overall model size are presented in Section 3.4
Training We train all models for a single epoch, corresponding to approximately 450K steps. For optimization, adopt the AdamW (Loshchilov & Hutter 2019) optimizer with B1 0.9, $2 = 0.999, a weight decay of 0.01, and a learning rate of 8 x 10 with a warmup of 32K steps.
3.2 EVALUATION
We evaluate our tokenizers from the perspectives of compression, reconstruction, and speech under- standing. We also evaluate SiTok on speech generation via zero-shot TTS, with results reported in Appendix
We assess the efficiency of the tokenizer in terms of token rate (TPS: tokens per second), frame rate (FPS: frames per second), and bitrate (BR). These metries directly reflect the trade-off between compression and representational capacity.
Compression
Reconstruction To evaluate speech reconstruction quality, we measure intelligibiility speaker sim- ilarity, and speech quality. Intelligibilityy is assessed using word error rate (WER), computed with whi spperllarge rge-v3 (Radford et al. 2023). Speaker similarity (SIM) is computed as the cosine similarity between M.
embeddings of the prompt and the generated speech (Chen al.) 2022). Speech quality is evaluated using the official UTMOS checkpoint (Sacki et al. 2022). We report these results on the SeedTTS test-en (Anastassiou et al. 2024) evaluation set.
Understanding We evaluate the learned representations on three speech understanding tasks: emotion recognition (ER), speaker verification (SV), and keyword spotting (KS), following the setup of the DASB benchmark (Mousavi et al. 2024). Additionally, following Yang et al. (2025), we train an LLM-based ASR (LLM ASR) model with a 1B parammetter LLM backbone, which takes the dis- crete speech tokens generated by the speech tokenizer as input and autoregressively predicts the corresponding text. We also report ASR results (CTC ASR) using the direct CTC decoder of our tokenizers. The ASR evaluation is conducted the LibriSpeech test-clean (Panayotov al.) 2015).
Evaluation Baselines We also compare our approach with a range of open-source speech tokeniz- ers, see more details about the baselines in the following sections.
3.3 RESULTS AND COMPARISON
In this section, we present comprehensive evaluation of our proposed speech tokenizer. We first report the main results for speech reconstruction Section 3.3.1 where we compare against a wide range of existing tokenizers under different compression settings. We then evaluate downstream performance in Section 3.3.2. covering diverse understanding tasks. Beyond these comparisons, we further analyze the effectiveness of semantic regularization in Section 3.3.3 the impact of scaling model size in Section 3.3.4, and efficient decoding strategies in Section 3.3.5 that improve inference speed without sacrificing quality. Finally, we conduct extensive ablation study 3.4 to isolate the contributions of different components, including loss design, codebook configurations, and frame rate choices, providing insights into the design principles of scalable speech tokenizers.
5

---

Published as conference paper at ICLR 2026
3.3.1 MAIN RESULTS FOR RECONSTRUCTION
Table 1: Main reconstruction results. "Decoder Finetuning" indicates that the encoder and VQ are frozen while the decoder is further trained for several steps. "Token CFG" denotes the use of classifier-free guidance, more details are shown in Section 3.4 "CN" means codebook number.
Model
FPS/TPS
CN
BR (kbps)
WER (4)
SIM (*)
UTMOS (f)
Ground Truth
2.14
0.730
3.53
SpecchTokenizer ZZang et al.
2023
50/100
2
1.00
7.98
0.468
2.47
BigCodec (Xin et al. 2024
80/80
1.04
3.25
0.615
3.59
DualCodec [Li a. 2025a
12.5/75
0.925
2.63
0.624
3.78
Wav Taa'mokee (J al 2024
75/75
0.90
6.65
0.483
3.36
Mimi (Défossc2 ct al. 2024
12.5/75
0.825
4.51
0.527
3.09
X-codec 2Cc
2025b
50/50
0.80
2.63
0.620
3.68
SemantiCodec (1.uu al. 2024
25/50
0.675
5.11
0.488
2.83
BiCodec (Wang et al. 2025a
50/50
0.65
3.05
0.612
3.68
Vevo Tokenizer (Zhang et al.
2025c
50/50
0.65
3.04
0.534
3,50
StableCodec (Parker ct al.
2024
25/25
0.40
11.1
0.410
3.87
FireRedTTS Tokemizer Cruo ct al
2024
25/25
0.35
3.35
0.597
3.40
CosyVoice
Voice Tokenizer
DDu et
2024a
25/25
0.30
5.63
0.465
3.65
SiTok (CN =
12.5/12.5
0.20
4.06
0.641
3.44
Decoder Finetuning
12.3/12.5
0.20
3.79
0,682
3,48
Token CFG
12.5/12.5
0.20
3.34
0.635
3.60
SiTok (CN 2)
12.5/25
0.35
3.17
0.658
3.44
SiTok (CN 4)
12,5/50
0.70
2.80
0.660
3.46
Table presents detailed comparison of our speech tokenizers against other baseline models on the reconstruction task. Our model demonstrates exceptional performance under highly challeng- ing setting. At its base configuration (single codebook), our tokenizer operates at extremely low bitrate of 0.2 kbps and token rate of only 12.5 Hz, which significantly lower than all compet- ing methods. Despite this extreme compression, it achieves highly competitive results. We also demonstrate that the model's performance can significantly enhanced with simple yet effective strategies. Decoder finetuning boosts speaker similarity to remarkable 0.682. Applying token classifier free guidance reduces WER to 3.34. In addition, increasing the number of codebooks via RVQ yields consistent improvements in both WER and similarity.
3.3.2 DOWNSTREAM UNDERSTANDING
Understanding Tasks Table shows that our tokenizer significantly outperforms all baselines several speech understanding tasks. In particular, we achieve substantial improvements LLM- based ASR (WER 4.95), and consistently surpass all baselines on ER, SV, and KS. LLM-based ASR
results for baseline models adopted from Yang al. (225 "CN" means codebook number and "CS" means codebook size.
Table 2: Main results for understanding tasks.
Model
FPS/TPS
CN/CS
BR (khps)
CTC ASR (L)
ASR (1)
ER (1)
SV (L)
KS (1)
DAC Rombach et al.
2022
50/150
3/1024
1.5
58.4
48.9
17.8
68.8
EnCodec Delossez
2022
3/024
77.2
Mimi Délossez et 2074
12.5/100
8/2048
1.1
23.1
54.3
19.7
92.2
Way
RawTokenzzerJn 20:4
40/40
1/4096
0.48
45.6
51.1
19.4
65.3
StableCodee/
25/25
1/46656
0.40
28.0
GLM4-Voice et 20244
12.5/12.5
1/16384
0,20
16.3
SiTbk (CN =
12.5/12.5
1/65536
0.20
9.50
4.95
63.5
13.8
96.9
SiTok (CN
12.5/50
4/16384
0.70
8.30
4.49
64.4
8.59
3.3.3 EFFECTIVENESS OF SEMANTIC REGULARIZATION
Table 3 demonstrates the impact semantic regularization on both reconstruction and understand- ing tasks, which is key enhancing reconstruction quality and learning meaningful representations

---

Published as a conference paper at ICLR 2026
for downstream understanding. Without regularization, the model shows severely degraded intelli- gibility (WER rising from 4.06 to 33.0) and poor downstream performance. In contrast, applying CTC-based regularization substantially reduces WER, improves similarity and speech quality, and boosts all understanding tasks. This highlights that CTC supervision anchors the quantized latent space to linguistic meaning, ensuring tokens both acoustically faithful and semantically infor- mative, especially for low-rate tokenizers.
Table 3: Effectiveness semantic regularization on reconstruction and understanding.
Reconstruction
Understanding
CTC Reg.
FPS/TPS
WER ())
ST)
UTMOS (1)
CTC ASR ())
ASR (L)
ER (1)
SV (L)
KS (t)
12,5/12.5
4,06
0.641
3.44
9.50
4.95
63,5
13.8
96.9
Yes
12.5/25
3.17
0.658
3.44
8.64
4.72
61.7
11.1
97.8
2.80
0.660
8.30
4.49
64.4
12,5/12.5
33.0
0.495
2.68
29.4
57.9
86.1
No
12.5/25
10.1
0,598
2.99
9.53
55.3
15.5
92.7
12,5/50
5.17
0.611
2.84
7.27
60.4
13.5
92.8
Observation: Applying semantic regularization to the quantized latent space crucial for both reconstruction and representation learning, particularly when operating at low token rates.
3.3.4 EFFECTIVENESS OF MODEL SCALING
Our model scaling experiments from the 0.63B "S" model the 1,61B "XL" model reveal clear trade-off between reconstruction fidelity and downstream task performance, shown in Table 4 While larger models consistently yield better reconstruction quality, with the XI. model achieving understanding tasks peaks with the 1.12B "L"
the best WER, SIM and UTMOS, performance model, which delivers superior results in ASR. The fact that the largest model does not uniformly outperform its smaller counterparts, and even shows degradation in tasks like SV, suggests that ex- cessive model capacity may prioritize fine- grained acoustic details over the abstract, discriminative features crucial for understanding. Therefore, tion, providing the most effective balance between high-quality synthesis and robust generalization. Further exploration of architectural designs is left for future work.
identify the "L" model the optimal configura-
Table 4: Results for model size scaling. We vary the number of encoder and decoder layers while keeping the CTC layers fixed to evaluate the impact on both reconstruction and understanding tasks.
Reconstruction
Understanding
Size
Enc.
Dec.
Params (B)
WER (1)
SIM (f)
UTMOS (t)
CTC ASR (1)
ASR ())
(f)
SV (L)
KS (t)
8
0.63
4.18
0.608
3.43
5.24
60.8
B
12
12
0.88
4.01
0.634
3.46
9.78
5.19
62.5
13.8
96.9
1.12
4.06
0.641
3.44
4.95
96.9
XL.
24
24
1.61
3.84
0.649
3.51
9.62
5.07
63.5
14.7
97.3
3.3.5
EFFICIENT DECODING
Shorteut Fine-Tuning Table 2 shows that directly reducing the number of inference steps leads to a clear degradation in intelligibility and audio quality. In contrast, shortcut fine-tuning substantially alleviates this issue, achieving much lower WER and higher speaker similarity numbers of diffusion steps. Moreover, this improvement also translates into a significant reduction in real-time factor (RTF): the model runs 0.041, 0.024, and 0.013 RTF for 16, 8, and 4 diffusion steps, respectively. These results demonstrate that shortcut fine-tuning effectively adapts the model faster sampling schedules while preserving reconstruction fidelity and enabling highly efficient inference.
with very small
Observation: Shortcut fine-tuning enables efficient low-step inference, retaining high intelligibility and similarity while substantially accelerating decoding.

---

Published as a conference paper at ICLR 2026
Shortcut Fine-Tuming
Shortcut Fine-Tuning
WER
SIM 1
UTMOS
11-
0.00
0.30
Score
0.5
0,90
Inference Steps
Inference Steps
Inference Steps
Figure 2: Impact of shortcut fine-tuning on different inference steps. We report WER, SIM, and UTMOS, Shortcut fine-tuning achieves consistently better intelligibility and similarity, especially at small step numbers.
Weight Diffussin Head diffusion inference. We provide the results in Appendix C.1
We also explore using light-weight diffusion heads to accelerate
3.3.6 SITOK FOR SPEECH GENERATION
Beyond speech understanding, an important use case of speech tokenizer is to serve as the mod- eling target for speech generation. To this end, further evaluate SiTok the zero-shot text-to- speech (TTS) task, assessing whether the learned discrete representations can effectively support high-quality speech synthesis. Experimental results show that SiTok is not only effective rep- resentation for speech understanding, but also enables high-quality and efficient speech generation. In particular, the extremely low token rate of SiTok significantly reduces the sequence length quired for generation, leading to faster inference while maintaining strong perceptual quality. These results suggest that SiTok provides unified representation that well suited for both speech un- derstanding and speech generation. Detailed experimental results for zero-shot TTS are reported in Appendix
3.4 ABLATION STUDY
To better understand the contributions of different design choices in tokenizer, we conduct ries of ablation studies, the results are shown in Table We systematically examine training objec- tives, regularization strategies, refinement mechanisms, codebook configurations, and frame rates. These analyses not only validate the effectiveness of our proposed components but also provide insights into the trade-offs between efficiency, reconstruction quality, and downstream performance.
Diffusion vs. Regression We investigate the choice of the reconstruction objective by comparing our proposed diffusion-basee objective (D) against a conventional regression- objective (R), such as an L1 loss on the mel-spectrogram. As shown in Table [] our baseline model trained with the diffusion loss significantly outperforms the regression- based counterpart across all key metrics. Specifically, the diffusion model achieves a substantially lower WER (4.06 vs. 4.66), higher speaker similarity (0.641 vs. 0.587), and better downstream ASR performance (4.95 vs. 6.06). To further explore if a diffusion decoder could salvage regression regression-trained model. we conducted an experiment where only the decoder was fine-tuned with the diffusion objective on top of a regression- prrressionn-petrannee model (R D). Therefore, by design, the understanding metrics for (R) and (R D) identical, while reconstruction metrics (WER/SIM/UTTMOS change. While this modestly improved speaker similarity, it failed to match the performance of the end-to-end diffusion model and even wors- ened the WER to 5.73. This indicates that the representations learned under the diffusion objective inherently superior for both high-fidelity reconstruction and downstream task transferability.
Observation: Adopting a diffusion- objective is critical for learning high-quality representations, significantly outperforming a standard regression objective in both reconstruction fidelity and downstream task performance.
8

---

Published as a conference paper at ICLR 2026
Table 5: Ablation study.
Reconstruction
Understanding
CTC CTC ASC (1)
UINNO (
CTC
Tok: CFG
WER (
SIME CD
ASR (L)
ER (1)
S5 (
KS (
3
3
8
FPS
D
0.1
246
32
X
12.5
4.06
0.641
3,44
9.50
4.95
63.5
13.8
96.9
R
0.1
240
32
x
12.5
4.66
0.587
3.28
12.2
6.06
63.3
13.6
95.2
Loss
R
0.1
210
32
X
12.5
5.73
0.634
3.35
12.2
6.06
63.3
13.6
95.2
216
x
12.5
33.0
0.495
2.68
29.4
18.9
86.1
0.02
246
32
12.5
5.05
0.607
3.44
12.2
7.41
58.3
13.5
97.2
CTC W.
0.5
246
32
X
12.5
8.81
0.614
3,20
11.0
7.87
64.2
16.6
91.2
D
246
32
X
12.5
10.1
0.585
3,38
10.5
8.90
62.1
13.9
96.8
D
0.1
213
32
X
12.5
5.48
0.640
3,39
11.7
5.72
55.7
16.3
95.7
D
0.1
214
32
X
12.5
4.30
0.641
3.33
11.5
5,40
58.7
16.0
96.4
0.1
215
32
X
12.5
4.26
0.648
3.43
10.5
5.33
61.2
15.0
96.4
247
32
12.5
3.94
0.651
3.39
10.6
5.09
97.3
214
X
12.5
3.17
97.8
CN
214
32
X
12.5
2.80
0.660
3,46
8,30
4,49
8.59
97.7
D
0.1
244
32
X
12.5
2.50
0.645
3,30
8.42
4.68
60.0
7.53
98.2
D
0.1
21
64
X
12.5
4.08
0.642
3,46
9.58
5.27
61.7
14.4
97.3
CD
D
0,1
214
128
X
12.5
3.85
0.642
3.36
9.14
4.59
59.7
14.3
97.3
0.1
214
256
X
12.5
5.04
0.641
3.30
10.9
5.54
64.1
16.5
95.9
0.1
216
32
12.5
3.34
0.635
3.60
9.54
4.89
62.4
14.0
96.8
Tok. CFG
244
32
12.5
2.56
0.645
3.58
8.34
4.67
61.7
98.2
0.1
210
32
6.25
23.0
0.428
3,10
15.8
12.7
20.7
88.5
FPS
0.1
210
32
x
25
3.05
0.688
3.72
9.19
4.45
7.28
97.8
CTC Loss Weight We analyze the impact of the CTC loss weight, which serves as semantic regularizer. The results clearly demonstrate that this regularization indispensable. Setting the weight to 0 leads to catastrophic performance collapse, with the WER soaring 33.0 and the downstream ASR error rate to 29.4, confirming that the model fails to learn any meaningful rep- resentations without textual supervision. Conversely, excessively high weight (e.g.. 0.5 or 1.0) also degrades performance across both reconstruction and understanding tasks, likely by forcing the model discard too much acoustic detail in favor of semantic content. Our experiments iden- tify a weight 0.1 as providing the optimal balance between enforcing semantic consistency and preserving high-fidelity audio reconstruction.
Observation: The CTC loss weight is critical hyperparameter; too low a value fails to enforce semantic consistency, while too high a value impairs reconstruction fidelity.
Reconstruction Refinement As shown in our main reconstruction results (Table D. both decoder finetuning and token classifier-free guidance (CFG) serve as powerful techniques to enhance recon- struction quality. Decoder finetuning, a training-time strategy, specializes the synthesis module on the fixed representations, leading to significant improvements in both intelligibility and particularly speaker similarity. Separately, applying token CFG at inference time provides complementary approach to boost fidelity. Our ablation study demonstrates that CFG consistently and substantially reduces WER, achieving a low of 2.56 in our CD = configuration, while also improving percep- tual quality (UTMOS). These findings indicate that both training- time adaptation and inference-time guidance are highly effective strategies for refining the final output.
Codebook Size We investigate the effect of the codebook size (CS), which controls the vocab- ulary of discrete tokens. We find enlarging the codebook from 2 to 2 consistently improves reconstruction quality, evidenced by steady decrease in WER from 5.48 3.94. Downstream task performance, particularly ASR, also benefits and reaches an optimum at a codebook size of 26 suggesting this size offers the best balance between representational power and generalization.

---

Published as a conference paper at ICLR 2026
Codebook Number We also evaluate increasing the number of codebooks (CN) using RVQ, which directly increases the bitrate. We find that scaling CN from to 8 yields substantial and consistent improvements across most metrics. The reconstruction WER drops dramatically from 4.30 to 2.50, while downstream performance on tasks like ASR and SV is also significantly boosted. This demonstrates effective trade-off between compression and fidelity within framework.
Observation: The vector quantizer's design offers a flexible trade-off between quality and complexity. Increasing the number of codebooks provides a direct path to higher fidelity and better downstream performance at the of bitrate.
Frame Rate vs. Performance Trade-off We also investigate two alternative frame-rate settings: 6.25 Hz and 25 Hz. We find that reducing the frame rate to 6,25 Hz significantly degrades both re- construction and downstream task performance, while increasing it to 25 Hz improves performance but doubles the frame rate. Therefore, we adopt 12.5 Hz as the default setting to balance efficiency and performance.
CONCLUSION
this work, we propose SiTok, a diffusion autoencoder-based speech tokenizer that enables end- to-end joint modeling of reconstruction and quantization for improved acoustic fidelity. We further introduce semantic regularization to learn effective, semantically rich representations for speech understanding, and explore shorteut fine-tuning techniques to significantly accelerate diffusion de- coding. Extensive experiments demonstrate that SiTok achieves strong performance on both speech reconstruction and diverse speech understanding tasks. In addition, conduct extensive ablation studies, providing insights into the key design choices.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work studies speech tokenizers with diffusion autoencoders. Our models are designed for aca- demic research and downstream tasks such speech understanding and speech language modeling. While tokenizers themselves are neutral, we acknowledge potential misuse in downstream systems (e.g., generating synthetic speech for impersonation) and encourage responsible and ethical use of our models.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide detailed descriptions of model architectures, training settings, and evaluation protocols in the main paper and Appendix.
USE OF LLMS
LLMs were employed for auxiliary purposes in this work, such grammar checking, polishing the manuscript. However, all technical contributions, model implementations, and experimental analyses were conducted by the authors. We acknowledge the use of LLMs where appropriate and ensure that their involvement does not compromise the originality of the work.

---

Published as a conference paper at ICLR 2026
REFERENCES
Philip Anastassion, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingx- ing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, Yuchen Liu, Zhengxi Liu, Lu Lu, Junjie Pan, Xin Wang, Yuping Wang, Yuxuan Wang, Zhen Wei, Jian Wu, Chao Yao, Yifeng Yang, Yuan- hao Yi, Junteng Zhang. Qidi Zhang. Shuo Zhang, Wenjie Zhang, Yang Zhang, Zilin Zhao, Dejian Zhong, and Xiaobin Zhuang. Seed-tts: A family of high- versatile speech generation mod- els. arXiv preprint, abs/2406.02430, 2024.
Alexei Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In NewrIPS, 2020.
Xiaoyu Bie, Xubo Liu, and Gaël Richard. Learning souree disentanglement in neural audio codec. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 2025.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matthew Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghi- dour. Audiolm: A language modeling approach to audio generation. IEEE ACM Trans. Audio Speech Lang. Process.,
2023.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujic Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. Wavlm: Large-scale self-supervised pre- training for full stack speech processing. IEEE J. Sel. Top. Signal Process., 1055)1105- 2022.
Chung-Choee Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random--rojection quantizer for speech recognition. In International Conference on Machine Learning, pp. 3915-3924. PMLR, 2022.
Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. w2v-bert: Combining contrastive learning and masked language modeling for self- supervised speech pre-training. In ASRU, pp. 244-250. IEEE, 2021.
Alexandre Défossez, Jade Copet. Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint, abs/2210. 0tbs/2200.13438, 2022.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dia- logue. arXiv preprint arXiv:2410.00037 2024.
Luca Della Libera, Francesco Paissan, Cem Subakan, and Mirco Ravanelli. Focalcodec: Low-bitrate speech coding via focal modulation networks. arXiv preprint arXiv:2502.04465. 2025.
Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report. arXiv preprint arXiv:2504. 18425, 18425, 2025.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijie Yan. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint, abs/2407.054077, 2024a.
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, and Jingren Zhou. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint, abs/2412.10017,, 2024b.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighte linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
12

---

Published as a conference paper at ICLR 2026
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. arXiv preprint S3l1:2400.066666, 2024.
Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. arXiv preprint arXiv:2410.2255,.222 2024.
Yitian Gong, Luozhijie Jin, Ruifan Deng. Dong Zhang, Xin Zhang, Qinyuan Cheng, Zhaoye Fei, Shimin Li, and Xipeng Qiu. Xy-tokenizer: Mitigating the semantic-acoustic conflict in low-bitrate speech codecs. arXiv preprint (rXiv:2506.23322 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2 2024.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Hao-Han Guo, Kun Liu, Fei- Yu Shen, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. Fir- credtts: A foundation set--to-ppeec framework for industry-level generative speech applications. arXiv preprint, abs/2409.00 00020,,,..3,... 2024.
Yiwei Guo, Zhihan Li, Hankun Wang, Bohan Li, Chongtian Shao, Hanglei Zhang, Chenpeng Du, Xie Chen, Shujie Liu, and Kai Yu. Recent advances in discrete speech tokens: A review. arXiv preprint arXiv:2502.0644 2025.
Haorui He, Zengqiang Shang. Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In SL7. IEEE, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-66 6851, 2020.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE ACM Trans. Audio Speech Lang. Process., 29:3451-3460, 2021.
Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, et al. Step-audio: Unified understanding and generation in intelligent speech interaction. arXiv preprint 2 19946, 2025.
Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, et al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint arXiv:2408. 16532, 2024.
Yidi Jiang, Qian Chen, Shengpeng Ji, Yu Xi, Wen Wang, Chong Zhang, Xianghu Yue, ShiLiang Zhang, and Haizhou Li. Unicodec: Unified audio codec with single domain-adaptiv codebook. 2025.
arXiv preprini
Zeqian Ju, Yuancheng Wang. Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Eric Liu, Yichong Leng. Kaitao Song. Siliang Tang, Zhizheng Wu, Tao Qin, Xiangyang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, and Sheng Zhao. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: j2Jad
//openreview. wwreerreviee.net//oommmmm
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved RVQGAN. In NewrlPS, 2023.
13

---

Published as a conference paper at ICLR 2026
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. Voice- box: Text-guided multilingual universal speech generation at scale. In Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference mation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL ht http:/aaaprsseeam http://pppppp..iiss.ecepeeeeeeep 2ttps///e/iieee/eeee////////
Neural Infor-
2d
html
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-11532, 2022.
Jiaqi Li, Xiaolong Lin, Zhekai Li, Shixi Huang. Yuancheng Wang, Chaoren Wang. Zhenpeng Zhan, and Zhizheng Wu. Dualcodec: A low-frame-rate, semantically-enhaneed neural audio codec for speech generation. arXiv preprint arXiv:2505. 13000, 2025a.
Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang, Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong, et al. Baichuan- A unified framework for end-to-end speech interaction. arXiv preprini 2XXi:2252.17233 2025b.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://cpenreviiee net forum? (oorum?id=PquMRDDES PPVPRPDDDCT
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301. 12503, 2023.
Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. Semanti- codec: An ultra low bitrate semantic audio codec for general sound. IEEE Journal of Selected Topics in Signal Processing, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR (Poster). Open- Review.net, 2019.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made simple. arXiv preprini arXiv:22309. 15505, 2023.
Pooneh Mousavi, Luca Della Libera, Jarod Duret, Artem Ploujnikov. Cem Subakan, and Mirco Ravanelli. basb-discrete audio and speech benchmark. arXiv preprint arXiv:2406.1423 2024.
Pooneh Mousavi, Gallil Maimon, Adel Moumen, Darius Petermann, Jiatong Shi, Haibin Wu, Haici Yang, Anastasia Kuznetsova, Artem Ploujnikov, Ricard Marxer, et al. Discrete audio tokens: More than survey! arXiv preprint arXiv:2506.1027 2025.
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R Costa-Jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, al. Spirit- Im: Interleaved spoken and written language model. Transactions of the Association for Compu- tational Linguistics, 13:30-52, 2025.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur: Librispeech: An ASR corpus based on public domain audio books. In ICASSP, pp. 5206-5210. IEEE, 2015.
Julian D Parker, Anton Smirnov, Jordi Pons, CJ Carr, Zack Zukowski, Zach Evans, and Xubo Liu. Scaling transformers for low-bitrate high-quality speech coding. arXiv preprint 2024.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching Yao Chuang, et al. Movie gen: A cast of media founda- tion models. arXiv preprint arXiv:2410.1372 2024.
14

---

Published as a conference paper at ICLR 2026
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakom. Dif- fusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10619-10629, 2022.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML, volume 202, pp. 28492- 28518, 2023.
Luis A Pérez Rey, Vlado Menkovski, and Jacobus W Portegies. Diffusion variational autoencoders. 2019.
arXiv preprint
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 106840060000 2022.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hi- roshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.021 022204...15,, 2022.
Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing syn- thesizers. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vi- enna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL Forum? RerummiddRc7daaaa /dWwL33.
net/
Hubert Siuzdak. Vocos:
Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. In ICLR. OpenReview.net, 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https::/openreviee: net fott/ffrum?dd
EXTIG12RRHH
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2xXiv22377009288 2023.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In NIPS, pp. 6306-6315, 2017.
Xinsheng Wang. Mingqi Jiang. Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang. Xiaoqin Feng. et al. Spark-tts: An efficient Ilm-based text-to-speech model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.0177 2025a.
Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freezc-omni: A smart and low latency speech-to-speech dialogue model with frozen 11m. arXiv preprint arXiv:241 /. /.00774, 2024.
Yuancheng Wang. Dekun Chen, Xueyao Zhang, Junan Zhang, Jiaqi Li, and Zhizheng Wu. Tadi- codec: Text-aware diffusion speech tokenizer for speech language modeling. In Advances in Neural Information Processing Systems, 2025b. Preprint available at arXiv:2508.1670
Yuancheng Wang. Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. Maskget: Zero-shot text-to-speech with masked generative codec transformer. In ICLR. OpenReview.net, 2025c.
Simon Welker, Matthew Le, Ricky TQ Chen, Wei-Ning Hsu, Timo Gerkmann, Alexander Richard, and Yi-Chiao Wu. Flowdec: A flow-basee based full-band general audio codec with high perceptual quality. arXiv preprint arXiv:2503.01485, 2025.
15

---

Published as a conference paper at ICLR 2026
Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. Bigeodec: Pushing the limits of low-bitrate neural speech codec. arXiv preprint arXiv:2409.05377, 2024.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang. et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20211 2025.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint, Abs/2412.151155 155 2024a.
Dongchao Yang, Songxiang Liu, Haohan Guo, Jiankun Zhao, Yuanyuan Wang, Helin Wang, Zeqian Ju, Xubo Liu, Xueyuan Chen, Xu Tan, et al. Almtokenizer: A low-bitrate and semantic- audio codec tokenizer for audio language modeling. arXiv preprint arXiv:2504. /0344, 2025.
Haici Yang, Inseon Jang, and Minje Kim. Generative de-quantization for neural speech codec via latent diffusion. ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1251-1255. IEEE, 2024b.
Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, et al. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 25697-25705, 2025a.
Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi Dai, et al. Llasa: Scaling train-time and inference-time compute for Ilama-based speech synthesis. arXiv preprint arXiv:2502.04122, 042000280, 2025b.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang. Jongheon Jeong. Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. ICLR. OpenReview.net, 2025.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEF/ACM Transactions Audio, Speech, and Language Processing, 30:495-501 2021.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glice-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint 0rX11:2412.02612,, 2024a.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong, and Jie Tang. Scaling speech-text pre- pre-training with synthetic interleaved data. In ICLR. OpenReview.net, 2024b.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural infor- mation processing systems, 32, 2019.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308.1669) 16692, 2023.
Xueyao Zhang, Yuancheng Wang, Chaoren Wang, Ziniu Li, Zhuo Chen, and Zhizheng Wu. Ad- vancing zero-shot text-to-speech intelligibility across diverse domains via preference alignment. In ACL. Association for Computational Linguistics, 2025a.
Xueyao Zhang, Junan Zhang, Yuancheng Wang, Chaoren Wang, Yuanzhe Chen, Dongya Jia, Zhuo Chen, and Zhizheng Wu. Vevo2: Bridging controllable speech and singing voice generation via unified prosody learning. arXiv preprint arXiv:2508. 16332, 2025b.
16

---

Published as a conference paper at ICLR 2026
Xueyao Zhang, Xiaohui Zhang. Kainan Peng, Zhenyu Tang, Vimal Manohar, Yingru Liu, Jeff Hwang, Dangna Li, Yuhao Wang, Julian Chan, Yuan Huang, Zhizheng Wu, and Mingbo Ma. Vevo: Controllable zero-shot voice imitation with self-supervised disentanglement. In ICLR. OpenReview.net, 2025c.
Yue Zhao, Yuanjun Xiong, and Philipp Krähenbühl. Image and video tokenization with binary spherical quantization. arXiv preprint arXiv:2406.07545. 2024.
17

---

Published as a conference paper at ICLR 2026
A MODEL ARCHITECTURE
Model 'he tokenizer's architecture consists of causal encoder, vector quantizer (VQ), diffu- sion decoder, and a causal auxiliary CTC decoder. For the causal encoder and the causal auxiliary CTC decoder, we utilize standard Llama-style Transformer blocks (Touvron et al. 2023 Grattation et al. 2024), incorporating RoPE positional encoding (Su et al. 2024) and the SiLU (Elfwing et al., 2018) activation function. The encoder is specifically implemented with 16 causal Llama decoder layers, and the auxiliary CTC decoder with 4 such layers. A consistent configuration, unless other- wise specified, applies across these components (and the diffusion decoder): hidden size of 1536, an intermediate size of 4096, and 16 attention heads. The VQ module employs default config- uration of 32 dimensions, featuring a codebook of 65,536 entries, with updates managed via an exponential moving average (EMA) (van den Oord et al. 2017j. The diffusion decoder is realized using 16 layers, adapted from the causal IJama decoder structure into a non-causal form. Diffu- sion step embedding is incorporated by substituting RMSNorm (Zhang & Sennrich, 2019) with an Adaptive RMSNorm variant. To study the effect of model capacity, we scale the number of encoder and diffusion decoder layers while keeping other architectural settings fixed. We experiment with four configurations: S (8 encoder / 8 decoder layers, 0.63B parameters), B (12/12, 0.88B), L (16/16 16, 1.12B), and XL (24/24, I.61B). Unless otherwise specified, we adopt the L configuration our default setting.
B RELATED WORK
Discrete Speech Tokenizer Speech tokenizers are foundational components for speech language models. Early approaches (Zeghdoorr et al.] [02:ssélssee et al. 2022, Kumar et al.,(002) primar- ily focused on audio compression, relying on residual vector quantization (RVQ) (Zeghidour al., 2021 Lee et al. 2022) and operating at high frame rates and bitrates, which are suboptimal for lan- guage modeling. More recent work has shifted toward tokenizers specifically designed for language modeling, emphasizing low frame rates (Défossez et al.] 2024 Li et al. 2025a. Della Libera et al.,
2025 semantically rich representations (Zhang et al. [203 Défossez et al.) 2024 Li et 2025a Wang et al., [202aa Ye et al. (2025a b. Liu et al. 2024; Guo et al. 2024 Du et al. 2024bb Zhang et al.) 2025c Jiang et al.] 2025| Bie et al. 2025). and simplified single-laye codebooks (Parker et al. 2024 Xin et al. 2024 Ji et al. 2024). Nonetheless, many of these tokenizers still struggle to
achieve even greater compression rates, for instance, 12.5 Hz with a single codebook. While TaDi- Codec (Wang et al. 2025b) successfully reduced the frame rate 6.25 Hz by leveraging diffusion autoencoder and incorporating text into its decoder, this text-aware design inherently restricts its ap- plicability, rendering it unsuitable for speech understanding tasks. In this work, our goal is to design a tokenizer which can jointly achieve a sufficient compression rate for efficient language model- ing, preserve high-quality audio reconstruction, and learn effective, semantic-rich representations for understanding speech.
Diffusion-Basee Speech Tokenizers Diffusion-based approaches (Ho al. 2020 Song et al.
2021) have emerged as a promising direction, showing strong scalability and robustness at low token rates. Yet. most existing methods still adopt two-stage design: tokens
first extracted from self-supervised speech models (Baevski et al., 2020 Chung et al., 2021] Hsu et al. 202T] Chen al.) 2022 Chiu et al.) 20222 Radford al., (023). and only then are waveforms reconstructed through diffusion. This separation limits joint optimization, as the quantizer is not trained end-to- end with the decoder. Recent efforts (Welker et al. 2025 Yang al. 2024b) apply diffusion to improve de-tokenization fidelity, but remain constrained to relatively high token rates and still relay on two-stage modeling. Pushing diffusion-based tokenizers to ultra-low bitrates (e.g., below 0.2 kbps or 20 tokens/s) in a compact, Lunguage-mode--fienndd framework therefore remains open and critical challenge. In this work, we address this challenge with SiTok, a diffusion based speech tokenizer that unifies vector quantization and reconstruction modeling in end-to-ent framework, while introducing semantic regularization to ensure the learned codes are both highly compressive and semantically rich for speech language modeling.
Hsu
18

---

Published as a conference paper at ICLR 2026
C MORE EXPERIMENTS AND RESULTS
C.1 LIGHT-WEIGHT DIFFUSION HEAD
A primary cause of inefficiency in diffusion inference is the need execute a forward pass through the entire model at each denoising step. Our preliminary experiments revealed that for our autoencoder-based tokenizer, using simple regression loss (e.g., LI L2) alone can recon- struct speech with acceptable intelligibility. albeit with poor perceptual quality. This indicates that the main body of the decoder capable of generating the fundamental structure of the speech, while the iterative diffusion process primarily serves to refine its quality and detail.
Based on this insight, we propose partitioning the decoder Da to decouple the main structure gener- ation from the iterative refinement. We divide the decoder into two components: substantial main body, D Po and smaller Light-weight Diffuusion Head, D Descss The main body consists of the Vreas
initial, deeper transformer blocks, while the head is composed of the final few blocks.
During the decoding process, the quantized embedding sequence Zg is first passed through the main body Denain only once to produce base representation Muase:
hhase
chreain (Zz).
This base representation then provides the foundational conditioning, which subsequently re- fined by the diffusion head into the final spectrogram. The iterative denoising process is performed exclusively by the light-weight head Dorey which takes the noisy spectrogram æt and the base representation buse as conditioning predict the velocity:
(z f, hbsse) = Deraa (2 t, hbssse
This architectural modification significantly reduces the computational overhead per inference step, as the majority of the decoder's parameters 'D De.... are utilized just a single forward pass. This approach allows for rapid inference while retaining the high-quality synthesis capabilities of the diffusion model.
In this work, apply the proposed light-weight diffusion head to base model architecture with 16 transformer decoder layers. Specifically, the first 12 layers (3/4 of the decoder) are used as the main body De..... while the last 4 layers serve the diffusion head D Danan During inference, the base representation hrsse is computed once by the main body, and only the light-weight head executed iteratively across diffusion steps. With 16 diffusion steps the default setting, the theoretical speedup approaches a 4x reduction in per-step computation compared to applying all 16 layers at every denoising step.
As shown in Table 6 this architectural modification yields nearly identical reconstruction perfor- mance to the full model. The light-weight head maintains comparable WER and perceptual metrics (SIM and UTMOS), while significantly reducing the computational cost. This demonstrates that most of the heavy-lifting for content and structure generation handled by the main body, and the lightweight head suffices to refine acoustic detail during diffusion inference.
Table 6: Ablation study of the light-weight diffusion head.
Reconstruction
Model
WER
SIM
UTMOS
Base
4.06
0.641
3.44
w. light head
3.97
0.610
3.46
C.2 ZERO-SHOP SHOT TTS WITH SITOK
also evaluate SiTok in a downstream speech generation task zero-shot text-to-speech (TTS) to further demonstrate the effectiveness and efficiency of SiTok for speech language modeling. This experiment verifies that the discrete representations learned by SiTok are not only suitable for construction and understanding, but also serve as strong generation interface for speech language models.
19

---

Published as a conference paper at ICLR 2026
Table 7: Zero-shot TTS results comparing AR-O-AR--TTS with representative AR-based TTS sys-
tems. FPS/TPS follow the definition in Section 3.2
RTF is measured on a single A100 GPU.
Model
FPS/TPS
WER ())
SIM (1)
RTF (J)
CosyVoice 2 (Du et al. 2024b)
25/25
2.89
0.66
0,455
SparkTTS Wang et al. Llasa Ye et al. 2025b)
2025a)
12.5/12.5
2.50
0.57
0.601
50/50
3.94
0.58
0.422
SiTok-AR-77S
12.5/12.5
2.46
0.64
0.234
We build 0.5B parameeerr LLM-based TTS model (denoted as SiTok-AR-TZS AR-TS),, initialized from . 0-0. 5B Yang et al. 2024a), which autoregressively predicts SiTok tokens from text. The model trained on TOOK hours of the Emilia (He al. 2024) dataset under standard AR-TTS training recipe. During inference, the predicted discrete token sequence is decoded by our diffusion decoder to obtain mel-spectrograms, we use a default decoding step of 16 in this experiment.
We follow the evaluation protocol in Section 3.2 and report WER and SIM on the SeedTTS test-en set. To further assess practical efficiency, we also report the real-time factor (RTF) measured on single NVIDIA A100 GPU, averaging 10 runs of synthesizing 10-second utterance. Results are We use some strong --badeed TTS models as baselines.
summarized in Table 7
The results show that SiTok-AR-TTS achieves competitive superior intelligibilitt and speaker similarity compared to strong baselines, while operating at a substantially lower inference cost. Interestingly, the WER obtained by SiTok-AR-TTS is lower than the reconstruction WER of SiTok itself. This trend is consistent with some recent zero-shot TTS systems Du et al. (2024a); Guo Guo al. 2024 Zhang al. (2025a), where the generated speech tokens directly conditioned on the Text.
Another key observation the considerable efficiency gain. Because SiTok operates at only 12.5 Hz, the autoregressive text-to-token decoding runs on sequence 2 to 4x shorter than those used by conventional neural codecs operating at 25 50 Hz. This reduction directly translates into faster inference for speech generation, and results in significantly lower RTF of 0.234, making SiTok particularly attractive for large-scale TTS or speech generation systems. Overall, these findings demonstrate that SiTok serves as highly effective interface for speech generation: its discrete representations not only support high-quality reconstruction and strong downstream understanding. but also enable efficient, high-fidelity TTS within unified speech tokenization framework.
Observation: SiTok provides strong zero-shot TTS performance with high intelligibility and similarity, while its extremely low token rate enables substantially faster inference compared to existing AR- basedd TTS systems.
C.3 COMPARISON WITH ALTERNATIVE QUANTIZATION METHODS
SiTok is not tied specific quantization design. principle, any quantization method, such as Finite Scalar Quantization (FSQ) (Mentzer et al. 2023) and Binary Spherical Quantization (BSQ) (Zhao al. 2024), can be integrated into our diffusion autoencoder. In this section, we compare our standard VQ module with a representative alternative, Fixed-Scalar Quantiza- tion (FSQ). We follow a commonly used FSQ configuration with per-dimension cardinalities whose Cartesian product yields a codebook size of 2
[2, 2, 2, 65536, identical to the codebook size employed by our VQ setup.
Table summarizes the results. Despite having the same codebook size, standard VQ achieves stronger performance across reconstruction quality and downstream understanding tasks. We at- tribute this performance gap to two properties of our large-scale training regime. First, the learnable embedding vectors in VQ provide greater representational flexibility than the fixed sealar partitions of FSQ, enabling richer modeling of fine- finengriined acoustic and semantic structure. Second, with large batch sizes, EMA updates, and diffusion-baat based optimization, SiTok exhibits stable codebook utilization exceeding 95% throughout training, meaning that FSQ's typical advantage, improved
20

---

Published as a conference paper at ICLR 2026
quantization stability, offers limited benefit in setting. As result, VQ emerges the more pressive and empirically effective choice, though our results confirm that SiTok remains compatible with broad family of quantization techniques.
Table Comparison VQ and FSQ within SiTok.
Model
WER ())
SIM (t)
UTMOS ())
CTC ASR (L)
ASR (1)
ER (*)
SV (L)
KS (f)
SiTok (VQ)
4.06
0.641
9.50
4.95
63.5
13.8
96.9
SiTok (FSQ)
5.23
0.629
3.44
10.02
5.33
62.0
14.1
96.9
D REPRODUCIBILITY STATEMENT
To support reproducibility and facilitate future research, provide comprehensive implementation details of SiTok this appendix. Specifically, we include (1) detailed architectural specifications (Appendix A and pseudo-code for the SiTok model (Appendix D.I1). (2) pseudo-code outlining the core end-to-end training loop of the diffusion autoencoder (Appendix (D.2), and (3) additional information regarding training hyperparameters, data preprocessing, and other implementation con-
siderations (Appendix D.3
We also confirm that will release the full inference code and pretrained model checkpoints (on public, research-onle datasets) to the research community upon publication, enabling researchers to reproduce our results and further build upon SiTok.
D.1 PSEUDO-CODE FOR SITOK
Listing 1: Pseudo-code for SiTok.
class SiTok:
def (
self, in dim=122,
hidden_
[lddenn_size1156,
intermmediatt_size=4000
encoder_layere-16,
dccoder_laycr===16,
ctc_dbcoder_layc::-4,
num_heads-16,
vq_emb_dim=in downsample_factorr=.
vocab_st socab_size=3210,,
# temporal stacking (reduce FPS to 12.5 Hz) self. stackinn stackIn (dwnsnadddaaamaeeeeeaeaee self.stack stackOut (downsample_factor,
transformer encoder (Llama-style causal model)
self.encoder = LlamaModel
hidder_size internediate_size,
encoder_ incoeer_layers, num num_heads, in_dim downsample_factor))
vector quantizer (Binary Spherical Quantization)
self self.vg_i lii (hidden_size, ve_emb_dim) self.vg
sel = niin (vq_emb_dimg hidden_size)
diffusion decoder (DiT-style transformer) self.decodes DiT(
21

---

Published as a conference paper at ICLR 2026
hidden_size, incermediate_sipe. decoder_layers, nun_heads, use_cond-rrue, uee_alif[stepp-Tree)
CTC semantic decoder (Llama-style causal model)
self. cere_deeceerrree LlamaModel
hidden_size, Internediate_siee
stc__ccoder_layors, num_heads, vocab_size)
forward: training-time outputs for loss computation
def (seward x, x x_mask)
"""SiTok forward for training losses. """
# 1) stack encode to continuous latents
h self.stack_in
h self.encoder (h, x_mask)
# 2) vector quantization to discrete speech tokens 2 = self.ve_in (h)
vq_info = self.vg(z) cond self. (2_q)
# 3) forward fiffusion (flow matching)
t - sample_unifa form()
t U 1) eps N(O, I] noisy mel
eps = handn_like (x)
felaad diffuse eps, x s self.stack_inix.:
# 4) diffusion decoder prodicts flow / velocity flow decodee (x_t, cond)
5) сTC semantic logits (for semantic regularization) cte_logits_ wtfffcccceecoderr (cond)
return
"x": X,
# mel
"noise": eps, "flox_pred": flow_pred,
# dissusion noise target
# for flow-matching loss
"ctc_logits"; etc_logits,
4 for CTC loss
"vq_loss": vg_info (acinnooo"ommii commiit"] VQ commitment loss
forward diffusion (flow matching target)
def fiff (self, x, eps, t):
"""Apply forward diffusion to obtain noisy sample x_t."MM ""* # x_t (1 alpha (t)) * eps alpha (t)
practice alpha(t) implements the flow-matching schedule.
x t) eps + + return x
inference helpers
def incode(self, x, mask)
"""Encode mel into quantized VQ embeddings / indices. ***
h self.stack_in(xt
h self.encoder (h, mask)
2 sn (h)
22

---

Published as a conference paper at ICLR 2026
2_9, indices self.vq 2_9, indices
def decode (self, z_q, prompt-None, steps=N)
"""Reverse diffusion to generate mel from vQ enbeddings. cond self.ve_ous (z_q) mel diffuuson_reverse decoode,,,,r, cond, prompt, stops) return self.stackkout (mel)
D.2 PSEUDO-CODE FOR TRAINING LooP
Listing 2: Pseudo-Code for Training Loop SiTok.
for batch in dataloader:
# 1) prepare mel features and masks
me1 racter (batch. speech)
[H, T,
x mask
batch. speech mask
text ids
batch.text: ids
# semantic supervision
text_mas mask
batch.text. mask
4 2) forward pass through SiTok
out = sitok. forward (x, x_mask)
x_gt
out ["x"]
hoise out ["noise"]
flow_pred
out ("fiow_pred"]
etc_logits
out ["ctc_logits"]
vo_s
out ["va_loss"]
# 3) diffusion (flow-matching) loss
#
target velocity flow_gt x_gt nolse
eps
diff_loss Li (flow_ flow_gt)
CTC semantic loss
ctc_loss = CTC Loss (ogi_s, text
total loss
total_loss diff_loss + vg_loss lambda_ctc * ete_loss
# 6) optimization
pptimizer.zoro_graa
total_loss backward))
cliip_gradeetss (si (ttt paramettret (),, max_norm=0.5)
optimizer.stepp
23

---

Published as a conference paper at ICLR 2026
D.3 MORE IMPLEMENTATION DETAILS
Data and Preprocessing We use 2 million hours of in-house data to train our models. The dataset covers multiple languages, with English accounting for the vast majority. We do not apply additional preprocessing to the speech data, such as splitting into shorter segments; instead, we train directly on the original utterance lengths paired with their transcripts. We use 50 Hz. 128-bin mel-spectrograms extracted at a 24K Hz sampling rate, with hop size of 480 samples (20 ms) and a window size of 1920 samples ms). The STFT computed with 7mm = 1920 using a Hann window, and mel filters span [fii fuax] = [0. 12,000] Hz. Finally, we apply global mean-variance normalization to the mel features using precomputed statistics (mean 4.92, variance 8.14). as both the input and reconstruction targets of our tokenizer, while first stacking every four consecutive frames to reduce the frame rate to 12.5 Hz for more efficient training. For waveform synthesis, we employ a Vocos- based (Siuzdak 2024) vocoder to convert the mel spectrograms back to audio waveforms at 24K Hz.
Training We train all models for single epoch, corresponding approximately 450K steps. For optimization, we adopt the A (Loshchilov & Hutter 2019) optimizer with & = 0.9. 82 0.999, a weight decay of 0.01, and learning rate of 8 x 10-5 with warmup of 32K steps. To maximize GPU utilization and stabilize training over varying utterance lengths, we employ dynamic batch size strategy: each GPU, pack utterances until the total duration reaches roughly 300 seconds of speech, corresponding to around 3750 tokens at our 12.5 Hz token rate. This ensures that each batch maintains consistent computational footprint while preserving full- utterance training without segmentation.
E LIMITATIONS
While SiTok demonstrates strong performance on both speech reconstruction and downstream speech understanding tasks, still falls short of continuous feature representations. Future work will focus on closing this performance gap. Furthermore, our diffusion-based decoder poses challenges for streaming generation. We are currently investigating fine-tuning strategies, such as chunk-wise AR diffusion, to enable low-latency streaming outputs.
24


Reviews:
[{"review_id": "XUoDxvMGjs", "reviewer": "Reviewer_xL4L", "summary": "This paper proposes Speech Diffusion Tokenizer (SiTok), a diffusion autoencoder-based speech tokenizer. It aims to achieve extreme compression, high-quality reconstruction, and effective semantic representations for speech language modeling simultaneously. SiTok addresses the limitations of existing speech tokenizers in balancing compression rate, reconstruction quality, and semantic representation by jointly learning semantic-rich representations and high-fidelity audio reconstruction.", "questions": "1. Why is there no demo page?\n2. Ablation study on CTC loss supervision:\n  The ablation study shows significant improvements in both WER and similarity metrics when CTC loss supervision is applied. While the improvement in WER is understandable as it directly relates to the semantic alignment of the speech tokens with the text, the substantial increase in similarity is puzzling. Typically, semantic and acoustic aspects involve a trade-off, and enhancing one might come at the cost of the other. Could the authors provide more detailed experimental insights into why the similarity metric shows such a marked improvement? This would help in better understanding the model's behavior and the interplay between semantic and acoustic features.\n3. Comparison with Tokenizers Trained for Understanding Tasks:\n  Many of the tokenizers compared in the understanding tasks seem to be trained primarily for reconstruction tasks. The performance metrics on understanding tasks may not be as convincing as they could be. Could the authors provide additional comparative experiments with tokenizers that are specifically trained for understanding tasks, such as S3Tokenizer? This would help in assessing the true effectiveness of SiTok in the context of speech understanding.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Performance: SiTok achieves superior performance in both speech reconstruction and understanding tasks at extremely low bit rates (0.2kbps) and token rates (12.5Hz), outperforming several strong baselines. This highlights its potential for efficient speech representation and modeling.\n2. Innovative Design: The combination of diffusion models and semantic regularization is a novel approach in speech tokenization. These innovations enable SiTok to learn representations that are both acoustically faithful and semantically meaningful.\n3. Flexibility: The model's performance can be further enhanced by adjusting parameters such as model size and codebook configurations, offering flexibility for different applications.", "weaknesses": "1. Lack of Direct TTS Task Evaluation: While the paper demonstrates strong performance on various speech understanding tasks such as ASR, ER, KS, and SV, it does not directly evaluate the proposed SiTok model on TTS tasks. This is a significant gap, as TTS tasks have unique requirements and challenges that may not be fully addressed by the current evaluation metrics.\n2. Lack of Detail on Multi-Codebook CTC Decoder Implementation: The paper does not provide specific details on which layer the CTC decoders are applied to in the multi-codebook setup. This lack of clarity can make it difficult for other researchers to reproduce and build upon the work.\n3. Real-time Challenges: Despite efforts to accelerate decoding, diffusion models' iterative sampling steps may still pose latency issues for real-time applications. Further optimization is needed to improve real-time performance.\n4. Resource Intensity: The large model size (1.6B parameters) requires substantial computational and storage resources for training and inference, potentially restricting its deployment on resource-constrained devices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C8XgoITTTq", "reviewer": "Reviewer_mPsW", "summary": "The paper introduces the Speech Diffusion Tokenizer (SiTok), a diffusion-based autoencoder that jointly optimizes vector quantization and waveform reconstruction. To align its discrete codes with linguistic information, SiTok incorporates semantic regularization via a CTC decoder. To enhance efficiency, the authors apply shortcut fine-tuning and other acceleration techniques to reduce diffusion steps without compromising quality. Trained on 2 million hours of speech with 1.6B parameters, the paper claims that SiTok is effective even at an extremely low token rate (12.5 Hz, 0.2 kbps) and outperforms previous baselines.", "questions": "- What is the real-time factor (RTF) and actual inference latency in milliseconds for the method compared to single-pass baselines?\n\n- How does task performance change if the CTC decoder is scaled proportionally (e.g., 24 layers for the XL model) rather than keeping it fixed at 4 layers?\n\n- Was Finite Scalar Quantization (FSQ) evaluated as an alternative to vector quantization (VQ)?\n\n- Can statistical significance tests be provided for the reported performance gap, given that the improvement is minimal?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper’s strength lies in its simple framework consists of a diffusion autoencoder, vector quantization, and CTC supervision. While not conceptually novel, it’s highly practical. The auxiliary CTC loss proves crucial (WER drops from 33.0 to 4.06, Table 3). Refinements like shortcut fine-tuning and classifier-free guidance further boost efficiency, reflecting thoughtful system design.\n\n- The experiments are comprehensive. Table 5 explores eight design factors (e.g., loss type, codebook settings, frame rate) and evaluates both reconstruction (WER, SIM, UTMOS) and understanding (ER, SV, KS, ASR) tasks.\n\n- At 0.2 kbps (12.5 Hz), SiTok delivers competitive performance in a highly compressed regime. With a single codebook, it achieves WER 4.06 and SIM 0.641", "weaknesses": "- The paper prefers mel-spectrograms over raw waveforms without strong justification. Modern architectures can handle raw waveforms efficiently, and prior speech tokenizers (e.g., EnCodec, WavTokenizer, BigCodec, Mimi) operate directly on them successfully. Claims that waveforms are inefficient or adversarially complex are unconvincing, especially given that adversarial methods have been applied effectively in prior work. Mel-spectrograms are lossy and require a separate vocoder (Vocos), yet no comparison to waveform-based models is provided.\n\n- The method relies on a single VQ codebook, despite evidence (Table 5) that residual VQ (RVQ) substantially improves performance (WER 4.06 → 2.50 with 8 codebooks). Leading tokenizers employ RVQ to avoid bottlenecks at low token rates. The paper provides no rationale for using a single codebook, ignores alternatives such as FSQ.\n\n- Diffusion-based speech tokenization is not novel; prior work includes CosyVoice, Vevo, FireRedTTS, NaturalSpeech 3, DiTAR, and OZSpeech. The claimed novelty of end-to-end training with CTC supervision is not clearly superior to established two-stage approaches.\n\n- The regression loss achieves a WER of 4.66 compared to 4.06 for diffusion, so the improvement is far from marginal. This raises questions about the practical benefit of using diffusion loss over a simpler L1 loss.\n\n- The paper is titled \"SCALING Speech Tokenizers\" but Table 4 shows scaling doesn't work: the 1.61B (XL) model performs worse than the 1.12B (L) model on key metrics (ASR: 5.07 vs 4.95; SV: 14.7 vs 13.8).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hCbQLwOobU", "reviewer": "Reviewer_5u8g", "summary": "This paper focuses on an important topic in the audio domain—the design of speech tokenizers. By scaling both the model size and the amount of training data, the authors build a highly performant tokenizer and provide a detailed discussion of its training strategies from multiple perspectives.", "questions": "It is unclear whether the authors plan to release the model checkpoints or pretrained weights. Without open-sourcing, the paper’s overall impact and reproducibility would be significantly limited. Clarification on this point would be appreciated.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The paper scales the speech tokenizer up to a 1.6B-parameter model trained on 2 million hours of speech data, effectively enhancing model performance and achieving state-of-the-art results.\n\n- The authors conduct comprehensive ablation studies on various aspects of the tokenizer’s training process and offer insightful observations on effective training techniques.", "weaknesses": "- The current version appears to lack qualitative analyses or examples of actual reconstructed speech. Since audio reconstruction fidelity is critical for evaluating a speech tokenizer, it would be highly valuable to include qualitative results (e.g., audio samples) to better demonstrate the model’s effectiveness. **If such results are not presented in the rebuttal, I would consider decreasing my score.**\n\n- The main improvements in this work seem to stem from scaling the model and training data, while many methodological components follow prior work. Nevertheless, the systematic analysis and strong pretraining effort provide meaningful progress for the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AhzTk1jzZw", "reviewer": "Reviewer_WeZq", "summary": "This paper proposes SiTok, a diffusion-based speech tokenizer that achieves high compression ratios while maintaining high-fidelity speech reconstruction. A CTC loss–based semantic regularization method is introduced to enhance the semantic richness of quantized representations. In addition, the paper explores shortcut fine-tuning to accelerate diffusion inference. Extensive experiments demonstrate that SiTok delivers superior reconstruction and downstream understanding performance at low bitrates, and comprehensive ablation studies confirm the effectiveness of each module.", "questions": "1.While the paper focuses on reconstruction quality and semantic richness of the speech tokenizer, has the model been evaluated on TTS downstream tasks? Such results could better demonstrate its applicability to\ngenerative scenarios.\n\n2.The reconstructed audio achieves remarkably high SIM scores, even at 200 bps. Could the authors clarify\nwhich component primarily contributes to this improvement? Or does the large-scale training data (2M\nhours) play a dominant role in boosting SIM performance?\n\n3.The encoder is causal, whereas the decoder is not. The authors briefly mention this limitation, but I am\ncurious whether converting the diffusion-based decoder into a causal one would lead to significant\ndegradation in reconstruction quality, as streaming applications are of growing interest to the community.\n\nHere are some suggestions:\n\n1.The training configuration is insufficiently detailed. It would be helpful to include information such as batch\nsize and the number of GPUs used.\n\n2.The paper frequently mentions decoder fine-tuning, which seems to correspond to a two-stage training\nscheme. The authors are encouraged to clarify and include these details in the training description.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.**Strong performance at extremely low bitrates.** The proposed SiTok achieves excellent results in both speech reconstruction quality and downstream understanding tasks at very low bitrates. In reconstruction evaluation, SiTok at 200 bps reaches a SIM score of 0.682, with its UTMOS and WER metrics are comparable to those of previous SOTA codec models. On downstream understanding tasks such as LLM based ASR, ER, SV, and KS, SiTok exhibits rich and effective semantic representations.\n\n2. **Comprehensive and well-analyzed ablation studies.** The paper provides extensive ablation experiments that validate the effectiveness of each core component, including model scaling, accelerated diffusion strategies, CTC loss weighting, and reconstruction refinement. The corresponding analyses are clear and insightful.\n\n3. **Exploration of acceleration strategies for diffusion-based tokenizers.** The exploration of shortcut fine-tuning and a light-weight diffusion head offers an solution to accelerate the diffusion decoding process while maintaining high reconstruction quality.", "weaknesses": "1.The overall design of the proposed speech tokenizer lacks novelty. (1) Prior works such as LaDiffCodec [1]\nand TadiCodec [2] have already explored diffusion-based decoders to enhance reconstruction quality. (2)\nSimilarly, models like Baichuan Audio Tokenizer [3] and XY-Tokenizer [4] incorporate ASR-based\nsupervision to enrich semantic representations after quantization, although they adopt LLM-based\ndecoders rather than CTC loss.\n\n2.While the paper emphasizes SiTokʼs strong reconstruction quality at high compression rates, it omits\nimportant objective metrics such as STOI and PESQ, which weakens the persuasiveness of the evaluation.\n\n3.Although several strategies for accelerating diffusion inference are explored, the paper lacks concrete\nmeasurements and comparisons of reconstruction speed, such as RTF or related latency metrics.\n\n[1] Yang, Haici, Inseon Jang, and Minje Kim. \"Generative de-quantization for neural speech codec via latent\ndiffusion.\" ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing\n(ICASSP). IEEE, 2024\n\n[2] Wang, Yuancheng, et al. \"Tadicodec: Text-aware diffusion speech tokenizer for speech language\nmodeling.\" arXiv preprint arXiv:2508.16790 (2025).\n\n[3] Li, Tianpeng, et al. \"Baichuan-audio: A unified framework for end-to-end speech interaction.\" arXiv\npreprint arXiv:2502.17239 (2025).\n\n[4] Gong, Yitian, et al. \"XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech\nCodecs.\" arXiv preprint arXiv:2506.23325 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eBbG7SoHsV", "reviewer": "Reviewer_z58S", "summary": "The paper proposes several improvements to audio tokenizer training. \nFlow matching objective replaces the adversarial training, CTC loss to enforce semantic preservation, decoder finetuning, token CFG.", "questions": "Table 5: why are the understanding metrics on Loss R / R+D identical? aren’t you training the encoder/decoder jointly?\n\nWhy did you choose to make the encoder causal, and the decoder bidirectional? \n\nHow do you project and predict audio? (e.g. project audio to the encoder, project the noise audio to the decoder, predict the audio)\n\nHow about adding the raw audio WER/Sim/UTMOS results?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Auxiliary CTC loss is a good idea and is shown to be very useful.\nThe flow matching simplifies and speeds up training, and introduces stochastisity to decoding, which might improve audio quality (rather than a deterministic decoder).\nDecoder finetuning and Token CFG are intuitive ideas, that show benefits in practice.\nClear writing and multiple ablations.", "weaknesses": "Data is inhouse and not described in detail. Harder to reproduce or generalize to experiments done with different data mixtures.\nResults are shown for just one dataset - SeedTTS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)