Title: TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Speech Tokenizer

Text-to-Speech

Speech Language Modeling

Diffusion Model
Abstract: Speech tokenizers serve as foundational components for speech language models, yet current designs exhibit several limitations, including: 
(1) dependence on multi-layer residual vector quantization structures or high frame rates,
(2) reliance on auxiliary pre-trained models for semantic distillation, and
(3) requirements for complex two-stage training processes.
In this work, we introduce the **T**ext-**a**ware **Di**ffusion Transformer Speech **Codec** (***TaDiCodec***), a novel approach designed to overcome these challenges.
TaDiCodec employs end-to-end optimization for quantization and reconstruction through a diffusion autoencoder, while integrating text guidance into the diffusion decoder to enhance reconstruction quality and achieve optimal compression.
TaDiCodec achieves an extremely low frame rate of **6.25 Hz** and a corresponding bitrate of **0.0875 kbps** with a **single-layer codebook** for 24 kHz speech, 
while maintaining superior performance on critical speech generation evaluation metrics such as Word Error Rate (WER), speaker similarity (SIM), and speech quality (UTMOS),
Notably, TaDiCodec employs a single-stage, end-to-end training paradigm, and obviating the need for auxiliary pre-trained models.
We also validate the compatibility of TaDiCodec in language model based zero-shot text-to-speech with both autoregressive modeling and masked generative modeling, demonstrating its effectiveness and efficiency for speech language modeling, as well as a significantly small *reconstruction-generation gap*.
To facilitate reproducibility and further research, we will make our source code and pre-trained checkpoints publicly available.
Audio samples are are available at https://tadicodec.github.io/. We release code and model checkpoints at https://github.com/AmphionTeam/TaDiCodec.


Peer reviews:
comments: SUMMARY:
The paper proposes a text-aware, diffusion-based codec model that uses a diffusion decoder to generate speech. It can be trained with a single stage and achieves a low bitrate compared to other methods. They also show that the model has a smaller reconstruction-generation gap using zero-shot TTS. From the zero-shot TTS result, auto-regressive models trained with the codec can achieve better performance than other prior work.

STRENGTHS AND WEAKNESSES:
The paper proposes an interesting approach for speech codecs. By adding text conditions and prompts to the diffusion decoder, it can achieve a lower bitrate than prior work. It also shows promising zero-shot TTS results. 

However, some claims from the paper seem wrong. Firstly, they refer the auto-regressive zero-shot TTS as "speech language models", which should include a broader range of models, such as Moshi-style text-speech LMs. So "improving the reconstruction-generation gap for SLMs" seems to be overclaiming. Secondly, while they show a promising result on the zero-shot TTS task, there is no evidence on whether it benefits from the text-aware and the prompt, as other approaches don't have that. It will be beneficial to have ablation study on the text-aware component. A minor point: the results part directly refer to the number in the table, which is hard to follow.

QUESTIONS:
1. Can the authors provide an ablation study on the text-aware component?
2. From the reconstruction demo on the demo page, it seems the codec has some subtle changes to the prosody. As the codec is trained as a generative model, it makes sense to involve some randomness. Is it possible to introduce a metric to quantify that?

LIMITATIONS:
They are put in the appendix. I'm not sure if it's okay.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces TaDiCodec, a speech tokenizer that achieves low frame rate and bitrate using a single-layer codebook. The key idea is the integration of text information into a diffusion-based decoder. Unlike existing approaches that require multi-stage training, external pre-trained models for semantic distillation, TaDiCodec uses a unified diffusion autoencoder that jointly performs quantization and reconstruction. The authors showed the tokenizer's effectiveness through speech reconstruction tasks and zero-shot text-to-speech.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- Incorporating text-aware decoding into the diffusion process, eliminating the need for multi-stage pipelines\n- Achieving 6.25 Hz frame rate with 0.0875 kbps bitrate while maintaining competitive reconstruction quality\n- WER of 2.28 on SeedTTS test-en and 1.19 on test-zh showed SOTA performance\n\n**Weaknesses**\n- Only compares with Encodec, DAC, FACodec, SpeechTokenizer, and SemanticCodec. Missing some recent works: WavTokenizer, StableCodec, Single-Codec, BiCodec, and DM-Codec\n- The paper lacks evaluation on established comprehensive benchmarks: Codec-SUPERB, DASB, or ARCH benchmarks. Any particular reasons to ignore these benchmarks?\n- Codebook utilization rate is not reported, which is crucial for understanding quantization efficiency\n- No analysis of model size or memory requirements compared to baselines\n- TaDiCodec uses 100k hours while some baselines use different amounts. Is it a fair comparison?\n- Only 5,000 samples for test-en and test-zh, which may be insufficient for robust statistical significance analysis\n- Insufficient Ablation Studies:\n    - No ablation on the effect of text quality (e.g., using noisy ASR transcripts vs. ground truth)\n    - No analysis of performance degradation when text-speech misalignment\n    - Missing ablation on different text encoder architectures or pre-trained models\n- Limited Analysis of Failure Cases:\n    - Performance on edge cases (very long utterances, extreme speaking rates)\n    - Comparison of failure modes with text-free tokenizers\n- No evaluation on speech understanding tasks. Why evaluated only evaluated on TTS tasks?\n\nPlease feel free to point me if I missed anything. Thanks!\n\nQUESTIONS:\n- How does the model perform when text input is noisy or contains errors (e.g., from imperfect ASR)? Have you evaluated robustness to text quality degradation?\n- Can you provide more insight into why text conditioning enables such extreme compression? Is there a theoretical formulation that explains the information-theoretic relationship between text and the required speech token rate?\n- Have you explored distillation methods (e.g., consistency models) to reduce the number of diffusion steps to 1-2 while maintaining quality? This could significantly improve inference efficiency.\n\nLIMITATIONS:\nThe authors adequately address limitations in Appendix G", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TaDiCodec, a novel speech tokenizer designed for speech language modeling that achieves an extremely low frame rate of 6.25 Hz and bitrate of 0.0875 kbps using a single-layer codebook. The key innovation is the use of an end-to-end diffusion autoencoder with text-aware decoding, eliminating the need for multi-stage training, external pre-trained models for semantic distillation, or complex adversarial objectives.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Significant Compression Achievement: TaDiCodec achieves an unprecedented compression rate (6.25 Hz frame rate, 0.0875 kbps bitrate) while maintaining competitive reconstruction quality, representing a major advance in speech tokenization efficiency.\n\n2. The paper includes extensive experiments across multiple languages, challenging test sets (articulatory, code-switching, cross-lingual), and both reconstruction and generation tasks.\n\n3. The paper is written very clearly and is easy to read.\n\nWeaknesses\n1. Limited Application Scenarios: Although the title mentioned speech language modeling, the codec appears to be specifically designed for TTS tasks, rather than for speech language models that unify both speech understanding and generation. Due to its reliance on additional text information for decoding, the speech tokens are difficult to use for speech understanding tasks—especially those related to text, such as ASR. In contrast, the speech tokenizers used in models like Moshi and GLM-4-Voice are capable of handling both speech understanding and generation tasks.\n\n2.  The comparison in Table 1 is not fair. Although the paper claims that the token rate has been compressed to an extremely low frame rate, the table is actually misleading. This is because codecs like EnCodec, DAC, and Mimi reconstruct audio directly from speech tokens, while some of these codecs require additional conditioning information for reconstruction. Such a comparison is therefore not fair. I believe that at least one more column should be added to indicate whether each codec requires extra information for reconstruction, such as: none, prompt speech, global features, or text condition, etc.\n\nQUESTIONS:\n1. Regarding the diffusion decoder, since text and prompt speech can already achieve zero-shot TTS through NAR diffusion models [1][2], I am curious about what information is actually being learned by the 6.25 tokens per second. Are these tokens encoding the residual information, maybe speech - text - global? Providing more analysis on this point would add valuable insights to the paper.\n\n2.  In Figure 3a, the WER of reconstruction is generally better than that of generation, which is reasonable because reconstruction is based on the ground truth, while generation relies on the model’s learned distribution of real data. Therefore, the reconstruction of gt should theoretically serve as an upper bound for generation. That’s why I am very curious about, for TaDiCodec (ours) (en), the generation WER is even better than reconstruction. I believe that discussing this unconventional phenomenon would be very interesting and insightful.\n\n\n [1] Shen K, Ju Z, Tan X, et al. NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers[C]//ICLR. 2024.\n\n[2] Le, Matthew, et al. \"Voicebox: Text-guided multilingual universal speech generation at scale.\" Advances in neural information processing systems 36 (2023): 14005-14034.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TaDiCodec, a novel speech tokenizer designed to address several key limitations of existing methods. TaDiCodec uses a text-aware diffusion autoencoder to perform end-to-end optimization of both quantization and reconstruction, relying only on a diffusion loss. This approach eliminates the need for multi-stage training, auxiliary pre-trained models for semantic distillation, and complex adversarial objectives. The method achieves an extremely low token rate of 6.25 Hz with a single-layer codebook, while demonstrating strong performance in both speech reconstruction and downstream zero-shot text-to-speech (TTS) tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The proposed method is well-motivated and intuitively designed. It effectively addresses several major challenges in speech tokenizer design, including achieving an ultra-low frame rate, removing the dependency on pre-trained semantic feature extractors, and simplifying the training pipeline to a single end-to-end objective based on diffusion loss.\n* The paper provides a comprehensive and extensive comparison with numerous state-of-the-art audio tokenizers and downstream TTS systems.\n* The experimental results are strong, demonstrating that TaDiCodec outperforms most baselines in both reconstruction quality and, more importantly, in downstream zero-shot TTS tasks.\n\nWeaknesses:\n* The paper lacks clarity on some key implementation details. For instance, the specific downsampling and upsampling methods used before and after quantization are not described. The paper states that it uses mel-spectrograms as the reconstruction target, but it does not specify which vocoder model was used to generate the final audio waveforms. This is a critical detail needed when evaluating it compared to other methods.\n* While the current architecture, which relies on bidirectional attention and an iterative diffusion decoder, is primarily suited for audio compression and offline zero-shot TTS, this limits its applicability for streaming use cases where audio chunks must be generated sequentially before the full text token stream is available.\n\nQUESTIONS:\n* Since the vocoder is not specified, it could be assumed that the RTF calculation excludes vocoder inference time. Please clarify this explicitly.\n* While the contribution of this work and its significance is not diminished without this discussion, to broaden the work's impact on the wider community, it would be beneficial for the authors to discuss future directions for adapting TaDiCodec for streaming scenarios and its integration with real-time language models.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}