TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling
Authors: Yuancheng Wang
Dekun Chen
Xueyao Zhang
Junan Zhang
Jiaqi Li
Zhizheng Wu
Venue: NeurIPS 2025 poster ()
Keywords: Speech Tokenizer
Text-to-Speech
Speech Language Modeling
Diffusion Model
Primary area: applications
TL;DR: A novel speech tokenizer with an end-to-end diffusion autoencoder and text-aware decoding, operating at 6.25 Hz and 0.0875 kbps

Abstract: Speech tokenizers serve as foundational components for speech language models, yet current designs exhibit several limitations, including: 
(1) dependence on multi-layer residual vector quantization structures or high frame rates,
(2) reliance on auxiliary pre-trained models for semantic distillation, and
(3) requirements for complex two-stage training processes.
In this work, we introduce the **T**ext-**a**ware **Di**ffusion Transformer Speech **Codec** (***TaDiCodec***), a novel approach designed to overcome these challenges.
TaDiCodec employs end-to-end optimization for quantization and reconstruction through a diffusion autoencoder, while integrating text guidance into the diffusion decoder to enhance reconstruction quality and achieve optimal compression.
TaDiCodec achieves an extremely low frame rate of **6.25 Hz** and a corresponding bitrate of **0.0875 kbps** with a **single-layer codebook** for 24 kHz speech, 
while maintaining superior performance on critical speech generation evaluation metrics such as Word Error Rate (WER), speaker similarity (SIM), and speech quality (UTMOS),
Notably, TaDiCodec employs a single-stage, end-to-end training paradigm, and obviating the need for auxiliary pre-trained models.
We also validate the compatibility of TaDiCodec in language model based zero-shot text-to-speech with both autoregressive modeling and masked generative modeling, demonstrating its effectiveness and efficiency for speech language modeling, as well as a significantly small *reconstruction-generation gap*.
To facilitate reproducibility and further research, we will make our source code and pre-trained checkpoints publicly available.
Audio samples are are available at https://tadicodec.github.io/. We release code and model checkpoints at https://github.com/AmphionTeam/TaDiCodec.

TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling
Yuancheng Wang, Dekun Chen, Xueyao Zhang, Junan Zhang, Jiaqi Li, Zhizheng Wu
The Chinese University of Hong Kong. Shenzhen yuanchengvangol ink. cuhk. edu, en, wuzhizhengocuhk. edu. on
Abstract
Speech tokenizers serve as foundational components for speech language models, yet current designs exhibit several limitations, including: I) dependence on multi- layer residual vector quantization structures or high frame rates, 2) reliance on auxiliary pre-trained models for semantic distillation, and 3) requirements for complex two-stage training processes. In this work, we introduce the Text-awed ewa--aware Diffusiio Trunsformer Speech Codec (TaDiCodec), a novel approach designed to overcome these challenges. TaDiCodec employs end-to-end optimization for quantization and reconstruction through diffusion autoencoder, while integrating text guidance into the diffusion decoder to enhance reconstruction quality and achieve optimal compression. TaDiCodec achieves an extremely low frame rate of 6.25 Hz and a corresponding bitrate of 0.0875 kbps with a single-layer codebook for 24 kHz speech, while maintaining superior performance on critical speech generation evaluation metrics such as Word Error Rate (WER), speaker similarity (SIM), and speech quality (UTMOS). Notably, TaDiCodec employs single-stage end-to-end training paradigm, and obviating the need for auxiliary pre-trained models. We also validate the compatibility of TaDiCodec in language model based zero-shot text-to-speech with both autoregressive modeling and masked generative modeling, demonstrating
effectiveness and efficiency for speech language modeling, as well as a significantly small coansstructionggeennene gap. We will open source our code and model checkpoints. Audio samples are are available at https: :/tt::/aadiii /oweiioeec.ccccococooooooccceeec thub. io/. We release code and model checkpoints at https: //gi thub. http//mphiionn http///mphooTTTaa/aaa......
Introduction
Recent advances have been made in both large language model (LLM)-based text-to-speech (TTS) systems [1, 2, 3, 4, 5, 6, 7, 8, 9] and spoken language models [10, 11, 12, 13, 14, 15, 16, 17, 18]. At the core these systems lies the speech tokenizer, which converts continuous speech signals into discrete token scquences, thereby enabling the application of textual LLM paradigms to speech modeling. Beyond this, speech tokenizers play a fundamental role in bridging the text and speech modalities, forming the basis for cross-mode learning, alignment, and generation.
However, most existing speech tokenizers are suboptimal for speech language modeling. Prior works (e.g., EnCodec [19], SoundStream [20], DAC [21]) primarily target speech signal compression and transmission, relying on multi-layer residual vector quantization (RVQ) and operating high frame rates and bitrates. Such configurations make modeling with language models challenging and inefficient. More recently, several studies [5, 6, 22, 23, 24] have explored techniques for single-layer speech tokenizers. However, these approaches still fall short in reconstruction quality compared to RVQ-based tokenizers and often maintain high token rates (typically exceeding 50 tokens per second). Moreover, they usually depend on complex loss designs and adversarial training. Additionally,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

many of these models primarily optimize for acoustic-level reconstruction, resulting in discrete representations that lack semantic richness, making them suboptimal for language model modeling and causing recoosttruction-geeenne
gap.
Recent studies [2, 4, 10, 11, 25, 26] emphasize that effective speech tokens for language modeling should exhibit low frame rates and semantic richness, which criteria that directly shape the design of modern speech tokenizers. To achieve this, several works [10, 25, 26, 27] decompose speech into semantic and acoustic tokens by distilling features from speech self-supervised learning (SSL) models [28, 29, 30, 31]. In this framework, semantic tokens exhibit improved alignment with textual representations, thereby facilitating more effective language modeling. However, preserving reconstruction quality often requires RVQ. along with intricate loss functions, adversarial objectives. and the integration of external SSL models. An alternative line of work, including systems such as Cosy Voice [3], SeedTTS [2], FireRedTTS [4], and Vevo [32], adopts a two-stage design: first quantizing SSL-deriveed features, then training separate diffusion model [33, 34, 35] to reconstruct speech conditioned on these tokens. While this design enables relatively low frame rates and supports single-layer token representation, it comes with several limitations: 1) Two-stage training: the pipeline introduces greater architectural complexity and reduced training efficiency compared to end-to-end approaches; 2) External dependency: it relies pre-trained SSL or supervised models for semantic feature extraction; and 3) Struggle with extreme compression: most systems fail to achieve ultra-low token rates (e.g., fewer than 20 tokens per second), which are critical for modeling efficiency and scalability.
To address the limitations of current speech tokenizers, we propose the Text-aware Diffusion Trans- former Speech Codec (TaDiCodec), novel model that achieves an exceptionally low frame rate of 6.25 Hz. using a single codebook, corresponding to a bitrate of 0.0875 kbps for 24 kHz speech. Despite this ultra-low rate, TaDiCodec delivers high-fidelity speech reconstruction and robust per- formance downstream speech language modeling tasks. Specifically: 1) TaDiCodec unifies quantization and reconstruction within an end-to-end diffusion autoencoder, removing the need for separate semantic distillation or complex adversarial objectives by relying solely on diffusion loss; 2) it enhances reconstruction quality and compression efficiency by incorporating text and prompt guidance into the diffusion decoder. Our design is motivated by the increasing availability of transcriptions from automatic speech recognition (ASR) systems [36, 37J. and the widespread use of paired speech-text data in generative applications. In zero-shot TTS scenarios, for instance, the target text is inherently available; in end-to-end spoken language systems, speech and text tokens are typically generated jointly [12, 13, 14, 15, 16, 17, 38].
Our experiments show that TaDiCodec achieves performance comparable to or better than existing speech tokenizers in both reconstruction and downstream speech generation, while maintaining significantly smaller gap between reconstruction and generation. In addition, adopts a much simpler pipeline and operates with much fewer tokens. We evaluate zero-shot TTS using TaDiCodec under both autoregressive and masked language modeling settings, achieving strong results in intelligibility, speaker similarity, speech quality, and overall training and inference efficiency. A comparison with other tokenizers is presented in Figure and Table 1.
The contributions of our work are summarized as follows:
We propose TaDiCodec, a novel speech tokenizer with a token rate of 6.25 Hz and bitrate of 0.0875 kbps, based on a diffusion autoencoder that jointly performs quantization and re- construction without adversarial training. external pretrained models for semantic distillation, or multi-stage training. This design enables efficient optimization and simplifies the speech tokenization pipeline.
We introduce text-aware and prompt-guided decoding into the diffusion process to facilitate ex- treme compression. By leveraging paired speech-text data, this approach enhances reconstruction quality and enables high intelligibility, speaker similarity, and speech quality under ultra-low token rates.
We build zero-shot TTS models using our tokenizer under both autoregressive and masked language modeling settings, achieving WERs of 2.28 and 1.19 on Seed7TS test-en and test-zh, respectively. Our models demonstrate notable improvements on challenging benchmarks such as articulatory, code-switching, and Poss-lingual test sets, and support real-time inference with RTFs ranging from 0.12 to 0.29 across different model sizes.

---

CenyVoice Takenizer
Cadec
CosyVoce takenieer
(0.175 Kbps)
0.70
Semoniitade
SIM (Higher is Better)
BigCodot
Neviblamizer
0.65
Mima
BigCodoc (1.06 Kapel
Bicostec
BiCadec
Toxan
0.60
0.55
CosyVoice To Rakniizer
WarTolenizer (0.9 Kaps)
0.50
Mimi (0.825
0.45
(0 695 Wagc)
UTMOS (Highar Better)
3.2
WER (Lower is Better)
2.8
Figure 1: Comparison between TaDiCodec and other speech tokenizers. We use a three- dimensional coordinate system display the performance across three dimensions: the x-axis represents WER, the y-axis represents UTMOS, and the z-axis represents SIM. The size of the markers proportional the kbps value.
2 Related Work
Discrete Speech Tokenizer Discrete speech tokenizers convert continuous speech into discrete tokens, enabling modern zero-shot TTS and speech language modeling. Early tokenizers [19, 20, 21] focused audio compression, relying on residual vector quantization (RVQ) [20, 39] and operating at high frame rates and bitrates, settings ill-suited for language modeling. Recent work has shifted toward designing tokenizers tailored for language modeling, emphasizing low frame rates [10, 26], semantic- rich representations [4, 5, 6, 10, 25, 26, 27, 32, 40, 41], and single-layer codebooks [22, 23, 24]. Diffusion-bass methods [33, 34] have gained popularity for their performance at low token rates and scalability. However, they typically follow two-stage pipeline: extracting tokens via self-supervised speech representations [28, 29, 30, 31, 36, 42], then reconstructing waveforms through diffusion. For example, [43, 44] apply diffusion to improve de-tokenization quality, but still operate at relatively high token rates. Achieving ultra-low bitrates (e.g., below 0.2 kbps 20 tokens/s) with compact, generative-friendly framework remains a major challenge.
Zero-shot TTS Modern zero-shot TTS systems typically operate discrete speech tokens using either autoregressive (AR) language modeling [1, 2, 3, 4, 5, 6, 8, 9, 41] or masked generative (language) modeling (MGM) [7, 45, 46, 47J. Some models [2, 3, 4, 7, 41] adopt an "AR diffusion" framework, where diffusion decoder enhances waveform quality based on predicted tokens. Zero- shot TTS is also foundational in recent end-to-end spoken language models. For example, Qwen2.5- Omni [18] uses a "talker" module to generate speech tokens from the text output a "thinker." Similar architectures [15, 16] decode speech directly from text, while others [11, 12, 14, 17] leverage TTS models to synthesize large-scale speech corpora for training dialogue agents.
3 Method
3.1 TaDiCodec
Speech Tokenization with Diffusion Transformer Autoencoder Some speech tokenizers adopt raw waveform signals as modeling targets. However, raw waveforms often contain a considerable amount of redundant information. In this work, we instead adopt the mel-spectrogram as both the input and reconstruction target for the tokenizer, given its compactness and ease of inversion to waveform using vocoder models [48, 49]. Formally, denote the input mel-spectrogram as

---

X E RTxd where 7 denotes the number of frames, corresponding to the number of waveform frames divided by the hop size. The tokenizer's encoder & transforms x into a sequence of latent embeddings, i.e,, (x)). These embeddings are then quantized by the vector quantization (VQ) module Q into a discrete token sequence q = ((E(x)) E zT where Tg is the length of the token sequence, typically equal to 7' divided by a predefined down-sampling factor. Each token di (for i E [0,T,) T))) corresponds to an index in a codebook. The decoder 2 subsequently reconstructs the mel-spectrogram as & = (g).. Previous speech tokenizers primarily adopted generative adversarial networks (GANs) [50] for training the system, typically operating on short speech segments (e.g., 1-3 seconds) and employing CNNs as the backbone. However, GANs often suffer from issues related to training stability and efficiency, and the reliance on CNN- based architectures and short- -eemmet training further constrains the model's ability to capture maagge dependencies, leading to a focus on only local acoustic patterns. To overcome these limitations, we use a fully Transformer-baser [51] architecture for both the encoder and decoder, and adopt diffusion loss for reconstruction training, enabling more stable optimization and improved modeling capabilities. Specifically, we adopt flow sample Gaussian noise and generate a noisy target X via a linear interpolation: x, = Ix + (1 -)), where E [0. 1] is randomly sampled noise level. The model is then trained predict the velocity field v. defined as the derivative of x, with dx, respect to 1, L..., v di =x-6 e. We provide more details about flow matching in Appendix B.
matching-basse -basin decoder [35, 52]. During training,
Binary Spherical Quantization For quantization, Binary Spherical Quantization (BSQ) [53], which does not rely on an explicit learnable code- book. We first apply downsampling to the en- coder output (x), followed by a linear projec- tion to obtain a low-dimensional latent sequence: h = Linear( (Downsamlle(((x))) E RT where Tg is the number of quantized frames and L is the latent dimension. Each vector h, E IR.4 of h is then projected onto the unit sphere: u, h, [T. Binary quantization is applied independently on each dimen- sion: è, VI. sign(u,), where sign(x) is the element- wise sign function. To enable gradient flow through the quantization step, adopt Straight-Through Estimator (STE): signsTE (x) sg(sign(x) x) + x, where sg(+) denotes the stop-gradient operation. The quantized latent sequence E RT,xL is then mapped back to the d-dimensional space and upsampled to the original temporal resolution: Upsample( "psaaple(Lincarr Linear(fi) (i) E I Txd. Each quantized vector h, corresponds to a discrete token index computed by:
Quantizer
Encoder E
DiT Decoder
-V
D
Ntext
Training speech tokenizer with diffusion autoencoder. We optimize tok- enization and reconstruction end-to-end with diffusion loss. The input x is passed through the encoder and quantizer to get (x)),, which is then conditioned and input into the DiT decoder to predict the velocity y corre- sponding to the noisy xT.
Figure 2:
I. k = Σ 1 2/-1
where 11-1 is the indicator functioo.. As noted in [53]. BSQ can be optimized without the need for a commitment loss [54], since its quantization error is theoretically bounded. This property enables end-to-end training of the system using only the diffusion loss. See Appendix C for further details.
Text-aware De-Tokenization Most existing speech tokenizers rely solely on speech features for reconstruction. However, in the context of speech language modeling, the corresponding text associated with the speech is oflen readily available. For example, in TTS, the target text is always known, and in most end-to-end spoken dialogue systems, text and speech tokens are generated jointly [10, 11, 12, 13, 14, 15, 16, 17, 18]. Motivated by this observation, we propose a text-aware de-tokenization strategy, which conditions the diffusion decoder on the corresponding text sequence Xlexi- To further improve reconstruction quality under the extremely low compression rate setting, we introduce a prompt mechanism into TaDiCodec, similar to prior works [7, 55, 56, 571. This mechanism enables the model better reconstruct speech when prompt is provided, making particularly suitable for speech generation scenarios such as zero-shot TTS and the decoding stage of spoken language models. Specifically, during training, we randomly sample a prefix x prompe from the input mel-spectrogram by drawing segment length / ~ Uniform(0. (0, 0.25L., where I. denotes the total number of frames in the mel-spectrogram. The prefix is preserved without any added noise, while the loss is computed solely on the noisy portion of the sequence. Table 4 shows this prompt

---

mechanism yields substantial improvements in reconstruction performance. We also experiment with removing text conditioning from the decoder and observe sigmificant performance degradation under extremely low token rate and bitrate settings. e.g., at frame rate of 12.5 Hz, the WER exceeds 10.
Notably, Unlike prior works 3, 4, 11, 32, 40, 58] that adopt two-stage pipeline: first training a VQ model and then a separate diffusion model for de-tokenization, our tokenizer jointly learns feature quantization and reconstruction in an end-to-end manner. The overall training objective of TaDiCodec can be formulated as:
Laitt B ee [[[(x e) Do (x)
(2)
where Eg and Dg are the encoder and decoder parameterized by 0 and ø. We ignore the prompt for simplification. We also find that continuing to train the decoder while freezing the encoder and VQ module can further improve performance.
3.2 Speech Language Modeling with TaDiCodec
Existing speech tokenizers often neglect their effectiveness in downstream speech language modeling tasks and suffer from a pronounced tokenizer to large-scale multilingual zero-shot TTS, adopting an "AR + Difffusion" paradigm: an autoregressive model first predicts speech tokens from text Xtext. which are then passed, along with the text, to TaDiCodec's 's diffusion decoder to generate speech. The AR model, parameterized by v, optimized to minimize the negative log-likelihood of the target token sequence conditioned on the input text and previously predicted tokens:
cennsstruccionngeeraattt gap. In this work, we apply our
Ta
LAR =-Eig log p(q: (rexii))
where q; is the i-th token of q. We also apply the non-autoregressive Masked Generative Modeling (MGM) [7, 59] for modeling speech tokens. See more details about MGM in the Appendix D.
4
Experiments
We first describe the implementation details and datasets (Section 4. We then present the speech reconstruction results of TaDiCodec in Section 4.2, including the main results (Section 1,2. Table 1), multilingual performance (Table 2). subjective evaluation results (Table 3), and ablation studies on tokenizer design (Section 4.2.2, Table 4). Section 4.3 reports the zero-shot TTS results of models built upon TaDiCodec (Table 5), along with results on model size scaling and training and inference efficiency (Table 6), and an analysis of the reconstruction -enerrationnnn gap (Figure 3).
4.1 Experimental Settings
Datasets We use the Emilia [60] dataset to train all of our models. Emilia is a multilingual and diverse in-the-wild speech dataset designed for large-scale speech generation. It contains 46.8K hours of English, 49.9K hours of Chinese, 1.6K hours of German, 1.4K hours of French, 1.7K hours of Japanese, and 0.2K hours of Korean.
Implementation Details We build TaDiCodec using standard Llama-style Transformer blocks [61], with bidirectional attention instead of causal attention. The base configuration employs an 8-layer encoder and 16-layer decoder, each with hidden size 1024, intermediate size 4096, and 16 attention heads. We further explore decoder variants; see Section 4.2.2 and Table 4 for details. We adopt RoPE positional embedding [62] and RMSNorm [63]. For the text-aware diffusion decoder, RMSNorm is modified to Adaptive RMSNorm to condition on the diffusion step embedding. Text tokens are adapted from pretrained LLM vocabulary [64, 65], and concatenated with speech features along the time axis before being input to the decoder. For vector quantization, we use BSQ [53] with latent size of 14, yielding codebook size of 2 16384. All models are trained on 80GB NVIDIA A100 GPUs using dynamic batching with 200 seconds of speech per batch. We train the tokenizer for 800K steps using AdamW W [66] with a learning rate of 7.5 x 10-5 and 32K warmup steps. TTS models are trained for 300K steps with a learning rate of 3 x 10-4 unless otherwise specified. AR models extend the vocabulary of pretrained textual L.MMs [3, and are trained with 0.2B, 0.5B, 3.0B, and 4.0B parameters, see Section 4.3 for analysis. For MGM models, follow the setup of [7].
5

---

Table I: The comparison between TaDiCodec and other speech tokenizers. TaDiCodec offers extremely high compression rate, achieving a 6,25 Hz frame and token rate and 0.0875 kbps bitrate without requiring additional pretrained models for semantic distillation. It achieves comparable
better reconstruction quality than other speech tokenizers, based on generation-relatel metrics.
System
Frame Rate
Token Rate
Bitrate (kbps)
Codebook
Semantic
Reconstruction Quality
Number
Distill Free
WER
SIM (T) UTMOS
Token rale less than
EnCodee [19]
75
150
1.5
5.36
0.48
1.54
(RVQ)(211
1.75
DAC (VQ) [21]
75
0.75
12.74
0.45
2,08
Speechokkenieer [27]
50
100
7.96
0.46
2.47
Mini [10]
12.5
100
1.1
3.99
0.57
3.21
DualCodec [26]
12.5
0.925
2.63
0.62
3,78
DualCodec [26]
B:Codee [6] 76 k61
50
50
0.65
X
3.05
0.61
3,68
X-code: 16 MMC
0.62
3,68
WarTokemizerr [23]
0.48
3,36
BigCodec [22] 16 Ml::
80
80
1.04
3.25
0.61
3,59
TAAE [24] kife
25
0,4
11.08
0.41
3.87
Two stage, Diffinsion decoder
SemantCodle [40]
25
50
0.675
5.11
0.49
2,83
Tokenizer [32]
3.50
FacRedTTS Tokenizer [+]]
25
0.35
3.35
0.59
3,40
Cosy Voice Tokenizer [3]
25
0.3
5.63
0.47
3,65
Cosy Voice Tokenizer
0.325
Token rate less than 20
Two stage, Diffusion decoder
Ints Tokemizer (68)
12.5
12.5
0.175
7.14
0.67
3.37
One stage, Diffission decoder
TaDiCodee
0.0875
TaDiCCdee (w. detj*
6,25
6.25
0.0875
2.73
0.69
3.73
"w. def" denotes continued training of decoder for 400K additional steps, encoder and VQ modile
Evaluation We evaluate our approach from two main perspectives: speech reconstruction using the proposed tokenizer (Section 4.2) and zero-shot TTS performance (Section 4.3). We assess intelligibility (WER), speaker similarity (SIM), and speech quality (UTMOS), Speaker similarity is computed as the cosine similarity between WavLM-TDDE TDNN embeddings of the prompt and generated speech [28]. WER is measured using whisper- -arge--v -v3 [36] for non- non-Chinese languages and
paraf -ar [37] for Chinese, following prior work 3, 7, 58]. Speech quality is evaluated addition to objective metries, we conduct subjective evaluation via Comparative Mean Opinion Score (CMOS). We do not report signal-level metrics (e.g., PESQ, STOI), as our focus on generation-oriente performance, line with [58, 67]. Further Evaluation details provided in Appendix F.
using the official UTMOS checkpoint.
4.2 Speech Reconstruction
4.2.1
Main Results
We report our main results on SeedTTS test-en [2] in Table We also evaluate our methods on multilingual test sets in Table 2. Subjective evaluation results are show in Table 3.
Baselines We compare with wide range baselines settings where the token rate is less than 150: 1) single stage with multi-layer codebook and adversarial training: EnCodec [19], DAC [21],
SpeechTokeniize [27], Mimi [10], DualCodec [26]: single stage with single-layer codebook and adversarial training: DAC (with single VQ). BiCodec [6], X-codee 2 [5], Wav Tav'eieee [23],
BigCodec [22], TAAE [24]; 3) two stage with diffusion decoder: SemantiCodec [40], Vevo Tok-
enizer [32], FireRedTTS Tokenizer [4], Cosy Voice [3] & Cosy CosyVoice 2 Tokenizer [41], Ints Tok- enizer [68]. provide more detailed description of these baselines in Appendix E.
Results Analysis 1) Compression: TaDiCodec demonstrates significantly higher compression rate compared to all baselines. operates at a token rate of 6.25 Hz with single-layer codebook. resulting in bitrate of 0.0875 kbps. Among the baselines, the closest in compression rate to TaDiCodec the Ints Tokenizer, which has double the token rate and bitrate of TaDiCodec. However, it performs worse in terms of WER (7.14 vs. 2.73) and UTMOS (3.37 vs. 3.73) and requires two-stage training and semantic distillation. All other baselines have token rate greater than 25 and bitrate of at least 0.3 kbps. Compared to other single-stage and distillation-fre free models, BigCodec has higher

---

Table 2: Results of multilingual speech reconstruction. In addition English, we evaluate five other languages: Chinese (zh), French (fr),, German (de), Japanese (ja), and Korean (ko).
System
Bitrate
en
zh
fr
de
ja
ko
(kbps)
WER
SIM
WER
SIM
WER
SIM
WER
SIM
WER
SIM
WER
SIM
Mini [10]
1.1
3.99
0.57
2.87
0.59
20.71
0.55
16.12
0.59
25.71
0.44
36.10
0.57
BiCodec [6] /6
0.66
0.57
11.98
0.64
0.49
29.39
FireRedTTS Tokenizer [4]
0.35
3.35
0.59
1.99
0.68
20.16
0.56
13.87
0.61
18.57
0.48
32.20
0.62
TaDiCodec (w. dct)
0.0875
2.73
0.69
0.94
0.75
20.29
0.69
11.77
0.73
20.22
0.59
26.80
0.74
WER (3.25 vs. 2.73) and lower SIM (0.61 vs. 0.69) than TaDiCodec, with bitrate of 1.04 kbps. Models with lower bitrates, such as TAAE, still have bitrates four times higher than ours and perform significantly worse in WER and SIM. Other single-layer codebook tokenizers like BiCodec, X-codec 2. and Way Tokenizer have bitrates 7.4, 9.1, and 10.3 times higher, respectively. 2) Reconstruction Quality: In terms of WER, TaDiCodec achieves score of 3.02 without decoder continued-training and 2.73 with fine-tuning, ranking just behind DualCodec and X-codec 2. which have scores of 2.57 and 2.63, respectively, but with bitrates 10.6 and 9.1 times higher. Table 4 shows that our setting with a bitrate of 0.175 kbps achieves the best WER. In terms of SIM, TaDiCodec with decoder achieves the best SIM of 0.69, while even without decoder continued continued-training, it reaches 0.67, surpassing all baselines except for the Cosy Voice 2 tokenizer. In terms of UTMOS, our model achieves scores of 3.68 and 3.73 (with and without decoder contimued-training)) ranking just behind DualCodec and AE, which have scores of 3.78 and 3.87. However, these models operate at much higher bitrates of 0.925 kbps and 0.4 kbps and demonstrate poorer performance in SIM.
Results for Multilingual As shown in Table 2, TaDiCodec achieves the best WER on English, Chinese, German, and Korean, with especially low WER on Chinese. It also outperforms all baselines in speaker similarity across all evaluated languages.
Subject Evaluation Result As shown in Table 3, our proposed system achieves the highest CMOS score among evaluated baselines. More details about subjective evaluation are shown in Appendix F.3.
4.2.2 Ablation Study
In this section, we explore several designs for TaDiCodec. For the ablation study, we report the results on SeedTTS test-en and SeedTTS test-zh. 1) Vector Quantization Scheme: Replacing BSQ with a standard VQ tokenizer (implemented following [21, 69] with explicit codebook of the same size as BSQ) leads to consistent degradation across all evaluation metrics. This indicates that BSQ more effectively preserves both speech quality and intelligibility. 2) Tokenizer Size Scaling: Reducing the decoder size to 160M results in substantial performance drops, particularly in English WER. In contrast, increasing the decoder size results in marginal improvements. These results also imply the existence of a scaling law for TaDiCodec, warranting further investigation in future work. 3) Prompt Mechanism: The introduction of the prompt mechanism substantially improves all three evaluation metrics. A plausible explanation is that the prompt serves as a global conditioning signal (e.g., speaker identity), thereby reducing the quantizer's 's burden encode such global information. 4) Inference Time Scaling: Increasing the number of inference steps to 50 yields marginal improvements, while reducing it to 10 leads to slight degradation. However, further reduction to 5 steps results in a noticeable drop in performance. Considering the rrade-off between efficiency and quality, using 10 32 steps appears be a reasonable operating range. We aim to achieve comparable performance with fewer inference steps (e.g., 1-2 steps) by leveraging techniques such as [70, 71, 72]. 5) Decoder Continued-training: We explore freezing the encoder and the VQ module while only continued-trainine reconstruction. This approach yields further improvements, with WER dropping from 3.02 2.73 for English and from 1.11 to 0.94 for Chinese. SIM also improves for both languages. 6) Diffusion vs. GAN: We also replace the diffusion loss with PatchGAN [73], but observe a noticeable performance drop in both intelligibility and speech quality.
the decoder for an additional 400K steps, focusing solely on
4.3 Zero-shot TTS
In this section, we present the zero-shot TTS results using TaDiCodee as the prediction target. We evaluate two different language modeling approaches: autoregressive (AR) and masked generative
7

---

Table 3: Subjective CMOS
Table 4: Ablation study.
scores. We randomly choose 40 samples from a in-the-wild data source. Comparisons be- tween different models also be found in demo page.
Recon, Send
Recon. Seed zh
System
WER
SIM
UTMOS
WER
SIM
UTMOS
TaDiCodec
3.02
0.67
1.11
bug wg
3.30
0.64
3.44
1.25
0.72
2.46
w. prompt - wo. prompt
8.63
0.52
3.26
5,42
0.59
2.28
System
CMOS
decoder size:
160M
7.96
0.63
2,02
decoder size: 320M - 480M
2.90
0.69
3.68
1.02
0.75
2.73
Ground Truth
+0.28
Mimi [10]
-1.79
frame rate: 6.25 hz 12.5 kz
2.57
0.69
3.58
1.09
0.75
2.68
WayTokenizer [28]
-1.33 20.25
Inference steps: 50
2.87
0.68
3.66
1.07
0.75
2.68
DualCodec [26]
-0.92 20.31
Inference steps: 10
3.85
0.67
3.65
1.23
0.74
2.69
X-codec 2 [5]
-1.07 20.19
Inference steps: 5
7.89
0.65
3.19
1,96
0.73
2.35
TaDiCodec
0.00
w. decoder continued-training
2.73
0.69
3.73
0.94
0.75
2.69
Table 5: The zero-shot TTS results. Beyond regular cases, we also evaluate challenging scenarios, including articulatory, code-switching, and cross-lingual settings.
System
Frame
Rade
Regular
ab
Articolatory
dh
<n
Code-switching
zh WER
Ah2en
Cross-iiggal
alieen
WER
SIM
WER
SIM
WER
SIM
WER
SIM
WER
SIM
SIM
WER
SIM
WER
SIM
Roveline rystews
MIAGCC []]
2,40
2.28
0.77
14.5
0.60
0.74
19.79
0.26
8.47
0.20
16:22
0.56
(56)
93.75
3.87
0.71
14.13
0.61
0.66
32.63
0.68
19.93
0.64
13.78
0.45
ARS [7]
0.75
15.08
0.68
24.07
0.71
59.71
3.76
0.70
24.
0.56
Cloy Voice [41]
0.66
1.29
0.76
8.63
0.66
0.74
0.67
7.59
0.53
RaaRedITS
1.27
0.45
0.34
9.04
0.45
[68]
2.55
12:75
19.46
SparkTTS [6]
1.78
10.19
0.57
15.12
16.36
0.58
Liasa [5]
8.02
0.64
11:36
0.55
21.20
26.47
0.89
9.18
0.41
1.51
0.75
10.67
0.63
8.97
0.71
9.05
0.65
4.75
0.45
TaDiE>doo-MGM (0le
1.60
10.78
11.08
0.45
1.19
0.75
8.33
0.63
8.74
16.09 7.67
0.64
2.91
0.45
modeling (MGM) and we denote our models as "TaDiCodec-AR" and "TaDiCodec-MGMT respec- tively. The results are reported on eight test sets, including SeedTTS test-en and SeedTTS test-zh, referred to as Regular en and Regular zh, which are widely adopted benchmarks for TTS evalua- tion [2, 5, 6, 7, 41, 56]. In addition, we report performance on more challenging test sets, proposed in [68], covering articulatory scenarios (such as repeated words and tongue twisters), code-switching, and cross-lingual settings. We provide more details about the evaluation datasets in Appendix F.I.
Baselines We compare with a wide range of open-source and state-of-the-ae baselines including: 1) AR-based Systems: ARS [7], Cosy Voice 2 [41]. FireRedTTS [4], Ints [68]. SparkTTS [6]. Llasa [5]; 2) NAR-based systems: MaskGCT [7] and F5-TTS [56]. We provide more detailed description of these baselines in Appendix E.2.
Main Results We report the main results of our models and baselines on eight test sets in Table 5.
Our models exhibit significant improvements in intelligibility while maintaining speaker similarity comparable to state-of-the-art zero-shot TTS systems. In terms of WER, TaDiCodec-AR achieves the best performance on the Regular en and Regular zh test sets, reaching 2.28 and 1.19 respectively, and outperforming all baselines. On more challenging test sets, TaDiCodec-AR demonstrates even more pronounced advantages, for example, reducing WER from 15.03 to 9.16 on Code-switching en, and from 4.88 to 2.91 on Cross-lingual en2zh. Notably. these improvements are achieved without any task-specifis optimization or reinforcement learning fine-tuning [74, 75] on WER, as done in work such [41]. For TaDiCodec-MGM it consistently outperforms or matches the performance of state-of-the oftte-oftthe-eee thethe--rt art NAR zero-shot TTS systems across all test sets. Even with only 10 inference steps, which is significantly more efficient, it achieves a WER of 1.69 on Regular zh, compared to 2.28 from Mas On more challenging test sets, such as Cross linssll en2zh, it reaches 4.66 (vs. 13.78 from FS-TTS), and on Code-swilching en, achieves 14.94 (vs. 35.35 from FS-TTS). terms of SIM, both TaDiCodec-AR and AaDiCodec-MGNN show clear advantages over recent systems such as FireRedTTS, SparkTTS, and Llasa. Their SIM scores are slightly lower than those of MaskGCT and Voice 2, which operate higher frame rates of 50 Hz and 25 Hz, respectively.
8

---

Table 6: Results and RTF analysis for TTS model size scaling.
System
Model Size
KTF
Regular
Articulatory
Code-snüiching
Cros-lingud
WER
SIM
WER
SIM
WER
SIM
WER
WER
SIM
WER
Baxaline
ConyWeise Veioz [+1]
0:58
0:47
2,89
0.66
1.29
0.56
5,63
0.66
7,60
0.74
28.32
0.99
18.90
0.75
9.98
0.67
7.59
0.53
SpaekT75 (6]
0:58
0.59
2.50
0.57
1.78
0.66
10.19
0.57
13:37
0.65
15.12
0.46
16.85
0.65
9.73
0.58
4.58
0:40
Lizsa [-]
0:44
3.94
0.58
9,02
0.54
13.36
11.59
21.20
0..5
17.56
0..0
26.95
0.39
20.42
0..9
9.18
0.41
TaISCodee-MGM
0:68
0::2
3.69
0.65
1.51
0.63
8.97
14.76
20:01
0.33
9.85
0.48
JalsColee-AA-4..1
0.38
0:30
7.68
0.64
1.48
0.54
16.00
0163
12.54
0.30
16.18
0.56
33.91
0.32
13.40
0.64
4.56
0.48
0.58
0:32
5.63
0.65
1.45
0.75
13.00
0.63
9.94
0.70
13.58
0.57
17.10
0.38
8.79
0.64
4.07
0.48
MADNCader-ARR38
3008
0.25
5.34
0.05
1,23
0.75
8.34
1..03
8,32
0.70
11.31
0.57
15.47
0.33
7,85
0..0
3.99
0.48
TaloCondt-ARR-48
Tall An-ee-AR--18
slln
8.0
0.29
0:35
2.25
0.65
1.39
0.75
8:23
0.63
8.74
0.70
9:16
0.57
16.09
0.73
7.07
0.64
2.1
0.48
Figure 3: Performance gap between reconstruction and generation. Each system includes both
English and Chinese variants. Bars represent WER and SIM for reconstruction and generation.
Ado
(a) WER gap between reconstruction and generation.
(b) STM gap hetween reconstruction and generation.
Model Size Scaling, Training and Inference Efficiency We demonstrate that our 6.25 Hz tokeniza- tion is not only effective but also significantly more efficient for both training and generation. We further explore how scaling the model size affects both performance and efficiency. Results are shown in table 6. As described in the implementation details, we train all our TTS models for 300K steps. We find that the models achieve the optimal evaluation results at around 200K steps. All models can be trained in approximately one day under our setup, which uses NVIDIA A100 GPUs with flash attention and b precision. For inference efficiency, measure using Real- Time Factor (RTF). We use a 5-second speech as a prompt to generate approximately 10 seconds of speech, sampling 5 times and taking the average. The experiments show that even with 4.0B parameters, achieve an RTF of 0.29 without any deployment acceleration. With vL..M [76), the 4.0B AR model can achieve an RTF of 0.13. Additionally, the O.6B TaDiCodec-MGG 0.12. We also observe a reasonable improvement in performance with increasing model parameters, especially on challenging test sets (Articulatory, Code-switching, and Cross-lingual). Notably, our 0.5B model already matches or surpasses many sfate-of-the-an the-art systems with an RTF of 0.22.
AR model can
model achieves an RTF of
Reconstruction and Generation Gap In Figure 3, present the performance gap between recon- struction and generation across multiple systems. Our proposed system, TaDiCodee, demonstrates a notably small performance gap: -1655% for English WER (generation better than reconstruction), -5.8% for English SIM, +26.5% for Chinese WER, and -0.0% for Chinese SIM. These results indi- cate that TaDiCodec is highly generation-friendy wwweerild-ieeeeeeeiiiiiiie during generation. In contrast, existing systems such as Mimi exhibit a much larger degradation (e.g.- -00.55% en WER gap and -265.9% zh WER gap), suggesting that they are less effective in transferring reconstruction capabilities to generation. This highlights the advantage of our design in ensuring consistency between reconstructed and generated outputs.
most of the reconstruction quality
5 Conclusion
In this work, we introduce TaDiCodec, novel speech tokenizer that injects textual information into the decoder and incorporates a prompt mechanism within an end-to-end diffusion autoencoder training framework. TaDiCodec achieves extremely low frame rate of 6.25 Hz and a corresponding bitrate of 0.0875 kbps, using a single-layer codebook for 24 kHz speech. Beyond reconstruction, apply TaDiCodec to zero-shot TTS using both AR and MGM, demonstrating its effecttiveness,

---

efficiency, and suitability for generation. These results highlight TaDiCodec as viable and innovative solution for speech language modeling.
Acknowledgment
The authors gratefully acknowledge the funding support from the National Natural Science Foun- dation of China (Grant No. 62376237), the Shenzhen Science and Technology Program (Grant No. ZDSYS2023062609 0002266091320000 1302006), the Shenzhen Research Institute of Big Data (Internal Project Fund, Grant No. T00120230002), and the 2023 Shenzhen Stability Science Program. We would also like to thank the anonymous reviewers and the Area Chair for their insightful comments and valuable suggestions, which helped improve our paper.
10

---

References
[1] Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02. 01111, 2023.
[2] Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-Its: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406. 02430, 2024.
[3] Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024.
[4] Hao-Han Guo, Kun Liu, Fei- Yu Shen, Yi-Chen Wu, Feng- Xie, Kun Xie, and Kai-Tuc Xu. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint wrxiv:2409.03283, 2024.
[5] Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi DAI, al. Llasa: Scaling train-time and inference-time compute for llama-based speech synthesis. arXiv preprint arXiv:2502.04/28 2025.
[6] Xinsheng Wang, Mingqi Jiang. Ziyang Ma. Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, et al. Spark-uts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens. arXiv preprint S.Xi::5003001710 2025.
[7] Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xucyao Zhang, Shunsi Zhang, and Zhizheng Wu. Maskget: Zero-shot text-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.0077: 2024.
[8] Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pictquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 1 11:1703- 1718, 2023.
[9] Puyuan Peng, Yao Huang, Daniel Li, Abdelrahman Mohamed, and David Harwath. Voice- craft: Zero-shot speech editing and text-to-speech in the wild. arXiv preprint 1697733403.66973 2024.
[10] Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.0000 000377 2024.
[1]] Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong, and Jie Tang. Scaling speech-text pre-training with synthetic interleaved data, 2024. URL https: //arxiv.org/abll abs/ 2411 17607.
[12] Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. 4locevoiice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02612.22 2024.
[13] Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang, Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong, et al. aaichuan-audiin A unified framework for end-to-end speech interaction. arXiv preprint arXiv:2502.172..9 2025.
[14] Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, et al. Step-audio: Unified understanding and generation in intelligent speech interaction. arXiv preprint arXiv:2502. 1946, 2025.
[15] Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. arXiv preprint urXiv:2409.0660 06666, 2024.
11

---

[16] Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie. Ke Li, Xing Sun, and Long Ma. Freeze-omni: smart and low latency speech-to-speede dialogue model with frozen 1lm. arXiv preprint urXiv2411.00077 2024.
[17] Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report. arXiv preprini arXiv:2504. 18425, 2025.
[18] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, et al. Qwen2. S-omni technical report. arXiv preprint 2rXiv:2503.20215. 2025.
[19] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210. 2022.
[20] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
[21] Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36, 2024.
[22] Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. Bigcodec: Pushing the limits of low-bitrate neural speech codec. arXiv preprint urXiv:2409.05377, 2024.
[23] Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang. Xize Cheng, Zehan Wang, Ruiqi Li, al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint 2rXiv:2408. 2024.
[24] Julian D) Parker, Anton Smirnov, Jordi Pons, CJ Carr, Zack Zukowski, Zach Evans, and Xubo Liu. Scaling transformers for low-bitrate high-quality speech coding. arXiv preprint arXiv:2411.198422 2024.
[25] Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, et al. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25697-25705, 2025.
[26] Jiaqi Li, Xiaolong Lin, Zhekai Li, Shixi Huang. Yuancheng Wang, Chaoren Wang, Zhenpeng Zhan, and Zhizheng Wu. Dualcodec: A wwwffrae-rate, semantically-cnhang neural audio codec for speech generation. In Proceedings of Interspeech 2025, 2025.
[27] Xin Zhang, Dong Zhang, Shimin Li. Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308. 16692, 2023.
[28] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre- training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 1665)1505--518, 2022.
[29] Alexei Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: framework for self-supervised learning of speech representations. Advances in neural information processing systems,
12460, 2020.
[30] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEF/ACM transactions on audio, speech, and language processing, 29:3451-3460, 2021.
[31] Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random-projection quantizer for speech recognition. In International Conference on Machine Learning, pages 3915-3924. PMLR, 2022.
12

---

[32] Xueyao Zhang, Xiaohui Zhang, Kainan Peng, Zhenyu Tang, Vimal Manohar, Yingru Liu, Jeff Hwang, Dangna Li, Yuhao Wang, Julian Chan, al. Vevo: Controllable zero-shot voice imitation with supervsse disentanglement, arXiv preprint arXiv:2502.07243, 2025.
[33] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-666 6851, 2020.
[34] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1, 1,1000111114555. 13456, 2020.
[35] Yaron Lipman, Ricky TQ Chen, Heli Ben- Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[36] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 18492-28511 PMLR, 2023.
[37] Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, et al. Funasr: A fundamental end-to-end speech recognition toolkit. arXiv preprint arXiv:2305. 1013, 2023.
[38] Heting Gao, Hang Shao, Xiong Wang, Chaofan Qiu, Yunhang Shen, Siqi Cai, Yuchen Shi, Zihan Xu, Zuwei Long, Yike Zhang, et al. Lucy: Linguistic understanding and control yielding early stage of her. arXiv preprint arXiv:2501. /6327, 2025.
[39] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11523-11532, 2022.
[40] Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. Seman- ticodec: An ultra low bitrate semantic audio codec for general sound. IEEE Journal of Selected Topics in Signal Processing, 2024.
[41] Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint 2rXiv:2412.101 2024.
[42] Yu-An Chung, Yu Zhang, Wei Han, Chung- Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244-250. IEEE, 2021.
Chiu, James Qin, Ruoming Pang, and
[43] Simon Welker, Matthew Le, Ricky TQ Chen, Wei-Ning Hsu, Timo Gerkmann, Alexander Richard, and Yi-Chiao Wu. Flowdec: A flow-based full-band general audio codec with high perceptual quality. arXiv preprint arXiv:2503.01448 2025.
[44] Haici Yang, Inseon Jang, and Minje Kim. Generative de- quantizzaiion for neural speech codec via latent diffusion. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1251-1255. IEEE, 2024.
[45] Zeqian Ju, Yuancheng Wang. Kai Shen, Xu Tan, Detai Xin, Dongchao Yang. Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprini a0Xiv:2403.03 2024.
[46] Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint 23XXx::3000506330 2023.
[47] Yifan Yang. Shujie Liu, Jinyu Li, Yuxuan Hu, Haibin Wu, Hui Wang, Jianwei Yu, Lingwei Meng, Haiyang Sun, Yanqing Liu, et al. Pssueo-autoregressive neural codec language models for efficient zero-shot text-to-speech synthesis. arXiv preprint 250i 10352, 2025.
13

---

[48] Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with scale training. arXiv preprint S1t1:22006046658, 2022.
[49] Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306. 00814, 2023.
[50] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 (3(11))139-144 2020.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 2022.
[53] Yue Zhao, Yuanjun Xiong, and Philipp Krähenbühl. Image and video tokenization with binary spherical quantization. arXiv preprint (CLV::2006075445 2024.
[54] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[55] Matthew Le, Apoorv Vyas, Bowen Shi. Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilin- gual universal speech generation at scale. Advances in neural information processing systems, 36, 2024.
[56] Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885, 2024.
[57] Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, et al. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot us. arXiv preprint arXiv:2406. 18009, 2024.
[58] Yuancheng Wang, Jiachen Zheng, Junan Zhang, Xueyao Zhang, Huan Liao, and Zhizheng Wu. Metis: A foundation speech generation model with masked generative pre-training. arXiv preprint arXiv:2502.033228 2025.
[59] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[60] Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang. Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. arXiv preprint arXiv:2407.0553 033661,,,,36, 2024.
[61] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
[62] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 568:127063, 2024.
[63] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
[64] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
14

---

[65] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:24/2. 15115, 2024.
[66] Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[67] Junan Zhang. Jing Yang, Zihao Fang, Yuancheng Wang. Zehua Zhang, Zhuo Wang, Fan Fan, and Zhizheng Wu. Anyenhance: A unified generative model with prompt-guie gompt-guidance and self-critic for voice enhancement. arXiv preprint arXiv:2501.15417, 2025.
[68] Xueyao Zhang, Yuancheng Wang, Chaoren Wang. Ziniu Li, Zhuo Chen, and Zhizheng Wu. Ad- vancing zero-shot text-to-speech intelligibility across diverse domains via preference alignment. arXiv preprint arXiv:2505.044 041 13, 2025.
[69] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint
2021.
[70] Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
[71] Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024.
[72] Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. arXiv preprint arXiv:2410. 12555, 2024.
[73] Ugur Demir and Gozde Unal. Patch-based image inpainting with generative adversarial net- works. arXiv preprint arXiv: /
2018.
[74] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53741, 2023.
[75] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[76] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principlex, pages 61 1-626, 2023.
[77] Stefan Elfwing, Eiji Uchibe, and Kenji Doya. weeighd-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
[78] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats fiffusioon-tokenizrr is key to visual generation. arXiv preprint arXiv:2310.05737, 2023.
[79] José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa Improved masked image generation with token-critic. In European Conference on Computer Vision, pages 70-86. Springer, 2022.
[80] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang. and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation. arXiv preprint 2408. 2024.
[81] Loie Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Dup- penthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, al. Seamless: Multilingual expressive and streaming speech translation. arXiv preprint arX0v:2312.05188, 2023.
15

---

[82] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309. 15505, 2023.
[83] Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in Neural Information Processing Systems, 35:28708-28720, 2022.
[84] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprini arXiv:2407.2178: 2024.
[85] Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A 2019.
puassiell-multilinggen speech corpus. arXiv preprint
[86] Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, al. Didispeech: A large scale mandarin speech corpus. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1968-6972. IEEE, 2021.
[87] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[88] Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint 12X11 2022.
16

---

A Implementation Details
A.1 Model Architecture
All our models follow the standard Transformer [51, 61] architecture, employ RoPE positional encoding [62] and the SiLU [77] activation function. The encoder and decoder of the tokenizer and MGM models use bidirectional attention, while the AR models adopt causal attention.
The M-DiCoddccAR-O..35 and TaDiCodec-AR-3B models are initialized from the textual LLMs [65], respectively, while TaDiCodec-AR- 5-mini iittiii-insuucttt [64].
Quen2. 5-0. 5....BB-. 50-55B-nnnn Instruct and Qwen2.
4B is initialized from Phi-3.5-mini
Table 7: Model configurations.
Num. hidden layers
Num. attention heads
Num. key value heads
Num.
Model
Hidden size
Intermediate size
parameters
TaDiCodec Encoder
8
1024
4096
16
16
-.16B
TaDiCodec Decoder
16
1024
4096
16
16
0.32B
TaDiCodoe-AR-O.2
672
24
0.25B
TaDiCodoc-AR-0055
896
4864
24
0.5B
TaDiCodo--RR-B
2048
11008
36
16
2
3B
TaDiCodse-AR-4B
3072
8192
32
TaDiCodec-MGM
1280
5120
16
0.6B
B
Flow Matching
We provide additional details of the flow matching framework used to train the diffusion decoder in TaDiCodec. Flow matching [35] defines a continuous transformation from a prior distribution (e.g.. Gaussian noise) to a target data distribution (e.g., mel-spectrograms) by learning time-dependent velocity field along an interpolated trajectory x,-
While multiple interpolation strategies can be used to construct x,, we adopt the optimal transport path formulation [35, 52], instantiated in this work simple linear interpolation. Specifically, given a clean mel-spectrogram x E Rxxd and noise sample e N0, I), we construct an intermediate sample as:
= xx + ()e,),, t)e, ~~ (0.
where is sampled uniformly from [0, 1], and x, denotes the noisy input at time 1. The corresponding ground-truft velocity the temporal derivative x,;
dx dt
=
(5)
The diffusion decoder Ds is trained to predict v, conditioned on the token sequence 4 Q(Ea(x))
and the associated text texts using the following objective:
Laitt = B(x.x.x))...
[|](x e) Do I.. x.xex))
(6)
Inference At inference time, start with noise sample x N(O, and solve the ordinary differential equation:
dx; D,
from / 0 to r using a simple Fuler ODE solver over discretized set of N time steps.
Flow matching provides a stable and interpretable training signal by directly supervising the instanta- neous direction in which noisy sample x, should evolve match the clean target In setting, it enables effective training of the speech tokenizer under low bitrate constraints.
C Binary Spherical Quantization
1 VL VL
Binary Spherical Quantization (BSQ) [53] optimizes over an implicit codebook CasQ = which corresponds the I-dimennsionl hypercube projected onto the unit sphere. Each corner Cx CBsQ represents unique discrete token & E {0. .O.....2* 24 1).
17

---

Given an encoder output (x),, we first obtain a low-dimensional latent sequence h E RTxx1. after linear projection. BSQ then projects each vector h, onto the unit sphere:
h
(8)
and performs binary quantization independently on each dimension:
0, VL sign(u,),
where sign(x) is the element-wise sign function, with sign(0) defined to ensure codewords lie on the unit sphere. To enable gradient-based training, BSQ uses the arrighh-Through Estimator (STE) for backpropagation:
signsTe(x) sg(sign(x) - -x) + +x,, where sg(.) denotes the stop-gradient operation. For each vector H,, the corresponding discrete token index is computed as:
(10)
Σ [h.,000 2'-1,
(11)
where [[]] is the indicator function. This efficient implicit code assignment scheme allows fast token computation and decoding via bitwise operations.
BSQ offers several appealing properties: it avoids the need for an explicit learnable codebook; its quantization error is bounded, allowing the entire system to be trained without a commitment loss [54].
In this work, use 14, resulting in a codebook size of 2 16384.
D Masked Generative Models
In this section, we provide a brief introduction to masked generative models (MGMs) [7, 59, 78]. Let [yr, (2,,,y2... V2...... Ynl denote discrete sequence of length n. At each time step define the masked input as X, = x ④ m, where m, [m,.1, m2.2...... mn,nl is a binary mask. Specifically, x; is replaced with special [MASK] token if m,,; = 1. and remains unchanged if m,, O. Each mask element m,,i independently sampled from a Bernoulli distribution with parameter y(). where (() E (0. 1] is a masking schedule function (e.g., y(1) sin x for E (0, The fully unmasked input is denoted by X = x.
MGMs are trained reconstruct the original sequence from partially observed inputs, conditioned on an optional context e (e.g., this paper, text Xtexr condition), by modeling the conditional distribution (x xr, c). The model parameters A are optimized by minimizing the expected marginal cross- cross-entropy over the masked tokens:
Lmask -xEx, m Σ
mys log Pe xx,
(12)
t=1
At inference time, MGMs generate tokens in parallel via iterative decoding. The process begins with a fully masked sequence xr. Assuming a total of S decoding steps, at each step j E .I.....S.. prediction to is sampled from (x xy-(--1- (..))--)......... c). Then, [n y(7 - j {) tokens are selected masked sequence x
based on confidence scores be remasked, resulting
The confidence score for 9/ in to is given by Po(y; XT-O-D- if the position was masked; otherwise, its score is set to 1, indicating that unmasked tokens will not be remasked. The [n y(T j D1 tokens with the lowest confidence scores are selected for masking.
Note that the method for computing confidence scores is not unique. For example, [79] propose Token-Critic, separate critic model trained to estimate token-wiss confidence, thereby guiding the sampling process. In addition, [79, 80] suggest that masked generative modeling can be interpreted simplified form of discrete diffusion.
In this work, we develop MGM models for text-to-token. Given the low token rate of 6.25 Hz. the task is relatively casy to model, and 10 25 inference steps sufficient to achieve good results.
18

---

E
Baselines
E.1 Speech Tokenizer
EnCodec [19] A Residual Vector Quantization (RVQ)-based neural audio codec operating at frame rate 75 Hz. We use two codebooks for inference, achieving bitrate of 1.5 kbps. We use the official checkpoint'.
DAC [21] An improved VQGAN-based [79] codec that projects latent features onto a low- dimensional space (e.g., 8 dimensions) prior to quantization. We reproduce two variants: one urilizing three codebooks at 25 Hz frame rate, and the other single codebook at 75 Hz frame rate. Both configurations operate at token rate of 75 Hz. and achieve bitrate of 0.75 kbps.
[27] Enhances first-layer speech tokens via semantic distillation using features from HuBERT [30]. This tokenizer operates 50 Hz and we two codebooks for inference. We use the official checkpoint".
Mími [10] Follows the design of SpeechTokenize but utilizes WavLM [28] for semantic distillation. The tokenizer employs eight codebooks, each size 2,048, at 12.5 Hz frame rate, resulting in a bitrate of 1.1 kbps. We use the official checkpoint
CoaeCodec [26] state-of-the-are low-frame-rate, semantically-enhanced neural audio codec designed for speech generation. DualCodec directly encodes SSL representations [42] into first-layer codec tokens. adopt a configuration with a 12.5 Hz token rate and a 8-layer codebook hierarchy. The first codebook contains 16,384 entries, while the remaining five each contain 4,096 entries, yielding bitrate 1.225 kbps. We use the official checkpoint".
tokenizer with a single-layer codebook. It discretizes audio into semantic tokens based on features from wav2vec 2.0 [29]. It operates at token rate 50 Hz with codebook size of 8,192. achieving bitrate of 0.65 kbps. We use the official checkpoint
BiCodec [6] A semantically-enhane
Maw2Vcc2-BERY [81] and an acoustic encoder for low-level acoustic features. Their outputs are concatenated prior to quantization. It operates at token rate of 50 Hz with codebook size of 65,536, yielding bitrate of 0.8 kbps. We use the official checkpoint
X-codec 2 [5] Employs a dual-encoder design: semantic encoder based
Tok'eizzr [23] A single-codebook tokenizer trained on 800K hours of mixed-domain audio. It operates at a 75 Hz token rate with codebook size of 4,096, resulting in a bitrate of 0.9 kbps. We use the official checkpoint'.
BigCodec [22] A single-codebook tokenizer with scaled model size. It integrates sequential modules into convolutional architectures and applies low-dimensional quantization to enhance code utilization. It operates at an Hz token rate with a codebook size of 8, 1,29,, yielding a bitrate of 1.04 kbps. We use the official checkpoint
TAAE [24] A transformer-bart based tokenizer using Finite Scalar Quantization (FSQ) [82] for speech tokenization. It operates at 25 Hz token rate with a codebook size of 46,656, resulting in bitrate of 0.4 kbps. We the official implementation"
SemantiCodee [40] Combines a semantic encoder (AudioMAE [83] with k-means clustering) and an acoustic encoder, featuring diffusion decoder for reconstruction. It operates at 50 Hz token rate, with codebook sizes of 16,384 (semantic) and 2,048 (acoustic), achieving a bitrate 0.675 kbps. We
the official implementation
19

---

Vevo Tokenizer [32] A two-stage tokenizer utilizing features from HuBERT [30], followed by VQ and diffusion decoder. employs single codebook of size 8,192 a 50 Hz token rate, resulting in a bitrate of 0.65 kbps. We use the official checkpoint
FereRedTTS Tokenizer [4] single-codehook tokenizer trained in two stages. Transforms speech into semantic embeddings via features from HuBERT [30], followed by a ResNet-based encoder and quantization. It uses 40 ms frame shift and a codebook size of 16,384. global embedding also incorporated, and decoding is performed using flow matching, Its implementation is available
CosyVoice Tokenizer [3] A single-codebook tokenizer trained in two stages. The encoder is initialized from an ASR model [36] and subsequently trained with a supervised loss. flow matching model is used to predict nnel-ppectror.ams. It operates at 25 Hz token rate and 0.3 kbps bitrate. Its code is available
Cosy Voice 2 Tokenizer [41] An improved version of Cosy Voice that replaces VQ with FSQ. It operates at a 25 Hz token rate and 0.325 kbps bitrate. official implementation is available
Ints Tokenizer [68] Combines the DualCodec [26] semantic encoder with a flow matching decoder, similar the Vosye variants. It uses single codebook with 16,384 entries at 12.5 Hz token rate, resulting in bitrate of 0.175 kbps. The resulting TTS model, Ints, demonstrates state-of-the-an intelligibility [68].
E.2 Zero-shot TTS
F5-TTS [56] An open-source flow matching- basching-based TTS systems. It follows E2 TTS [57] and uses a flow matching transformer [35, 55] to convert the text to acoustic features directly [56].
MaskGCT [7] An open-source large-scale MGM-based TTS system that eliminates the need for explicit alignment information between text and speech supervision, as well as phone-level duration prediction. We use the official code and checkpoint which is trained on Emilia [60].
ARS [7] Introduced as AR baseline by [7]. and referred as "AR SoundStorm" in the original paper [7]. It adopts a cascaded architecture, including the AR text-to-token and the NAR MGM codec-to-waveform [46].
Cosy Voice 2 [41] An open-source, large-scale zero-shot TTS system built upon an AR model initialized from Qwen2. 5.0 1--..BB--11 IB-Instruct, which predicts speech codes extracted by the Cosy Voice 2 tokenizer.
FireRedTTS [4] An open-source, large-scale AR-based zero-shot TTS system, which predicts speech codes extracted by the FireRedTTS tokenizer.
Ints [68] An open-source, large-scale zero-shot TTS system built upon an AR model initialized from
Phi-3 mini-instruut.. which predicts 12.5 Hz speech codes extracted by the Ints tokenizer.
SparkTTS [6] An open-source, large-scale zero-shot TTS system built upon an AR model initial- ized from Qwen2. 5B- Instrucct which predicts speech codes extracted by the BiCodee [6].
Llasa [5] An open-source, large-scale zero-shot TTS system built upon an AR model initialized from Llama3. 2 1B [84], which predicts speech codes extracted by the X-codee 2 [5].
F
Evaluation
F.1
Test Sets
SeedTTS test-en We adopt test set introduced in Seed-TTS [2], consisting of 1,000 samples drawn from English public corpora, including the Common Voice dataset [85]. We refer to this set as
11
htnttpp://gittubb.omm http:///hh..aaaaaaaaaaaaaaaaaaa.
httppp//ttt....ocooeo/////////tt 15
20

---

"Regular en" and use for zero-shot TTS evaluation (Table and Table 6). Additionally, it is used for evaluating the performance of our tokenizer.
SeedTTS test-zh We adopt test set introduced Seed-TTS, comprising 2,000 samples drawn from Chinese public corpora, including the DiDiSpeech dataset [86]. We denote it as "Regular zh" for zero-shot TTS evaluation.
Articulatory en, Articulatory zh These sets are introduced in [68] and contain tongue twisters and repeated texts. For Chinese, the SeedTTS test-hard set is used directly. For English, reference speech prompts taken from SeedTTS rest-en, while the corresponding articulatory texts constructed using Deepseek- V3 [87] to match the style of SeedTTS test-hard. Each set contains 400 samples. An example:
Prompt text: Salmon is one of the most popular fish and very delicious, though usually not sustainablc.
Target text:
A big black bug bit big black bear, but the big black bear bled black blood from the bite.
Code-switching en, Code-switching zh These sets are introduced in [68]. consist of target texts that mix English and Chinese. Based SeedTTS test-en and test-zh, the reference speech prompts kept unchanged, while Deepseck-Ve V3 is employed to convert the texts into a sode-switching format. Each set contains 500 samples. An example:
Prompt text:
D. F H PU
Target text:
1:E execution Z Ogilvie A1 followers itt rounded up 3 put in jail.
Cross-lingual zh2en, Cross-lingual en2zh These sets are introduced in [68]. two types of cross- lingual samples are constructed: zh2en and en2zh, each comprising 500 samples. The zh2en set pairs Chinese reference speech from SeedTTS test-zh with English target text from SeedTTS test-en, while the en2zh set follows the reverse configuration. Each set contains 500 samples. An example:
Prompt text: US
N
Target text:
The recluse from Lithuania and his compatriot making up stories about mermaids and fays.
Multilingual test sets We additionally construct four multilingual test sets to evaluate tokenizer reconstruction in non-English languages, including French (fr), German (de), Japanese (ja), and Korean (ko). For each language, we randomly sample 300 utterances from Common Voice [85].
F.2 Objective Evaluation
Frame Rate, Token Rate, Bitrate Frame rate means the speech is compressed into how many frames per second (measured in Hz), while each frame may contain multiple tokens; token rate refers to how many discrete tokens are produced per second; bitrate indicates the total amount of information retained, computed token rate multiplied by the number of bits per token (measured in kbps), and reflects the overall compression level of the tokenizer.
For example, suppose a speech tokenizer operates frame rate of 25 Hz, meaning the input audio is compressed into 25 frames per second. If each frame contains 2 codebook tokens (i.e., layers of quantization), and each codebook has size of 2048 (requiring 11 bits per token since 2 2048),
then:
Token Rate 25 frames/sec x 2 tokens/frame 50 tokens/sec
Bitrate 50 tokens/sec bits/token 550 bps 0.55 kbps
21

---

This means the speech is represented with bitrate of 0.55 kbps, indicating high compression level while retaining discrete structure for downstream modeling.
WER Word Error Rate (WER) is employed to assess the intelligibility of reconstructed or gen- erated speech. We adopt two automatic speech recognition (ASR) models for WER computation: whisper- -arge--3 [36] and paraformer-za [37]. The former is used for non- -hnnneeeee ut- terances, while the latter applied Chinese speech, following established practices in recent studies [2, 7,
SIM Speaker similarity (SIM) is computed as the cosine similarity between speaker embeddings extracted from the prompt and the generated utterance. We use the WavLM-TDDN model [28] for speaker embedding extraction, following [1, 2, 7, 45,
UTMOS
Speech naturalness and perceptual quality are evaluated using UTMOS [88], a Mean Opinion Score (MOS) prediction system. UTMOS combines ensemble learning of strong and weak learners: the strong learners are fine-tuned self-super learning (SSL) models with architectural enhancements, while the weak learners apply lightweight regression SSL features. We use the official UTMOS checkpoint
F.3 Subject Evaluation
We conduct subjective evaluation of speech tokenizers in terms of audio quality using the Compara- tive Mean Opinion Score (CMOS):
System Interface: Users listen to two speech samples, A and B, compare their speech quality.
Instruction: Participants are asked, "Which speech has better audio quality?".
Five response options: A +2 (Sample A has much better audio quality). A (Sample has slightly better audio quality), Tie (Both have equal audio quality), +1 (Sample B has slightly better audio quality), and B +2 (Sample B has much better audio quality).
Evaluation Criteria:
Figure shows shotscreen of the evaluation system.
Litden Audio
Acoucsment
Figure Shotscreen of the subjective evaluation system.
We randomly select 40 samples from an in in-the-wild dataset. Each the six systems: Ours, X-codec 2, DualCodec, Wav Taykninne,,, Mimi, and Ground Truth, generates all 40 samples. For evaluation, each baseline system compared against ours, resulting a total of 40 x 200 sample pairs. Each pair is evaluated by two human listeners.
G Limitations and Future Work
TaDiCodec achieves an extremely low frame rate of 6.25 Hz and a corresponding bitrate of 0.0875 kbps using single-layer codebook for 24 kHz speech compression, while demonstrating strong
17 httttp/://uuggggggg....o////////
verification httppp////naiggggooooo/////////p
22

---

performance in both reconstruction and text-to-speech tasks in terms of intelligibilityy speaker similarity, and speech quality, There remains room for improvement and several promising directions for future work: 1) TaDiCodec employs diffusion autoencoder for tokenization and de-tokenization, which involves multiple steps during inference. Compared to GAN-based tokenizers, this results in higher decoding latency. Future work may explore distillation or more powerful generative models to enable single-step inference while maintaining performance. 2) While TaDiCodec has shown its effectiveness for speech language modeling through zero-shot TTS, it is worth further evaluating its applicability in speech understanding and dialogue systems. 3) TaDiCodec currently requires text input for the decoder. It would be valuable explore unified models that can transcribe, tokenize, and reconstruct speech simultaneously, enabling one model for joint understanding, compression, and reconstruction.
H Broader Impacts
Our model enables high-quality speech modeling, which can benefit applications such as personalized speech interfaces. speech restoration, and accessibility tools. However, it also poses risks of misuse, including voice spoofing and unauthorized impersonation. These risks are particularly concerning in scenarios involving biometric authentication or deceptive media. To prevent misuse, we advocate for the development of reliable deepfake detection tools, watermarking methods for synthetic speech, and reporting mechanisms flag suspected abuse.
23

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We clarify the scope and contribution of our paper in the abstract and the last two paragraphs of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
24

---

Justification:
We discuss the limitations in Appendix G.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the implementation in Section 4.1 and Appendix A. And we will release our code and model checkpoints.
Guidelines:
25

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release our code and model checkpoints at httts:///itthtt com/ AmphiionTeam/Taaicode.. We open-source datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guides ht/gguudd//ooduubnn
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / mnips.cc/publle hgtgguudesssse (CodeSubmiii wttpsSSumissiionne.. aasionoooccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
26

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details about experimental setting and evaluation in Section 4.1, Appendix A and Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For subjective evalution, we provide the confidence interval.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide the information in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
27

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //turri//........////// wwtppclcc//eeeeeeeeeecccec Couuulc/Ethhiie chchiielleeddene
Answer: [Yes]
Justification: We do conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts in Appendix H.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
28

---

Answer: [Yes]
Justification:
We discuss it in Appendix H.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the original papers for all data and models we used.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We release our code and model checkpoints at https:///ithub. com/ AmphionTTeam/TaiiCoom
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
We provide details about subject evaluation in Appendix F.3.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not address potential risks to research participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: We only use LLMs for editing.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M policy (https /ttp///eenppp.eee/e cc/ 2roonfrrencee/20255 /LLM) for what should or should be described.
30


Reviews:
[{"review_id": "seZofpW3zx", "reviewer": "Reviewer_DCzz", "summary": "This paper introduces TaDiCodec, a novel speech tokenizer designed to address several key limitations of existing methods. TaDiCodec uses a text-aware diffusion autoencoder to perform end-to-end optimization of both quantization and reconstruction, relying only on a diffusion loss. This approach eliminates the need for multi-stage training, auxiliary pre-trained models for semantic distillation, and complex adversarial objectives. The method achieves an extremely low token rate of 6.25 Hz with a single-layer codebook, while demonstrating strong performance in both speech reconstruction and downstream zero-shot text-to-speech (TTS) tasks.", "questions": "* Since the vocoder is not specified, it could be assumed that the RTF calculation excludes vocoder inference time. Please clarify this explicitly.\n* While the contribution of this work and its significance is not diminished without this discussion, to broaden the work's impact on the wider community, it would be beneficial for the authors to discuss future directions for adapting TaDiCodec for streaming scenarios and its integration with real-time language models.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The proposed method is well-motivated and intuitively designed. It effectively addresses several major challenges in speech tokenizer design, including achieving an ultra-low frame rate, removing the dependency on pre-trained semantic feature extractors, and simplifying the training pipeline to a single end-to-end objective based on diffusion loss.\n* The paper provides a comprehensive and extensive comparison with numerous state-of-the-art audio tokenizers and downstream TTS systems.\n* The experimental results are strong, demonstrating that TaDiCodec outperforms most baselines in both reconstruction quality and, more importantly, in downstream zero-shot TTS tasks.\n\nWeaknesses:\n* The paper lacks clarity on some key implementation details. For instance, the specific downsampling and upsampling methods used before and after quantization are not described. The paper states that it uses mel-spectrograms as the reconstruction target, but it does not specify which vocoder model was used to generate the final audio waveforms. This is a critical detail needed when evaluating it compared to other methods.\n* While the current architecture, which relies on bidirectional attention and an iterative diffusion decoder, is primarily suited for audio compression and offline zero-shot TTS, this limits its applicability for streaming use cases where audio chunks must be generated sequentially before the full text token stream is available.", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we want to thank for your carefully reading and providing the constructive comments! Below we address the concerns mentioned in the review.\n\n`Weakness 1: The paper lacks clarity on some key implementation details. For instance, the specific downsampling and upsampling methods used before and after quantization are not described. The paper states that it uses mel-spectrograms as the reconstruction target, but it does not specify which vocoder model was used to generate the final audio waveforms. This is a critical detail needed when evaluating it compared to other methods.`\n\nThank you for your suggestion! We are happy to provide more implementation details. Since our tokenizer operates at a frame rate of 6.25 Hz, while both the input and output mel-spectrograms are at 50 Hz, we perform downsampling before quantization and upsampling after quantization to match the temporal resolutions.\n\nTo keep it simple, we use basic interpolation for both operations. Let the embedding before quantization be denoted as `z`, and the embedding after quantization as `z_q`. The following pseudocode illustrates this process using `torch.nn.functional.interpolate`:\nimport torch.nn.functional as F\n\n```python\n# Downsample from 50 Hz to 6.25 Hz before VQ\nz_downsampled = F.interpolate(z, scale_factor=1/8, mode=\"linear\", align_corners=True)\n# Quantize\nz_q = vq(z_downsampled)\n# Upsample back to 50 Hz after VQ\nz_q_upsampled = F.interpolate(z_q, scale_factor=8, mode=\"linear\", align_corners=True)\n```\n\nWe find this simple approach to be effective and sufficient for aligning the representations temporally across stages. \n\nFor the vocoder, we use the Vocos [1] architecture with increased model capacity (around 120M parameters) and adopt the loss functions and GAN design from DACodec [2]. Unlike traditional vocoders that require upsampling, the Vocos decoder directly predicts the STFT spectrogram, which leads to higher inference efficiency compared to BigVGAN. Additionally, we find that it achieves strong reconstruction quality for both speech and singing.\n\nWe will include this information in the appendix of the paper, and we commit to releasing our models and training code to further support transparency and reproducibility.\n\n`Weakness 2: While the current architecture, which relies on bidirectional attention and an iterative diffusion decoder, is primarily suited for audio compression and offline zero-shot TTS, this limits its applicability for streaming use cases where audio chunks must be generated sequentially before the full text token stream is available.`\n\nThank you for the insightful comment! This is indeed one of the directions we plan to explore in future work. From an engineering perspective, even without modifying the model architecture, we can perform chunk-wise inference by generating a fixed number of text tokens, predicting the corresponding speech tokens, and decoding them with TaDiCodec, without waiting for all text tokens to be generated.\n\nA more promising direction is to design the decoder itself to be chunk-wise during training, supporting autoregressive diffusion over both text and speech tokens. Concretely, the current decoder input follows the pattern:\n\n```\n[text tokens, speech tokens]\n```\n\nWe propose to restructure it as:\n\n```\n[text_chunk_1, speech_chunk_1, ..., text_chunk_n, speech_chunk_n]\n```\nThis interleaved organization pattern of text and speech tokens has also been adopted in many speech LLMs, such as GLM4-Voice [3] and the streaming version of CosyVoice 2 [4].\n\nwhere the length of speech tokens in each chunk is set to a fixed ratio relative to the text tokens. Within each chunk, diffusion is used for generation, while causal attention is applied across chunks. This approach has been validated in prior works such as [5, 6].\n\nOur preliminary experiments also show that introducing a slight shift (or look-ahead) of approximately 1 second between speech tokens and the target mel-spectrogram improves chunk-wise streaming inference. Under a 1-second chunk size, the model can achieve real-time performance with minimal degradation in quality.\n\n`Question 1: Since the vocoder is not specified, it could be assumed that the RTF calculation excludes vocoder inference time. Please clarify this explicitly.`\n\nFor the vocoder, we use the Vocos [1] architecture with increased model capacity (around 120M parameters) and adopt the loss functions and GAN design from DACodec [2]. Unlike traditional vocoders that require upsampling, the Vocos decoder directly predicts the STFT spectrogram, which leads to higher inference efficiency compared to BigVGAN. Additionally, we find that it achieves strong reconstruction quality for both speech and singing. It is worth noting that in the RTF calculation for TTS, we observe that the vocoder contributes minimally to the overall inference time across all baselines. The majority of the computational cost lies in the language model and the flow-matching decoder.\n\n`Question 2: While the contribution of this work and its significance is not diminished without this discussion, to broaden the work's impact on the wider community, it would be beneficial for the authors to discuss future directions for adapting TaDiCodec for streaming scenarios and its integration with real-time language models.`\n\nThanks for your suggestion! We discuss this question in Weakness 2.\n\nThanks again for your valuable comments, we would be grateful if we could hear your feedback regarding our answers. We would be happy to answer and discuss if you have further comments.\n\nReference:\n\n> [1] Siuzdak H. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis[J]. arXiv preprint arXiv:2306.00814, 2023.\n\n> [2] Kumar R, Seetharaman P, Luebs A, et al. High-fidelity audio compression with improved rvqgan[J]. Advances in Neural Information Processing Systems, 2023, 36: 27980-27993\n\n> [3] Zeng A, Du Z, Liu M, et al. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot[J]. arXiv preprint arXiv:2412.02612, 2024.\n\n> [4] Du Z, Wang Y, Chen Q, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models[J]. arXiv preprint arXiv:2412.10117, 2024.\n\n> [5] Liu Z, Wang S, Inoue S, et al. Autoregressive diffusion transformer for text-to-speech synthesis[J]. arXiv preprint arXiv:2406.05551, 2024.\n\n> [6] Ju Z, Yang D, Yu J, et al. MoonCast: High-quality zero-shot podcast generation[J]. arXiv preprint arXiv:2503.14345, 2025."}, {"review_id": "xf5B88AZmP", "reviewer": "Reviewer_rwbf", "summary": "This paper introduces TaDiCodec, a novel speech tokenizer designed for speech language modeling that achieves an extremely low frame rate of 6.25 Hz and bitrate of 0.0875 kbps using a single-layer codebook. The key innovation is the use of an end-to-end diffusion autoencoder with text-aware decoding, eliminating the need for multi-stage training, external pre-trained models for semantic distillation, or complex adversarial objectives.", "questions": "1. Regarding the diffusion decoder, since text and prompt speech can already achieve zero-shot TTS through NAR diffusion models [1][2], I am curious about what information is actually being learned by the 6.25 tokens per second. Are these tokens encoding the residual information, maybe speech - text - global? Providing more analysis on this point would add valuable insights to the paper.\n\n2.  In Figure 3a, the WER of reconstruction is generally better than that of generation, which is reasonable because reconstruction is based on the ground truth, while generation relies on the model’s learned distribution of real data. Therefore, the reconstruction of gt should theoretically serve as an upper bound for generation. That’s why I am very curious about, for TaDiCodec (ours) (en), the generation WER is even better than reconstruction. I believe that discussing this unconventional phenomenon would be very interesting and insightful.\n\n\n [1] Shen K, Ju Z, Tan X, et al. NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers[C]//ICLR. 2024.\n\n[2] Le, Matthew, et al. \"Voicebox: Text-guided multilingual universal speech generation at scale.\" Advances in neural information processing systems 36 (2023): 14005-14034.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Significant Compression Achievement: TaDiCodec achieves an unprecedented compression rate (6.25 Hz frame rate, 0.0875 kbps bitrate) while maintaining competitive reconstruction quality, representing a major advance in speech tokenization efficiency.\n\n2. The paper includes extensive experiments across multiple languages, challenging test sets (articulatory, code-switching, cross-lingual), and both reconstruction and generation tasks.\n\n3. The paper is written very clearly and is easy to read.\n\nWeaknesses\n1. Limited Application Scenarios: Although the title mentioned speech language modeling, the codec appears to be specifically designed for TTS tasks, rather than for speech language models that unify both speech understanding and generation. Due to its reliance on additional text information for decoding, the speech tokens are difficult to use for speech understanding tasks—especially those related to text, such as ASR. In contrast, the speech tokenizers used in models like Moshi and GLM-4-Voice are capable of handling both speech understanding and generation tasks.\n\n2.  The comparison in Table 1 is not fair. Although the paper claims that the token rate has been compressed to an extremely low frame rate, the table is actually misleading. This is because codecs like EnCodec, DAC, and Mimi reconstruct audio directly from speech tokens, while some of these codecs require additional conditioning information for reconstruction. Such a comparison is therefore not fair. I believe that at least one more column should be added to indicate whether each codec requires extra information for reconstruction, such as: none, prompt speech, global features, or text condition, etc.", "was_revised": true, "final_justification": "I appreciate the authors’ response in addressing my concerns, and I am willing to retain my current positive score.", "rebuttal": "Dear reviewer, we want to thank for your carefully reading and providing the constructive comments! Below we address the concerns mentioned in the review.\n\n`Weakness 1: Limited Application Scenarios... In contrast, the speech tokenizers used in models like Moshi and GLM-4-Voice are capable of handling both speech understanding and generation tasks.`\n\nIn this work, we primarily focus on exploring the use of diffusion-based speech tokenizers for TTS and as output representations for speech language models. The motivation stems from the observation that, in TTS scenarios and most speech language models, the corresponding text of the target speech is typically available. For example, in TTS, the target text is always known, and in most end-to-end spoken dialogue systems, text and speech tokens are generated jointly. Under this setting, we aim to push the limits of compression in speech tokenization.\n\nWe are actively investigating how diffusion speech tokenizers can be used for both speech generation and understanding. One promising direction is to inject textual information into the speech tokens during training, either through explicit ASR supervision or by enabling the encoder to generate an additional set of tokens aligned with the text—thereby eventually removing the decoder's dependence on textual input. This remains an exciting area for future work. In this paper, however, we focus on the text-aware decoder setting, where the decoder is conditioned on textual information during generation.\n\n`Weakness 2: The comparison in Table 1 is not fair... I believe that at least one more column should be added to indicate whether each codec requires extra information for reconstruction, such as: none, prompt speech, global features, or text condition, etc.`\n\nThank you for your suggestion. We will revise the table in the paper accordingly and add an additional column to clarify the extra inputs required by each codec. We will also compute an adjusted token rate upper bound that accounts for these additional conditions, to provide a clearer and more accurate comparison.\n\n`Question 1: Regarding the diffusion decoder... Are these tokens encoding the residual information, maybe speech-text-global? Providing more analysis on this point would add valuable insights to the paper.`\n\nThank you for the insightful question. Prior works such as [1, 2] predict mel-spectrograms or continuous latent features directly from text and prompts, but still require an additional duration predictor to explicitly align text and speech. Meanwhile, studies like [3, 4] have shown that it is possible to predict speech features without explicit alignment modeling. However, all of these approaches directly model complex continuous acoustic representations, which are often difficult to capture comprehensively the target distribution.\n\nQualitatively, these models can be viewed as a **special case of TaDiCodec with a token rate of zero**. That is, they first predict text, encode the audio into \"discrete tokens with sequence length of zero\", then decode from text and \"zero length tokens\". It is evident that such approaches struggle to accurately reconstruct the original speech, especially in more expressive scenarios. Our demo page demonstrates that our model is capable of reconstructing challenging speech types such as singing, whispering, and speech with significant prosodic variation, with minimal timbre or pitch distortion.\n\nReturning to the TTS and language modeling setting, we argue that predicting TaDiCodec tokens and decoding them is more tractable than directly predicting high-dimensional continuous features. This is because the distribution over compressed discrete tokens is easier to model, and the tokens can be decoded into high-quality audio. Our TTS results, which outperform strong baselines such as F5-TTS, further support this claim.\n\nWe also provide analyses to better understand what TaDiCodec learns. Since our codec achieves high-fidelity reconstruction, the learned tokens encode residual factors such as prosody, speaker information, and text-speech alignment. This is evident from our demo page, which showcases reconstructions of singing and other expressive speech styles. Furthermore, because our system is trained end-to-end without explicit disentanglement constraints, the tokens naturally capture some semantic information as well. This behavior reflects the tradeoff governed by the information bottleneck, which is influenced by factors such as codebook size and token rate. Our preliminary experiments show that with a codebook size of 16,384, the TTS model achieves excellent performance, suggesting that the alignment between text and speech tokens can be easily learned under this capacity. However, when the codebook size is reduced to 4,096, we observe that while the reconstruction quality remains high, TTS performance drops significantly. We attribute this to a loss of semantic content in the tokens, which makes it harder for the language model to learn the alignment between text and speech tokens.\n\nWe also provide additional evaluations of our tokenizers on the DASB benchmark to show that TaDiCodec learns acoustic information. Due to time constraints, we select two representative tasks: emotion recognition and speaker verification. Following the official DASB protocol, we train models using our codec embeddings (after projection) and compare the performance against SpeechTokenizer as reported in the original DASB paper:\n\n| Model           | ER (Acc) | SV (EER) |\n|-----------------|----------|----------|\n| TaDiCodec       | 54.5     | 16.7     |\n| SpeechTokenizer | 49.1     | 20.4     |\n\n`Question 2: In Figure 3a, the WER of reconstruction is generally better than that of generation...  I believe that discussing this unconventional phenomenon would be very interesting and insightful.`\n\nInteresting question! This phenomenon has also been observed in some advanced TTS systems. We believe the underlying reason lies in certain biases introduced by the way WER is computed. For most test sets, paired speech and text are provided, and in theory, the ground-truth speech should yield a WER of zero. However, this is often not the case in practice, as ASR models learn to predict from an averaged distribution, and may not perfectly align with the real ground-truth audio.\n\nIn contrast, the generated speech tokens are guided directly by the text, which in some cases may be more easily captured by the ASR model, resulting in lower WER.\n\n**A similar phenomenon is also observed between the reconstructed audio from TaDiCodec and the ground-truth speech**. For example, on the Chinese test set SeedTTS-zh, we find that TaDiCodec-reconstructed speech yields a better WER than the original ground-truth speech (0.94 vs. 1.254). One possible explanation is that TaDiCodec uses a text-aware decoder, which may lead to better alignment between the predicted speech and the corresponding text. From the perspective of an averaged ASR distribution, this alignment could make the generated speech more recognizable to the ASR model.\n\nThanks again for your valuable comments, we would be grateful if we could hear your feedback regarding our answers. We would be happy to answer and discuss if you have further comments.\n\nReference\n\n> [1] Shen K, Ju Z, Tan X, et al. NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers[C]//ICLR. 2024.\n\n> [2] Le, Matthew, et al. \"Voicebox: Text-guided multilingual universal speech generation at scale.\" Advances in neural information processing systems 36 (2023): 14005-14034.\n\n> [3] Eskimez S E, Wang X, Thakker M, et al. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts[C]//2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024: 682-689.\n\n> [4] Chen Y, Niu Z, Ma Z, et al. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching[J]. arXiv preprint arXiv:2410.06885, 2024."}, {"review_id": "73SVPwvxOX", "reviewer": "Reviewer_rmsF", "summary": "This work introduces TaDiCodec, a speech tokenizer that achieves low frame rate and bitrate using a single-layer codebook. The key idea is the integration of text information into a diffusion-based decoder. Unlike existing approaches that require multi-stage training, external pre-trained models for semantic distillation, TaDiCodec uses a unified diffusion autoencoder that jointly performs quantization and reconstruction. The authors showed the tokenizer's effectiveness through speech reconstruction tasks and zero-shot text-to-speech.", "questions": "- How does the model perform when text input is noisy or contains errors (e.g., from imperfect ASR)? Have you evaluated robustness to text quality degradation?\n- Can you provide more insight into why text conditioning enables such extreme compression? Is there a theoretical formulation that explains the information-theoretic relationship between text and the required speech token rate?\n- Have you explored distillation methods (e.g., consistency models) to reduce the number of diffusion steps to 1-2 while maintaining quality? This could significantly improve inference efficiency.", "limitations": "The authors adequately address limitations in Appendix G", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- Incorporating text-aware decoding into the diffusion process, eliminating the need for multi-stage pipelines\n- Achieving 6.25 Hz frame rate with 0.0875 kbps bitrate while maintaining competitive reconstruction quality\n- WER of 2.28 on SeedTTS test-en and 1.19 on test-zh showed SOTA performance\n\n**Weaknesses**\n- Only compares with Encodec, DAC, FACodec, SpeechTokenizer, and SemanticCodec. Missing some recent works: WavTokenizer, StableCodec, Single-Codec, BiCodec, and DM-Codec\n- The paper lacks evaluation on established comprehensive benchmarks: Codec-SUPERB, DASB, or ARCH benchmarks. Any particular reasons to ignore these benchmarks?\n- Codebook utilization rate is not reported, which is crucial for understanding quantization efficiency\n- No analysis of model size or memory requirements compared to baselines\n- TaDiCodec uses 100k hours while some baselines use different amounts. Is it a fair comparison?\n- Only 5,000 samples for test-en and test-zh, which may be insufficient for robust statistical significance analysis\n- Insufficient Ablation Studies:\n    - No ablation on the effect of text quality (e.g., using noisy ASR transcripts vs. ground truth)\n    - No analysis of performance degradation when text-speech misalignment\n    - Missing ablation on different text encoder architectures or pre-trained models\n- Limited Analysis of Failure Cases:\n    - Performance on edge cases (very long utterances, extreme speaking rates)\n    - Comparison of failure modes with text-free tokenizers\n- No evaluation on speech understanding tasks. Why evaluated only evaluated on TTS tasks?\n\nPlease feel free to point me if I missed anything. Thanks!", "was_revised": true, "final_justification": "Some of my concerns were addressed, including the additional DASB results, robustness to ASR transcript quality, and clarification on codebook utilization. I appreciate the clarification regarding missing baselines (e.g., DM-Codec) and the provided rationale for excluding Single-Codec.\n\nThat said, several issues remain insufficiently addressed:\n\n* **Benchmarking**: While DASB was partially evaluated, no results were shown for Codec-SUPERB. ARCH was a typo. Thank you for the clarification. I was referring to comprehensive evaluation suites such as HEAR or similar holistic benchmarks that assess general-purpose capabilities.\n\n* **Model comparison fairness**: Training data discrepancies across baselines (100k+ vs. others) remain a concern. Simply stating that other models \"likely\" use similar data pipelines is speculative and does not eliminate the concern regarding fair comparison.\n\n* **Evaluation scope**: The evaluation still focuses only on speech reconstruction and generation. It omits important speech understanding tasks, which are essential to validate tokenizers for broader use beyond TTS.\n\n* **Failure case analysis**: The discussion on long utterances and singing is useful, but mostly anecdotal. A more systematic quantitative analysis would strengthen the claims.\n\n* **Ablations**: Still missing key ablations, such as on text-speech misalignment and architectural alternatives. While the authors argue for a minimal design, this design choice itself warrants ablation to verify its effectiveness and generalizability.\n\n* **Clarification on DM-Codec**: I respectfully disagree with the authors’ claim that DM-Codec's performance is “comparable to SpeechTokenizer.” DM-Codec demonstrates strong performance across both generation and understanding tasks, even at lower token rates, depending on the configuration. Its RVQ depth does not necessarily imply a higher effective bitrate, as token pruning and entropy-based compression techniques are often used. The dismissal without deeper analysis seems premature.\n\n**Conclusion**: This is a promising direction and a well-written paper with a clean design and competitive results on TTS tasks. However, the limited evaluation scope, lack of comprehensive benchmarking, and insufficient ablation weaken the overall strength of the contributions. I would increase my rating to **borderline accept**. While I recognize some contributions, I do not see enough depth to champion this paper during AC discussions, specifically following the NeurIPS acceptance bar.\n\nThanks again.", "rebuttal": "Dear reviewer, we want to thank for your carefully reading and providing the constructive comments! Below we address the concerns mentioned in the review.\n\n`Weakness 1: Missing some recent works: WavTokenizer, StableCodec, Single-Codec, BiCodec, and DM-Codec.`\n\nIn Table 1 of our paper, which presents the main results of our tokenizers, we compare against several representative baselines, including **WavTokenizer, BiCodec, StableCodec (TAAE)**, as well as  some other recent models such as **DualCodec, x-codec 2, Mimi, BigCodec, Vevo Tokenizer, FireRedTTS Tokenizer, CosyVoice 1/2 Tokenizer, and Ints Tokenizer**.\n\nFor Single-Codec, we did not include results as the model weight and inference code are not publicly available. We did not reproduce the model as our benchmark already includes a sufficiently diverse set of strong baselines.\n\nRegarding DM-Codec, we were not previously aware of this work and will add the appropriate citation in our revision. While it adopts 8 layers of RVQ (which means it has a much higner token rate), its performance is comparable to that of SpeechTokenizer.\n\n`Weakness 2: The paper lacks evaluation on established comprehensive benchmarks: Codec-SUPERB, DASB, or ARCH benchmarks...`\n\nOur work primarily focuses on designing speech tokenizers tailored for speech generation with language models or as the output of speech language models. Under this setting, we explore how to compress speech representations as much as possible. So, our evaluation mainly targets speech reconstruction and speech generation with language modeling.\n\nTo address the reviewer's suggestion, we additionally evaluate our codec on the DASB benchmark [1]. Due to time constraints, we select two representative tasks: emotion recognition and speaker verification. Following the official DASB protocol, we train models using our codec embeddings (after projection) and compare the performance against SpeechTokenizer as reported in the original DASB paper:\n\n| Model | ER (Acc) | SV (EER) |\n|-|-|-|\n| TaDiCodec | 54.5 | 16.7 |\n| SpeechTokenizer | 49.1 | 20.4 |\n\nFurthermore, we could not identify a benchmark named **ARCH** that is publicly available or specifically related to speech codec evaluation. Could you please provide more details about it? We would be happy to look into it!\n\n`Weakness 3: Codebook utilization rate is not reported.`\n\nThank you for the suggestion. We adopt BSQ for quantization, which significantly alleviates the codebook collapse issue. In our experiments, with a codebook size of 16,384, our method achieves a utilization rate of over 99%.\n\n`Weakness 4: No analysis of model size or memory requirements compared to baselines.`\n\nWe provide detailed model configurations in Appendix A.1, along with an ablation study on tokenizer model size in Table 4, and analysis of model size and real-time factor (RTF) for TTS in Table 6.\n\nSince the core design of our work leverages a text-aware module and a diffusion autoencoder to enable high-quality reconstruction at an extremely low token rate (6.25 Hz), while maintaining efficient generation within the full speech generation with language modeling pipeline, we believe that the results and analysis presented in Table 6 sufficiently support our claims regarding efficiency and model compactness.\n\n`Weakness 5: TaDiCodec uses 100k hours while some baselines use different amounts. Is it a fair comparison?`\n\nIndeed, we are not able to ensure that all compared tokenizers are trained on exactly the same dataset. Our model, TaDiCodec, is trained on the Emilia dataset consisting of 100k hours of speech. A significant portion of the baselines are also trained on Emilia or its supersets (which implies they use more training data including Emilia), or they are trained on datasets larger than 100k hours collected with similar data pipelines. These include: Mimi , DualCodec (Emilia), X-codec 2 (Emilia + MLS), Vevo Tokenizer (Emilia), FireRedTTS Tokenizer (200k+ hours), CosyVoice 1/2 Tokenizer (170k+ hours), and Ints Tokenizer (Emilia).\n\n`Weakness 6: Only 5,000 samples for test-en and test-zh, which may be insufficient for robust statistical significance analysis.`\n\nWe follow standard benchmarks commonly used in prior work on speech generation, and **provide a detailed discussion of our evaluation datasets in Appendix F**.\n\nSpecifically, for tokenizer evaluation, we use SeedTTS test-en (1,000 samples) and test-zh (2,000 samples), along with multilingual test sets in French, German, Japanese, and Korean, each consisting of 300 utterances randomly sampled from Common Voice. \n\nFor TTS evaluation, in addition to SeedTTS test-en and test-zh, we include 400 samples from Articulatory-en, 400 from Articulatory-zh, 500 from Code-switching-en, 500 from Code-switching-zh, 500 from Cross-lingual-zh2en, and 500 from Cross-lingual-en2zh. While many previous works only evaluate on SeedTTS test-en/test-zh or LibriSpeech.  In fact, the total number of evaluation samples used in our work exceeds that of most related papers. \n\n`Weakness 7: Insufficient Ablation Studies.`\n\nThank you for the valuable suggestion! We are more than happy to provide additional analysis as follows:\n\n**The effect of text quality and performance degradation when text-speech misalignment**:\n\nWe use ground-truth text during inference. To assess the robustness of our system to imperfect text input, we additionally provide results using transcripts generated by Whisper-large-v3. On the SeedTTS test-en set, Whisper-large-v3 achieves a low WER of 2.15. We observe a slight degradation in WER when using ASR transcripts (from 2.73 to 3.24), while SIM and UTMOS remain nearly unchanged. This aligns with our motivation: modern ASR systems are already powerful enough for high-quality text transcription. Moreover, in the speech generation setting, the target text is typically predefined or user-specified, and thus always accessible. Therefore, using ground-truth text for evaluation does not affect the fairness or validity of our TTS results.\n\n| System | WER  | SIM  | UTMOS |\n|-|-|-|-|\n| TaDiCodec (with ground-truth text) | 2.73 | 0.69 | 3.73   |\n| TaDiCodec (with Whisper-large-v3 text)      | 3.24 | 0.69 | 3.72   |\n\n**Ablation on different text encoder architectures**:\n\nIn fact, our work does not involve any dedicated text encoder for either the tokenizer or the TTS models. We follow a minimal design principle: for both the tokenizer and the TTS system, the text is simply treated as a prefix of the speech token sequence and fed jointly into a standard Transformer architecture using self-attention. No additional modules are used to separately process the text input (e.g., via a dedicated text encoder). While incorporating such components may offer potential improvements, this direction is orthogonal to our core design goals.\n\n`Weakness 8: Limited Analysis of Failure Cases.`\n\nIn our demo page, we showcase several challenging cases such as singing and utterances with significant pitch variation, demonstrating that our method performs better than other baselines (with low token rates) in these scenarios.\nAs for very long speech samples, we observe that performance degrades when the model processes utterances longer than 40 seconds in a single pass. The reason is the training data primarily consists of utterances shorter than 30 seconds. However, this issue can be mitigated by chunking long utterances into segments (e.g., 30 seconds each), which is straightforward to implement.\n\n`Weakness 9: No evaluation on speech understanding tasks. Why evaluated only evaluated on TTS tasks?`\n\nWe provide a detailed explanation in our response to Weakness 2.\n\n`Question 1: How does the model perform when text input is noisy or contains errors (e.g., from imperfect ASR)? Have you evaluated robustness to text quality degradation?`\n\nWe provide a detailed explanation in our response to Weakness 7.\n\n`Question 2: Can you provide more insight into why text conditioning enables such extreme compression? Is there a theoretical formulation that explains the information-theoretic relationship between text and the required speech token rate?`\n\nWe're happy to provide more high-level insights. This is closely related to the **question raised in question 1 of reviewer rwbf**.\n\nTaDiCodec is inspired by recent NAR TTS systems such as E2TTS, which have demonstrated that it is possible to generate speech features directly from text using diffusion models. However, these models operate on high-dimensional continuous acoustic representations, which are often difficult to model accurately and comprehensively.\n\nIn contrast, TaDiCodec introduces discrete tokens that encode essential information beyond semantics, including prosody, speaker characteristics, and text-speech alignment, which are critical for accurate reconstruction. Additionally, the powerful generative capacity of the diffusion decoder allows it to synthesize fine-grained acoustic details that do not need to be explicitly encoded in the tokens. This design enables TaDiCodec to achieve high-quality reconstruction and more accurately capture the target speech distribution.\n\nDue to space limitations, we provide a more detailed discussion in our response to question 1 of reviewer rwbf. Please refer to it for further insights.\n\n`Question 3: Have you explored distillation methods (e.g., consistency models) to reduce the number of diffusion steps to 1-2 while maintaining quality? This could significantly improve inference efficiency.`\n\nIt's a very promising direction, and we are actively exploring this problem by finetuning the decoder through distillation. We plan to share results in future work.\n\nThanks again for your valuable comments, we would be grateful if we could hear your feedback regarding our answers. We would be happy to answer and discuss if you have further comments.\n\nReference\n\n> [1] Mousavi P, Della Libera L, Duret J, et al. Dasb-discrete audio and speech benchmark[J]. arXiv preprint arXiv:2406.14294, 2024."}, {"review_id": "wdZg0FaeiU", "reviewer": "Reviewer_WHX2", "summary": "The paper proposes a text-aware, diffusion-based codec model that uses a diffusion decoder to generate speech. It can be trained with a single stage and achieves a low bitrate compared to other methods. They also show that the model has a smaller reconstruction-generation gap using zero-shot TTS. From the zero-shot TTS result, auto-regressive models trained with the codec can achieve better performance than other prior work.", "questions": "1. Can the authors provide an ablation study on the text-aware component?\n2. From the reconstruction demo on the demo page, it seems the codec has some subtle changes to the prosody. As the codec is trained as a generative model, it makes sense to involve some randomness. Is it possible to introduce a metric to quantify that?", "limitations": "They are put in the appendix. I'm not sure if it's okay.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper proposes an interesting approach for speech codecs. By adding text conditions and prompts to the diffusion decoder, it can achieve a lower bitrate than prior work. It also shows promising zero-shot TTS results. \n\nHowever, some claims from the paper seem wrong. Firstly, they refer the auto-regressive zero-shot TTS as \"speech language models\", which should include a broader range of models, such as Moshi-style text-speech LMs. So \"improving the reconstruction-generation gap for SLMs\" seems to be overclaiming. Secondly, while they show a promising result on the zero-shot TTS task, there is no evidence on whether it benefits from the text-aware and the prompt, as other approaches don't have that. It will be beneficial to have ablation study on the text-aware component. A minor point: the results part directly refer to the number in the table, which is hard to follow.", "was_revised": true, "final_justification": "I accepted the author's rebuttal and would like to retain my rating.", "rebuttal": "Dear reviewer, we want to thank for your carefully reading and providing the constructive comments! Below we address the concerns mentioned in the review.\n\n`Weakness 1: However, some claims from the paper seem wrong. Firstly, they refer the auto-regressive zero-shot TTS as \"speech language models\", which should include a broader range of models, such as Moshi-style text-speech LMs. So \"improving the reconstruction-generation gap for SLMs\" seems to be overclaiming.`\n\nThank you for your suggestion. We acknowledge that the term speech language modeling can have broader interpretations depending on context. In our work, we use speech language modeling to specifically refer to generation speech tokens with language models, particularly in the context of text-conditional speech generation. This definition has also been adopted in prior works on speech tokenizers [1, 2, 3, 4], which similarly evaluate their models primarily on language-model-based TTS tasks.\n\nWe fully understand your concern about potential overclaiming, and we commit to clarifying our definition of speech language modeling in the revised version of the paper to avoid overclaiming.\n\nAdditionally, we are actively exploring the use of TaDiCodec (as the output) for building Moshi- or GLM4-Voice-style dialog speech LLMs, where both text and speech tokens are generated jointly. These models can decode speech outputs via the TaDiCodec decoder, which also motivates our tokenizer design.\n\n`Weakness 2: while they show a promising result on the zero-shot TTS task, there is no evidence on whether it benefits from the text-aware and the prompt, as other approaches don't have that. It will be beneficial to have ablation study on the text-aware component.`\n\nWe compare all baseline TTS models in a zero-shot setting, where a prompt is provided for each sample to guide the generation. In fact, most of these baselines  require prompts for their language models and diffusion decoders (if such components are present). In our case, we additionally provide text to the decoder, as this is a core part of our design.\n\nFor the ablation study on the text-aware component, we provide additional experiments demonstrating its impact in our response to Question 1.\n\n`Question 1: Can the authors provide an ablation study on the text-aware component?`\n\nWe have conducted additional experiments to study the impact of the text-aware decoder. Since the core design of our tokenizer is to achieve high-quality reconstruction under extremely low token rates with the diffusion autoencoder and the guidance of text information, we started with the most basic setting: a GAN-based speech tokenizer. As shown below, the GAN-based model struggles to reconstruct intelligible speech at such low token rates and suffers from unstable training behavior compared to diffusion-based models.\n\nWe then replaced the tokenizer with an end-to-end trained diffusion autoencoder without any textual conditioning. We observe that both speech quality and similarity improve significantly, and the WER also decreases noticeably, though it still remains relatively high at this stage.\n\nFinally, we introduced the text-aware decoder, which further improved performance across all metrics. Introducing the text-aware decoder significantly reduces the representational burden on the tokens, allowing them to focus on learning residual information beyond the text, while still potentially capturing some textual content, under very low token-rate settings. This design finally makes accurate reconstruction.\n\n| System                        | WER | SIM  | UTMOS |\n|------------------------------|---------|------|-------|\n| GAN                          | 65.7    | 0.38 | 2.97  |\n| Diffusion                    | 22.5    | 0.59 | 3.23  |\n| Diffusion + text-aware decoder | 2.73    | 0.69 | 3.73  |\n\n`Question 2: From the reconstruction demo on the demo page, it seems the codec has some subtle changes to the prosody. As the codec is trained as a generative model, it makes sense to involve some randomness. Is it possible to introduce a metric to quantify that?`\n\nWe provide additional evaluation metrics to measure prosodic deviation between the generated speech and the ground truth. Specifically, we compute the F0 Correlation (FPC) over all samples in the test set to assess overall prosody preservation.\n\nTo quantify the randomness introduced by the generative model, particularly due to the initial gaussian noise in flow matching models, we generate each sample five times and compute the standard deviation of FPC across these generations. We use the average standard deviation across the dataset as an indicator of prosodic variability introduced by sampling stochasticity.\n\n| Model               | Bitrate | Mean FPC | Mean FPC Std (generating 5 times) |\n|---------------------|---------|----------|------------------------------------|\n| DualCodec           | 1.225   | 0.77     | 0.00                              |\n| CosyVoice Tokenizer | 0.65    | 0.67     | 0.028                              |\n| Vevo Tokenizer      | 0.175   | 0.68     | 0.031                              |\n| TaDiCodec           | 0.0875  | 0.71     | 0.039                              |\n\n\nWe can observe that DualCodec achieves the highest mean FPC and zero mean FPC std, which can be attributed to its discriminative modeling approach and highest bitrate, allowing it to preserve prosody more precisely. In contrast, CosyVoice Tokenizer, Vevo Tokenizer, and TaDiCodec yield similar lower mean FPC scores,while TaDiCodec exhibits a higher mean FPC Std, likely due to its lower bitrate.\n\nBy the way, from a perceptual perspective, we observe that minor variations in prosody tend to occur more across different samples, rather than across multiple generations of the same sample. That is, the differences introduced by stochastic sampling (of the  initial gaussian noise) are generally subtle.\n\nThanks again for your valuable comments, we would be grateful if we could hear your feedback regarding our answers. We would be happy to answer and discuss if you have further comments.\n\nReference:\n\n> [1] Zhang X, Zhang D, Li S, et al. Speechtokenizer: Unified speech tokenizer for speech large language models[J]. arXiv preprint arXiv:2308.16692, 2023.\n\n> [2] Ji S, Jiang Z, Wang W, et al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling[J]. arXiv preprint arXiv:2408.16532, 2024.\n\n> [3] Ye Z, Sun P, Lei J, et al. Codec does matter: Exploring the semantic shortcoming of codec for audio language model[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(24): 25697-25705.\n\n> [4] Li J, Lin X, Li Z, et al. DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation[J]. arXiv preprint arXiv:2505.13000, 2025."}]

Author rebuttal:

Meta-review: This paper introduces TaDiCodec, a text-aware diffusion autoencoder for speech tokenization that addresses major limitations of existing methods by enabling end-to-end optimization of quantization and reconstruction without multi-stage training, pre-trained semantic models, or adversarial objectives. The method achieves a low token rate of 6.25 Hz and demonstrates strong performance in both speech reconstruction and downstream zero-shot TTS tasks. The paper is well-motivated, experimentally solid (even if it could be improved), and shows improvements over prior work.  Overall , the rebuttal has addressed most concerns and the paper is technically solid and impactful within its scope, and merits acceptance.
Decision: Accept (poster)