Hierarchical Semantic-Acoustic Modeling via Semi-Discrete Residual Representations for Expressive End-to-End Speech Synthesis
Authors: Yixuan Zhou
Guoyang Zeng
Xin Liu
Xiang Li
Renjie Yu
Ziyang Wang
Runchuan Ye
Weiyue Sun
Jiancheng Gui
Kehan Li
Zhiyong Wu
Zhiyuan Liu
Venue: ICLR 2026 Poster (2026)
Keywords: text-to-speech synthesis
diffusion language model
semi-discrete representations
voice cloning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose an end-to-end hierarchical diffusion-autoregressive model that generates expressive and holistic speech via semi-discrete residual representations, eliminating the need for discrete speech tokenizers.

Abstract: Generative models for speech synthesis face a fundamental trade-off: discrete tokens ensure stability but sacrifice expressivity, while continuous signals retain acoustic richness but suffer from error accumulation due to task entanglement. This challenge has driven the field towards multi-stage pipelines that rely on pre-trained discrete speech tokenizers, but these create a semantic-acoustic divide, limiting holistic and expressive speech generation.  We resolve these dilemma through hierarchical semantic-acoustic modeling with semi-discrete residual representations.Our framework introduces a differentiable quantization bottleneck that induces natural specialization: a Text-Semantic Language Model (TSLM) generates semantic-prosodic plans, while a Residual Acoustic Model (RALM) recovers fine-grained acoustic details.This hierarchical semantic-acoustic representation guides a local diffusion-based decoder to generate high-fidelity speech latents. 
Critically, the entire architecture is trained end-to-end under a simple diffusion objective, eliminating dependency on external discrete speech tokenizers. Trained on over 1 million hours of speech, our 0.5B-parameter model achieves state-of-the-art zero-shot TTS performance among open-source systems, demonstrating that our approach delivers expressive and stable synthesis. Audio samples are available at: https://voxcpm.github.io/VoxCPM-demopage/.

Published as a conference paper at ICLR 2026
HIERARCHICAL SEMANTIC-ACOUSTIC MODELING VIA SEMI-DISCRETE RESIDUAL REPRESENTATIONS FOR EXPRESSIVE END-TO-END SPEECH SYNTHESIS
Yixuan Zhou', Guoyang Zeng", Xin Liu?, Xiang Li', Renjie Yu', Ziyang Wang", Runchuan Ye Weiyue Sun?, Jiancheng Gui Kehan Li', Zhiyong
Liu
Shennhen International Graduate School, Tsinghua University ModelBest Inc
*Department of Computer Science and Technology, Tsinghua University yx-zhou23@mai 1s wtt.eemml.....ts....t...ea www.nghua.dd.....
ABSTRACT
Generative models for speech synthesis face fundamental trade-off: discrete tokens ensure stability but sacrifice expressivity, while continuous signals retain acoustic richness but suffer from error accumulation due to task entanglement. This challenge has driven the field towards multi-stage pipelines that rely on pre- trained discrete speech tokenizers, but these create a semantic-acoustic divide, limiting holistic and expressive speech generation. We resolve these dilemma through hicrarchical semantic-acoustic modeling with semi semi-discrete residual rep- resentations. Our framework introduces differentiable quantization bottleneck that induces natural specialization: Text-Semantic Language Model (TSLM) generates semantic-prosodic plans, while Residual Acoustic Model (RALM) recovers fine- fine-graine grained acoustic details. This hierarchical semantic-acoustic repre- sentation guides local diffusion-based decoder to generate high-fidelity speech latents. Critically, the entire architecture is trained end-to-end under a simple dif- fusion objective, eliminating dependency on external discrete speech tokenizers. Trained on over 1 million hours of speech, our 0.5B 0.5B-parameter model achieves state-of-the-ard zero-shot TTS performance among open-source systems, demon- strating that our approach delivers expressive and stable synthesis. Audio samples are available at: hhttps://voxcpos
io/ tttphhh..oooooooPP-deeeee.apppp
INTRODUCTION
The pursuit of modern text-to-speech (TTS) systems has evolved beyond intelligibility toward the synthesis of genuinely human-like audio, capable of conveying subtle emotions, speaker identity, and contextual nuances (Shen et al. 2018 Ping et al. 2017 Ren al. 2020 Li al. 2019). This leap is critical for applications like empathetic virtual assistants and immersive digital avatars, and hinges on core technical challenge: simultaneously capturing the fine-grained acoustic details that define vocal richness and the long-range semantic structures governing intelligibility and natural prosody.
Inspired by the success of large language models (LLMs), dominant paradigm frames TTS
sequence modeling task over discrete tokens from pre-trained neural audio codecs (e.g., EnCodec DDéfossez et al. 2022 Autoregressively or Non-autoregresssiely predicting these tokens from text or phonemes (Borsos et al. 2023a Kharitonov et al., 2023, Chen et al., 2025 Wang et al.] 2025c Peng et al. 2024) offers excellent scalability and in-context learning capabilities. However, this approach faces a fundamental "quantization ceiling". as the compression process irreversibly dis- cards subtle acoustic details. To mitigate this quality loss,
TTS systems (Du al.) 2024a b) 2023 Zhou et al. (2025) Casanova et al.) 2024] adopt multi-stage hybrid pipelines. Here, an LLM generates discrete tokens which condition separate diffusion-based decoder. While improv- ing fidelity, this solution creates stark semantic-acoussio divide: the LLM operates in an abstract, discrete space unaware of acoustic reality, while the diffusion model performs local refinement with-
"Corresponding author.

---

Published as a conference paper at ICLR 2026
out high-level context. This fragmentation prevents end-to-end optimization and limits holistic and expressive speech synthesis.
Alternatively, other approaches directly model continuous speech representations to avoid quan- tization loss. Early systems like Tacotron 2 (Shen et al. 2018) and more recent models such as MELLE (Meng al. 2024) generate mel-spectrograms autoregressively. However, predicting con- tinuous targets under standard regression losses often yields over-smoothed and low-diversity out- puts. To address this, recent innovations have explored replacing the regression objective with denoising process to model the distribution of the next continuous representations, spanning both
non-autoregressive paradigms (Shen al.] 2023) Le et al.] 2023 Chen et al., 2024) and autore- gressive methods(Li et al. [2024] Jia et al., 2025 Peng et al. 2025). Among these, autoregres- sive approaches have often demonstrated superior performance in capturing natural prosody and expressive variation. This innovation successfully enhances the detail and diversity of generated continuous representations. However, a more fundamental issue persists: in fully continuous au- toregressive model, the tasks of high-level semantic-prosodit planning and fine-grained acoustic rendering are conflated within single learning objective. The model is forced to simultancously solve two disparate tasks requiring different inductive biases- in a continuous output space. This entanglement presents a significant challenge to the modeling capacity of a single LLM, as it must learn to be both global planner and local renderer without inherent architectural bias sepa- rate these functions. We argue that this conflation is a root cause of instability. The model's 's focus is inevitably pulled towards fitting low-level acoustic textures, which compromises its ability to main- tain high-level semantic coherence, leading to the well-known problem of error accumulation over long sequences (Pasini al. 2024).
In this work, we introduce a unified, end-to-end framework that resolves this trade-off through hi- erarchical semantic-acone is that holistic and expressive speech synthesis requires explicit architectural separation between semantic-prosodie planning and acoustic rendering, yet should remain within a cohesive, end-to- end trainable system. The core innovation is differentiable Finite Scalar Quantization (FSQ) (Mentzer et al. 2024) bottleneck that induces natural specialization: (I) Text-Semantic Language Model (TSLM) generates semantic-prosodic plans stabilized through quantization, focusing lin- guistically meaningful patterns; and (2) a Residual Acoustic Language Model (RALM) recovers fine-grained details lost during quantization, specializing in acoustic refinement. This hierarchical design enables each component to excel at its respective role while maintaining differentiability, and both of them will be used to guide a local diffusion decoder generate high-fidelity speech latents. Critically, the entire hierarchical model is trained end-to-end under a simple diffusion ob- jective, seamlessly integrating planning and rendering without pre-trained tokenizers. Our main contributions are as follows.
modeling with semi-discrete residual representations. Our key insight
We propose an end-to-end hierarchical architecture that introduces an internal semi-discrete bottleneck resolve the expressivity-stabiiie trade-off. This mechanism implicitly addresses task entanglement in continuous models by inducing a beneficial separation between semantic- prosodic planning and fine-grained acoustic modeling within a single, unified framework.
We introduce residual learning strategy that, in conjunction with the bottleneck, enables a holistic yet specialized modeling process. Unlike fragmented multi-stage pipelines, our approach achieves functional separation without architectural fragmentation, simplifying the training pipeline and eliminating dependency on external discrete speech tokenizers.
We demonstrate the efficacy of our approach through large-scale training on over I million hours of bilingual speech. The resulting model, VoXCPM-0.5B, achieves state-of-the-arr zero- shot TTS performance among open- systems, validating its practical strength.
We provide extensive ablation studies that conclusively validate the semi-discrete residual rep- resentations the crucial component for robust, expressive, and long-form synthesis. We will release code and models to support future research.
2 RELATED WORK
2.1 DISCRETE
TTS
The discrete token paradigm has emerged as a dominant approach in modern TTS, leveraging the success of large language models. This method converts speech into discrete representations using
2

---

Published as a conference paper at ICLR 2026
neural audio codecs such EnCodec (Défossez et al. 2022) and DAC (Kumar al. 2023) through residual vector quantization (RVQ). M (Borsos et al. 2023a) and VALL-E (Chen et al. 2025)
pioneered this direction by framing TTS an autoregressive sequence prediction task over discrete acoustic tokens. Subsequent developments include SoundStorm (Borsos et al. 2023b), which in- troduced non-autoregressivve generation for improved efficiency, and Spear-TTS (Kharitonov al.) 2023), which focused on multilingual capabilities with minimum supervision. VoiceCraft (Peng al. 2024) and XTTS (Casanova et al. 2024) further advanced zero-shot TTS with in-context learning.
Recent advancements have focused on enhancing the scalability, controllability and zero-shot adap- tation. Voice (Du et al.) (024a) proposed supervised semantic tokens for improved zero-shot performance, while its successors, Cosy Voice 2 and (Du al. 2024b, 2025) incorporated text- based LLM initialization and streaming synthesis for Humann-pritty quality and low latency. In- dexTTS (Deng et al.] 2025) and IndexTTiS2 (Zhou et al.] 2025) introduced precise duration and emotion control in autoregressive token generation, enabling applications with strict timing and ex- pressivity requirements. SparkTTS (Wang et al. 2025b) utilized single-stream decoupled speech to- kens for modeling efficiency, and FireRedTTSS (Guo et. 2024) along with its update FireRedTTS-2 (Xie et al., 2025) established frameworks for industry- generative speech, including long-fom multi-speaker dialogue. Despite these progresses, discrete approaches suffer from inherent quanti- zation artifacts, limiting acoustic fidelity and prompting hybrid solutions.
2.2 CONTINUOUS REPRESENTATION TTS
To circumvent quantization losses in discrete models, continuous representation approaches directly model speech features such as spessppetsogramss or audio latents. Early systems like Tacotron
(Shen et al.] 2018) established the encoder- framework for text-to-mel mapping, while Fast- Speech (Ren et al., 2020) introduced explicit duration modeling for alignment stability. Recent de- velopments have integrated diffusion processes to enhance detail and diversity. Non-autoregressive models like NaturalSpeech 2 (Shen al. 2023) and VoiceBox (Le et al., (023) apply diffusion directly on continuous representations. FT-TTS (Chen et al. 2024) advanced flow-matching for effi- cient synthesis. Autoregressive paradigms, often superior in prosody and variation, include MELLE (Meng et al.] 2024) for mel-spectrrogrm generation. Innovations like ARDIT (Li et al., (0244) use an autogressive diffusion transformer for TTS, unifying semantic coherence and acoustic natural- ness via parameter sharing. DiTAR (Jia et al. 2025) extended this with patch-based design: causal LM for inter-patch stability and a bidirectional local diffusion transformer for intra-patch re- finement. Voiee (Peng et al. 2025) employed next-token diffusion for long-fon multi-speaker synthesis. More recent models such as CLEAR (Wu et al., (025) and FELLE (Wang et al.) (025a) focus on latent autoregressive modeling with token-wise coarse-to-fine hierarchies, while MELA- TTS (An et al.) 2025) and KALL-E (Zhu et al.) 2024) combine joint transformer-diffusion with next-distribution prediction for improved efficiency and quality. Despite these advances, continuous models often entangle high-level semantic planning with low-level acoustic rendering, leading to instability in long sequences without explicit separation.
2.3 HIERARCHICAL AND RESIDUAL MODELING IN TTS
Hierarchical and residual approaches decompose TTS into layered tasks to balance stability and ex- pressivity. HierSpeech++ (Lee et al. 2025) employed variational inference for semantie-acoustic
mapping. HALL-E Nishimura al. 2025) uses hierarchical neural codecs with LLMs for minute-
long synthesis. MARS6 (Baas et al.] 2025) builds robust encoder-decoder transformers with hierar- chical tokens. DiffStyleTTS (Liu et al. 2024) applies diffusion for hierarchical prosody modeling- HAM-TTS (Wang et al., 2024) introduces hierarchical acoustic modeling with data augmentation for zero-shot TTS. QTTS (Han et al.) 2025) features hierarchical parallel architectures for residu- ally quantized codes. These methods address flaws in prior paradigms: implicit designs lack regu- lated bottlenecks, lokenizer-dependen models suffer discrete losses, and fragmented stages hinder end-to-end optimization. However, few fully integrate explicit residual designs with semi-discrete bottlenecks in a unified framework, as proposed in our work, to achieve implicit disentanglement without external dependencies.
3

---

Published as a conference paper at ICLR 2026
Speech
Z
Text Hidden States: Has
Acoustic Embeddings: E
Scaler Semantic Hidden: In
Residual Acoustic Language Model (RALM)
FBQ
FSQ
FSG
Text Semantic Language Model (TSLM)
Tokenization
"VeNO" it TTS model."
Figure I: Overall architecture of VoxCPM. The model hierarchically generates speech by first pro- cessing audio latents through LocEnc, then producing a semi-discrete speech skeleton with the TSLM and FSQ. refining acoustic details with the RALM, and finally generating high-fideligy latent output with the LocDiT.
METHODOLOGY
3.1
CORE DESIGN MOTIVATION
Generative speech synthesis faces fundamental tension between expressivity and stability. Dis- crete tokenization methods (e.g., discrete speech tokenizers with language models) ensure stable autoregressive generation but irreversibly discard fine- fine-grained acoustic details through quantization. Continuous approaches preserve full fidelity but suffer from error accumulation in long sequences due to information entanglement, often leading to catastrophic failure in intelligibility.
Critically, we identify key limitation in existing discrete tokenization approaches: methods that directly use FSQ VQ to obtain discrete codebooks for language modeling face an inherent scal- ability challenge. As the dimensionality increases to capture richer acoustic information, the code- book size grows exponentially, creating an unmanageably large and sparse vocabulary that language models struggle to predict accurately.
We hypothesize that an effective solution should structurally separate the modeling of stable semantic- semantic-roosodic content from fine- fineggrinedd acoustic details while maintaining differentiability for end-to-end training. Our key insight is to introduce a differentiable quantization bottleneck that naturally induces this separation through scalar quantization, splitting information into discrete- like skeleton for content stability and continuous residual components for detail expressivity.
Unlike multi-stage TTS systems composed of seperate LM and diffusion that treat quantization as means to obtain discrete prediction targets, our approach uses quantization solely as a regularization mechanism constrain the hidden state space. This distinction allows to avoid the vocabulary explosion problem while still benefiting from the stabilizing effects of discrete representations.
3.2 MODEL OVERVIEW
VoxCPM employs hierarchical autoregressive architecture that generates sequences of continuous
speech latents Z = {z)..., ZM} conditioned on input text tokens T = {t..... t where each
2, E PxD represents a patch of P frames with D-dimensional VAR latent vectors. The generation

---

Published as a conference paper at ICLR 2026
process follows:
(Z II T Z<.) 1=1
The core innovation lies in our hierarchical conditioning mechanism with residual representation learning. It made up local audio encoder (LocEne), a text-semantic language model (TSLM), residual acoustic language model (RALM) and local diffusion transformer decoder (LocDiT). A stop predictor attached to the output the TSLM to determine the endpoint generation. As shown in Figure I each patch generation involves:
2 LocDiT(h
h
FSQ(TSLM(T) E&D) RAAM(-)
stable skeleton
residual details
represents historical audio context aggregated by lightweight LocEnc that compresses VAE latent patches into compact acoustic embeddings. The hierarchical backbone produces conditioning signal h!!iall that encapsulates both semantic content from TSLM (with FSQ) and acoustic details from RALM. This signal guides the LocDiT to generate the current latent patch through denoising diffusion process. The entire model is trained end-to-end with gradients flow- ing through all components, including the FSQ bottleneck via straight-through estimation, ensuring coordinated optimization toward holistic speech synthesis.
where E LocEnee22 2i)
3.3 HIERARCHICAL SEMANTIC-ACOUSE MODELING
Our hierarchical modeling approach is designed to implicitly separate semantic-prosadt planning from fine-grained acoustic synthesis, addressing the fundamental expresityy trade-off through structured representation learning.
3.3. TEXT-SEMANTIC LANGUAGE MODEL (TSLM)
The Text-Semantic Language Model forms the main part of our hierarchical architecture, responsi- for capturing high-level linguistic structure and generating contextually appropriate speech pat- terns. Unlike conventional TTS systems that typically operate on phoneme sequences, our approach leverages pre-trained text language model (MiniCPM-4 (Team al. (025)) as its initial back- bone, enabling richer contextual understanding and more natural prosody prediction directly from raw text. By processing both text tokens and historical audio context, the TSLM learns to gener- semantic content and prosodic structure that evolve naturally throughout the underlying linguistic meaning rather than simply mapping phonemes to acoustic features. The TSLM produces continuous semantic-prosod representations that encode both the content to be spoken and how it should be prosodically realized, serving as input to the subsequent quantization stage.
utterance, reflecting
3.3.2 SEMI-DISCRETE REPRESENTATION LEARNING VIA FSQ
At the core of approach lies the Finite Scalar Quantization (FSQ) layer, which projects the tinuous hidden states from the TSLM onto structured lattice to create semi-discrete representa- tion. The FSQ operation transforms each dimension of the continuous vector through deterministic scalar quantization:
ISLM
= A cclpp round
L, L
(3)
where A is the quantization step size, the clipping range, and round maps values to discrete levels. This transformation creates structured discrete representation while maintaining differen- tiability through the -traight-through estimator during backward passes.
The FSQ layer acts bottleneck, analogous to the first layer of Residual Vector Quantization (RVQ), which captures a coarse semantic- prosodic skeleton (e.g.. content, intonation patterns). We term this representation "semi- dimii-iicretee it employs significantly larger dimensionality than stan- dard FSQ to ensure sufficient informational capacity. Unlike RVQ, where the first layer is a predic- tion target and subsequent layers model finer details, FSQ bottleneck serves an intermediate,
5

---

Published as a conference paper at ICLR 2026
differentiable inductive bias within the continuous data flow. It encourages the model to priori- tize modeling stable, high-level components (the remantic-prosodif skeleton) by providing clear learning signal for what information should be preserved through the bottleneck. This structured approach mitigates error accumulation by reducing the modeling burden on the TSLM, allowing to focus the major components of the speech.
3.3.3 RESIDUAL ACOUSTIC MODELING
To recover the fine-grained acoustic information attenuated by quantization, we introduce the Resid- ual Acoustic Language Model (RALM). This module specializes in reconstructing those subtle vocal characteristics that conventional discrete methods sacrifice for stability. It processes the quantization residuals along with contextual information to recover speaker identity, spectral fine structure, and micro-prosodic variations:
h residual
RALM
ISLM text
H Ecs)
(4)
Here, the RALM conditions its predictions on both the TSLM hidden states of the text part H the
representation of speech part H and the historical acoustic embeddings E This residual learning approach creates a natural division of labor: the TSLM+FSQ pathway fo- cuses on content stability and prosodic coherence, while the RALM pathway specializes in acoustic expressivity and speaker characteristics.
The final combined representation hnfmal = h FSQ h besidual thus encapsulates both semantic stability and acoustic expressivity, creating a comprehensive signal that guides the subsequent local diffusion process.
3.3.4 LOCAL DIFFUSION TRANSFORMER DECODER
The Local Diffusion Transformer (LoeDiT) serves as our high-fidelity synthesis module, generat- ing continuous latent patches conditioned on the hierarchical representation h!fmall produced by the preceding modules. Following DiTAR (Jia et al. 2025), we employ a bidirectional Transformer architecture that enables full receptive field modeling within each patch. To enhance generation consistency, we incorporate the previous patch z,-1 as additional conditioning context, which has been empirically validated to significantly improve output quality by framing the task outpainting rather than independent patch generation. Besides, we mask the LM guidance in LocDiT condition with specific probability ratio, for enabling classifier-fres guidance (CFG) during inference.
3.4 TRAINING OBJECTIVE
The entire model is trained end-to-end using flow-matching objective that directly optimizes the quality of the generated speech latents. We adopt the conditional fow-matching formulation for its training stability and sampling efficiency:
CFM = Et, 2,, e | h. z;-1) 1) di
d (arz) + or)))
(5)
where z; az + ore the noisy latent time t, with N/ I). and vø the velocity field predicted by the LocDiT.
Simultaneously, binary classification loss is applied to train the model to predict the end of speech sequence:
Ssup
E Exseequence
BCE (h (hh55) ,
(token is the last) )]
where sp is a stop-logit projection layer, and BCE denotes the binary cross-entropy loss.
The gradients from this loss are backpropagated through the entire autoregressive hierarchy, includ- ing the FSQ layer (via straight-through estimation), the TSLM and the LocEne. This end-to-end optimization under the combined objective L Lrm + ALsupp allows each component to learn its specialized seme-smiiiic planning, stabilization, and acoustic refinement- in a coordinated man- ner, guided by the unified objective of accurately modeling the continuous speech latents.

---

Published as a conference paper at ICLR 2026
3.5 CAUSAL AUDIO VAE
To enable efficient streaming synthesis, we employ a causal Variational Autoencoder that operates in computationally efficient latent space. VAE is pre-trained separately using a composite objective that combines reconstruction loss in the Mel-spectrogram domain, adversarial training with multi- period and multi-scale discriminators, and a minimal KL-divergence term to regularize the latent space. The use of a latent space rather than raw audio waveforms significantly reduces computational requirements while preserving perceptual quality. The causal nature of the VAE ensures that both encoding and decoding operations can be performed in a streaming fashion, making the entire system suitable for real-time applications where low latency critical. The detailed implementation of AudioVAE can be found in Appendix D
4
EXPERIMENTS AND RESULTS
4.1 EXPERIMENTAL SETUP
Datasets We conducted experiments on two primary datasets: 1) Large-scale Bilingual Corpus:
To explore the best performance, we collected an internal large-scale, bilingual dataset totaling over I million hours, mainly comprising of Chinese and English speech. 2) Emilia Dataset: For com- parisons and ablation studies, we used the publicly available Emilia dataset (He et al. 2024) (95K hours). All audio was resampled to 16kHz mono, processed with source separation, voice activity detection (VAD), and automatic speech recognition (ASR) system obtain text-audio alignment.
Implementation Details We implemented VoxCPM using the Megatron framework, with 0.5B- parameter configuration, comprising a 24-layer Text-Semantic Language Model (TSLM), initial- ized from the pre-trained MiniCPM-4-0.55 (Team et al., 2025 and a randomly initialized 6-layer Residual Acoustic Language Model (RALM).. We trained two models for comparisons: 1) VoxCPM was trained with internal large-scale bilingual corpus for 500K steps using 40 NVIDIA H100 GPUs; 2) VoxCPM-Emilia was trained on the Emilia dataset for 200K steps using 24 H100 GPUs. Both used the AdamW optimizer with a peak learning rate of x 10-4 and a (WSD) schedule (Hu et 2024). All ablation studies followed the same 200K-step training pro- tocol on 8 H100 GPUs using the Emilia dataset, employing a fixed learning rate (i.c., without the WSD schedule) of x 10-4 to ensure consistent comparison.
Evaluation Metrics and Benchmarks We employed comprehensive subjective and objective eval- uations. Objective metrics included Word / Character Error Rate (WER CER) for intelligibility, speaker embedding cosine similarity (SIM) for voice cloning, and DNSMOS for overall quality. Subjective evaluation involved Mean Opinion Score (MOS) tests rated by 20 native speakers on naturalness (N-MOS) and speaker similarity (S-MOS) using 5-point scales. Models were assessed on two challenging benchmarks: 1) SEED-TTS-EVE
focusing on general TTS intelligibility and similarity in English and Chinese, including "Hard" set with complex sentences; 2) CV3-EVAL, derived from Cosy Voice 3 competition, emphasizing expressive and in-the-wild voice cloning.
Baselines We compared VoxCPM against wide range of ofte--of-the-arr open- TTS systems, including CosyVoice
Voice series (Du et al.] (2024a MaskGCT (Wang et al., 2025c). F5-TTS CCen al., 2024), Spark (Wang al. (025b). FireRedTTS series (Guo al. 2024 Xie al. 2025), IndexTTSS 2 (Zhou et al. 2025). HiggsAudio v2 and so on. All baseline results were obtained using official implementations with default settings, or as reported in their original papers. Details of compared models see Appendix E
4.2 MAIN RESULTS: COMPARISON WITH STATE-OF-THE-ART TTS
As shown in Table I VoxCPM achieves state-of-the-ee performance among open-source models on the SEED-TTS SEED-TTS-EVAL benchmark. It attains an English WER of 1.85% and a Chinese CER of 0.93%, surpassing strong competitors like IndexTTS2 and Cosy Voice2. Concurrently, VoxCPM maintains high speaker similarity, with SIM scores of 72.9% (EN) and 77.2% (ZH). This demon- strates that the proposed semi-discrete bottleneck effectively balances intelligibility and expressivity by hierarchical semantic-acoustic modeling, mitigating the instability common in continuous mod- els while preserving details often lost discrete models. The VoxCPM-Emin variant, trained on a
httts:://uught wttp::/ttppppppp//tttaatcccotctt -ww.0
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance on Seed-TTS-eval Benchmark
Hard
Model
Params
Data/hrs
WER
SIM
CER
SIM
CER
SIM
FS-TTS
0.3B
100K
2,00
67.0
1.53
76.0
8.67
71.3
MaskGCT
1B
100K
2.62
71.7
2.27
77.4
Cosy Voice
0.3B
170K
4.29
60.9
3.63
72.3
11.75
Cosy Voice2
0.5B
170K
3.09
65.9
1.38
75.7
6.83
72.4
SparkTTS
0.5B
100K
3.14
57.3
66.0
FiseRedTTS
0.5B
248K
3.82
46.0
1.51
63.5
17.45
62.1
FireRedTTS-2
1.4M
1.95
66.5
1.14
73.6
Qwen2.5-Oman
7B
2.72
63.2
75.2
7.97
OpenAudiiosslmiiii
0.5B
2M
1.94
55.0
1.18
68.5
23.37
64.3
IndexTTS 2
1.5B
55K
2.23
70.6
1.03
76.5
7.12
75.5
Vibe
1,58
3.04
68.9
1.16
74.4
HiggsAudio-v2
3B
10M
2.44
67.7
1.50
74.0
$5.07
65.6
VoxCPM-
0.5B
100K
2.34
68.1
1.11
74.0
12.46
69.8
VoxCPM
0.5B
1.8M
1.85
72.9
0.93
77.2
8.87
73.0
Table Performance on CV3-eval Benchmark. *denotes close-sourced systems.
CV3-EVAL
CV3-Hard-7Ht
CV3-Hard-EEN
Model
ZH-CER
EN-WER
CER
SIM
DNSMOST
WER
SIM
DNSMOS
FS-TTS
5,47
8,90
SparKTTS
5.15
Gov--Soviis
7.34
12.5
CosyVoice2
4.08
12.58
72.6
3.81
11.96
66.7
3.95
OpenAudio-sI-mini
4.00
5,54
18.1
58.2
3.77
12.4
55.7
3.89
IndexTTS2
3,58
4.45
12.8
74.6
3.65
8.78
74.5
3.80
HiggsAudio-v2
Cosy Voice3 0.5B
3.89
5.24
14,15
78.6
3.75
9.04
75.9
3.92
Cosy Voice3-1
9.77
76.1
VoxCPM-Emilia
4.47
5.23
22.2
62.6
3.47
10.00
62.6
3,68
VoxCPM
3.40
4.04
12.9
66.1
3,59
7.89
64.3
3.74
smaller public dataset, delivers competitive results (EN-WER: 2.34%, ZH-CER: 1.11%). When compared VoxCPM-Emilia against competitive AR models trained on similar data scales (e.g., Voice, SparkTTS, FireRedTTS). our model consistently outperforms them stability and similarity. While some NAR models like MaskGCT achieve higher objective metrics, AR models typically excel in prosodic naturalness and expressiveness the subjective evaluations (as shown in Table 3. This highlights the data efficiency and architectural robustness of our approach, as the FSQ bottleneck stabilizes the learning of comantic-acoustee representations even with less training data. Notably, while DiTAR's phoneme-based approach shows slightly better stability, VoxCPM's use of BPE tokens with pre-trained LLM initialization provides superior text understanding capa- bilities and eliminates dependency external phonemizers. Besides, our hierarchical design with residual acoustic modeling reduces the fundamental limitation direct continuous token modeling, evidenced in ablation studies.
On the CV3-EVAL benchmark (Table 2). designed to evaluate expressive and in-the-wild perfor- mance, VoxCPM excels with ZH-CER of 3.40% and an EN-WER of 4.04%. Its robustness is further confirmed on the challenging CV3 Hard-Test set, where it achieves an EN-WER of 7.89%, However, VoxCPM achieves a relatively lower DNSMOS score compared to others, as the prompt audios used in CV3-Hard inherently have a low DNSMOS (around 3.5), which demonstrates its faithful cloning of the recording environment and vibe. These results underscore the model's capability to handle complex, realistic inputs, strength attributed to the RALM's role recovering ime-grained acoustic details subsequent to the TSLM- FSQ-based semantic-prosodit modeling.
outperforming even close-sourced Cosy
Subjective evaluations (Table 3 further validate the objective findings, with VoxCPM achieving competitive performance across both languages. On English tests, VoxCPM obtains the highest scores in speaker similarity and good results in naturalness. For Chinese, while VoxCPM trails In- dexTTS 2 in naturalness, it achieves slightly superior speaker similarity. We found that many prompt audios from the Seed- -eed-TTS-- TTS-Eval test set contain disfluencies or exhibit a monotonous tone, which inherently constrains the naturalness of our cloned outputs. This pattern suggests that VoxCPM
8

---

Published as a conference paper at ICLR 2026
cels voice cloning consisteney, while IndexTTS 2 may have advantages in prosodic naturalness for Chinese. VoxCPM-Emin shows competitive speaker similarity but relatively lower naturalness, highlighting the impact of training data scale.
Table 3: Subjective Evaluations in terms of Naturalness and Speaker Similarity. Note: We select the competitive baseline models for subjective comparison based objective results.
ZH
EN
Model
N-MOS
S-MOS
N-MOS
S-MOS
MaskGCT
3.20 0.11
3.77 0.11
3.84
4.00
0.10
CosyVoice
4.14
3.97 0.30
IndexTTS 2
4.25
4.00
4.03 0.10
4.16
VoxCPM-Emilla
0.19
3.91
3.91
4.10 0.09
VoxCPM
0.10
4.11
4.11 0.09
+.18
4.3 ABLATION STUDY: EFFECT OF THE SEMI-DISCRETE BOTTLENECK
As shown in Table 4 the ablation studies on the FSQ bottleneck dimensionality (More details can be found in Appendix Table 9 provide critical insights. The catastrophic performance degradation of the purely continuous model (w/o FSQ), especially on hard cases (ZH-CER: 24.92%), validates our core hypothesis: entangling semantic planning and acoustic rendering in continuous space leads to instability. Without the inductive bias imposed by FSQ, the model struggles to separate these tasks, resulting in error accumulation complex utterances.
The optimal performance observed at FSQ levels (FSQ-d128/d256) reveals a key trade-off. Lower dimensions (e.g.. FSQ-d4) over-constrain the representation, limiting prosodic capacity. Higher di- mensions (e.g.. FSQ-d10224 provide insufficient discretization strength, allowing task entanglement persist. The peak FSQ-d256 indicates the bottleneck creates effective "summary space": discrete enough to stabilize long-range semantic plamning yet continuous enough to retain crucial prosodic and speaker information, thereby enforcing beneficial division of labor within the model.
4.4 ABLATION STUDY: EFFECT OF RESIDUAL ACOUSTIC MODELING
As shown in Table the ablation studies about the residual language modeling validate our core ar- chitectural innovations. Notably, the purely continuous variant (w/o RALM: TSLM LocDiT) analogous to DiTAR's approach-shows significantly degraded performance, particularly on challenging cases. The performance gap persists across different TSLM configurations, confirm- ing that the challenge is fundamental to the learning objective rather than parameter allocation. This conclusively demonstrates the advantage of our explicit separation between semantic and acous- tic modeling. To isolate the benefits of our hierarchical architecture from simple capacity effects, we compared single-stream models with identical initialization foundation but varying depths: [w/o RALM: M24 layers, LM init.)] vs. [w/o RALM: TSLM(30 layers, partial LM init.)]. The gains from increasing capacity is marginal (e.g., EN-WER from 4.34 to 4. 4.12), while the gains from in- troducing hierarchical structure is larger in default setting (e.g., EN-WER 2.98). This confirms that the hierarchical TSLM/RALM separation is the dominant factor for stability and expressiveness, significantly outweighing the benefit of merely increased capacity. Furthermore, the hicrarchical design exhibits superior performance compared to its single-stream counterpart, even under fully random initialization ([Hierarchical, w/o LM init. in TSLM] vs. [w/o RALM: TSLM(30 layers, random ini This simultaneous improvement demonstrates that the hierarchical residual design provides intrinsic benefit for a specialized division of labor, which is effective even without the strong semantic grounding from pre-trained parameters.
Secondly, the critical role of residual acoustic input further evidenced by the substantial degrada- tion when ablating original acoustic embeddings (w/o Ezs in RALM), highlighting that the RALM requires fine-graine mne-grained acoustic information to accurately recover acoustic details. Finally, the best performance of the default setting demonstrates the effectiveness of the residual connection. By summing the TSLM and RALM hidden states, the model explicitly delegates semantic-prosodae planning to the TSLM and acoustic refinement the RALM, achieving optimal integration.
Finally, the effect of pre-training initialization is intuitive: removing pre-trained rre-trained text LM initializa- tion results in significant degradation in intelligibility (e.g., EN-WER 2.98 5.24). This confirms

---

Published as a conference paper at ICLR 2026
that the pre-trained knowledge is critical for establishing the TSLM's initial capabilitt and stabil- ity. We observe that models trained from random initialization shows a marginal increase in SIM, suggesting that without the strong linguistic guidance from the pre-trained LLM, the model might implicitly allocate more capacity to acoustic feature modeling.
Table 4: Ablation Studies about FSQ bottleneck dimensions and core architecture designs.
Model Setting
EN
ZH
ZH-hard
WER
SIM
CER J.
SIM
CER & SIM
default setting (w/ FSQ: d256x9)
2.98
62.6
1.77
70.4
18.19
w/ FSQ: d4s9
5.18
59.3
4,05
68.0
19:55
62.3
w/ FSQ: d128s9
3.43
62.2
1.67
70.7
16.76
65.7
w/ FSQ: d1024e9
3.07
62.0
2.38
69.8
20.38
64,7
w/o FSQ: d1024s00
3.67
62.1
2.30
69.6
24.92
63.5
Hierarchical, wfo L.M init. TSLM
5.24
63.4
2,41
70.9
24.66
65.6
w/o RALM: TSLM (24 layers, LM init.) - LocDiT
4.34
61.8
3.05
69.4
25.00
63.8
RALM: TSLM
layers, random init.) LocDiT
62.6
3.46
69.8
30.40
w/o RALM: TSLM (30 layers, partial LM init) LocDiT
4.12
62.0
3.07
69.6
26.20
63.1
in RALM: TSLM ALM LocDiT w/n Aonstaal in condition: TSLM FSQ LoaDiT
4,94
68.1
27.17
61.7
3.86
58.3
3,05
67.6
23.65
61.7
4.5 ANALYSIS AND DISCUSSION
Analysis of Hierarchical Representations T-SNE visualizations (Appendix Figure t 5 confirm the specialized roles of TSLM and RALM. TSLM-FSQ outputs form cemantic-prosodeee structures closely tied text content, whereas RALM residuals exhibit strong speaker-related variations for acoustic rendering. This functional specialization validates the efficacy of our hierarchical residual modeling. More details and analysis can be found in Appendix H.
Expressive Synthesis Capabilities Beyond quantitative metrics and visualizations, VoxCPM shows good expressive synthesis capabilities directly from text benfiting from the architecture design and training data. When not using prompt speech, the model tends to express suitable style from con- textual cues, also shown in F.
We strongly recommend readers to listen our demo samples.
Scalability and Efficiency The performance improvement from foxCPM-Emilin to VoxCPM high- lights the architecture's scalability with increased data. The hierarchical design allows larger models to effectively utilize increased capacity for learning complex patterns. In terms of inference effi- ciency, VoxCPM-0.5B achieves real-time factor (RTF) of 0.17 on a single NVIDIA RTX 4090, confirming practical deployment feasibility.
CONCLUSION
In this work, resolve the fundamental trade-off between expressivity and stability in text-to- speech synthesis by introducing a unified, end-to-end framework based on hierarchical semantic- acoustic modeling with semi-discrete residual representations. Our approach leverages differen- tiable quantization bottleneck to induce a natural separation of concerns: a text-semantic language model captures high-level semantic proasdic-poooodic structure, while a residual acoustic model recovers fine-grained details. This eliminates the dependency on external discrete speech tokenizers and mit- igates the error accumulation that plagues purely continuous autoregressive models. Extensive ex- periments demonstrate that our model achieves state-of-the- the-art zero-shot TTS performance among open-source systems, excelling both intelligibility and speaker similarity. The success of Vox- CPM validates that learning structured, regularized latent spaces provides principled foundation for expressive generative audio modeling.
6 ACKNOWLEDGEMENT
This work is supported by National Natural Science Foundation of China (62076144) and Shenzhen Science and Technology Program (JCYJ202208 0000020081100000 1800001033).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Keyu An, Zhiyu Zhang, Changfeng Gao, Yabin Li, Zhendong Peng. Haoxu Wang. Zhihao Du, Han Zhao, Zhifu Gao, and Xiangang Li. Mela-tts: Joint transformer-diffusiin model with representa- tion alignment for speech synthesis. arXiv preprint arXiv:2509. 14784, 2025.
Matthew Baas, Pieter Scholtz, Arnav Mehta, Elliott Dyson, Akshat Prakash, and Herman Kamper. Mars6: A small and robust hierarchical-coder sext-to-speech model. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2025.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31:2523-2533 2023a.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint urXiv:2305.06636 2023b.
Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Al- jafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, et al. Xtts: a massively multilingual zero-shot text-to-speech model. arXiv preprint arXiv:2406. 04904, 2024.
Sanyuan Chen, Chengyi Wang, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. IEEE Transactions on Audio, Speech and Language Processing, 2025.
Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. CoRR, 2024.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi High fidelity neural audio compression. arXiv preprint arXiv:2210.134 2022.
Wei Deng, Siyi Zhou, Jingehen Shu, Jinchao Wang, and Lu Wang. Indextts: An industrial-level con- trollable and efficient shot text-to-speech system. arXiv preprint arXiv:2502.05512, 2025.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint (xXv:2407.05500) 2024a.
Zhihao Du Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412. 101 17, 2024b.
Zhihao Du Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, et al. Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training. arXiv preprint arXiv:2505.17589 2025.
Hao-Han Guo, Yao Hu, Kun Liu, Fei- Yu Shen, Xu Tang, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. Fireredtts: A foundation texteth-speech framework for industry-level generative speech applications. arXiv preprint ar arXiv:2409.0322 2024.
Yichen Han, Xiaoyang Hao, Keming Chen, Weibo Xiong, Jun He, Ruonan Zhang, Junjie Cao, Yue Liu, Bowen Li, Dongrui Zhang, et al. Quantize more, lose less: Autoregressive generation from residually quantized speech representations. arXiv preprint 1rXiv:2507.12197 2025.
Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang. Jiaqi Li, Peiyang Shi, al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In 2024 IEEE Spoken Language Technology Workshop (SLT), pp. 885-890. IEEE, 2024.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06339. 2024.

---

Published as a conference paper at ICLR 2026
Dongya Jia, Zhuo Chen, Jiawei Chen, Chenpeng Du, Jian Wu, Jian Cong, Xiaobin Zhuang, Chumin Li, Zhen Wei, Yuping Wang, et al. Ditar: Diffusion transformer autoregressive modeling for speech generation. arXiv preprint ar arXiv:2502.03990 2025.
Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High- fidelity text-to-speech with minimal supervision. Transactions of the Association for Computa- fional Linguistics, 11:1703-1718, 2023.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Systems, 36:27980-27993, 2023.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36: 36:14005-14034, 2023.
Hang-Hoon Lee, Ha- Ha-Yeong Choi, Seung-Bing Bin Kim, and Seong-Wii Whan Lee. Hierspeech++: Bridg- ing the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis. IEEE Transactions on Neural Networks and Learning Systems, 2025.
Naihan Li. Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 6706-6713, 2019.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37: 56424-56445, 2024.
Jiaxuan Liu, Zhaoci Liu, Yajun Hu, Yingying Gao, Shilei Zhang, and Zhenhua Ling. Diffstyletts: Diffusion- based hierarchical prosody modeling for exxttoo-spechh with diverse and controllable styles. arXiv preprint arXiv:2412.03388, 2024.
Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, et al. Autoregressive speech synthesis without vector quantization. arXiv preprint DrxXiv:400700855:1 2024.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made simple. In The Twelfth International Conference on Learning Representattinss 2024.
Yuto Nishimura, Takumi Hirose, Masanari Ohi, Hideki Nakayama, and Nakamasa Inoue. Hall-e: Hierarchical neural codec language model for minute-long zero-shot text-to- eex--to-ppeech synthesis. In The Thirteenth International Conference on Learning Representations, 2025.
Marco Pasini, Javier Nistal, Stefan Lattner, and George Fazekas. Continuous autoregressive models with noise augmentation avoid error accumulation. arXiv preprini arXiv:2411. 2024.
Puyuan Peng, Po- Yao Huang, Shang-War Wen Li, Abdelrahman Mohamed, and David Harwath. Voice- craft: Zero-shot speech editing and text-to-speech in the wild. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 12442- 12462, 2024.
Zhiliang Peng, Jianwei Yu, Wenhui Wang. Yaoyao Chang, Yutao Sun, Li Dong, Yi Zhu, Wei- jiang Xu, Hangbo Bao, Zehua Wang, et al. Vibevoice technical report. arXiv preprint 2508. 19205, 2025.
Wei Ping, Kainan Peng, Andrew Gibiansky, Serean O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: Scaling text-to-speech with convolutional sequence learning. arXiv preprint arXiv: 17X.vv1710.0765... 2017.
12

---

Published as a conference paper at ICLR 2026
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations, 2020.
Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang. Rj Skerrv-Ryan, et al. Natural tts synthesis by con- ditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 1779-4783. IEEE, 2018.
Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, Jiang Bian, et al. Natural- speech Latent diffusion models are natural and zero-shot speech and singing synthesizers. In The Twelfth International Conference Learning Representations, 2023.
MiniCPM Team, Chaojun Xiao, Yuxuan Li, Xu Han, Yuzhuo Bai, Jie Cai, Haotian Chen, Wentong Chen, Xin Cong, Ganqu Cui, et al. Minicpm4: Ultra-efficient Ilms on end devices. arXiv preprint SCKV:22506009000 2025.
Chunhui Wang.
Chang Zeng. Bowen Zhang, Ziyang Ma, Yefan Zhu, Zifeng Cai, Jian Zhao, Zhonglin Jiang, and Yong Chen. Ham-tts: Hierarchical acoustic modeling for token-based zero- shot text-to-speech with model and data scaling. arXiv preprint 2rXiv:2403.05989 2024.
Hui Wang, Shujie Liu, Lingwei Meng, Jinyu Li, Yifan Yang. Shiwan Zhao, Haiyang Sun, Yanqing Liu, Haoqin Sun, Jiaming Zhou, al. Felle: Autoregressive speech synthesis with token-wise coarse-to to-fin flow matching. arXiv preprint arXiv:2502.11128, 2025a.
Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Lingin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, al. Spark-tts: An efficient Ilm-based text-to-speech model with single-stream decoupled speech tokens. arXiv preprint 2rxiv:250.00770, 2025b.
Yuancheng Wang. Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xucyao Zhang, Shunsi Zhang, and Zhizheng Wu. Maskget: Zero-shot text-to-speech with masked generative codec transformer. In The Thirteenth International Conference on Learning Representations, 2025c.
Chun Yat Wu, Jiajun Deng, Guinan Li, Qiuqiang Kong, and Simon Lui. Clear: Continuous la- tent autoregressive modeling for high-quality and low-latency speech synthesis. arXiv preprint arXiv:2508. 19098, 2025.
Kun Xie, Feiyu Shen, Junjie Li, Fenglong Xie, Xu Tang, and Yao Hu. Fireredtts-2: Towards long conversational speech generation for podcast and chatbot. arXiv preprint arXiv:2509.0200 02020, 2025.
Shu-wen Yang, Po-Han Chi, Yung- Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Tine Ting Lin, et al. Superb: Speech processing universal performance benchmark. Interspeech 2021, 2021.
Siyi Zhou, Yiquan Zhou, Yi He, Xun Zhou, Jinchao Wang, Wei Deng. and Jingchen Shu. Indextts2: A breakthrough in emotionally expressive and conttlll-c auto-rogressive zero-shot text- to-speech. arXiv preprint arXiv:2506.21611 2025.
Xinfa Zhu, Wenjie Tian, and Lei Xie. Autoregressive speech synthesis with next-distribution pre- diction. arXiv preprini 2xXv:22412.16846, 2024.
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
In accordance with ICLR policy, we disclose that large language model (LLM), specifically Gemini-2.5 and DeepSeek, was used as a general-purpose assistive tool during the writing of this paper. The LLM was employed solely for the purpose of text polishing and refinement, specifically to assist with improving grammar, enhancing word choice, and increasing the overall readability and fluency of the manuscript. However, the model played role in the intellectual scientific contributions this work, including research ideation, experimental design, result analysis, or con- clusion drawing. All such core content is solely the product of the human authors, who take full responsibility for the research presented.
B REPRODUCIBILITY STATEMENT
Inference codes are available at codes.zip. Training was conducted using the Megatron framework on 40 NVIDIA H100 GPUs for the main VoxCPM model and H100 GPUs for Vox Ablation studies used 8 H100 GPUs. Full details about hyperparameters are in Section D
C ETHICS STATEMENT
Since zero-shot TTS model achieves high-quality speech synthesis with the ability to closely mimic speaker characteristics, it carries potential risks of misuse. These risks include, but are limited to, spoofing voice authentication systems or impersonating a specific speaker without their consent, Our experiments were conducted under the assumption that the use of any reference speaker's voice is authorized and intended for legitimate synthesis purposes. To mitigate these risks, we strongly advocate for the development of robust synthesized speech detection algorithms. Fur- thermore, we believe it is crucial to establish clear ethical guidelines and reporting mechanisms for the responsible deployment of such technology.
D IMPLEMATATION DETAILS OF VOXCPM
D.1 MODEL. ARCHITECTURE
VoxCPM consists of a 24-layer TSLM (initialized from MiniCPM-4-0.5B) and a 6-layer RALM. The FSQ layer uses 256 dimensions with 9 scalar levels. The diffusion decoder has 4 layers, opti- mized for high-efficacy latent generation, as shown in Table
For Audio VAE, it operates at 25 Hz. frame rate, designed to be compatible with the streaming nature of VoxCPM. The VAE's architecture is inspired by DAC, with both its encoder and decoder implemented using stacked Causal Convolutional Networks (Causal CNNs). For 16 kHz single- channel audio, the encoder achieves a 640x downsampling factor through a series of strided convo- lutions with a stride sequence of [2, 5, 8, 8), compressing the audio into a 25 Hz latent representation. The decoder then reconstructs the original waveform by upsampling from this latent representation. The training objectives consist of an adversarial (GAN) loss, Mel-spectrogram loss, and KL divergence loss, with the latter's weight set to 5e-5.
D.2 TRAINING CONFIGURATION
Both VoxCPM and VoxCPM-Emilia employed the Warmup-Stable-Decay (WSD) training strategy, which we found essential for optimal convergence. Specifically, the decay phase with annealing to a very low learning rate (combined with batch size doubling) significantly enhances model per- formance, particularly for zero-shot speaker similarity, demonstrated in Table I0] All ablation studies followed the same 200K-step training protocol on H100 GPUs using the Emilia dataset, employing fixed learning rate of 10-1 to ensure consistent comparisons.
14

---

Published as a conference paper at ICLR 2026
Table The model architecture and parameters of VoxCPM-0.5B.
Module
Configuration
Parameters
LocEne
layers, 1024 hidden dim, 4096 FFN dim 24 layers, 1024 hidden dim, 4096 FFN dim 256 dimensions, quantization levels layers, 1024 hidden dim, 4096 FFN dim layers, 1024 hidden dim, 4096 FFN dim
59M
TSLM
433M
FSQ
0.5M
RALM
89M
LocDiT
64M
Stop Predictor
3 layer MLP, 1024 hidden dim, 2 output dim (that is, TSLM and RALM work in 12.5Hz token rate)
IM
patch-size
Audio
16kHz waveform 25Hz latents (downsampling [2, 5, 8, 8))
75M
Table 6: Training configurations for VoxCPM variants.
Model
Phase
Learning Rate
Tokens/Batch
Steps
GPUs
VoxCPM
Stable
x 10-1 10-1 + 5 x 10-6
4,096
400K
40 x H100
VoxCPM
Decay
8,192
100K
x H100
VoxCPM-Emilia
Stable
x 10-4
4,096
150K
24 x H100
VoxCPM-Emilia
Decay
1 10 + x 10-6
8,192
50K
H100
VoxCPM-ablation
Stable
1 10
4,096
200K
x H100
E DETAILS OF BASELINES
We compared VoxCPM against several state-of-the-art TTS systems, focusing on open-source mod- els from the SEED-TTS-EVAL and CV3-EVAL benchmarks. Below, we describe the key baselines, their architectures, training data, and how evaluation samples were generated.
Cosy Voice Series (Du al. 2024a 2025): A family of TTS models leveraging su- pervised semantic tokens and multi-stage pipelines. Cosy Voice (Du et al., 2024a) uses transformer-based architecture with S3Tokenizer, trained on approximately 170K hours of multilingual speech data (primarily English and Chinese). Coic22 (Du et al.) (024b) enhances this with text-based LLM initialization, and introduces streaming syn- thesis for low-latency synthesis. CosyVoice3 (Du al. 2025) further leverages in-the- wild speech data, and proposes a novel speech tokenizer to capture prosody, with pow- erful post-training techniques. We used officially released checkpoints from https: //github. wtt./oom////u/ue wtw.A/uddo.iMMM/oyoooo..ooo with default settings to generate sam- ples.
MaskGCT (Wang et al. 2025c): A non-autoregressive transformer model that employs masked generative transformers to predict discrete speech tokens derived from neural audio codec and SSL tokens. It is trained the Emilia dataset and achieves strong performance on zero-shot TTS. Besides, it can control the duration precisely. Samples were generated using the official implementation provided at htttts:://gtttii com/ open-mmlab/ ab/ hwwpaaaaaaa/mmaaaaaaaaam/aa
F5-TTS (Chen al. 2024): A non-autoregressive model utilizing flow-matching for eff- cient speech synthesis. It operates on continuous mel-spectrogram representations, also trained on Emilia dataset. We used the official codes and checkpoint from https: //github. com SWi www./wd/ddn.
TTS to generate evaluation samples.
SparkTTS (Wang et al.) 2025b): A languae model-based model using single-stream de- coupled speech tokens improve modeling efficiency. It is trained on 100K hours of bilingual (English and Chinese) dataset VoxBox, enabling it capability to achieve con- trollable speech synthesis. Samples were generated using the official implementation at ccom/SparkAadio///5 Saaaadii/ppaaaTTTtaa TTS
FireRedTTS Series (Guo et al. 2024 Xie et al. 2025): FireRedTTS presents a founda- tion TTS framework for industry- level generative speech application. FireRedTTS2 fur-
15

---

Published as a conference paper at ICLR 2026
ther employs it for long-for formffom multi-speaker dialogue. FireRedTTS-2 improves scalabil- ity with a refined transformer architecture, trained on 1.IM hours monologue speech data and 300k hours of mplti-speaker dialogue data. We used official checkpoints from hstps://githuse com /F
for sample generation.
IndexTTS2 (Zhou et al. 2025): autoregressive large-scale TTS model with precise duration control. It also achieves disentanglement between emotional expression and speaker identity, enabling independent control over timbre and emotion. Samples were generated using the official checkpoint from index-tts
www.ccmmiinnee ttnn/ineex-tts
Higgs Audio v2: A powerful audio foundation model pretrained on over 10 million hours of audio data and diverse set of text data. It proposes unified audio tokenizer captures both semantic acoustic features. We used the official implementation at https:// www.//ooooooooooooooooo boson withiggs-aasseeeee audio to generate samples.
github. com
Voice (Peng et al. 2025): novel framework designed for generating expressive, long- form, multi-speaker conversational audio, such podcasts, from text. It presents contin- uous speech tokenizers (Acoustic and Semantic) operating at an ultra-low frame rate of 7.5 Hz. Because now the codes are unavailable at phttps: //qi //gtttttaaa. thab. wwt.mmocooooooooccooooooooooooot VibeVoice we use the official results reported in the paper.
OpenAudio-sl-mini: A compact and powerful TTS model (0.5B parameters) using dual- AR architecture and online Reinforcement Learning from Human Feedback (RLHF). Samples were generated using the official implementation at sulhttps:/ /ttpt:///ghhbb com/
fishaudio/ fish- speech
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 COMPREHENSIVE COMPARISONS ON SEED-TTS-EVAL BENCHMARK
In this section, we present comprehensive comparisons covering both open-source and close-source TTS systems. As shown Table 7 VoxCPM not only outperforms open-soure, competitors but also achieves results comparable to several proprietary systems, despite their significantly larger parameter counts and training data scales.
Table 7: Performance on Seed-TTS-eval Benchmark, including close-sourced systems
EN
ZH
Hard
Model
Params Open-Source
WER
SIM
CER |
SIM
CER
SIM
MegaTTS3
0.5B
X
2.79
77.1
1.52
79.0
DiTAR
0.6B
X
1.69
1.02
75.3
Cosy Voice3
0.5B
X
2.02
71.8
1.16
78.0
6.08
75.8
Cosy Voice3
1.5B
2.22
72.0
1.12
78.1
5.83
75.8
Seed-TTS
2.25
76.2
1.12
79.6
7.59
77.6
MiniMax MiniMax--peech-02
X
1.65
0.83
78.3
F5-TTS
0.3B
2.00
67.0
1.53
76.0
8.67
71.3
MaskGCT
1B
2.62
71.7
2.27
77.4
Cosy Voice
0.3B
4.29
60.9
3.63
72.3
11.75
70.9
Voic22
0.5B
3,09
65.9
1.38
75.7
6.83
72.4
SparkTTS
0.5B
3.14
1.54
66.0
FireRedTTS
0.5B
3.82
46.0
1.51
63.5
17.45
62.1
FireRedTTS-2
1.95
66.5
1.14
73.6
Qwen2.5-Ommi
7B
2.72
63.2
L.70
75.2
7.97
74.7
OpenAudio-sl-miin
0.5B
1.94
55.0
1.18
68.5
23.37
64.3
IndexTTS 2
1.5B
2.23
70.6
1.03
76.5
7.12
75.5
Voie
1.5B
3,04
68.9
1.16
74.4
HiggsAudio-v2
3B
2.44
67.7
1.50
74.0
55.07
65.6
VoxCPM-Emilia
0.5B
2.34
68.1
1.11
74.0
12.46
69.8
VoxCPM
0.58
1.85
72.9
0.93
77.2
8.87
73.0
16

---

Published as a conference paper at ICLR 2026
F.2 EFFECT OF LM GUIDANCE ON LOcDIT
To explore CFG influence and select the best inference setting, we tested different CFG value, that is, the LM (the sum of TSLM-FSQ hidden and RALM hidden) guidance LocDiT. We found that CFG value=2.0 could achieve optimal balance across all metrics. Higher weights (23.0) degraded intelligibility significantly.
Table 8: Effect of LM guidance on LocDiT, tested on Wedd-TTS-eval Benchmark.
EN
ZH
ZH-hard case
CFG Value
WER 4
SIM 1
CER
SIM 1
CER
SIM f
(w/o CFG)
16.32
55.1
14.47
61.5
56.87
43.0
1.5
1.86
72.1
1.16
77.0
9.60
73.9
2.0
1.85
72.9
0.93
77.2
8.87
73.0
3,0
2.16
71.4
1.12
74.7
13.22
65.0
12.78
60.7
17.23
59.4
48.46
39.9
F.3 EFFECT OF THE SEMI-DISCRETE BOTTLENECK
This section shows more comprehensive investigation about FSQ bottleneck dimension, as shown in T
be found that when removing scalar quantization in the bottleneck layer, the per- formance tends to degrade, showing that semi-discrete latent space maybe more stable and robust than fully continuous space. Besides, we investigated the impact between FSQ and VAE regu- larization as the bottleneck in VoxCPM. We replaced the FSQ bottleneck with continuous VAE bottleneck and trained under the same setting. We specifically chose dimension of 16 for the continuous bottleneck (16d- VAE), as high-dimensional continuous latents (e.g., 256d) empirically suffer from posterior collapse or inefficient compression due to optimization challenges. Under the same 16-dimensional setting, the FSQ model achieves significantly better robustness than the VAE. The VAE bottleneck nearly doubles the error rate on challenging cases (CER=28.) 17%) compared to the FSQ counterpart (CER=14.42%) This confirms that the semi-disen mini-iiccrete nature of FSQ provides an indispensable inductive bias for stability that continuous regularization lacks. While the con- tinuous 16d-VA VAE achieves slightly higher English speaker similarity than 16d-FSQ (likely because continuous vectors can encode finer acoustic details), this comes at the cost of intelligibility. Our default 256d-FSQ strikes the best balance, maintaining high expressivity without the optimization instability associated with high-dimensional continuous bottlenecks.
Table 9: FSQ dimension selection study on the Emilia dataset. Note: The 256-dim configuration was selected for the final VoxCPM configuration with the understanding that larger training datasets needs more powerful modeling capabilities.
ZH
ZH-hard case
Model Setting
WER
SIM 1
CER 4
SIM
CER +
SIM 1
w FSQ: d4s9
5.18
59.3
4.05
68.0
19.55
62.3
w FSQ: d16s9
3.22
60.4
1.87
70.5
14.42
66.2
w FSQ: d64s9
3.22
61.1
2.14
69.8
17.48
65.1
w FSQ: d128s9
3.43
62.2
1.67
70.7
16.76
65.7
w FSQ: d256s9
2.98
62.6
1.77
70.4
18.19
64.9
W FSQ: d1024s9
3.07
62.0
2.38
69.8
20.38
64.7
w/o FSQ: d1024sco
3.67
62.1
2.30
69.6
24.92
63.5
w VAE: d16
3.56
62.1
1.94
69.7
28.17
62.7
F.4 EFFECT OF TRAINING PHASE ON PERFORMANCE
As mentioned in D.2 the two-phase Warmup-Stable-Des Decay (WSD) learning rate schedule is critical for achieving optimal model performance. The initial Stable phase allows the model converge reliably strong baseline. The subsequent Decay phase then essential for refining the model,
17

---

Published as a conference paper at ICLR 2026
particularly for improving its zero-shot voice similarity capabillitiss Furthermore, doubling the batch size (from 4K to 8K tokens) during the Decay phase a necessary complement to the reduced learning rate. A larger batch size provides more accurate and stable estimate of the gradient di- rection, which is crucial for effective and stable optimization when using very low learning rates. This strategy prevents the noise from small-batch gradients from destabilizing the fine-tuning pro- cess, enabling the model to make consistent improvements in both intelligibility (lower WER) and speaker similarity (higher SIM).
Table 10: Performance across training phases.
EN
ZH
ZH-Hard Case
Phase
WER 4
SIM
CER
SIM f
CER 4
SIM t
Stable
2.05
69.7
0.99
75.1
13.22
68.6
Decay
1.85
72.9
0.93
77.2
8.87
73.0
F.5 VISUAL ANALYSIS OF HIERARCHICAL REPRESENTATIONS
To validate our core hypothesis of learned implicit semantic-acoustic disentanglement, we conducted a t-SNE visualization of the internal representations our hierarchical model. The resulting distri- butions, shown in Figures 2 and 3 empirically confirm the specialized roles of the TSLM and the RALM. Figure 2 iillutrates the model's behavior in a zero-shot voice cloning task, where each color corresponds to a distinct utterance from unseen speaker. The TSLM-FSQ outputs form stable, speaker-agnostic remantic-prosodee structure, while the RALM residuals cluster by speaker identity, confirming their specialized roles in content planning and acoustic refinement.
Figure 3 further demonstrates the VoxCPM's capability to infer appropriate prosody and style di- rectly from text, when using any speech prompt. When processing different text genres (news, poetry, conversation), TSLM-FSQ representations cluster by semantic category, showing that the pre-trained language model backbone effectively infers appropriate prosodic patterns directly from text content. For example, embeddings for "news" group together, separate from "story-telling" or "rap-lyrics." The RALM outputs display greater authin-category variation, indicating its role in adding fine-grained acoustic nuances to the semantic-prosodie plan.
To investigate whether the observed cemantic-acoustio specialization originates from the hierarchical structure design or the pre-trained LLM weights, we conducted a controlled experiment. We trained a VoxCPM variant with fully random TSLM initialization and performed t-SNE visualization of its latent spaces for both zero-shot cloning (Figure 4 and text-to-speech (Figure 5) tasks.
The visualizations reveal that even in the absence pre-trained semantic knowledge, the latent space exhibits similar functional division of labor. The TSLM- TSLM-FSQ hidden states still show rele- vance to semantic content and weak speaker relevance (as seen in Figure 5) while the RALM hidden states display strong speaker-dependent clustering (as in Figure specialization is slightly less distinct compared to the L.LM initialized model, the consistency con- firms that the residual semi-discrete bottleneck structure itself is the key inductive bias that enables the model to self-adapt and learn a specific semantic-acoustic division of labor. The pre-trained lan- guage model parameters primarily serve significantly sharpen, stabilize, and enhance this inherent modeling capacity.
Although this seaf-learned
F.6 FUSION STRATEGIES ABLATION
The LocDiT module is responsible for generating the final speech latents by conditioning on the combined semantic-acoustic representation derived from the TSLM (via FSQ) and the RALM. In the default VoxCPM configuration, we adopt a simple element-wise summation of the TSLM and RALM output hidden states as the fusion mechanism, primarily for its intuitively aligning with the "residual" nature of the design and good performance. To validate this choice, we conducted an ablation study comparing summation against several other common feature fusion strategies (con- catenation, gating, and attention based fusion). The results are presented in Table II It shows that while concatenation slightly improves speaker similarity, the simple summation strategy achieves
18

---

Published as a conference paper at ICLR 2026
distribution TSLM-FSQ semi-discrete
distribution hidden
1822
MD
35
1833
483
183
4810
-10
-20
-30
Figure 2: The T-SNE visualization of latent space distributions in zero-shot voice cloning task.
distribution semii-iiscree Prompt Speach)
distribution RALM
Prompt Speech)
acetre
pedtry
-20
10ME
Figure 3: The T-SNE visualization of latent space distributions in text-to-speech task, without prompt speech.
the best balance of intelligibility (lowest WER and CER) and acoustic quality. The significantly higher error rates observed with attention-basee fusion suggest that relying on the model to dynam- ically learn the fusion weights introduces optimization instability in the continuous latent space.
Table II: Ablation different fusion strategies for combining TSLM and RALM outputs LocDiT.
Fusion Strategy
EN- (L)
EN-SIM (t)
ZH-CER (L)
ZH-SIM (t)
Sum (Default)
2.98
62.6
1.77
70.4
Concatenation
3.16
64.1
2.88
71.3
Gating
3.75
63.6
2.04
70.2
Attention
4.92
59.9
5.43
66.5
19

---

Published as a conference paper at ICLR 2026
distribution TSLM-FSQ semi-discrete hidden Speech)
distribution
3300740
3330334
02004530
02004530
33001/0
32301131
1200220
3230220
03112300
1200970
28
12305200
-20
LOME Cinaraie
Figure 4: The T-SNE visualization of latent space distributions (without pretrained LLM parameters initialization for TSLM) in zero-shot voice cloning task.
The distribution of TSLM-FSQ semi-discrete hidden (wio Prompt Speech)
The distribution RALM residual hidden (wio Prompt Speech)
documenay
decumenten
soepyiiling
46
need
-20
1958
Figure 5: The T-SNE visualization of latent space distributions (without pretrained LLM parameters initialization for TSLM) in text-to-speech task, without prompt speech.
F.7 PROBING ANALYSIS FOR SEMANTIC-ACOUSEE MODELING
To verify the core hypothesis that the FSQ bottleneck induces a functional separation of labor be- tween the TSLM (semantic focus) and RALM (acoustic focus), we conducted Layer-wise Probing Experiments following the SUPERB YYagg al., 2021) setting. We trained lightweight linear classifiers on the frozen internal hidden states extracted from four key locations: LocEne, TSLM FSQ, and RALM. We assessed the capture of linguistic-content using phoneme recognition (PR) and automatic speech recognition (ASR), measured by Phoneme Error Rate (PER) and WER, re- spectively. The lower values indicate better linguistic content preservation (higher intelligibillity. Acoustic-timbre modeling was evaluated via the Automatic Speaker Verification (ASV) task, mea- sured by Equal Error Rate (EER). The lower values indicate better speaker identity modeling (higher acoustic/timbre distinguishability)..
The probing experiment results offer evidence to architectural claims:
20

---

Published as a conference paper at ICLR 2026
Acoustic Features Aggregation: The LocEnc is responsible for aggregating the low-level continuous acoustic latents from VAE. Consequently, it reflects the weak seman- tie relevance (PER=59.122 12%% WER=65.79%) and carries fundamental acoustic information like timbre (EER=15.38%).
Semantic Specialization:
The TSLM representation achieves the lowest PER (45.60%) and WER (60.43%), validating its role in capturing the stable linguistic and prosodic content.
Acoustic Filtering: The FSQ representation exhibits the highest EER (19.25%) This is a critical finding: the quantization bottleneck effectively acts acoustic filter, actively reducing correlation with speaker identity and prioritizing content-prosodic stability.
Residual Recovery: Conversely, the RALM representation achieves the lowest EER (13.24%) This conclusively proves that the RALM specializes in recovering the fine- grained acoustic residuals and speaker identity that were filtered by the FSQ
The slight rise content error post-ESQ PPR=500.0%% WER=62.37%) confirms the expected lossy nature of quantization and the subsequent shift in the RALM's focus from pure semantic modeling toward acoustic refinement. These findings empirically validate our premise: VoxCPM successfully inductive bias.
enforces natural division of labor using the FSQ
Table 12: Layer-wise probing results on internal hidden states of VoxCPM.
Hidden State Location
PR (PER)
ASR (WER) 4
ASV (EER) 4
LocEnc output
59.12
65.79
15.38
TSLM last hidden (Pre-FSQ)
45.60
60.43
18.70
FSQ output
50.90
62.37
19.25
RALM last hidden
53.49
64.85
13.24
F.8 EFFECT OF MODEL. SCALABILITY
We conducted a model scalability analysis to empirically demonstrate the robustness of the hier- archical VoxCPM architecture across varying capacities. We trained larger variants of VoxCPM, namely IB and 3B parameters, on the Emilia dataset for 200k iterations.
The specific architectural configurations used for the larger models
follows:
VoxCPM-1B: This model consisted of a 28-layer TSLM (h 2048), a 7-layer RALM (h = 2048), and 6 layers for both the LocEne and LocDiT modules (h 1024).
VoxCPM-3B: This model was configured with a 32-layer TSLM 2560), an 8-layer RALM (h 2560), and 8 layers for both LocEnc and LocDiT (h 1024).
As shown in Table 13 increasing the model capacity consistently improved performance across all core metrics (EN- WER, EN-SIM. ZH-CER, and ZH-SIM). Specifically, the 3B variant achieved the best intelligibility (2.60% WER) and speaker similarity (66.7% SIM). This validates that the multi-component, hierarchical design of VoxCPM remains fully scalable and effectively leverages increased parameter counts.
Table 13: Scalability analysis of VoxCPM variants trained on the Emilia dataset.
EN
ZH
Model Size
WER
SIM
CER
SIM 1
0.5B
2.98
62.6
1.77
70.4
IB
2.95
65.9
1.82
72.0
3B
2.60
66.7
1.78
72.3
21

---

Published as a conference paper at ICLR 2026
F.9 THE COMPARISON OF REAL-TIME FACTOR AND STREAMING SYNTHESIS DETAILS
We evaluated the Real-Time Factor (RTF) of VoxCPM in comparison to other TTS models on a sin- gle NVIDIA RTX 4090 GPU. VoxCPM achieves remarkably low RTF of 0. 17,7, significantly faster than Voice2 (0.52) and SparkTTS (0.80); due to its efficient 12.5 Hz token rate and lightweight components. The throughput for the LM backbone in VoxCPM is approximately 73 tokens/s.
Table 14: The Real-Time Factor of some TTS systems.
TTS System
RTF
Voice 2
0.52
SparkTTS
0.80
IndexTTS 2
0.85
VoxCPM 0.17
Furthermore, VoxCPM supports streaming synthesis with theoretical first-packet latency be- low 100 ms: the LocDiT generates patches in under 10ms with 10 iterations, owing to its short local context and lightweight parameters, while the causal AudioVAB enables incremental processing. To ensure smooth playback, the last three latents are buffered, resulting in 80 ms audio chunks per step.
G FURTHER DISCUSSION
G.1 THEORETICAL GROUNDING: INFORMATION BOTTLENECK PRINCIPLES
To better explain the success use of FSQ bottleneck in VoxCPM, provide intuitive theoreti- cal grounding information bottleneck principles: The FSQ layer functions as a capacity-constrained bottleneck between the high-capacity TSLM and the downstream RALM. Mathematically, the scalar quantization inherent in FSQ acts as lossy compression filter. To minimize the end-to-end diffu- sion loss, the TSLM is heavily penalized if attempts to encode high-variance, fine-grained acoustic details (like timbre specific residuals), as these would be significantly distorted by the scalar quan- tization step. Consequently, the TSLM naturally forced to prioritize encoding low-variance, stable neatur the semantic and global prosodic structure (content, intonation). This mechanism effectively offloads the task of modeling high-frequency, complex acoustic residuals to the RALM.
Empirical results from our probing experiments (Appendix F.7 validate this induced division of labor: TSLM representations demonstrate relatively high semantic accuracy (low PER and WER) but acoustic- agnossic (high speaker EER), while RALM exhibits the opposite behavior. This confirms the FSQ's effectiveness as an internal inductive bias for hierarchical task separation.
G.2 ANALYSIS ON QUANTIZATION CEILING
Discrete tokenizers like S3Tokenizer map speech signals to a fixed, finite codebook of size K.imposing rigid, finite ceiling on expressive information (the "quantization ceiling"). Our Audio VAE operates in 64-dimensional continuous latent space, which retains significantly richer acoustic detail and avoids the critical precision loss inherent in mapping to finite, low-dimensional set.
Continuous Latents Diserete Codebooks:
Internal FSQ Bottleneck Capacity: Inside VoxCPM, the FSQ bottleneck serves as a regularizer, not a prediction target for a small codebook. We use a high--iiig linee-iimnsional FSQ (e.g.. d256s9), offering vast representational space far exceeding traditional VQ/FSQ codebooks. Our FSQ ablation study (Table shows that when we constrain the FSQ capacity mimic a traditional small codebook (e.g., FSQ d4s9, equivalent to 6561 vocab), the clone similarity (SIM) significantly degrades, proving that full information preservation is essential for the model's performance ceiling.
VAE Reconstruction Quality: To quantify the potential information bottleneck imposed by the VAE, we evaluated its reconstruction quality on the SEED-TTE TTD--TS--RE TTS-Eval dataset. The VAE is designed to be reconstruction-orienned with a low KL-penalty, ensuring maximum information retention. As shown in Table [5] the VAE-reconstructed audio exhibits high fidelity across objective metrics. The minor difference between the Ground Truth (GT) and Reconstruction confirms that the continuous
22

---

Published as a conference paper at ICLR 2026
latent space effectively preserves necessary acoustic detail, distinguishing it from the significant information loss typically associated with low-bitrate discrete tokenizers.
Table 15: AudioVAE reconstruction performance comparison against Ground Truth (GT) on SEED
TTS-Eval.
Metric
Mel Loss (1)
STOI (1)
WER ())
SIM (1)
DNSMOS (1)
GT Recon
0.000
1.000
1.21
75.5
3.96
0.924
0.948
1.33
73.6
3.92
H EVALUATION METRICS AND QUESTIONNAIRES
To ensure robust subjective evaluation, we developed an automated interface for assessing gener- ated speech samples. For each evaluation item, participants interact with three components: the Evaluation Interface (containing the audio sample to be rated), the Questionnaire, and the Rating Guidelines. Evaluations were conducted by 20 people for both English and Chinese samples.
H.1 NATURALNESS MOS
Evaluation Interface: A single audio clip accompanied by its corresponding text prompt.
Questionnaire: To what extent does the speech sound natural and human-like? Does convey en- gagement with the content, does it resemble artificial voice lacking contextual understanding?
Rating Guidelines:
5: Indistinguishable from human speech, highly natural and engaging-
4: Surpasses expectations for synthetic speech, very human-like.
3: Meets expectations for Al-generated speech, reasonably natural.
2: Below average, with noticeable artificial qualities.
1: Clearly synthetic, lacking human-like expression.
H.2 SPEAKER SIMILARITY MOS
Evaluation Interface: A reference audio clip alongside the audio clip to evaluated.
Questionnaire:
Focusing solely on voice characteristics (disregarding content and audio quality), how closely does the evaluated voice match the reference voice?
Rating Guidelines:
5: Nearly identical to the reference voice, as if spoken by the same person.
4: Highly similar to the reference voice, with minor differences.
3: Moderately similar, sharing some vocal traits.
2: Largely dissimilar, with few shared characteristics.
1: Completely distinct, bearing resemblance to the reference voice.
23


Reviews:
[{"review_id": "7j5oLtpD3B", "reviewer": "Reviewer_ada6", "summary": "This manuscript proposes a novel end-to-end TTS framework, VoxCPM, to address the fundamental trade-off between discrete and continuous speech representations, where discrete representations maintain stability but sacrifice expressivity, while continuous representations retain acoustic richness but are prone to error accumulation. The hierarchical semantic-acoustic modeling framework is well-motivated and addresses some limitations in current TTS systems. \n\nThe proposed VoxCPM is with the hierarchical semantic-acoustic architecture, consisting of a Text-Semantic Language Model (TSLM), a Residual Acoustic Language Model (RALM), and the Local Diffusion Transformer (LocDiT). The key innovation is the differentiable Finite Scalar Quantization bottleneck, which serves not as a prediction target, but as an intermediate regularization mechanism to induce natural task separation. \n\nThe manuscript demonstrates that their 0.5B-parameter model achieves state-of-the-art zero-shot TTS performance with extensive experiments and ablations.", "questions": "1. You sum TSLM and RALM outputs. Did you experiment with other fusion strategies (concatenation, gating, attention)?\n\n2. The paper states FSQ is \"analogous to the first layer of Residual Vector Quantization (RVQ)\". Did you experiment with using a learned vector quantizer (e.g., a single VQ layer) as the bottleneck instead of the non-learned FSQ? It seems a learned codebook might create an even more efficient semantic-prosodic skeleton. And what happens if you replace FSQ with VQ-VAE or other quantization methods?\n\n3. Please discuss the potential quality ceiling imposed by their pre-trained causal VAE  and how it relates to the \"quantization ceiling\" they criticize in discrete tokenizers.\n\n4. Please try your best to provide more theoretical analysis as mentioned in \"Weaknesses\".\n\n4. The VAE is also external tokenizer. \n\n5. For citation formatting, please ensure consistent citation style (some entries lack year information)", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The major contribution, using a differentiable FSQ bottleneck as an internal inductive bias rather than a predictive target, is a elegant and well-motivated solution to the task entanglement problem. By regularizing the hidden state, it forces the TSLM to focus on stable semantic-prosodic structures while offloading fine-detail modeling to the RALM.\n\n2. The paper provides compelling quantitative results. VoxCPM achieves state-of-the-art performance among open-source models on two challenging benchmarks, SEED-TTS-EVAL and CV3-EVAL. The ablation studies conclusively validate the authors' core hypotheses.\n\n3. While the best-performing model (VoxCPM) is trained on a large, proprietary dataset , the authors have responsibly run all critical ablation studies on the public Emilia dataset.", "weaknesses": "1. While the empirical results are strong, the paper lacks theoretical justification for why FSQ specifically induces semantic-acoustic disentanglement\n\n2. No formal analysis of the \"quantization ceiling\" or error accumulation mechanisms, and how the proposed architechture can effectively address them.\n\n3. The claim that FSQ acts as an \"inductive bias\" needs more rigorous theoretical grounding and mathematical proof.\n\n4.  The main SOTA results are from the VoxCPM model trained on a 1-million-hour internal, bilingual dataset. While the authors provide VoxCPM-Emilia, trained on a public dataset, the top-line performance is dependent on this massive, inaccessible dataset. This limits the direct reproducibility of the primary claims, although the architectural insights from the Emilia-based ablations remain sound. Some baselines (F5-TTS, GPT-Sovits) may use significantly less data, making direct comparison problematic.\n\n5. The paper doesn't discuss whether the performance gains are primarily from architecture or data scale\n\n6. The entire model generates latents from a pre-trained Causal VAE. This VAE is itself a form of compression. The paper criticizes discrete tokenizers for their \"quantization ceiling\", but doesn't discuss the potential information bottleneck or quality ceiling imposed by its own VAE. The quality of this VAE is critical to the final output, but it's only briefly detailed. \nThe author also claimed that the voxCPM do not rely on a external tokenizer. However the VAE is itself a tokenizer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o50gBaJIRD", "reviewer": "Reviewer_emjo", "summary": "The paper introduces a hierarchical AR framework based on semi-discrete residual representations to mitigate the trade-off between stability and expressivity observed in recent LLM-based TTS systems using speech tokenizers. The proposed architecture comprises a Text-Semantic Language Model that captures high-level linguistic and prosodic structure, and a Residual Acoustic Language Model that reconstructs fine-grained acoustic details. It further incorporates a differentiable FSQ bottleneck with LocDiT and a causal VAE to achieve unified and high-fidelity speech generation.", "questions": "Q1. Does the observed separation in Appendix F.5 - Figure 2 truly result from the FSQ bottleneck, or is it primarily due to the pre-trained initialization of the semantic LLM? How would the t-SNE visualization change if the TSLM were trained from random initialization?\n\nQ2. In most recent LLM-based TTS systems, increasing model capacity generally improves both speaker similarity and intelligibility. However, in the ablation study (Table 4), deeper TSLM layers slightly increase SIM but noticeably degrade WER. It would be helpful if the authors could clarify the reason for this. observation. Specifically, how can the effects of structural disentanglement be distinguished from capacity related trade-offs, and might the observed degradation reflect limitations in training dynamics rather than an inherent need for the RALM module?\n\nQ3. The paper states that the TSLM uses MiniCPM-4-0.5B as its backbone, while the overall model size is also reported as 0.5B parameters. Could the authors clarify how this total is computed? Specifically, how many parameters are allocated to the AudioVAE and the RALM modules?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper presents a clear and well-structured framework that effectively balances stability and expressivity through hierarchical semantic-acoustic modeling. The proposed combination of TSLM, RALM, and the FSQ bottleneck is conceptually coherent and technically sound. \n- The writing is clear and easy to follow, and the experimental results are well organized and convincingly support the main claims.", "weaknesses": "- The paper introduces FSQ as a semi-discrete bottleneck, but its function appears closer to latent feature regularization than true discrete representation learning. Because FSQ is not used for token prediction, it mainly constrains feature space rather than modeling discrete structure. A comparison with VAE-based latent regularization would help determine whether FSQ provides meaningful representational advantages or simply smoother feature constraints. \n- The authors claim the system as an  end-to-end framework, but it cannot be considered truly  end-to-end because it relies on a pre-trained audio VAE to extract continuous speech latents.\n- The paper provides limited details about the Causal Audio VAE, even though the design of latent speech tokens can significantly affect overall performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kFjgpJeFIq", "reviewer": "Reviewer_kJR8", "summary": "The paper introduces VoxCPM, a framework that addresses speech expressivity and intelligibility challenges. Codec-based methods enable stable autoregressive generation but lose fine-grained acoustic details due to quantization, while continuous approaches accumulate errors over long sequences. VoxCPM combines a Text-to-Speech Language Model (TSLM) for generating semantic and prosodic plans with a Residual Acoustic Language Model (RALM) that restores fine acoustic details. Guided by these components, a local diffusion-based decoder produces high-fidelity speech latents. Experiments on two datasets show that VoxCPM achieves better results with fewer model parameters.", "questions": "- Regarding the acoustic embeddings (E<i), what specific acoustic features (pitch variations, timbre, voice quality) does it learn?\n\n- Could the author clarify how the proposed framework manages long audio sequences?\n\n- RALM is reported to recover “speaker identity” implicitly, yet the model has no explicit conditioning on speaker embeddings or style tokens. Could the authors clarify how speaker identity is actually captured?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a differentiable FSQ bottleneck that divides semantic–prosodic planning from acoustic rendering, maintaining full end-to-end trainability and overcoming the limits of both discrete and continuous methods.\n\n- The paper reduces computational cost by using a VAE-based model that works in a compact latent space rather than on raw waveforms. A causal design allows for low-latency, real-time streaming synthesis.\n\n- The paper shows that VoxCPM outperforms other open-source models on the experimented dataset using 0.5B parameters, outperforming larger models such as IndexTTS2 (1.5B) and HiggsAudio-v2 (3B).", "weaknesses": "- The paper combines existing components (pre-trained LLM, FSQ, diffusion decoder) without explaining why this combination works beyond T-SNE visualizations. What specific features do TSLM vs. RALM capture? Missing are probing experiments, attention analysis, layer-wise studies, or comparisons with alternative quantization methods. The paper should provide a few fundamental insights to verify their claims.\n\n- The model does not use any explicit alignment mechanism, such as attention or duration modeling, to synchronize text tokens with acoustic frames. Instead, it depends on implicit correlations learned during training, which can result in mispronunciations, timing instability, or prosodic drift in longer utterances.\n\n- The ablation studies don’t justify critical design decisions, including the 24-layer/6-layer split between TSLM and RALM, the contribution of pre-trained LLM initialization quality (e.g., comparing MiniCPM-4 vs. other pre-trained models), and the necessity of the FSQ bottleneck itself by comparing against simpler hierarchical architectures without quantization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EUKdFef3dE", "reviewer": "Reviewer_Nvjv", "summary": "The paper presents VoxCPM, a hierarchical model for zero-shot TTS. The authors point out that existing discrete token approaches, while stable, tend to lose subtle acoustic nuances, whereas fully continuous models preserve detail but often become unstable over longer utterances. VoxCPM addresses this by introducing a semi-discrete differentiable bottleneck that separates stable discrete, potentially linguistic and prosodic information, from finer acoustic details, avoiding the scalability issues of large discrete codebooks while keeping the system trainable end-to-end.\n\nThe model generates speech in stages: a text-semantic language model first captures content and prosody, this representation is regularized through a scalar quantization layer to form a coarse conditioning, and a residual acoustic module restores missing details such as timbre and micro-prosody. A local diffusion transformer then synthesizes the final latent audio segments, conditioned on both text and previously generated context. The latents are derived from a causal VAE.\nTraining is done jointly with a flow-matching objective.\n\nThe proposed method is technically sound and well-motivated, addressing a central bottleneck in hierarchical speech synthesis. The proposed semi-discrete, differentiable quantization + residual refinement framework feels like a natural evolution of current VALL-E-style architectures toward more continuous, end-to-end differentiable systems of hierarchical speech synthesis.", "questions": "See weaknesses above.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Cohesive hierarchical design: The decomposition into semantic (TSLM+FSQ), residual (RALM), and generative (LocDiT) components is conceptually elegant and grounded in recent hierarchical TTS models.\n\n2. Innovative FSQ usage: Using scalar quantization as a differentiable inductive bias (rather than discrete target) is novel—addresses VQ scalability while maintaining stability.\n\n3. End-to-end differentiability: Gradients flow through quantization and all components, unlike most multi-stage systems.\n\n4. Strong performance across benchmarks\n\n5. The t-SNE plots of the TSLM and RALM show a clear division of roles of the two stages.", "weaknesses": "1. Complex multi-stage hierarchy:\nThe full pipeline—LocEnc → TSLM → FSQ → RALM → LocDiT—adds several interacting components and training dependencies. Despite being end-to-end, this design could be challenging to scale or fine-tune efficiently.\n\n2. Limited analysis of latency and efficiency:\nThe paper emphasizes real-time feasibility via a causal VAE but provides no measured latency or throughput results. It remains unclear how the diffusion-based LocDiT performs under streaming constraints.\n\n3. FSQ interpretability and scalability:\nWhile the ablation results show that the FSQ layer improves stability, plays a distinct role from the RALM, and remains largely speaker- and acoustics-agnostic, the claim that FSQ representations correspond to linguistic content is not directly supported by experimental evidence. This interpretation is plausible, but could be strengthened through probing analyses—such as phoneme prediction or ASR accuracy tests—to confirm that the FSQ indeed captures linguistic or prosodic structure. In addition, the paper does not discuss how well these semi-discrete representations transfer or scale when fine-tuned on different datasets, even within the same language domain.\n\n4. The diagram should include the mathematical notation to make it clearer which token/color corresponds to which notation in the equation and text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eRBm0NnVI1", "reviewer": "Reviewer_MsWh", "summary": "The paper proposes a cascaded TTS model that predicts semantic tokens, refine with added residual latents and finally decoded into speech latents with a latent diffusion model.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. While there are many similar cascaded architecture that predicts semantic tokens, refine with acoustic tokens/latents and finally decode into speech latents/waveform, there is some (but limited) novelty in using FSQ as a bottleneck layer directly after LLM output, rather than as in a pre-trained speech tokenizer. This does give some stability and flexibility.\n2. There is ablation study that investigate the impacts of the design choices.", "weaknesses": "1. While the model does seem to perform better in term of WER, the speech naturalness performance seem to be a mixed bag. In Table 2, the model does not perform as well in DNSMOS metrics. Also in Table 3 the model underperforms/perform on-par on naturalness. In the evaluation section the authors tends to spent more time on the positive results but discuss little about the negative results. I also want to ask why some of the models evaluated in Table 2 are not presented in Table 3?\n2. The paper doesn't evaluate the expressiveness nor the controllability of the proposed model. Some competitors (e.g. CozyVoice and Higgs Audio) are focusing on these aspects, and it's an important direction in the field of TTS. I wonder if the authors have some results in this area?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)