VibeVoice: Expressive Podcast Generation with Next-Token Diffusion
Authors: Zhiliang Peng
Jianwei Yu
Wenhui Wang
Yaoyao Chang
Yutao Sun
Li Dong
Yi Zhu
Weijiang Xu
Hangbo Bao
Zehua Wang
Shaohan Huang
Yan Xia
Furu Wei
Venue: ICLR 2026 Oral (2026)
Keywords: Text-to-Speech; Podcast Generation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: VibeVoice can synthesize long-form speech for up to 90 minutes (in a 64K context window length) with a maximum of 4 speakers, capturing the authentic conversational vibe and surpassing open-source and proprietary dialogue models.

Abstract: Generating long-form, multi-speaker conversational audio like podcasts poses significant challenges for traditional Text-to-Speech (TTS) systems, particularly in scalability, speaker consistency, and natural turn-taking. We present VibeVoice , a novel model designed to synthesize expressive, long-form speech with multiple speakers in a zero-shot manner. A core component of our approach is the continuous speech tokenizers operating at an ultra-low frame rate of 7.5. This tokenizer effectively preserves audio fidelity while significantly boosting computational efficiency for processing long sequences. To facilitate training on authentic conversational dynamics, we have developed an annotation pipeline that generates pseudo transcriptions and turn-taking labels for extensive podcast data. Leveraging this data and our efficient tokenizer, VibeVoice  employs the next-token diffusion framework. This enables VibeVoice  to: (1) synthesize long-form speech (up to 30 minutes) with up to 4 speakers, surpassing the typical 1-2 speaker limits of many prior models; and (2) achieve a high degree of naturalness in turn-taking, pacing, and the rendition of subtle non-lexical cues (such as breaths and lip smacks), which are crucial for listener immersion and capturing the authentic vibe of expressive conversations.

Published as a conference paper at ICLR 2026
VIIEE: EXPRESSIVE PODCAST GENERATION WITH NEXT- TOKEN DIFFUSION
Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, Yutao Sun; Li Dong" Yi Zhu, Weijiang Xu, Hangbo Bao, Zehua Wang, Shaohan Huang, Yan Xia, Furu Wei Microsoft Research
http://akaa.ms/Gennraall
ABSTRACT
Generating long-form, multi-speaker conversational audio like podcasts poses significant challenges for traditional Text-to-Speedd -ett---eecch (TTS) systems, particularly in scalability, speaker consistency. and natural tur-takking We present VIBEVOICE, a novel model designed to synthesize expressive, long-form speech with mul- tiple speakers in zero-shot manner. core component of our approach is the continuous speech tokenizers operating at ultra-low frame rate of 7.5. This tokenizer effectively preserves audio fidelity while significantly boosting computational efficiency for processing long sequences. To facilitate training on authentic conversational dynamics, we have developed an annotation pipeline that generates pseudo transcriptions and turntakking labels for extensive podeast data. Leveraging this data and our efficient tokenizer, VIBEVOICE employs the next-token diffusion framework. This enables VIBEVOICE long-form speech (up to 90 minutes) with up to 4 speakers, surpassing the typical 1-2 speaker limits of many prior models; and (2) achieve high degree of naturalness in turn-taking, pacing, and the rendition of subtle non-lexical cues (such as breaths and lip smacks), which are crucial for listener immersion and capturing the authentic vibe of expressive conversations. Code and checkpoint are available
(1) synthesize
Voice
INTRODUCTION
Recent advancements in Text-to-Speech (TTS) synthesis have achieved remarkable success in gener- ating high-fidelity, natural-sounding speech for single speakers in relatively short utterances (Wang al. 2023a Anastassiou et al.) (2024] Le et al. 2023] Chen et 2024a b Du al. 2024 JJia al.) Ye et al. [025 Wang et al. (025). However, a frontier remains unexplored: the scalable synthesis of long-ford form, multi-speaker conversational audio, such as podcasts, multi-participant audiobooks, and extended dialogues. While commercial interest in this area is evident, with closed-source applications like Google's NotebookL.M offering podcast generation capabilities, the underlying technical approaches often remain opaque. 'he demand for automatically generating such complex audio content from text is rapidly increasing, yet it poses substantial, open research challenges that lie beyond the capabilities demonstrated by most publicly available TTS systems.
2025
Generating convincing multi-speaker conversations introduces unique hurdles. While traditional systems technically produce multi-speaker, long-form audio by concatenating individually synthesized utterances (as demonstrated in our experiments). achieving naturalness in speaker interaction remains significant challenge. Key difficulties include maintaining vocal consistency for each speaker across extended dialogues and capturing the authentic conversational flow, encompassing natural turn-taking, pacing, and subtle non-lexical cues (e-g., breaths, lip smacks) crucial for listener immersion. While previous work like MoonCast (Ju et al. (2255 has demonstrated the feasibility of synthesizing podcasts, the demand for more adaptable, efficient, and
architectures that can generate these interactions end-to-end persists.
To address these limitations, we present VIBEVOICE, as illustrated in Figure novel framework developed for the scalable synthesis of expressive, long-form, multi-speaker speech. Our approach
Core contrihutors, Contact person: fuwei microsoft.com

---

Published as a conference paper at ICLR 2026
min
Tokenizer Encoder Decoder
A
Diffusion Head
Discrete Text Taken
Continuous Acoustic / Semaantic Latent
D
<End>
VibeVoice
A
< <tart>
Thanks for heving
1
Hello, uh, I'm ...
S
User Input: Voice Text Scripts
Figure Scalable and expressive podcast synthesis with VOIEE. Voice prompts and text scripts provide initial input. VIBEVOIe processes hybrid context features, and its hidden states condition token level Diffusion Head (D), which predicts acoustic VAE for speech segments, subsequently recovered by acoustic decoder (A).
tackles these challenges through synergistic combination of efficient audio representation, authentic data preparation, and an integrated generative architecture.
A cornerstone of VIBEVOICE is its efficient hybrid speech representation strategy, derived from specialized acoustic and semantic tokenizers, both operating an ultra-low frame rate of 7.5 Hz. The Acoustic Tokenizer aggressively compresses audio while preserving remarkable reconstruction fidelity, and Semantic Tokenizer extracts linguistic content. This decoupled design, with both components leveraging the same highly efficient frame rate, allows for optimized acoustic and semantic feature extraction. These features are then combined to form the rich, yet compact, hybrid input essential for managing long-form content within our generative model. In addition, to VIBEVOICE
generate long conversational speech, we propose to use hybrid speech representation Moreover, developed data processing pipeline that curates and annotates podcast data. This provides rich, naturalistic training material, enabling VIBEEOICE to learn realistic intonation, turn-taking, and subtle expressive cues, thereby enhancing perceived audio realism.
VIBE VOICE leverages an end-to-end LLM-based architecture with a diffusion head, drawing inspiration from LatentLM (Sun et al. 2024). The LLM (Yang et al. 2024) handles robust textual understanding and dialogue flow, while the diffusion head (Li et al. 2024b) ensures high-fidelity acoustic generation. This framework achieves scalability through effcient design, employing specialized tokenizers to create a rich yet compact hybrid input for the LLM, and a streamlined diffusion process for well-structured acoustic tokens. This architecture enables VIBE VOICE to synthesize expressive, multi-speaker podcasts with coherence, intelligibility, and engaging conversational dynamics.
2 METHOD
2.
CONTINUOUS SPEECI TOKENIZERS
We employ two separate tokenizers as input to learn both acoustic and semantic features. It is worth noting that while we use the term 'tokenizer' to highlight its role in generating the sequence for the LLM, Acoustic Tokenizer operates continuous latent space (via o~VA rather than a discrete codebook. In our experiments, generating long-form speech benefits from this separate design.
2

---

Published as a conference paper at ICLR 2026
Acoustic Tokenizer adopts the principles of a Variational Autoencoder (VAE) (Kingma & Welling
2014), specifically drawing inspiration from the a- VAE variant proposed in Latent (Sun et al,
2024 to mitigate potential variance collapse issues of VAEs when used in autoregressive modeling settings. The process involves an encoder network, parameterized by ø. which maps the input audio the parameters of latent distribution, primarily the μ.. Notably, variance is pre-defined distribution (O,0, C,)) in V-VA, rather than learnable distribution in VAE (Kingma & Welling 2014), as demonstrated in Figure 2 A latent vector then sampled using the reparameterization
trick. Following the a- --V approach to ensure robust variance for autoregressive modeling, we can formulate this as:
/ = (
μ + a @ E, where E NO,, 0~ ~(0,Co)
à (ecoder,(z))
Semantic Tokenizer mirrors the hierar- chical architecture of the Acoustic Tok- enizer's encoder, but without VAE com- ponents, its objective is deterministic content-centric feature extraction. The main difference is the training objective, which uses Automatic Speech Recogni- tion (ASR) the proxy task. During training. its output is decoded by several Transformer decoder layers to predict text transcripts, aligning the semantic encoder's s representations with textual semantics. Refer to Figure 2 for the comparisons.
Proxy task: Speech Reconstruction
Enc
Dec
-Ifi-
N (0.
(0.1))
Praxy task: Automatic Speech Recognition
Enc
Decodar
Hello, I'm...
Figure 2: Acoustic Tokenizer (upper) reconstructs waveform by o-VAE, while Semantic Tokenizer (lower) uses ASR as its proxy task.
2.2 VIBEVOICE
VOICE employs a Large Language Model (LLM) as its core sequence model, integrated with specialized audio encoding and diffusion- baafusion-bbsed decoding modules to achieve scalable, high-fidelity multi-speaker speech synthesis. The overall inference architecture is depicted in Figure
2.2.1 1 INPUT REPRESENTATION
Voice Prompt Feature for each speaker k {1...., the provided voice prompt waveform un is encoded solely by the Acoustic Tokenizer encoder. This component, analogous to the o-VAE described in LatentLM (Sun et al. 2024), maps the raw waveform into a sequence of continuous Th
Here, Za,k = [=], .... a E I xda, where Tę is the sequence length for speaker k's prompt and d, the dimension of the acoustic features. Z,, capture the essential timbral and prosodic characteristics required for speaker.
VAE features Za,k = AcousticvAE
Text Script Embedding of the dialogue script, consisting of turns W for each speaker kt is tokenized into subword units. Note that script does not include the transcript of the acoustic prompt. These are then mapped to text embeddings E3 E R xdum using a standard embedding layer, where TV is the script length for speaker and dum is the LLM's hidden dimension.
Prompt Sequence X is formed by concatenating the acoustic-only voice prompt features and the text script embeddings, interleaved with speaker identifiers (Speaker,) X [Speaker,, ..., Speaker,- y. Za Speaker, F,, Sp....Speek n: E., <S>). where <S> a special token of start-of-speech. {2a, )==1.... are passed through projection layer match the LLM's hidden dimension dum-
2.2.2 SPEECH GENERATION WITH HYBRID SPEECH REPRESENTATION
The acoustic feature preserves fine fine-grained acoustic details, while the semantic feature captures higher-level linguistic information. To fully exploit the advantages of both, we propose to use hybrid representation in VOICE. As illustrated in Figure VIBEVOICE leverages both acoustic and
3

---

Published as a conference paper at ICLR 2026
semantic representations throughout the generation process. Specifically, each generation step, the generated speech segment is predicted as
+ 1
VIBEVOICE(X, 2,.0....... 2p,e),
V:+1 AccusticDoo(aa.t11
(2)
where yi+11 denotes the waveform of speech segment generated at the (i + 1)-th step, and zp, E Rdd represents the hybrid speech representation. The latter is obtained by combining acoustic and semantic encodings:
pp,i = Wa + W. Semanticraa
where Wa Rdmxxd. and W. E Rd Xd, are learnable projection matrices. As semantic features are more close to the text prompt, involving semantic features as the input can stabilize the generation process, especially for long-form speech.
The final generated speech is obtained by concatenating the generated segment at each decoding step: Y concat .).)))))) As each speech segment is generated purely based on historical context, VIBEVOICE naturally a streaming speech generation model.
2.2.3 DIFFUSION-BASE ACOUSTIC LATENT VAE GENERATION
To synthesize speech, VIBE employs a light diffusion head (Li al. 2024b) conditioned on the LLM's current hidden states. Specifically, the Diffusion Head in Figure hidden state h, as condition and predicts the corresponding acoustic VAE za,5.
takes an LLM
Training: The diffusion process (Ho et al. (020) learns to reverse a forward process that gradually adds Gaussian noise (0, 7) the clean acoustic Za,i over Tain discrete timesteps t:
( Vāiza,i + a
(4)
where oe = 111 is cumulative product of terms derived from a predefined noise schedule {Bg 1 Tai A noise prediction network сp, constituting the lightweight Diffusion Head (Li et al. 2024b), is trained to estimate the injected noise E given the noisy feature (t), the timestep t, and the conditioning LLM hidden state h;, The objective is minimize the L2 loss:
Lpar = E..,,....h,lll ||c co(za, (t), h2)ll
(5)
Crucially, the diffusion process trained to predict only the acoustic VAE Zo,i- Concurrently, when the LLM generates the conditioning hidden state h, for the diffusion head, it is also tasked with predicting whether the current speech segment should conclude, effectively determining if a diffusion termination token (e.g., <E>) should be emitted at this step.
Inference: During synthesis at step the Diffusion Head utilizes Classifier Guidance (CFG) (Ho & Salimans| 2022) enhance conditioning. The process starts with random noise and iteratively denoises it. At each denoising step I, two predictions of the noise are made by the network <g: (1) The conditional prediction, using the current predicted token's 's hidden state h;; and (2) The unconditional prediction, where represents a context derived solely from a special start-of-sqquence token <S>, effectively an unconditional or minimally conditioned generation prompt. The final noise estimate used for the denoising update is a linear combination of these two:
غ غ= (=)) hes>) + w (E)( h.) - c0(2a, hes=))
(6)
where is the guidance scale. while >> 1 amplifies the conditioning from. This guided noise estimate / then used to take a step towards the less noisy feature. This iterative process. typically accelerated using an efficient sampler like 3PM-Solver++ (L.1 al. 2025), eventually yields an estimate of clean acoustic features.
0 recovers purely unconditional generation (based on <S>),
2.3 DATA PREPARATION
The development of the proposed VOICE relies speech data with consistent range annotations, including transcriptions and speaker turn labels. However, previous methods have primarily focused generating short audio segments Yuu et al., 2024 He et al. 2024), which
4

---

Published as a conference paper at ICLR 2026
not directly applicable to multi-speaker, long-form audio annotation. To address this limitation, we propose a novel automatic annotation pipeline tailored for extended speech data. The pipeline consists three steps, including Segmentation and Transcription, Diarization, Quality Filtering. Detials and evaluations are available in Appendix A
Note that, compared previous data processing pipelines (Yu et al. 2024 He al. 2024 Ding et al. 2025), our pipeline does not include speech enhancement. This decision is based on our observation that while speech enhancement can effectively reduce noise, it often introduces distortion to the speech signal. In particular, emotionally expressive cues such as interjections or prosodic elements are prone degradation, which negatively impacts the naturalness of the audio.
EXPERIMENT
3.1 TOKENIZER SETUP
Acoustic Tokenizer is architected with a mirror-symmetrie encoder-decoder structure. The encoder employs hierarchical design with stages of modified Transformer blocks (Vaswani et al. 2017) (using ID depth-wise causal convolutions instead of self-attention) for efficient streaming processing, Six downsampling layers achieve cumulative 3200x downsampling rate from 24kHz input, yielding 7.5 tokens/frames per second. Each encoder/decoder component has approximately 340M parameters. The training objective follows the DAC methodology (Kumar et al., 2023), including its discriminator and loss designs. Key hyperparameters and further training specifics are detailed Appendix
Semantic Tokenizer's encoder mirrors the architecture of the Acoustic Tokenizer for structural consistency. It trained exclusively on an ASR proxy task using a cross-entropy loss function. This objective compels the encoder capture explicitly semantic and phonetic information. Upon completion of pre-training, the recoder-whide serves only to facilitate this training process- discarded. The keys hyperparameters are the same Acoustic Tokenizer.
3.2 VIBEVOICE SETUP
We instantiated VOIC''' core Large Language Model (LLM) using 1.5B and 7B parameter versions of Qwen2.5 (Yang al. 2024). The diffusion head (Li al., 2024b) comprises layers (approx. 123M parameters for 1,5 version). During VOICE training, the pre-trained acoustic and semantic tokenizers remained frozen, with only the LLM and diffusion head parameters being learnable. We employed curriculum learning strategy for the LLM input sequence length, progressively increasing from 4,096 65,536 tokens over 110k training steps (4,096 tokens for the first 40k steps, then 16,384 tokens for steps 40k-80k, then 32,768 tokens for steps 10k-100k and finally 65,536 tokens for steps 100k- The 7B version discarded the final phase due to resource limitations. VIBEVOICE was trained on approximately 80 billion tokens from an internal pseudo-labeled podcast audio collection (Section 2.3). Training a 1.5B model took approximately 170 hours on 64 AMD Instinct MI300X GPUs by using unscaler training engine (Lin et al. 2024). Comprehensive training hyperparameters are listed in Appendix F During inference, the CFG scale and denoising step are set to 1.3 and 10 respectively. Inference time comparison at Appendix E
3.3
EVALUATION SETUP
3.3.1 VIBEVOICE EVALUATION
Recognizing the limitations of existing public benchmarks for the podcast generation task (e-g., the MoonCast evaluation set (Ju et al., 2025), which contains only four samples), we developed VIBEVOICE-Ewa pipeline (described in Section 2.3 and Appendix (A) to better capture diverse podcast scenarios, and it consists of 108 podcast samples with durations ranging from 1 to 30 minutes. Details can be found in Appendix G
Objective Evaluation:
This dataset was curated using data processing
We employed Word Error Rate (WER) and widely used speaker similarity metric (SIM-O) (Anas-
tassiou et al.] 2024 Ju et al. 2025) as objective metrics to assess the quality of podcast generation. All generated podcasts were annotated using our custom data pipeline. The only difference that
5

---

Published as a conference paper at ICLR 2026
Subjective
Objective
Model
Realism
Richness
Preference
Average
WER-W
WER-N
SIM-O
Cosyvoice2 et et al. 2024
3.45
3.86
0.68
Mooncast (Ju
2.81
3.29
0.562
SesameAlLabge abbsab-C (SesameAllas Llbss 2025
2.89 m11
3,03
2.75 +ile
2.89
2,66
3.05
0.685
Higgs Audio V2 (oosnn Al 2125
295 211
3.19 1:14
2,83 1
2.99 2013
5.94
5.97
0.543
Elevenlabs v3 alphia Ellevenlabs]
3.34 .101
3.48
3,38 1112
3.40
2.39
2.47
0.623
Gemini 2.5 preview iis Google
3.55 =120
3.78 +1
3.65 +1:15
3.66
1.73
2.43
VIBEVOICE-IL5B VIBEVOICE-7B
3.59
=006
3.59
#::0
3,44
3.54
2006
1.11
1.82
0,548
3.71
-wve
3,81
+4.7
3.75
+014
3.76 2003
1.29
0.692
Table I: Human subjective and objective evaluation results. WER-W means using Whisper while WER-N means using Nemo. For all subjective metrics and SIM-O, higher scores are better. For WER, lower scores are better. Best results are in bold. The first phase subjective evaluation of Cosyvoice2, Mooncast and VIBEVOICE-1.5B can be found in Appendix
use the ground-truth number of speakers the number of cluster centers in the diarization process. Since the diarization results may not align perfectly with the input text prompts, we assign speaker labels based on which cluster center has the highest similarity to the speaker embedding of each speech prompt, and then compute SIM-O accordingly.
Subjective Evaluation: In the subjective evaluation, used eight podcast samples. These samples were based on two-speaker dialogue texts generated by a large language model, covering diverse topics such as technology, art, and travel. The lengths of the samples ranged from 3 to 15 minutes, with an average duration of approximately 7.3 minutes.
For subjective evaluation, we recruited 24 human annotators provide Mean Opinion Scores (MOS) across three dimensions: Realism (how natural and human-like the speech sounds, including prosody, emotion, and the smoothness of speaker turns), Richness (the expressiveness of the speech in terms of tone and emotion, including variation and adaptation to context), and Preference (overall listener enjoyment and subjective preference, reflecting naturalness, pleasantness, and engagement). The evaluation covered six models with all eight test samples, meaning that each annotator listened to approximately six hours of audio in total (roughly equivalent to reviewing 2, 2,160 short utterances of 10 seconds). Detailed definitions for each dimension and the precise 5-point scoring rubrics provided to evaluators are available in Appendix H
3.3.2 TOKENIZER EVALUATION
The fidelity of audio reconstructed from acoustic tokens a critical indicator of the tokenizer's efficacy in preserving essential acoustic information, particularly under high compression rates. To quantify this, we employed a suite of widely recognized objective metrics: PESQ (Rix et al. 2001 to measure overall speech quality, STOI (Taal al., 2010)) assess speech intelligibility, and UTMOS (Sacki et al., 2002)., which yields scores highly correlated with human evaluations. These evaluations were conducted on both the LibriTTS test-clean and test-other datasets (Zen al. 2019) and LibriSpeech test-clean dataset (Panayotov et al. (215)).
4 RESULTS
4.1 PODCAST GENERATION COMPARISON
We report subjective evaluation results in Table I and objective evaluation on the VIBEVOICE-Fval dataset in Table 2 comparing VIBEVOICE with baseline models and ablating different tokenizer configurations and model sizes.
human-led subjective assessments, VIBEVOICE-7B attains the highest average score (3.76). It consistently outperformed all other models across the dimensions of Realism (3.71). Richness (3.81), and listener Preference (3.75). Notably, surpassed strong proprictary models such Google's Gemini 2.5 Pro preview TTS (3.66 average) and Elevenlabs v3 alpha (3.40 average), indicating its superior ability to generate natural and engaging conversational audio that is preferred by human listeners. From objective standpoint, VOICE, also excels.
Comparison with Top-tier Models:

---

Published as a conference paper at ICLR 2026
Model
Seq-
Speaker
Speakers
Speakers
Speakers
Leng.
WER-W;
SIM-OI
WER-W.
SIM-O
WER-WI
SIM-OT
WER-W.
SIM-O
WER-WL
SIM-OT
(0--12 min) Set
Conyvoice2
Concat
3.14
0.79
3.5
0.73
5.33
0.69
5.83
0.70
4.27
0.73
MoonCaat
40K
7.2
0.51
7.9
0.63
17.24
11.52
0.482
10.42
0.551
Tokenizer Ablation
Acoustic (1.5B)
16K
13.74
6.22
Hybrid (1.58)
16K
1.93
0.66
0.79
0.39
2.50
0.64
1.68
0.64
1.84
0.64
VIBEVOICH-L.SD
32K
1.32
0.60
0.91
0.60
4.86
0.56
1.46
0.59
2.11
0.59
VIBEVOICE-LL58
64K
0.63
0.63
1.92
0.59
1488
0.58
1.34
0.58
1:22
0.60
VIREVOICE-7B
32K
0.75
0.68
0.66
0.75
VIBEVOICT-EAad Long (12-:30 min) Set
Conyvice22-
-Concat
0.75
4.34
0.71
0.70
4.95
0.74
MoonCast
0U677
VIREVOICR-L.9B
32K
0.87
0.57
2.08
0.61
1.06
0.57
1.33
0.54
1.33
0.57
VIBEVOICE-L.5D
64K
1,80
0.63
1.59
0.62
0.97
0.60
1,80
0.56
1.55
0.59
VIBEVOICE-7B
32K
1.08
0.79
1.55
0.77
0.84
0.73
1.51
0.71
1.24
0.75
Table 2: Results of VIBEVOICE and baseline models on the VIBEVOiCH-Eval dataset for multi- speaker podcast generation. Results are presented for short (0~-12 min) and long (12~30 min) duration subsets, across varying speaker counts. Seq. Leng. denotes the LLM training sequence length. "*" denotes using subset (12-13 min), and "4" denotes using the successful cases only (with 3 retries) due to MoonCast crashes on long and multi- Spenkerppeee 3) generation.
While the 1.5B model delivered the lowest Word Error Rate (WER-W: 1,11), demonstrating superior intelligibility, the VIBE VOICE-7B model achieved the highest speaker similarity score (SIM-O: 0.692) among all evaluated systems. This highlights its exceptional capability to preserve the unique vocal characteristics of target speaker while maintaining high content accuracy.
Scalability in Long-Form and Multi-Speaker Synthesis: A key contribution of VIBEVOICE its capacity to generate coherent, long-form audio with multiple speakers, significant challenge for existing TTS systems. As shown in Table 2. VIBEVOICE demonstrates remarkable stability and consistency in extended scenarios. For long-duration podcasts (12-30 minutes), maintains a very low WER-W of .24 and a high SIM-O of 0.75. This contrasts sharply with baseline unable to process the full long-form test and frequently
models; for instance, MoonCast crashed, as noted in the table. Furthermore, based on supplementary analysis, VIBEVOICE is architected support up four distinct speakers for durations of up to 30 minutes. This capability substantially exceeds that of prior work like MoonCast, which is limited to two speakers and up to 10 minutes of audio. This enhanced scalability is a direct result of our novel next-token diffusion framework and low frame rate continuous speech tokenizer, which operates at 7.5 Hz.
4.2 ABLATION STUDIES
Tokenizer Configurations:
The choice to employ a hybrid tokenizer that fuses separate acoustic and semantic representations is critical to model performance. The "Tokenizer Ablation" results in Table that an "Acoustic-only" model, while achieving competitive speaker similarity (SIM- O: 0.68), suffers from significantly degraded multi-speaker WER (6.22 overall). This indicates that while acoustic features alone can preserve speaker timbre, they lack the necessary semantic guidance to maintain content coherence interactive dialogues. The final "Hybrid" approach consistently strikes a superior balance, dramatically improving intelligibility (WER: 1.84) while maintaining strong speaker identity (SIM-O: 0.64).
Model Scale: Scaling the model from 1.5B to 7B parameters yields substantial and comprehensive performance gains. This is demonstrated by clear listener preference for the larger model, with the average subjective score (Table increasing from 3.54 to 3.76. This perceptual enhancement mirrored by dramatic improvements objective metrics (Table 2}. where the overall WER- W dropped from 2. to 0.66 and the SIM-O score rose from 0.59 to 0.75. These results confirm that a larger model significantly boosts both perceived audio quality and technical accuracy in speaker preservation.
CFG and DDPM Steps: Figure B illustrates the ablation of Classifier- Free Guidance (CFG) scale and DDPM (Ho et al. 2020) denoising steps inference performance. For WER (Figure Ba optimal results (e.g., WER of 1.55) are achieved with 10 denoising steps and a CFG scale of 1.25. Using fewer steps (e.g., 5) significantly degrades WER, while excessive steps show diminishing

---

Published as a conference paper at ICLR 2026
WER J
SIM- Similarity
0.60
3.5
3.11
2.60
2.31
2.38
1.35
0.601
0.558
0.517
0.495
0..66
CFC
CTC
0.601
0.571
0.535
0.511
2.5
0.54
2.33
1.55
3.03
2.79
2.0
125
0.603
0.577
0.553
0.525
-0.52
-..50
15
Step
Stop
(a) WER Heatmap
(b) SIM-O Heatmap
Figure 3: Ablation of CFG and DDPM steps on WER and SIM-O. Heatmaps show the effect of
DDPM steps (x-axis) and classifier-fre guidance (CFG) scale (y-axis) on SIM-O and WER scores.
Token Rate
Test-clean STOI
Test-other STOI
Model
N₂
PESQ
UTMOS
PESQ
UTMOS
Ground-Truth
4.056
3.483
Encodee (Déoosezz et al.
2022
600
2.72
0.939
3.04
2.682
0.924
2.657
DAC (Kumar
2.738
3.433
2.595
0.908
2.945
Encodae (Delowsz el a
2022
300
2,052
0,901
2.307
2.052
0.884
2,088
Specch
Zilan
2023
1.931
0.878
3.563
1.737
0.837
3.018
DAC (Kumar
1.246
0.771
1.494
1.245
0.751
1.499
Wav Toxemzer Ur 2025
75
2.373
0,914
4.049
2.261
0.891
3.431
NayTokenizer er 2025
1.703
0.862
3.602
1.662
0.834
3.055
(Acoustic)
3.068
0.828
4.181
2.848
0.823
3.724
Table 3: Objective evaluation of reconstruction quality on the LibriTTS test-clean and test-other datasets. Ng denotes the number of quantizers; VIBEVoIee Rate indicates the number of tokens/frames generated per second of audio. Higher PESQ, STOI, and UTMOS scores indicate better performance. Best results are in bold.
a single continuous oAVE Token
returns. For SIM-O Similarity (Figure 3b). the highest scores (around 0.6) are obtained with few as 5 DDPM steps across all tested CFG scales, with slight decrease observed as steps increases. Additional visualizations are provided in Appendix
4.3 RECONSTRUCTION COMPARISON
Table 3 details the objective acoustic reconstruction performance of our tokenizer against existing methods (Défossez al.] 2022 Kumar al.] 2023 /Zhang al., 2023 Ji et al. 2025)
the LibriTTS dataset (Zen et al, 2019). Results on LibriSpeech (Panayotov et al.] 2013) can
be found in Appendix Our acoustic tokenizer, uniquely operating an ultra-low 7.5 Hz, achieves leading PESQ and UTMOS scores on both test-clean (PESQ: 3.068, UTMOS: 4.181) and test-other (PESQ: 2.848, UTMOS: 3.724) subsets. This demonstrates its capacity for high-fidelity, perceptually excellent audio reconstruction despite aggressive compression, which is a key factor for VIBEVOICE 's scalability with long-for form audio. Regarding acoustic tokenizer, while its PESQ and UTMOS scores are strong, its STOI scores (test-clean: 0.828, test-other: 0.823) are comparatively lower. This likely due to our primary training data (podcasts) not undergoing extensive noise reduction, which can impact STOI's intelligibility focus more than PESQ/UTMOS's assessment of overall perceived quality.
5 RELATED WORK
Zero-shot Text-to-Speech (TTS) aims to synthesize speech in target speaker's voice using only brief voice prompt from that speaker, without requiring speaker-specifit model minettuning.. Existing approaches for single-speaker zero-shot TTS can be broadly categorized into multi-stage (Wang)
8

---

Published as a conference paper at ICLR 2026
et al. 2023a Anastassiou al.
2024 and one-stage methods (Le et al. 2023 Chen al. 2024a b
Jia et al. 2025 Ye et al. 2025 Wang et al., 2025] [hhnn et al. 2023 Ju et al. 2024 Borsos et al,
2023 Zhu et al. 2024). Multi- stage models typically first predict coarse- grassndd representations,
such as semantic tokens (Wang et al., 2023a Anastassiou et al., 2024), and subsequently employ a non-autoregressive model (Wang al. 2023a) generative diffusion model (Anastassiou al.)
2024) for coarse-to-fine acoustic feature generation. In contrast, one-stage methods simplify this
pipeline by directly predicting discrete codes (Wang al. 2025 Ye al. 2025), Mel spectrograms
(Meng et al. 2024). or continuous VAE latent features (Jia et al. (025).
While these methods have achieved considerable success for individual utterances, extending zero- shot capabilities to complex, interactive scenarios like multi- dialogues and podcasts presents significant further challenges. Accurately modeling natural nullt--ppeaker dynamies-inchun seamless turn-taking, consistent inter-speakes prosody, and subtle paralinguistic cues is paramount
(Schuller al.] 2013) Zhang et al.) 20199] Nguyen et al., 2023) Mitsui et al.) [023 Zhang et al.]
2024). Furthermore, capturing the spontaneity inherent in natural human conversations, such as filler words, hesitations, and informal speech patterns, remains important aspect for high-fidelity
synthesis (Li al., c)
Podcast Generation inherently grapples with a range of complex challenges, including preserving distinct speaker identities and emotional states over long durations, ensuring natural transitions and interactions among multiple participants, and maintaining a sense of spontaneity in dialogue. A pio- neering effort in this direction is NotebookL.M] though it lacks accompanying academic paper or technical report and supports only two fixed speaker voice. Another related work, SpeechSSM (Park) al. 2024), can generate audio up to 16 minutes in length but limited single-speaker scenarios. While recent advances such as MoonCast (Ju et al. 2025), FireredTTS2 (Xie et al. 2025). Higgs Audio V2 (Boson All 2025), TTSD (OpenMOSS Team (025) and DSM-TTS (eeghdoorr it al. 2025) have made notable progress in addressing these multifaceted challenges, there remains
a pressing need for more scalable and flexible solutions to podcast generation, such as generation length, number of speakers and generation stability.
Speech Tokenizer has become a cornerstone in modern speech processing, enabling efficient representation of audio for diverse applications. Early works with discrete representations, pioneered by VQ- (van den Oord et al.] 2017) and significantly advanced by subsequent neural audio codecs like SoundStream (Zeghidour et al. 2021) and Encodec (Défossez et al.] 2022)
demonstrated the feasibility of high-quality speech reconstruction with low-bit rate discrete tokens. Then, significant research efforts have been dedicated improving reconstruction fidelity through advancements in discriminator architectures (Wu et af.] 2023 Kumar et al.) 2023] Ahn et al. 2024).
alongside efforts to further improve compression ratios (Ji et al. 2025 Shechtman & Dekel
2024)
and frame rate (Zeng et al.) (2024). Increasingly, there is a trend towards enhancing the semantic properties of the tokenizer's latent space better support generative tasks. Some approaches aim
to enrich semantic content within a unified encoder structure (Zhang et al., 2023 Défossez et al.,
2024), while others explore separate encoders (Wang et al.) 2025] Ye et al. 2025).
Our tokenizer builds upon these directions with two key design choices: firstly, it operates at an ultra-low frame rate of 7.5 Hz. to achieve substantial compression and efficiency; secondly, explicitly decouples the acoustic and semantic tokenization pathways, employing distinct tokenizers that specialize in their respective domains to ensure both high-fidelity audio reconstruction and rich semantic grounding for the synthesis process.
6 CONCLUSION
We introduced VIBEVOICE, a novel framework for zero-shot, expressive, long-form, multi-speaker podcast generation. By integrating efficient hybrid speech representations from specialized ultra-low frame rate (7.5 Hz) acoustic and semantic tokenizers with an end-to-end LLM--bsed next-token diffusion architecture, and leveraging a curated podcast dataset, VIBEE achieves state-of-the--rt performance. It scalably synthesizes high-quality audio for up to 90 minutes with up to speakers, demonstrably surpassing existing baselines in both subjective perceptual
https://nctebook ge/

---

Published as a conference paper at ICLR 2026
soality-iincuddin naturalness, coherence, and realism and objective metrics like WER, thereby significantly advancing the capabilities of conversational TTS.
REFERENCES
Sunghwan Ahn, Beom Jun Woo, Min Hyun Han, Chanyeong Moon, and Nam Soo Kim. Hilcodec: High-fidelity and lightweight neural audio codec. IEEE Journal of Selected Topics in Signal Processing, 2024.
Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02453 2024.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.09636, 2023.
Boson AL Higgs Audio V2: Redefining Expressiveness in Audio Generation. gi github... com hatmm/booonaaaae heaihhiige aoniggs-auddi 2025.
https://
Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, pp. 160-172. Springer, 2013.
Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 17:24081-241255 2024a.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavim: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):
1505-1518, 2022.
Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.0688. 06885, 2024b.
Alexandre Défossez, Jade Copet. Gabriel Synnaeve, and Yossi Adi High fidelity neural audio compression. arXiv preprint arXiv:2210.13438 2022.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dialogue. Technical report, Kyutai, September 2024.
Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report. arXiv preprint arXiv:2504.18425 2025.
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang. Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint 1rXiv:2412.10117, 2024.
Elevenlabs. Elevenlabs
v3
alpha.
itt.://cca..ioooco//oooccoccsc 1sf
eleven-vee v3-avlvll-p- a3
Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Yan. Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition. In Interspeech, pp. 2063-2067. ISCA, 2022.
Google. Gemini 2.5 Pro Preview TTS. https://ll..oogll.ddd//eenninni
Haohan Guo, Kun Liu, Feiyu Shen, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kaituo Xu. Fireredtts: A foundation text-th- framework for industry-Level generative speech applications. CoRR, abs/2409.03283, 2024.
10

---

Published as a conference paper at ICLR 2026
Haorui He, Zenggiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In 2024 IEEE Spoken Language Technology Workshop (SLT), pp. 885-890. IFFE, 2024.
Jonathan Ho and Tim Salimans. arXiv:2207. 125988 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, Ziang Zhang. Xiaoda Yang, Rongje Huang, Yidi Jiang, Qian Chen, Siqi Zheng, and Zhou Zhao. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. In The Thirteenth Internationaa Conference on Learning Representations, 2025. t wtwpr//wwwww...tteeeoooo/oooaoae
net htt///orummmmoo
Dongya Jia, Zhuo Chen, Jiawei Chen, Chenpeng Du, Jian Wu, Jian Cong, Xiaobin Zhuang, Chumin Li, Zhen Wei, Yuping Wang, et al. Ditar: Diffusion transformer autoregressive modeling for speech generation. arXiv preprint (XXi::5522.039300 2025.
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang. et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint wrXiv:2403.03000 2024.
Zeqian Ju, Dongchao Yang, Jianwei Yu, Kai Shen, Yichong Leng, Zhengtao Wang, Xu Tan, Xinyu Zhou, Tao Qin, and Xiangyang Li. Mooncast: High-quality zero-shot podcast generation. arXiv preprint arXiv:2503. 14345, 2025.
Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, 2014.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved rvggan. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 27980-27993, 2023.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadcokar, and Wei-Ning Hsu. Voicebox: Text-guided multilingual universal speech generation at scale. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
Hanzhao Li, Xinfa Zhu, Liumeng Xue, Yang Song, Yunlin Chen, and Lei Xie. Spontts: modeling and transferring spontaneous style for tts. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12171-12175. IEEE, 2024a.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprini arXiv:2406.1 1838, 2024b.
Weiqin Li, Peiji Yang, Yicheng Zhong, Yixuan Zhou, Zhisheng Wang, Zhiyong Wu, Xixin Wu, and Helen Meng. Spontaneous style text-to-speech synthesis with controllable spontaneous behaviors based on language models. arXiv preprint arXiv:2407. 13509, 2024c.
Zhiqi Lin, Youshan Miao, Quanlu Zhang, Fan Yang. Yi Zhu, Cheng Li, Saeed Maleki, Xu Cao, Ning Shang, Yilei Yang. Weijiang Xu, Mao Yang, Lintao Zhang, and Lidong Zhou. nnscaler: Constraint-guided parallelization plan generation for deep learning training. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 347-363, 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++; Fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Research, pp. 1-22, 2025.
11

---

Published as a conference paper at ICLR 2026
Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, and Furu Wei. Autoregressive speech synthesis without vector quantization. CoRR, abs/2407.08551, 2024.
Kentaro Mitsui, Yukiya Hono, and Kei Sawada. Towards human-like spoken dialogue generation between ai agents from written dialogue. arXiv preprint urXiv:2310.01088, 2023.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomascllo, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 250-266, 2023.
OpenMOSS Team. MOSS-TTSD.
com http:///.......oomm///////////// TTSD 2025.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brishane, Queensland, Australia, April 19-
24, 2015, pp. 5206-5210. IEEE, 2015.
Se Jin Park, Julian Salazar, Aren Jansen, Keisuke Kinoshita, Yong Man Ro, and RI Skerry-Ryan. Long-form speech generation with spoken language models. arXiv preprint arXiv:2412. 18603, 2024.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pp. 28492-28511 PMLR, 2023.
Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE international conference on acoustics, speech, and signal processing. Proceedings (Cat. No. 01CH37221), volume 2. pp. 749-752. IEEE, 2001.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint
WrXiv:2204.02152 2022.
Björn Schuller, Stefan Steidl, Anton Batliner, Felix Burkhardt, Laurence Devillers, Christian MüLler, and Shrikanth Narayanan. Paralinguistics in speech and language-sttte-off-teee and the challenge. Computer Speech & Language, 27(1)44-3, 2013.
SesameAlLabs. SesameAlLabs CSM Model. httt:a///gstttub..cm///eesssseee 2025.
Slava Shechtman and Avihu Dekel. Low bitrate high-quality rvggan- discrete speech tokenizer. In Annual Conference of the International Speech Communication Association, 2024.
Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu,, Yichong Leng, Lei Hc, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09/ 16, 2023.
Yutao Sun, Hangbo Bao, Wenhui Wang, Zhiliang Peng, Li Dong, Shaohan Huang, Jianyong Wang. and Furu Wei. Multimodal latent language modeling with next-token diffusion. arXiv preprini arXiv:2412.08635. 2024.
Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. A short-time objective intelligibility measure for time-frequency weighted noisy speech. In 2010 IEEE international conference on acoustics, speech and signal processing, pp. 4214 4217. IEEE, 2010.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Neural Information Processing Systems, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 6000-6010, 2017.
12

---

Published as a conference paper at ICLR 2026
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang. Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models zero-shot text to speech synthesizers. CoRR, abs/2301.02111, 2023a.
Hongji Wang, Chengdong Liang, Shuai Wang, Zhengyang Chen, Binbin Zhang, Xu Xiang, Yanlei Deng, and Yanmin Qian. Wespeaker: research and production oriented speaker embedding learning toolkit. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023b.
Xinsheng Wang, Mingqi Jiang. Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, al. Spark-tts: An efficient Ilm- text-to-speech model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.01710, 2025.
Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Shunsi Zhang, and Zhizheng Wu. Maskget: Zero-shot text-to-speech with masked generative codec transformer. CoRR, abs/2409.00750, 2024.
Yi-Chiao Wu, Israel D Gebru, Dejan Marković, and Alexander Richard. Audiodec: An open-source streaming high-fidelity neural audio codec. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing ( (ICASSP), pp. 1-5. IEEE, 2023.
Kun Xie, Feiyu Shen, Junjie Li, Fenglong Xie, Xu Tang, and Yao Hu. Fireredtts-2: Towards long conversational speech generation for podcast and chatbot. arXiv preprint Sitiv:2559.020222. 2025.
Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. Bigcodec: Pushing the limits of low-bitrate neural speech codec. arXiv preprint arXiv:2409.05377 2024.
Hainan Xu, Fei Jia, Somshubra Majumdar, He Huang, Shinji Watanabe, and Boris Ginsburg. Efficient sequence transduction by jointly predicting tokens and durations. International Conference on Machine Learning, pp. 38462-38484, PMLR, 2023.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint ar1i:2441.151115, 2024.
Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi DAI, et al. Llasa: Scaling train-time and inference-time compute for ilama-based speech synthesis. arXiv preprint arXiv:2502. 04128, 2025.
Jianwei Yu, Hangting Chen, Yanyao Bian, Xiang Li, Yi Luo, Jinchuan Tian, Mengyang Liu, Jiayi Jiang, and Shuai Wang. Autoprep: An automatic preprocessing framework for in-the-wild speech data. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1136-1140. IEEE, 2024.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
Neil Zeghidour, Eugene Kharitonov, Manu Orsini, Václav Volhejn, Gabriel de Marmiesse, Edouard Grave, Patrick Pérez, Laurent Mazaré, and Alexandre Défosse7. Streaming siguence-to-sequenc learning with delayed streams modeling. arXiv preprint arXiv:2509.08753, 2025.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint 1904v:1904.0280 2019.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Gloce-voice: Towards intelligent and human like end-to-end spoken chatbot. arXiv preprint arXiv:2412.026/22 2024.
13

---

Published as a conference paper at ICLR 2026
Leying Zhang, Yao Qian, Long Zhou, Shujie Liu, Dongmei Wang, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Lei He, et al. Covomix: Advancing zero-shot speech generation for human-like like multi-talker conversations. Advances in Neural Information Processing Systems, 37: 100291-100317, 2024.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint Stti::2308..669999 2023.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv: 1911:00110000500 2019.
Xinfa Zhu, Wenjie Tian, and Lei Xie. Autoregressive speech synthesis with next-distribution prediction. arXiv preprint arXiv:2412.16846 2024.
A DATA PREPARATION PIPELINE
The data preparation pipeline contains the following steps:
Segmentation And Transcription First, we use the silero voice activity detection (VAD) tool to segment long audio recordings into shorter clips, each with a maximum duration 30 seconds. Next, we apply Whisper-large-v--uur punctuation, along with word-level timestamps. Since punctuation marks (specifically (!?!]) are closely related to speaker turns, we further perform semantic- based re-segmentation by splitting the original audio at their end timestamps. This approach results in more accurate segmentation of speaker boundaries.
(Radford et al.) 2023) to generate transcriptions with
Diarization We use the vblinkp model from the WeSpeaker toolkit (Wang et al. 2023b) to perform speech diarization. Specifically, the audio segments obtained from the previous step first divided into frames using window length of 1.5 seconds and a hop size of 0.75 seconds. Speaker embeddings are then extracted for each frame, and HDBSCAN (Campello et al. 2013) clustering is applied to assign speaker labels. To enhance clustering consistency, we further merge clusters whose centroids have a cosine similarity greater than 0.67. Finally, we re-segment the audio based on the frame-level speaker labels, yielding the final speaker turn annotations.
Quality Filtering To further ensure the accuracy of the annotations, we apply the following post- filtering strategies. First, we employ secondary ASR model (Xu et al.) 2023) to re-transcribe the final segments. If the word error rate (WER) between the primary and secondary ASR outputs exceeds 20%, the corresponding segment is considered unreliable. If more than 30% of the segments long audio file are marked as unreliable, the entire audio discarded. Second, we filter out audio files in which the total duration of speech is less than 60% of the overall audio length. Third, discard audio files that contain more than 4 distinct speakers.
Table summarizes the open-source toolkits utilized in our data preparation (Section 2.3 and evaluation (Section (.3.T). p.. We gratefully acknowledge all contributors to these open-source projects for their valuable and generous work.
B MORE TOKENIZER ABLATIONS
To validate the necessity of our hybrid representation, we analyze the Coupled Tokenizer, an alternative design explored during development. This architecture allows us to investigate the effects of encoding semantic and acoustic features within single, shared latent space, providing direct contrast to our final disentangled hybrid approach.
As shown in Figure the Coupled Tokenizer employs single encoder map input waveforms into a shared latent distribution μ.. This unified representation serves dual objectives: (1) speech reconstruction via an Acoustic Decoder, and (2) transcript prediction via Semantic Decoder. This joint optimization aims to capture both acoustic fidelity and semantic content within a unified bottleneck.
14

---

Published as a conference paper at ICLR 2026
Model
URL Link
Data Processing Pipeline
Silero VAD
hgtes////ghhhuuooo///sssssaaaaaa Whisper-largo-v--urr
Nemo ASR
Wespeaker embedding
http:///we.....ggoooo/o//oa00002 ver
Evaluation
WER toolkit
aiddoo/eraalatte.ssr..ft..
SIM-O toolkit
htt.: /blob/
main/cal sim.sh
Table 4: List of models and tools used our data processing pipeline.
Proxy task 1: Speech Reconstruction
Acoustic
-
Decoder
I
Enc
Semantic
Hello, I'm...
(O, Cg) (O.11)
Decoder
Proxy task 2: Automatic Speech Recognition
Figure 4: An illustration of the Coupled Tokenizer architecture. A single encoder produces shared latent representation which is utilized for both speech reconstruction (Acoustic Decoder) and ASR (Semantic Decoder). This design contrasts with our final Hybrid architecture (Figure which employs separate encoders to decouple semantic and acoustic representations.
We evaluated this design on both reconstruction quality and downstream podcast generation performance. Our findings are follows:
1. Reconstruction Quality: As shown in Table 7 the reconstruction performance of the Coupled Tokenizer (STOI: 0.72, PESQ: 1.92) is substantially lower than that of the acoustic encoder from our Hybrid model (STOI: 0.75, PESQ: 2.98). This suggests that, at low framerates such as 7.5 Hz, forcing single latent space to encode both abstract semantics and acoustic information fundamentally undermines its ability to retain the fine-grained acoustic details needed for high- fidelity reconstruction.
2. Podcast Generation Performance: The results of integrating the Coupled Tokenizer into our generation framework are detailed in Table 5 While the Coupled model improves content stability over the Acoustic-only baseline (reducing overall WER from 6.22 to 3.55), this improvement comes at a severe cost to speaker identity. The model yields an overall speaker similarity (SIM-O) of just 0.45-the lowest of all settings. This is a direct consequence of its degraded acoustic modeling capability; as evidenced by its poor reconstruction scores (Table 7). the shared latent space fails to preserve the unique acoustic details of speaker's voice.
These results clarify the architectural trade-offs summarized Table The Acoustic-only baseline relies entangled representations; while efficient for single speakers, it fails to disentangle speaker identity from content in multi-speaker scenarios, leading to semantic collapse (WER spikes from 1.06 13.74). The Coupled model introduces semantic constraints to mitigate this but over-compresses the representation, sacrificing speaker fidelity for content stability. Consequently, our final Hybrid architecture proves necessary: by explicitly disentangling semantic structure from acoustic realization, it achieves the optimal balance, maintaining both high intelligibility and robust speaker identity across complex multi-speaker sessions.
15

---

Published as a conference paper at ICLR 2026
Tokemizer
Soq- Leng.
Speaker
Speakers
Speakers
Speakers
WKR-W]
SIM-0"
WER-W]
SIM-O7
WRR-W]
SIM-OT
WRR-W]
SIM-O1
WER-W|
SIM-O7
Acoustic
16K
1.06
0.7
6.15
0.7
13.74
0.70
6.46
0.64
6.22
0.68
Coupled
16K
6.79
0.43
1.18
0.47
1.4
0.48
2.6
0.45
3.55
0.45
Hybrid (Final)
16K
1,93
0.66
0.79
0.59
2.50
0.64
1.68
0.64
1.84
0.61
Table Ablation study on tokenizer architectures for podcast generation (no more than 12 minutes). The final Hybrid approach is evaluated against Acoustic-only and Coupled baselines. The results show that our Hybrid approach achieves the best overall balance of WER and higher SIM-O, especially in multi-speaker scenarios.
C MORE BENCHMARK RESULTS
We evaluate VIBE VOICE on the SEED test sets (Anastassiou et al. 2024), widely used benchmark
composed of short utterances. For evaluation, approximately 1,000 English samples and 2,000 Chi- nese samples are drawn from the Common dataset, denoted test-en and test-zh, respectively. We compute word error rate (WER) using Whisper-large-v3 for test-en and Paraformer (Gao et al.,
2022) for test-zh. For speaker similarity (SIM), adopt WavLM-large (Chen et al. 2022) model.
Table presents the results on the SEED test sets. Although our model primarily trained on long-form speech, demonstrates strong generalization on short-utterance benchmarks. In addition, by employing lower frame rate, our model substantially reduces the number of decoding steps required to synthesize second of speech.
test-zh
test-en
Model
Frame Rate
CER( %
SIM
WER( %)
SIM f
MaskGCT Wang
2024
50
2.27
0.774
2.62
0.714
Sced-TTS
Anastassiou
2024
1.12
0.796
2.25
0.762
FircRedTTS (Guo ct al.
2024
25
1.51
0.635
3.82
0.460
Voice 2 Du cf al.
7024)
25
1.45
0.748
2.57
0.652
Spark TTS Wang
2025
50
1.20
0.672
1.98
0.584
VIBEVOICE-
7.5
1.16
0.744
3,04
0.689
Table Results on the SEED test sets.
D MORE RECONSTRUCTION RESULTS
The acoustic reconstruction performance the LibriSpeech test-clean set (Table largely mirrors and reinforces the conclusions drawn from our LibriTTS evaluations (Table 3).
Our dedicated VIBEVOICE Acoustic Tokenizer, operating at exceptionally low 7.5 fps with single quantizer (N, 1), achieves a UTMOS score of 4.19. This highly competitive, even surpassing models like BigCodec (Xin al. 2024) (UTMOS 4.11 at 80 fps) and achieving similar perceptual quality to X-codec2 (Ye et al. 2025 (UTMOS 4. 13 at 50 fps) and BiCodec (Wang et al.) 2025) (UTMOS 4.18 at 50 fps), despite those operating nearly 7 times underscores its remarkable efficiency in delivering high perceptual quality with significantly fewer tokens per second, trend consistent with its leading PESQ and UTMOS performance on LibriTTS.
frame rate. This
Conversely, the Coupled tokenizer (demonstrated in Figure 4 demonstrates a notable degradation in reconstruction quality (UTMOS 3.3 LibriSpeech performance, while still reasonable for such an extreme frame rate, is significantly lower than the dedicated acoustic model. These findings further solidify our rationale for employing separate acoustic and semantic tokenizers in the final VIBE architecture to preserve acoustic fidelity.
4.19 for the acoustic variant). This
E VIBEVOICE INFERENCE TIME
To understand the computational cost associated with generating speech using VIBEVOICE, we profiled the inference time of its core components single NVIDIA A6000 GPU. Table
16

---

Published as a conference paper at ICLR 2026
Model
Frame Rate
Ng
STOI
PESQ
UTMOS
Ground Truth
4.09
DAC (Kumar al.
2023
600
12
0.95
4.01
4.00
Encodec (Défossez ct al.
2022
600
0.94
2.77
3,09
Encodec DDéosseez et 2022
150
0.85
1.56
1.58
DAC Kumar et al. 2023
100
2
0.73
1.13
1.29
Tokeneer Zhang et al.
Tokenizer
2023
100
0.77
1.25
2.28
Mimi (Défossez et al. 20244
100
8
0.91
2.25
3.56
BigCodee (Xin et al. 2024)
80
0.93
2.68
4.11
Wav Tokenizer (Ji et al. 2025
75
0,90
2.13
3.79
Mimi (éfoossež ct al. (0244)
75
0.89
1.99
3.38
SpeechTokenizer Tokenizer (Zhang al.
2023
50
0.64
1.14
1.27
Mimi (Déoossez et al. 2024)
50
0.85
1.64
3.03
Wav Takeeizneerr (Ji et 2025
40
0.85
1.62
3.57
X-codec2 (Ye et al. 2025)
50
0,92
2.43
4.13
BiCodec (Wang et al.
2025
50
0.92
2.51
4.18
VIBEVOICE (Coupled)
N/A
0.72
1.92
3.3
VIBEVOICE (Acoustic)
7.5
N/A
0.75
2.98
4.19
Table 7: Reconstruction results on LibriSpeech test-clean set. Ng denotes the number of quantizers; VOICE uses a single continuous a VAE.. Higher PESQ. STOI, and UTMOS scores indicate better performance.
presents breakdown the inference cost in milliseconds (ms) per generated segment for different VOIEE model sizes (1.5B and 7B LLM parameters) and varying numbers of DDPM diffusion steps (1 and 10) for the diffusion head. The measurements include the time taken by the Large Language Model (LLM), the Diffusion Head, the Acoustic Decoder, and the Semantic Encoder, along with Real-Time Factor (RTF).
Inference Cost (ms)
Model
Model Size
Diffusion
RTF
LLM
Diffusion Head
Acoustic Decoder
Semantic Encoder
MoonCast
1.5B
N/A N/A
N/A N/A
Higgs Audio
3B
0.72
47.87
2.69
14.96
14.71
0.62
VIBEVOICE
10
53.80
22.76
14.89
14.59
0.83
58.71
3.25
14.75
14.52
0.70
7B
71.73
28.87
14.80
14.52
0.97
Table 8: Inference cost (ms) and Real-Time Factor (RTF) comparison between VIBEVOICE and baseline models. Measurements were conducted on a single NVIDIA A6000 GPU with batch size of 1.
As observed from Table g the LLM component constitutes the significant portion of the inference time, which is expected given its size and autoregressive nature. For the 1.5B model, the LLM takes approximately 48-54 ms, while for the larger 7B model, this increases to around 59-72 ms. The inference time of the Diffusion Head scales linearly with the number of DDPM steps; increasing the steps from to 10 results roughly 8-9x increase
processing time. The costs for the Acoustic Decoder and Semantic Encoder remain consistently low and efficient at around 14-15 ms.
Crucially, the Real-Time Factor (RTF) for VIBEVOICE remains consistently below 1.0 across all tested configurations, ranging from 0.62 to 0.97, signifying faster-than-real than-real-time capability. RTF of 1.43, indicating tlweerthan--ral-tiie processing. While Higgs Audio V2 achieves a competitive RTF of 0.72, it relies on a parallel generation strategy (processing 3-second audio chunks concurrently). VIREVOICE achieves
In contrast, the baseline MoonCast exhibits
17

---

Published as a conference paper at ICLR 2026
comparable superior efficiency within a strietly streaming framework, making it well-suited for real-time applications and the generation of long-form audio without accumulating latency.
F TRAINING PARAME-PR
Table 9 summarizes hyperparameters used for training the Tokenizer (both Acoustic and Semantic variants) and the main VIBEVOICE model.
Stage
Hyper-parameters
Values
Stages
7
Blocks per stage
[3, 3, 3
Downsampling ratios
[2. 2, 4, 4, 5,
Adam 3
(0.8, 0.99)
Adam
1e-6
LR
3 x 10-4
Weight decay
0.01
Tokenizer
Gradient norm clip
1000
Training steps
300,000
Acoustic Tokenizer
VAE dim
64
VAE C,
0.5
Batch size
160s
Sample size
4s
Semantic Tokenizer
Batch size
1024 tokens
Diffusion head layers
Diffusion head FFN ratio
3
Diffusion beta schedule Diffusion train steps Diffusion loss weight
Cosine
1000
5
Adam B
(0.9, 0.95)
VIBEVOICE
Adam e Batch size
1e-8
4M
Learning rate
1 x 10
Learning schedule
Cosine
Warmup steps
500
Weight decay
0.1
Gradient norm clip
2
Training steps
110,000
Table 9: Summary of training hyperparameters for Tokenizer and VIBEVOICE model stages.
G VIBEVOICE-
EVALUATION
To rigorously evaluate the capabilities of VIBEVOICE in generating long-form, multi-speaker podcasts, particularly its scalability with varying speaker counts and durations, we curated dedicated evaluation set named VIBEEOICE-Eval This dataset
constructed using internal data processing pipeline (as described in Section 2.3 of the main text) from podcast sources not included in the VOICE training set, ensuring fair assessment of generalization.
Table 10 details the distribution of the VIBEVOICE Eval dataset. comprises a total of 108 distinct podcast segments, amounting to approximately 28.9 hours of audio. The dataset is stratified by the number unique speakers present in each segment, ranging from to speakers. As shown, the average duration of segments tends to increase with the number of speakers, with 4-spekker segments
18

---

Published as a conference paper at ICLR 2026
Speaker Numbers
Samples Count
Average Duration (s)
Total Duration (h)
30
878
7.32
2
35
905
8.8
3
27
984
7.38
4
16
1210
5.38
Total
108
962
28.9
Distribution of the VIBEVOICE-Eval dataset, stratified by the number of speakers per sample. Durations are provided in seconds (average) and hours (total).
Table 10:
averaging over 20 minutes (1210.81 seconds). This diverse composition allows for comprehensive evaluation of model performance across different levels of conversational complexity and length.
To provide a qualitative illustration of the content within the V'BCEVOICEEE dataset, below is an excerpt from sample segment featuring 3 speakers. This example showcases typical conversational turn-taking and dialogue flow present in the evaluation materials:
Dialogue Example
Speaker O: Oh, look. There's Hank and Trash Truck.
Speaker 1: Hi, Walter.
Speaker Hey, Hank. Hey, Trash Truck.
Speaker 1: Hi, Donnie.
Speaker O: Look at me, Hank. I'm swinging. And you guys thought this rope wasn't 't strong enough to hold a bear. Mm-hmm. Sure, you can have turn. bet it's even strong enough to hold trash truck. Speaker Nuh-uh, trash truck. I'm next. Now, come on, Walter, catch up. Oh, my turn. Yes! Speaker Oh. Okay.
Speaker 2: Yes, okay, I'm ready. Now shove me. Somebody shove me. want to be shoved. Higher!
Higher! Woohoo!
Speaker 2: Oh, I can the top of Trash Truck's head from up here! Ah, ha, ha! Yes! Woo! Speaker Okay, Donnie. think Trash Truck's 's turn now.
Speaker 2: No. no, no. It's still my turn. Yes.
Speaker 1: Well, at school, each person gets five pushes for turn. Then, it's the next person's 's turn.
Speaker 2: What's five?
Speaker don't know. Sounds complicated.
Speaker I: It's number. Every time you're re pushed on the swing, we count number. Walter, you keep pushing and Trash Truck and will count.
Speaker 2: Trash Truck can count?
Speaker 1: Yeah, Trash Truck
count. Show him, Trash Truck.
Speaker Wow, trash truck sure knows his numbers, mean, hawks. Wow.
Speaker 1: Okay, your turn's 's up. Donnie.
Speaker 2: Oh, alright. Okay.
Speaker O: Oh, okay. Who's next?
Speaker 2: Get on up there, trash truck. Come on, buddy.
Speaker See? told you, rope is strong.
H SUBJECTIVE EVALUATION
The annotation interface is designed as the following:
We conducted subjective evaluations engaging multiple human listeners to assess the quality of generated podcasts. Listeners rated samples Realism, Richness, and Preference. We used two popular and competitive open-source model: AesameAlLabs. CSM (SesameAlLabs 2025) and Higgs Audio V2 (Boson All 2025). and two proprietary dialogue models: Elevenlabs v3 alpha (Elevenlabs) and Gemini 2.5 pro preview its (Google).
5-point
across three key dimensions:
Below are the definitions and detailed scoring standards for the six dimensions used our subjective evaluation:
19

---

Published as a conference paper at ICLR 2026
Prorppt Aedio
Desin
mativi mitur
R&B
Samplez
Figure 5: Annotation interface of subjective evaluation.
Realism
Measures whether the speech sounds naturally generated, rather than mechanical overly deliberate. A high score indicates that the speaking style is closer to the natural rhythm, pauses, intonation, etc., of real human communication.
Definition:
Scoring Standard:
1 point (Very Poor):
Speech is completely unnatural, sounding very mechanical,
stiff, or robotic.
2 points (Poor):
Speech is noticeably unnatural, with clear problems in rhythm, pauses, intonation, or tone, sounding quite awkward.
3 points (Average): Speech naturalness is acceptable, with occasional unnatural elements (e.g., improper pauses, flat intonation), but generally acceptable.
4 points (Good): Speech is quite natural and fluent, close human speech, but may have minor shortcomings in some details.
5 points (Very Good): Speech sounds very natural and fluent, as if real human is speaking, with excellent naturalness.
Richness
Definition: Measures whether the speech conveys variety and expressiveness in tone and emotion. This includes natural emotional variation according to context, and whether suitable background elements (e.g., natural BGM or sound effects) are incorporated.
Scoring Standard:
point (Very Poor): Completely flat and monotonous; no emotion or variation; mechanical delivery.
2 points (Poor): Very limited emotional expression; stiff tone; fails to create sense of atmosphere or scene.
3 points (Average): Some variation in tone or emotion, but insufficient to convey natural expressiveness; weak emotional coloring.
4 points (Good): Emotionally expressive and relatively natural; noticeable changes in tone and intonation; conveys basic emotions and atmosphere.
20

---

Published as a conference paper at ICLR 2026
5 points (Very Good): Emotionally rich and natural: tone, intonation, and emotional variation match the context; may include well-timed natural sound effects background elements, creating strong immersion.
Preference
Measures the listener's overall subjective preference toward the speech. reflects naturalness, pleasantness, attractiveness, and listenability -not just objective clarity or similarity.
Definition:
Scoring Standard:
Strong dislike; voice is grating, mechanical, unpleasant; almost unbearable to continue listening.
1 point (Very Poor):
2 points (Poor): Dislike: noticeable problems such as stiffness, monotony, or unnaturalness; poor listening experience.
3 points (Average): Neutral; acceptable but not engaging; no strong preference or dislike.
4 points (Good): Like; natural and fluent; comfortable to listen to; provides some enjoyment.
5 points (Very Good): Strongly like; highly attractive and enjoyable; pleasant enough to listen to for a long time.
FIRST-PHASE EVALUATION RESULTS
Subjective
Objective
Model
Spontancity
Coherence
Intelligibälity
Quality
Similarity
Realism
WER
SIM-O
Cosyvicc22
3.15
g.0
3.83_1.22
3.17
3.15
3.10
MoonCast (a aueaa 2025
3.17=1.05
3.67 3.67
4.00-c.11
3.04e0.9
3.19e1.14
3.02+1.12
281
0.56
3,86
3.89.0.0 20.94
guss
2.66
3.78.0.m 20:00
Table 11: Subjective and objective evaluation podcast generation. For all subjective metrics and SIM-O, higher scores indicate better performance. For WER, lower scores are better.
To verify the effectiveness of proposed VIBEVOICR framework, we conducted our first-phase experiment comparing Cosy Voice2, MoonCast, and ViBEVOICE-1.55 through both subjective and objective evaluations.
For the subjective evaluation, 20 human evaluators were invited to assess the quality of the generated podcasts. Each evaluator rated samples on 5-point scale across six key dimensions: Spontaneity, Coherence, Intelligibility, Quality, Similarity (to the voice prompt), and Realism.
For every text input, we generated cight podcasts with each the three models, yiclding an average audio length of 7.3 minutes per sample. In total, each evaluator listened to approximately 2.9 hours of audio rooghlly equivalent to reviewing 1,000 short utterances 10 seconds each.
From Table I1
observe that:
(1) VIBEVOICE-L.5B outperforms both Cosy Voice2 and MoonCast across all subjective evaluation metrics as well as in WER.
(2) Compared with Cosy Voice2, which can only generate relatively short utterances (<30s) and produces long podcasts by concatenating these clips, both MoonCast and VIBe VOICE (with end-to- end podcast generation) demonstrate significantly better coherence, highlighting the effectiveness of E2E podcast generation for maintaining contextual and prosodic consistency.
(3) Another notable finding is that although Cosy Voice2 achieves a higher score than VIBEVOICE the objective SIM-O metric, human evaluators consistently rated VIBEVOICE having higher
21

---

Published as a conference paper at ICLR 2026
similarity. One possible explanation is that SIM-O primarily captures timbre similarity but neglects prosodic features such as intonation, rhythm, and expressiveness. In natural conversations, speakers often exhibit subtle variations in voice due to emotional shifts or pitch dynamics. These variations are typically perceived by human listeners as consistent with the original speaker, but are penalized by SIM-O. When SIM-O exceeds certain threshold, it becomes less sensitive to these human- perceived consistencies, making it insufficient for evaluating speaker identity retention in long- form, multi-turn speech synthesis.
SPECTRAL ANALYSIS OF DIFFUSION INFERENCE STEPS
(a) Voice Prompt
(b) Step-10
(b) Step=
Figure 6: Spectrogram analysis of the denoising process. (a) The Voice Prompt exhibits natural environmental textures (background noise, reverb). (c) At Step=50. the model aggressively strips these non-speech components, yielding a clean but spectrally divergent signal. (b) Step=10 offers the optimal balance, preserving the environmental "atmosphere" necessary for high speaker similarity.
Diffusion-based decoding inherently acts a progressive denoiser. While increasing inference steps (T)) typically refines signal detail, also biases the output toward idealized, "clean" speech distribution. However, for "in-the-will podcast data, speaker's identity is perceptually entangled with their acoustic environment (e-g., room reverberation, microphone response, and ambient noise floor).
Figure 6 visualizes how this denoising behavior affects acoustic consistency. The voice prompt (a) contains visible background spectral textures characteristic of natural recordings. At T 50 (c), model aggressively suppresses these non-speech components, producing audio that technically cleaner (akin to studio recording) but spectrally mismatched from the reference. This "over-cleaning" explains the counter-intuitive drop in SIM-O scores at higher steps: the metric penalizes the absence of the reference's environmental noise. In contrast, generation at T 10 (b) retains these stochastic elements, maintaining the acoustic "atmosphere" the prompt and thus achieving higher similarity scores.
LLM USAGE STATEMENT
A large language model (LLM), namely Gemini 2.5 Pro, was employed solely for grammar checking of the manuscript. It was also used to generate dialogue text for the subjective evaluation of VIBEVOICE.
The L.M did NOT contribute to the research design, training data collection, analysis, or interpretation of results. The authors bear full responsibility for the accuracy and integrity of all content presented in this paper.
22


Reviews:
[{"review_id": "mMoh4SJfh1", "reviewer": "Reviewer_h5UP", "summary": "The paper presents VibeVoice, a framework for zero-shot, expressive, long-form, multi-speaker podcast generation. It introduces two ultra-low frame rate latent (acoustic and semantic) operating at 7.5 Hz and integrates them into a large language model with a next-token diffusion mechanism for scalable, high-fidelity audio synthesis. The system is capable of generating podcasts up to 90 minutes with up to four speakers.", "questions": "1. I wonder to know that should we feed the text scripts at the beginning of sequence all at once (like VibeVoice), or in an interleaved manner (e.g., alternating segments of text and audio, as in FireRedTTS2)? Which one is better?\n\n2. The paper frequently refers to “tokenizers,” but the representations used are actually continuous latent variables, not discrete tokens. It might be clearer to use terminology such as latent encoders or continuous representation extractors to avoid confusion.\n\n3. In Figure 3(b), the SIM-o metric decreases as the number of diffusion steps increases. Why does this happen? Intuitively, more denoising steps might be expected to yield smoother or more consistent outputs—so the observed negative correlation deserves explanation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles an important and challenging task—long-form podcast generation, which combines the difficulties of expressive multi-speaker speech synthesis, discourse coherence, and long-context modeling. Addressing such a complex problem is both timely and valuable for advancing speech generation research.\n2. The proposed approach is methodologically reasonable and well-motivated. The reduction of frame rate to 7.5 Hz effectively shortens the sequence length for long-context modeling. The design choices—including (1) the use of continuous latent representations rather than discrete tokens, (2) the hybrid modeling of acoustic and semantic features, and (3) the introduction of a diffusion head to improve latent prediction—all contribute to handling the challenges of long-form expressive audio generation.\n3. The evaluation includes a variety of subjective metrics.", "weaknesses": "1. Unclear Training Data Source and Scale: The training data is vaguely described as an internal pseudo-labeled podcast collection without specifying its source, composition, or duration. While the paper mentions ~80 billion training tokens, this could roughly correspond to around 3 million hours of audio, which is an enormous scale. Clarifying the dataset origin and scale is critical for assessing the reproducibility, fairness, and ethical validity of the work.\n\n2. Lack of Analysis on Frame Rate (7.5 Hz) Choice. The paper repeatedly emphasizes the importance of the 7.5 Hz frame rate but does not justify why this particular rate is optimal. It would be highly informative to include experiments at alternative frame rates (e.g., 12.5 Hz or 15 Hz) to show how longer sequences affect LLM modeling difficulty, as well as lower rates (e.g., 3.75 Hz) to measure the trade-off between sequence compression and degradation in WER, expressiveness, or audio quality.\n\n3. Limited Discussion on Scaling and Comparability. Since most compared baselines (e.g., ElevenLabs, Gemini) are black boxes with unknown data and architectures, the fairness of comparison is somewhat constrained. It would be helpful if the authors could supplement scaling experiments, such as: (1) Varying the amount of training data to show scaling behavior. (2) Increasing model capacity (e.g., from 7 B to 30 B) to explore potential performance gains. Such results would make the conclusions more robust and generalizable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "deagjNjDGo", "reviewer": "Reviewer_1JL1", "summary": "This paper introduces a new model VibeVoice  for long-form zero-shot text-to-speech. \nVibeVoice can generate up to 90 minutes of speech from a text script and a voice-condition, supporting up to 4 speakers and outperforms SOTA models on podcast generation (as shown by quantitative and qualitative metrics).\n\nThis method relies on key design choices:\n- a separation of acoustic and semantic tokens to get the best of both worlds between text intelligence and acoustic reconstruction\n- ultra low frequency (7.5 Hz) tokens (this speeds up LLM inference time)\n- next-token-diffusion objective (continuous targets) as opposed to the standard cross-entropy objective (discrete targets).\n\nThe authors show extensive ablatation experiments to validate their design choices and compare their generations to state of the art TTS models. They provide detailed training details, code and sample outputs (long-form generated conversations) for reproducibility.", "questions": "clarifications:\n- what is the diffusion training schedule and loss weighting that was used? Same question regarding inference schedule\n- what number of diffusion steps was used for all metrics reported in comparison table?\n- on the tokenizer ablation , it looks like using hybrid only shows benefits at 2+ speakers, while acoustic is better at 1 speaker. Could you elaborate on this?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- SOTA model quality on long-form podcast generation\n- Great level of detail on all fronts : tokenizer training, LLM training, data preprocessing, inference settings\n- original contribution of concatenating acoustic and semantic tokens for LLM input - with a soft target (diffusion). They do ablations on those design choices to validate them\n- extensive ablation studies on design choices (tokenizer choice, base model size, diffusion inference settings)\n- extensive evaluation on the model, with a new benchmark VIBEVOICE-Eval  + subjective human evaluation", "weaknesses": "- The WER metric on the CFG figure is not very stable, there are no clear trends (as opposed to SIM-O which shows clear trends). This questions the robustness of WER metric in the comparison figures to other models.\n- This paper compares to other models in terms of performance but not speed, the  inference time appendix does not show other models as comparison", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QS7aIb9lYt", "reviewer": "Reviewer_WPLm", "summary": "This paper proposes a method for generating long-form, multi-speaker audio podcasts. It introduces a continuous hybrid (semantic and acoustic) speech tokenizer that achieves high-quality audio reconstruction at a frame rate of 7.5 Hz. Building upon this, VibeVoice employs next-token diffusion to enable high-quality audio generation.", "questions": "Why does the acoustic tokenizer perform well with a single speaker, but exhibit a significant drop in WER when a second speaker is introduced? Could this be due to insufficient training on multi-speaker data?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors design a continuous hybrid (semantic and acoustic) speech tokenizer that achieves high-quality audio reconstruction at 7.5 Hz.  \nThey leverage next-token diffusion to enable high-quality audio generation.  \nAn efficient pipeline for processing raw podcast data is established.  \nThe method achieves impressive results compared to top-tier models.", "weaknesses": "The proposed method in this paper draws heavily on the design of LatentLLM. Moreover, the designs of both the semantic tokenizer and the acoustic tokenizer have already been extensively explored in numerous audio-related works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w2obpg5IkO", "reviewer": "Reviewer_Tqnw", "summary": "This paper presents **VibeVoice**, a system designed for **long-form, multi-speaker conversational text-to-speech (TTS)** synthesis. The work addresses three major challenges in the field, **scalability**, **speaker consistency**, and **natural turn-taking with long-context stability**, particularly in the context of podcasts and other extended dialogues.\n\nVibeVoice adopts a **hybrid acoustic–semantic representation**, where speech and text are encoded separately and later fused. The semantic stream is modeled using a fine-tuned **Qwen2.5 language model**, while the acoustic stream is generated via a **next-token diffusion model** that predicts continuous latent acoustic features at **7.5 Hz**.\n\nIn addition, the authors introduce a **customized data annotation pipeline** that includes segmentation, speaker diarization, and automatic filtering of low-quality samples.\n\nEmpirical results demonstrate improved **turn pacing**, **context-aware prosody**, and **speech naturalness** over long durations (up to 45 minutes), outperforming several leading commercial and open-source speech generation systems in both **subjective and objective evaluations**.", "questions": "1. Could the authors clarify the **key differences between VibeVoice and MoonCast**?  \n2. In the demo, the model appears capable of **singing**. Is this behavior intentional or emergent? Additionally, the singing quality seems suboptimal, are there potential ways to improve this aspect?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. **Comprehensive System for Long-Form Conversational TTS**  \n- The paper presents a well-integrated end-to-end system capable of generating long, multi-speaker conversations with smooth and natural conversational flow, a significant advancement for practical TTS applications.\n\n2. **Innovative Representation and Generation Mechanism**  \n- The use of a **7.5 Hz hybrid acoustic–semantic representation** effectively reduces context length for speech modeling.\n- The **autoregressive next-token diffusion** architecture is a sound and well-motivated design choice for achieving high-quality, coherent speech synthesis.\n\n3. **Robust Data Preparation Pipeline**  \n- The proposed data pipeline, combining segmentation, diarization, and transcript filtering, demonstrates careful system design and contributes to improving training data quality.\n\n4. **Thorough and Convincing Evaluation**\n- The model is extensively evaluated through both subjective and objective metrics.  \n- The **subjective evaluation includes 6 hours of audio** and **over 24 evaluators**, which makes the results convincing for an academic paper.  \n- The **blind subjective evaluation setup (Appendix G)** appears well-designed and appropriate for assessing perceptual quality.", "weaknesses": "Overall, I did not find any obvious weaknesses. There are minor typographical issues. For example, in **Table 3**, the last row appears to incorrectly include an *Nq* value even though the proposed speech tokenizer does not use a quantizer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "URRtcMhSV4", "reviewer": "Reviewer_eyTk", "summary": "VibeVoice is a next-token diffusion multispeaker long-context TTS model that benefit from the following innovations: 1) a 7.5 Hz framerate tokenizer; 2) an annotation pipeline that generates pseudo transcription and turn-taking labels.", "questions": "1. why do we use a $\\sigma\\text{-VAE}$ as opposed to just an Autoencoder for acoustic feature?\n2. In appendix G, subjective eval interface is annotated with Chinese, are human listeners in the subjective evaluation native speakers of English?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. this work studies an important problem - long-form conversation generation, and is one of the few papers in this field. The idea of using continuous features leads to significantly reduced sequence length which enables long context.\n2. the data annotation pipeline is an important contribution", "weaknesses": "1. a big contribution of the the paper as noted in the abstract is data pipeline, but it's described in appendix, and there is not evaluation on the design choices of the data pipeline\n2. the writing could be improved, for example, it should be noted that in the prompt part, the speech features are just short segments, while the text are the entire conversation (including both prompt and what's to be generated?)\n3. missing RTF", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E7mhtulzD2", "reviewer": "Reviewer_jDkA", "summary": "The paper introduces VibeVoice, a novel zero-shot Text-to-Speech (TTS) model designed for synthesizing long-form, multi-speaker conversational audio (e.g., podcasts, up to 30 minutes with up to 4 speakers). This addresses the limits of traditional TTS in handling scalability, speaker consistency, and natural turn-taking.\n\nA core technical element is an ultra-low frame rate (7.5 Hz) continuous speech tokenizer that significantly boosts computational efficiency for long sequences while preserving audio fidelity. Using an extensive new dataset with pseudo transcriptions and turn-taking labels, VibeVoice employs a next-token diffusion framework. This enables the model to achieve a high degree of naturalness in turn-taking, pacing, and subtle non-lexical cue rendition (like breaths), crucial for authentic, immersive conversational output.", "questions": "1. VibeVoice claims to synthesize long-form audio with up to four speakers. Given the high risk of speaker timbre drift and identity confusion in long-context, zero-shot TTS, what explicit architectural mechanisms are implemented to ensure robust and consistent speaker similarity for all four voices?\n\n2. How to prepare and collect data for more than two speakers? For transcription, it is unclear whether to rely only on pseudo-transcription, or should real data also be used? Furthermore, why doesn't pseudo-transcription negatively impact model performance?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper addresses an important problem in speech generation. Synthesizing long-form, multi-speaker conversational audio is highly relevant for real-world applications such as podcast generation, enhanced virtual assistants, and dialogue systems. The scenario is undoubtedly innovative and highly applicable", "weaknesses": "The core weaknesses of this work lie in a lack of sufficient technical novelty and inconsistent experimental evaluation.\n\n1. Methodology: The next-token diffusion (or next-distribution) framework, a central component of VibeVoice, has been previously introduced in the literature [1]. The paper does not clearly articulate the incremental technical novelty VibeVoice contributes beyond adapting this existing framework.\n\n2. Scenario: The goal of synthesizing speech for multi-speaker (more than two people) scenarios, particularly for long conversational contexts, is also not entirely new. Prior work [3,4], has specifically designed and discussed methods for long conversational speech generation for podcasts and similar applications for more than 2 people [2].\n\n3. While the paper addresses long-context dialogue generation, it fails to include a fair and direct comparison against key existing models that tackle similar challenges in multi-speaker and long-form synthesis [3,4]. This makes it difficult to ascertain the true performance gain of VibeVoice.\n\n4. The 7.5Hz tokenizer appears to be the core technical contribution, yet no sufficient details are provided to fully evaluate its novelty or effectiveness\n\n5. There is very little detail about the data preparation pipeline.\n\nMoreover, there are many inconsistent and incomplete experimental evaluation:\n\n1. Baseline Inconsistency: The choice of baselines across the evaluation tables is inconsistent. The paper should use a uniform set of relevant baselines across all comparative tables (e.g., Table 1 and Table 2) to provide a clear and comprehensive performance picture.\n\n2. Missing Subjective Metrics: Table 1, which appears to focus on multi-speaker/turn-taking quality, lacks subjective (human) evaluation metrics for some systems. \n\n3. Missing Single-Speaker Results: The evaluation in Table 2 is incomplete as it lacks results for the single-speaker scenario. \n\nFinally, the paper fails to include a complete and up-to-date list of references for all cited or discussed works. This is an essential requirement for a conference paper.\n\n[1] Zhu, Xinfa, Wenjie Tian, and Lei Xie. \"Autoregressive speech synthesis with next-distribution prediction.\" arXiv preprint arXiv:2412.16846 (2024).\n\n[2] Xie, Kun, et al. \"Fireredtts-2: Towards long conversational speech generation for podcast and chatbot.\" arXiv preprint arXiv:2509.02020 (2025).\n\n[3]Zeghidour, Neil, et al. \"Streaming sequence-to-sequence learning with delayed streams modeling.\" arXiv preprint arXiv:2509.08753 (2025).\n\n[4] Ju, Zeqian, et al. \"MoonCast: High-quality zero-shot podcast generation.\" arXiv preprint arXiv:2503.14345 (2025).\n\n[5] Zhang, Leying, et al. \"CoVoMix: Advancing zero-shot speech generation for human-like multi-talker conversations.\" Advances in Neural Information Processing Systems 37 (2024): 100291-100317.\n\n[6] Mitsui, Kentaro, Yukiya Hono, and Kei Sawada. \"Towards human-like spoken dialogue generation between ai agents from written dialogue.\" arXiv preprint arXiv:2310.01088 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)