MoonCast: High-Quality Zero-Shot Podcast Generation
Authors: Zeqian Ju
Dongchao Yang
Kai Shen
Yichong Leng
Zhengtao Wang
Songxiang Liu
Xinyu Zhou
Tao Qin
Xiangyang Li
Jianwei Yu
Xu Tan
Venue: NeurIPS 2025 poster ()
Keywords: zero-shot speech generation
podcast generation
Primary area: applications
TL;DR: a solution for high-quality zero-shot podcast generation

Abstract: Recent advances in text-to-speech synthesis have achieved notable success in generating high-quality short utterances for individual speakers. However, these systems still face challenges when extending their capabilities to long, multi-speaker, and spontaneous dialogues, typical of real-world scenarios such as podcasts. These limitations arise from two primary challenges: 1) long speech: podcasts typically span several minutes, exceeding the upper limit of most existing work; 2) spontaneity: podcasts are marked by their spontaneous, oral nature, which sharply contrasts with formal, written contexts; existing works often fall short in capturing this spontaneity. In this paper, we propose MoonCast, a solution for high-quality zero-shot podcast generation, aiming to synthesize spontaneous podcast-style speech from text-only sources (e.g., stories, technical reports, news in TXT, PDF, or Web URL formats) using the voices of unseen speakers.   To enable long audio generation, we employ a language model with parameter, data, and context scaling to process sequences in an innovative format designed for modeling entire multi-speaker, multi-turn speech interactions. To enhance spontaneity, we observe that ASR transcripts capture spontaneous speech details (e.g., filler words indicating hesitations, and specific punctuation and spaces reflecting breathing pauses), suggesting that these transcripts can serve as a partial indicator of speech spontaneity. Building upon this assumption, we utilize a script generation module to generate scripts incorporating these spontaneous elements. Experiments show MoonCast outperforms baselines, with notable improvements in contextual coherence and spontaneity.

MaonCast: High-Quality Zero-Shot Podcast
Generation
Zeqian Ju
Dongchao Yang"
Kai Shen²
Yichong Leng²
Zhengtao Wang?
Songxiang Liu?
Xinyu Zhou²
Tao Qin
Xiangyang Li!
Jianwei Yu²t
Xu Tan
Abstract
Recent advances in text-to-speech synthesis have achieved notable success gener- ating high-quality short utterances for individual speakers. However, these systems still face challenges when extending their capabilities to long, multi-speaker, and spontaneous dialogues, typical of real-world scenarios such as podcasts. These lim- itations arise from two primary challenges: 1) long speech: podeasts typically span several minutes, exceeding the upper limit of most existing work; 2) spontaneity: podcasts are marked by their spontaneous, oral nature, which sharply contrasts with formal, written contexts; existing works often fall short in capturing this spontaneity. In this paper, we propose MoonCast, a solution for high-quality zero-shot podcast generation, aiming synthesize spontaneous podcast-style speech from text-only sources (e.g., stories, technical reports, news in TXT, PDF, or Web URL formats) using the voices of unseen speakers. To enable long audio generation, a language model with parameter, data, and context scaling to process sequences in an innovative format designed for modeling entire multi-speaker, multi-turn speech interactions. To enhance spontaneity, we observe that ASR transcripts capture spontaneous speech details (e.g., filler words indicating hesitations, and specific punctuation and spaces reflecting breathing pauses), suggesting that these transcripts can serve partial indicator of speech spontaneity. Building upon this assumption, we utilize a script generation module to generate scripts incorporating these spontaneous elements. Experiments show MoonCast outperforms baselines, with notable improvements in contextual coherence and spontaneity.
employ
Introduction
Recently, significant advancements in large language models (LLMs) and speech codec technologies have substantially enhanced the performance of text-to-speech (TTS) synthesis, improving its natu- ralness, expressiveness, and tonal richness. These advancements have led to widespread adoption in industries such as customer service and short video production. As TTS technology continues to evolve, there is growing demand for generating long-duration podcast content from text-only sources, such as news and technical reports. Podcast speech requires not only extended audio lengths but also highly spontaneous expressions, often involving multiple speakers and dynamic interaction.
The limitations of previous efforts in generating high-quality podcasts stem from key challenges. First, long-context audio modeling presents challenges. Podcasts typically span over several minutes, featuring numerous utterances from multiple speakers. This requires the system generate not only realistic individual speech but also smooth transitions between utterances. Furthermore, high-quality
University Science and Technology China Moonshot "The Chinese University of Hongkong "Microsoft Research. Correspondence to: Jianwei Yu <tomasyu @ foxmail.coms and Xu Tan <tanxu2012 aaainn001@@@mml.commmmmm comi.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

podcast generation must account for the contextual coherence of each speaker, encompassing aspects such as prosody and timbre. Second, podcasts are highly spontaneous, typified by the fluid and casual flow of human conversation. They often contain human-like details, including filler words such as "um", occasional hesitations, and minor mistakes. In the multiple-speaker scenario, the system must also account for the interactions between speakers. However, the TTS community has largely focused on improving short individual utterance generation, with limited efforts exploring long-context, spontaneous scenarios. Specifically, academic research has primarily focused on short conversational speech Nguuyn et al. [2023] Mitsui et al. 2023), but these efforts often face difficulties when applied to longer, more complex podcast scenarios, particularly in capturing spontaneity and naturalness within inter-sentence interactions. Recently, industrial solutions like NotebookL.M have emerged to facilitate podcast creation from various knowledge sources. However, these solutions often lack transparency in their technical details, limiting their adaptability.
To overcome these limitations, we propose high-quality podcast generation system MoonCast. On one hand, to improve contextual coherence, we enable the holistic, zero-shot generation of multi-speaker,, multi-turn conversations, supported by innovative sequence format. To manage such extensive context, we adopt a language model-based speech modeling approach, scaled with approximately 500k hours of training data, 2.5B parameters, and a 40K token context length. In addition, we employ a chunk-wise autoregressive speech detokenizer for effective inference in the long-context scenario.
On other hand, to improve spontaneity, we build upon a novel observation: certain spontaneous speech details are often reflected within their corresponding automatic speech recognition (ASR) transcripts, such as hesitations linked to filler words, breathing pauses marked by specific punetuation or spaces, and ner-verbal sounds identified as onomatopoeic words. This observation prompts our novel assumption: ASR transcripts can act as a partial proxy for speech spontaneity. We further validate this assumption through an experiment designed to identify the impact of varying script spontaneity on a fixed text-to-speech model. The results show that the presence of spontaneous details in the script significantly impacts the spontaneity of the generated speech. This validated assumption informs our core design principle: train the model to generate spontaneous speech conditioned on ASR transcripts, and during inference, use input text designed to emulate ASR transcript characteristics to elicit spontaneous output. Accordingly, our audio modeling module is trained on large-scale dataset of spontaneous speech from diverse sources, with corresponding ASR transcripts generated through a data preparation pipeline. A three-stage curriculum learning approach is also employed to facilitate robust training on spontaneous speech data and progressively equip the model with zero-shot, long-context, and spontaneous speech generation capabilities. Finally, in our script generation module, we provide the LLM with demonstrations and detailed instructions to help emulate ASR transcript style and incorporate spontaneous elements into the scripts.
With this design, we can generate a spontaneous podcast of up to ten minutes from text-only input sources in a comprehensive manner. The experimental results show that the proposed system consis- tently outperforms the concatenate basclines in terms of intelligibility, coherence, and spontaneity for multi-lingual podcast generation. Specifically, MoonCast achieves subjective evaluation improve- ments of 0.39 in spontaneity, 0.28 in coherence, 0.05 in intelligibilitty 0.13 in speech quality and 0.25 in speaker similarity for Chinese, and 0.68 in spontaneity, 0.62 in coherence, 0.15 in intelligibillty and 0.05 in speech quality for English podcast generation. We invite readers to listen audio samples at h/tttps://nooccaatttts: .iw.aaat.emmoi..m......... io for a more intuitive experience.
We open-source MoonCast, including the prompts for script generation and the audio modeling module for speech generation, to support future research.
2 Background
2.1 Zero-Shot TTS
Zero-shot text-to-Speech synthesis aims to synthesize speech that mimics the characteristics of a target speaker using only a brief prompt speech, without requiring additional fine-tuning [Shen et al. 2023
hrcpps:/noteeooolln wwwp:oowwwmomomo...ooo.om.oomm www.gooo....oooo com/
Refer to Appendix
http::///gtthub..om/////////////
2

---

Knowlodge Sources
Sgontaneous Scripts.
Spontancous Audio
Scritt Generation Module
Audio Moddingg
Module
Figure 1: The overall pipeline of the proposed system.
Ju et al. 2024 Chen et al.
2024]. Recent advancements in zero-shot TTS can be broadly categorized
into two types based on the representation: discrete code or continuous latent. For the code-based
method, VALL-E [Wang et al. 2023] utilizes neural codec language models and achieves high fidelity
in zero-shot TTS. Seed-TTS [Anastassiou al. [224]] and CosyVoice
Voice (Du et al.) 2024a b leverage
single semantic codebook to reduce the difficulty on discrete code generation. Also, discrete diffusion
can be leveraged to enable code generation in non-autoregressive manner [Borsos et al.) 2023
Ju et al. 2024 For the latent-based method, Naturalspeech 2 (Shen et al. [023] leverages latent diffusion to predict the latent a speech codec conditioned on a short prompt speech. VoiceBox [Le] al. 2024) utilizes flow matching to model Mel-spectrogram of a speech. In this paper, we adopt pipeline that combines both the code-based and latent-based methods. Rather than focusing on speech that contains only single speaker, we consider zero-shot two-speaker podcast speech generation.
2.2 Dialogue and Conversation Generation
Most works in zero-shot TTS focus on the speech synthesis of one speaker. However, many scenarios such as dialogues, conversations, and podcasts require the TTS model to be able to synthesize speech with multi-speaker at the same time. Generating spoken dialogues (Schuller et al. [013 that include natural turn-taking, laughter, and other paralinguistic cues iZhang et al.) 2020 Adiwardana et al.)
2020 Lewis al.) 2020 Xu 2021 is non-trivial. Recent work has explored various approaches to address this challenge. DGSLM [Nguyen et al., 2023] uses a dual-tower transformer architecture to capture the turn-taking dynamics and non-verbal vocalizations in spoken dialogues, aiming to generating naturalistic spoken dialogues. Built on the top of dGSLM, CHATS [Mitsui et al.) 2023 makes the generated dialogues smooth turntakking To enable dialogue generation with diverse timbre, CoMMM (Zhang et al.) [024] proposes zero-shot dialogue generation to support zero-shot, multi-speaker, multi-round dialogue speech generation. Thanks to the progress in large language model (Achiam et al. 2023 Yang et al.) 2024), we can generate speech dialogue with more spontaneous content [Lu et 2025 Despite these advancements, most prior works rely on datasets of approximately 2000 hours (such as the Fisher dataset [Cieri 2004)) and are limited to generating dialogues of less than 90 seconds. These limitations stem from the challenges in maintaining coherence and naturalness over longer contexts. In this paper, we address these limitations by proposing a long-context text-to-semantic autoregressive architecture to model the inter-sentence prosody, speaker change, and paragraph-level spontaneity.
interactive and fluid by incorporating backchannels, laughter, and
2.3 Spontaneous TTS
Spontaneous TTS refers to the synthesis of speech that mimics natural, conversational speaking styles, as opposed more formal read speech. aims to generate speech with characteristics such as filler words (c.g., "um" and "uh"", diverse rhythms, and natural prosody variations (Yan al. 2021 Li al. 2024b). Spon (Li al.) [024a] proposes a neural bottleneck to help TTS model better model and transfer spontaneous style. Other works (Li et al., 2024b) utilizes LLM to systematically categorize the spontaneous behaviors and then uniformly model these behaviors in TTS model. BaseTTS [Lajszczak et al.) 2024] finds out that the spontaneity can come from emergence. Once the TTS model has been trained on a large number of speech data [Anastassiou et al. 2024]. it can acquire emergent abilities, such as expressing emotions. Along this direction, in our paper, we further find out that the spontaneity of the generated audio is significantly influenced not only by text-to-speech modeling but also by the script text itself.
3

---

3 Method
3.1 Overall
In this section, we describe the method for podcast speech generation. Conceptually, podeast consists of multi-speaker, multi-turn spoken dialogues in spontaneous manner. Unlike traditional zero-shot speech synthesis methods that focus on single speaker and fixed textual inputs, we divide the podcast generation process into two stages: 1) spontaneous script generation, which converts input knowledge sources into spontaneous text for podcast creation, and 2) spontaneous podcast speech generation, which involves multiple speakers and turns following the generated script. In this paper, we focus on generating two-speaker podcasts.
The overall system pipeline is illustrated in Figure !I To generate a spontaneous podcast, we first employ an I.L.-powered podcast script generation module to produce podcast scripts from input knowledge sources. Subsequently, we utilize a long context audio modeling module to generate podcast speech according the scripts, using unseen speakers' voices. In specific, for the novel task of podcast speech generation, represent each audio frame single discrete semantic audio code, thereby decomposing the task into text-lo-semantic generation and semantic-to-audio reconstruction sub-tasks, using the discrete semantic code sequence the intermediate representation. As shown in Figure 2 we use a speech semantic codec for speech tokenization, a texl-to-semantic model for semantic code modeling. flow-mate matching based speech detokenizer for semantic-to reconstruction, and pre-trained vocoder for wel-toowwveeoomm reconstruction.
4 Method
4.1
Audio Modeling Module
4.1.1
Long-Context Two-Speaker Text-to-Semantic Model
The zero-shot two-speaker podcast generation task aims to synthesize each turn of the podcast using the corresponding speaker's voice, based on the provided reference speech from speakers. A significant challenge arises from the length of speech code sequences. For example, with a 50 Hz single-layer speech codec (i.e., using single code to represent 20 ms speech frame), common codec setting, a 5-minute podcast corresponds to a sequence length of 15,000, Additionally, unlike single-speaker zero-shot speech synthesis, this task must also ensure contextual coherence and smooth transitions between individual speech segments.
We holistically model the multi-speaker, multi-turn podcast ensure superior contextual coherence. To effectively manage the extensive context, we utilize a language model-based speech modeling approach. This model is scaled with approximately 500,000 hours of training data, 2.5 billion parameters, and a 40,000-token context length. By inputting raw ASR transcripts as conditioning for speech modeling, we facilitate the model's ability to learn spontaneous patterns directly from their natural occurrence within the conversational speech transcripts used for training.
Sequence Design. We design an innovative sequence format for the novel task of zero-shot podcast generation, handling multi-speaker, multi-turn interactions while preserving the continuity of speech as coherent whole. To achieve this, we adopt full-texttto-flll-uudh interleaving approach, rather than interleaving on a per-turn basis. Specifically, as shown in Figure 2 from the same speaker to ensure alternating turns between speakers, and incorporate a special speaker change token after prompts and each podcast turn. This token indicates the change of speaker, thereby enhancing speaker robustness. Formally, we denote the prompt speech codes by 8f and the corresponding text by lt for speaker Sr, where i {1, 2}. The podcast is represented as list [sskk,, ty, s,)] consisting of M dialogue turns, where spkj. ty and s, correspond to the speaker, the script text and the speech for the turn j. where E {1...., M, J. To construct the two-speaker data sequence, we start from creating four sub-sequences by prepending the speaker identifier to each prompt and podcast turn: prompt text TP = {S , ,2, prompt speech SP = {8t ,²),, podcast text T = {spki, 41 ...... spkM, LM)). ). and podcast speech S {si,.... }. These sequences are concatenated in the order {TP, 1TP.77 ST .S.. ). We use the language model estimate the probability p(STT", TT, TP). During training, compute the average cross-entropy loss for each turn, inclduing
we merge adjacent segments

---

Podcust Speech
1--
Text Token
Audio Token
FM
FM
Speaker Change Token
O
O00
Text-to-Semantic Model
Prompt Text
Podcast Text
SC
SC
Prompt Speech
Figure 2: Overview of the audio modeling module. Speaker represented by blue, and speaker 2 by green. 'SC" denotes the speech Semantic Codec, and 'FM' denotes the Flow-Matching based speech detokenizer.
both speech codes and the special speaker change token. During inference, the predicted speaker- change token allows us to select the appropriate prompt speech for reconstructing the speech.
Curriculum Learning. Given the limited availability of high-quality long-form spontaneous speech data, we employ the curriculum learning technique to progressively enhance the model's capabilities across three distinct stages, gradually increasing the complexity of the training data to optimize learning efficiency. 1) In the first stage, we segment the entire audio from all data sources according to the annotations. Each segment contains a single-turn utterance from only one speaker. Specifically, we do not explicitly specify speech prompts, but instead implicitly assume that any prefix of the sequence serves as the prompt for the remainder. We train the model on these individual segments to initially develop its zero-shot TTS capability. 2) In the second stage, we begin modeling entire audio sequences involving two speakers and multiple turns. Given that non-conversatioon audiobooks, typically involve less interaction between speakers and feature simpler text with fewer spontaneous details, we start from these data sources, which are easier to learn from. Specifically, use the first turn from each speaker the prompt and scale the context length to 40,000 tokens (equivalent to 800 seconds in our setting) to accommodate long-context scenarios. This approach aims to enhance the model's 's consistency in speaker representation and robustness in long-context, two-speaker scenarios. 3) In the final stage, we refine the model's 's ability to generate spontaneous speech using conversational data from sources such as podcasts, which feature two speakers and multiple turns. These sources are characterized by dynamic, natural interactions between speakers and the presence of spontaneous speech elements, essential for capturing the nuances of real-world conversations. Similar to the second stage, we use the first turn from each speaker as the prompt while maintaining the context length 40,000 tokens to support long-context modeling. By exposing the model to this type of data, we aim to improve its ability to generate spontaneous speech.
scenarios, such
4.1.2 Chunk-wise Autoregressive Speech Detokenizer
To decode the generated speech codes and produce the final podcast, several naive approaches may come to mind, each with its own limitations. One approach might involve reconstructing the entire sequence at once. However, this method faces two major challenges. First, waiting for all tokens to be generated can be prohibitively slow, especially for long speech segments typical in podcast scenarios. Additionally, the large memory footprint required for processing such long sequences often exceeds the available GPU memory. Another naive method is to split the speech into fixed-length segments, reconstruct each segment individually, and then concatenate them. While this approach mitigates the memory issue by reducing the sequence length, it introduces new problem: discontinuities the boundaries between chunks. These discontinuities can lead to less fluent and consistent speech, as each segment is generated independently without considering the context of adjacent segments.
To address these limitations, we propose more efficient solution: chunk-wise autoregressive detokenizer. This method divides the speech tokens into small chunks (e.g., 3 seconds per chunk),
5

---

enabling more efficient processing of long speech segments. By processing the sequence in smaller, manageable chunks, we significantly reduce computational overhead and memory requirements. Additionally, we apply chunk-wise causal mask, which allows each chunk to access the history of previously generated speech chunks. This approach not only improves the fluency and consistency of the generated speech but also ensures more stable boundaries between chunks, effectively addressing the continuity issues that arise from direct chunking.
Flow Matching Model. Our detokenizer is based on a DiT [Peebles and Xie 2023 based flow- matching model, which conditions on speech codes and generates the mel-spectrogram from random Gaussian noise. Firstly, we take the chunk i and all previous chunks < where the chunk i is for generation and chunks à the prompt for clarification (i E [O. where N is the chunk amount). The chunk i's mel- -ppecroogran is M. and speech codes C, (Mc; and Ces for previous chunks < i as well). The flow-matching approach involves the forward process to add noise to the data, and the backward process to remove the noise in reverse. In training, we apply forward process to obtain the noised data (t) t M, + (1 - (1 mmin))) M by mixing the sampled gaussian noise M ~ N 1) with clean data M, at timestamp / [o, 1J. where min is a hyper-parameter. The flow-matching model fo. parameterize by 0, is adopted to learn the mapping (o(M. = (t, /dt = M; (1 min M with the condition C, In addition, the MMs, and Czi are adopted as the prompts for in-context learning. At inference, we start backward process from another sampled Gaussian noise M,(0) ~ N 1) and recover the clean data through the ODE: dt,(t)) fo dt)
Ma-g M
Mi,, M M. Mi,
MC-1
M,1
M,
M
M
Figure 3: The attention mask design for chunk-wise autoregressive speech detokenizer. M, means the clean mel-spectrogram and M/, means noisy mel-spectrogram. Yellow means allow attend, and gray means not allowed to attend. Attention is conducted among row-wise in figure.
Chunk-wise Causal Mask. To facilitate efficient training, we first segment the long speech into chunks and apply a chunk-wise causal attention mask [Liu et al. 2024]. This mask allows to access information from both the current noisy and previous clean chunks, thereby enabling the butch-parallel training of all noisy chunks within a single dataloader batch. As shown in Fig. 3 we assume the prompt chunk as M, (clean mel-spectrogramm) and the chunk for generation as M; (noisy mel-spectrogram). During training, we put all chunks M, and MC in a whole sequence, thus there 2NN chunks. The attention mask follows: 1) there is mask in the current chunk; 2) for the left half chunks where M, E [0, N) (i.e., lower rows in Fig. 3. we apply attention mask where M; can only attend the chunks M, where [0, i]: 3) for the right half chunks where M; [N, 2N) (i.e., upper rows in Fig. 3). we apply attention mask where chunk MC can only attend to clean chunks M, where j [0, i I] and noisy chunk M; itself.
Following this design, during inference, when the language model outputs a chunk, we use the flow-mater flw-matchiigg model to generate the corresponding mel-spectrogram M. We then apply prefilling for this chunk with M and a timestep of 0.999 to generate the kv-cache for efficient inference.
4.2 L.M--ooereed Script Generation Module
In this section, we present an LLM-powered podcast script generation module, enabling users to create rich and diverse scripts from different knowledge sources. This module consists of three components: (1) Content analysis: For any type of user input (e.g., Web URL., PDF), we combine I.LMs to recognize the content in the input. For example, if the user's input is Web URL, we use the search function in ChatGPT to retrieve the content from the link. (2) Briefing document generation:

---

In our preliminary experiments, we find that directly asking LLM to generate scripts based on the original content often results ill-suited and vague scripts, which leads to the loss of significant information. To address this issue, we propose generating a briefing document first, which covers the key points in the original content, Specifically, the briefing document includes five components: the title with authors, an abstract, main topics, key citations and conclusion. Each component includes an additional paragraph to explain technical terms, concepts, or methods that might confuse readers unfamiliar with the field. (3) Scripts generation: Based on the briefing document, use LLM to generate a podcast script that features coherent logic and comprehensive content. Specifically, We guide the LLM in three key areas: podcast structure, format, and content. For structure, ask the LLM to create engaging openings and closings that set the tone and effectively wrap up the podcast. Regarding format, the script must be in JSON format and feature two speakers: host who controls the pace of the conversation and a guest who primarily introduces the content of the document. In terms of content, the script includes key citations and explanations of technical terms in coherent manner, ensuring logical connections between topics and maintaining a moderate information density. To infuse the text with spontaneity and replicate the ASR transcript characteristics in training, we guide the LLM to incorporate spontaneous details such as filler words (e.g., 'um', 'uh', 'like', 'you know', "so'), response words (e.g.. 'yeah', 'right', 'okay"), ) repetitions and informal grammar. Moreover, we provide formatting tips, such as using spaces and commas within sentences to indicate pauses, and also offer a specific example of an ASR transcript as the demonstration.
Experiments and Results
5.1
Experimental Settings
In this section, we present a overview of the experimental setup, including detailed descriptions of the data preparation, the model architecture, and the evaluation setting.
5.1.1 Data Preparation
We conduct our experiments on a large-scale internal Chinese and English audio dataset comprising approximately 1.0 million hours of audio from diverse sources, including podcasts, audiobooks, and audio clips from shows. Following previous works (Yu et al.) 2024 He et al., [222], we apply a data processing pipeline to process these audio source. The final dataset comprises 300,000 hours from Chinese audiobook sources, 15,00 55,000 hours from Chinese conversational sources, and 200,000 hours from English conversational sources. Refer to Appendix B for more details.
5.1.2 Model Details
Speech Semantic Codec. For the speech semantic codec, both the encoder and decoder consist of 12 ConvNext blocks, each with a kernel size of and a hidden size of 384. The 1024-dimansional SSL feature is projected into an 8-dimensional space for quantization using an 8 8192-entry codebook. We train the codec for 200,000 steps.
Tex--to-Semantic Model For the -extani-emantic model, we use 2.5B- parameter, 16-layer Llama- style Transformer with hidden size of 3072 and 24 attention heads. We train it using the Megatron framework on 64 A100 80GB GPUs with tensor parallelism degree of 8, over a maximum sequence length of 40k, a batch size of 600, and for 2,000 steps in each curriculum learning stage. We use top-k value of 30, top-p value of 0.8, and temperature of 0.8 for inference. We use Byte-Pair Encoding (BPE) for text tokenization. The model undergoes curriculum learning in three stages. In the first two stages, it is trained Chinese data support zero-shot long-context speech generation. In the third stage, we mix both Chinese and English conversational data to handle multilingual spontaneous generation tasks.
Speech Detokenizer For the speech detokenizer, we adopt a -8B-paraeeter, 10-layer Dit-style Transformer with a hidden size of 2048 and 16 attention heads. During training, the chunk size is dynamically set between 0.5 and seconds to support flexible inference. For inference, we specifically use a chunk size of 3 seconds to achieve better quality. The backward ODE for each chunk is solved using 30 steps with the torchdyn toolkit (Poli et al.). In addition, we adopt 250M-parameter BiAVGAN [Lee et al. 2022 to reconstruct waveforms from melsspeccss.sams.
7

---

Table 1: The performance comparison on the Chinese podcast generation. Bold for the best result, and underline for the second-best result.
Subjective
Objective
Models
Spontancity (t)
Coherence (f)
Intelligibility (t)
Quality (f)
Similarity (t)
SIM-O (t)
WER ())
Cosyvoice2
3.68
3.5
4.18
3.94
3.94
0.85
2.40
Concat Baseline
3.94 +0.26
3.98
4.38 +0.15
4.00 4..3.
3.94
-0.21
0.86
1.90
MoonCast
4.33 20.17
g.26
4.43 20.12
4.13
4.19
40.15
0.77
2.15
Table 2: The performance comparison on the English podcast generation. Bold for the best result, and underline for the second-best result.
Suhjective
Objective
Models
Spontancity (1) Coherence (f)
Intelligibility (?)
Quality (t)
Similarity (1)
SIM-O
WER ())
Cosyvoice2
3.86 20.24
3.88
4.46 s0.14
4.25 4..1
4.40
0.73
2.77
Concat Baseline
3.73
3.71 +0.1
3.93
3.74
3.96
0.75
2.56
MoonCast
4.54 20.16
2.
4.61
4.30
4.25
0.53
1.81
5.1.3 Evaluation Details
Evaluation Dataset. For podcast generation, curate evaluation dataset comprising two knowledge sources in PDF format and two in web URL format, encompassing domains such as computer science paper economics paper: technology blogs and news articles To verify the importance spontaneous text, we select seven two-speaker Chinese podcasts, with speakers not present in the training data, totaling 125 turns, assess the impact of scripted text on generation quality. For both datasets, we use 5-10 seconds of speech as the prompt for each speaker.
Model Comparison. We employ concatenation baseline, whose text-to-semantic model is trained exclusively on single-speaker, single-turn data while other models remain the same. also utilize Cosyvoice2 [Du et al. 2024b), powerful open-sourced multi- lingual single-speaker zero-shot TTS model, as another base For these two baselines, we first generate each dialogue turn individually in zero-shot manner, and then concatenate these turns to form the complete podcast.
We employ both subjective and objective metrics for a comprehensive evaluation. For the subjective evaluation, we involve ten evaluators to assess three specific aspects of the generated podcast: the entire audio, transitions between segments and individual segments. Specifically, we consider 1) spontaneity of the entire generated podcast and 2) coherence of transitions between segments. Additionally, for individual segments, we focus on three metries: 3) intelligibility, 4) speech quality and 5) speaker similarity. For the objective evaluation, employ SIM-O to assess speaker similarity and the Word Error Rate (WER) evaluate robustness. detail, we apply the embedding model assess speaker cosine similarity between generated samples and the prompt speech. We average the SIM-O scores for each round according to the audio length. We use FunASR for Chinese speech transcription and NeMo ASR toolki @ for English. Note that we select NeMo ASR instead of Whisper because the Whisper model tends to suffer from hallucination issues.
Evaluation Metric.
pretrained WEWLM-
5.2 Experimental Results
In this section, first evaluate MoonCast by comparing it with existing baselines on the podcast generation task, thus confirming its superior performance. Subsequently, we empirically validate key assumption of MoonCass the spontaneity of the generated audio is significantly influenced by the spontaneity of the script text itself.
httpe//arrav.oor//fff//Oo0
https://www.nr http:///ww...aaaa.....o/////////

---

Table 3: The influence of spontaneous scripts for podcast generation. Bold for the best result, and underline for the second- best result.
Subjective
Objective
Models
Spontancity (t)
Coherence (f)
Intelligibility (f)
Quality (t)
Similarity (t)
SIM-O (t)
GT
4.73g0.00
4.63
4.57
4.48
4.37go.11
0.83
GT script
4.17
3.83 2.80
3.97
2.97
4.00
0.68
Written Script
3,22+0.00
3.53+0.12
4.27 20.31
3.62+0.12
3.67=0.18
0.68
Spontancous Script
1.03 20.30
2.00
4.53
4.03
4.03
0.72
5.2.1 Evaluation on Podcast Generation
To assess the efficacy of MoonCast, we evaluate podcast quality by comparing it with the two single- speraker baseline using the collected input knowledge sources. We report the evaluation results of the Chinese and English podcast generation in Table [nd We make the following observations: 1) MoonCast consistently surpasses the two concatenation baselines in terms of spontaneity, coherence, intelligibility and quality metrics for both Chinese and English podcast generation. Thus result demonstrates that the long-context two-speaker audio modeling captures contextual dependencies, thereby validating the effectiveness of our proposed method generating high-quality results. 2) Despite the inherent systematic errors in the ASR model when handling proper nouns and filler words, MoonCast still achieves WER of 2.15 for Chinese and 1.81 for English podcast generation, further demonstrating the robustness of the proposed system. 3) Furthermore, we observe certain degree of discrepancy between the SIM-O and subjective similarity metrics, possibly because the single speaker embedding used by the SIM-O score may not fully capture all speaker characteristics, such as temporal features like prosody. Additionally, the relatively lower SIM-O score observed for English podcast generation may be attributed to exclusive of English conversational sources, which tend to be more prone to diarization errors.
5.2.2 Impact of Spontaneous Script
To investigate the impact of spontaneous script texts on the generation of spontaneous podcasts, we compare the generated speech using three types of input podcast scripts: 1) GT script: the ground-truth script obtained through our data preparation pipeline from the collected, unseen podcast speech. 2) Written script: We ask LLM to filter out spontaneous details from the GT script, resulting in the corresponding written version. 3) Spontaneous script: We ask LLM to reintroduce spontaneous details to the written script, resulting the corresponding spontaneous version. To ensure a fair comparison, the same text-to-speech model is consistently applied across all script variations. The comparative results of the generated audio against the ground-trut -tuut audio are presented in Table WER results are excluded due to recognition errors inherent in the ASR-derived transcripts. Our findings reveal several key insights: 1) The GT script, being the most spontaneous, achieves the highest spontaneity score. This score significantly decreases (-0.95 compared to the GT script) when spontaneous details are removed in the written script. Upon reintroducing these details in the spontaneous script, the score partially recovers, approaching that of the GT script (-0.14 compared to the GT script). This underscores the critical role of spontaneity in podcast text quality. 2) Generally, written scripts exhibit larger training-inference mismatch compared spontaneous scripts (both GT script and spontaneous script settings), often resulting in poorer performance. This evidenced by consistent performance deficit exceeding 0.3 across metrics of spontaneity, coherence, quality and similarity, further emphasizing the importance of spontaneous scripting. 3) The system consistently achieves commendable sim-o and intelligibility scores across various settings, demonstrating its robust capability for tongecontext generation. Nonetheless, we note that the intelligibility of the GT script marginally affected by recognition inaccuracies in ASR transcripts. 4) Even with the use of the GT script, there remains a noticeable disparity in the quality of our generated audio compared to the GT audio, highlighting potential areas for future research. We hypothesize that several factors contribute to this performance gap: First, the data preparation pipeline may not be perfect, as the GT script still contains some recognition and diarization errors. Second, the GT audio contains rich and diverse spontancous non-speech details, such throat clearings.

---

Conclusion
Our work presents novel solution for high-quality zero-shot podcast generation, addressing the key challenges of long speech duration and spontaneity that limit traditional text-to-speech systems. By adopting long-context language model-based audio modeling approach and integrating a podcast generation module, MoonCast effectively synthesizes spontaneous, podcast-style speech from text- only sources using unseen speakers' voices. Experiments demonstrate that MoonCast outperforms existing baselines significantly in terms of contextual coherence, and spontaneity. This approach advances the state-of-the-ard text-to-speech for long and spontaneous dialogues, paving the way for more realistic and engaging podcast generation. We discuss our limitations and future work in Appendix G
References
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint Si1i::2203.0877444 2023.
D. Adiwardana, M.-T. Luong, D. R. So, J. Hall, N. Fiedel, R. Thoppilan, Z. Yang, A. Kul- shreshtha, G. Nemade, Y. Lu, et al. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977, 2020.
Anastassiou, J. Chen, J. Chen, Y. Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430 2024.
A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33: 12460, 2020.
L. Barrault, Y.-.. Chung, M. C. Meglioli, D. Dale, N. Dong, M. Duppenthaler, P.-A. Duquenne, B. Ellis, H. Elsahar, Haaheim, et Seamless: Multilingual expressive and streaming speech translation. arXiv preprint arXiv:2312.05117, 2023.
Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour. Audiolm: a language modeling approach to audio generation. Xiv preprint arXiv:2209.0311 03343, 2022.
Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, and M. Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.096 09636.....66...... 2023.
S. Chen, S. Liu, L. Zhou, Y. Liu, X. Tan, J. Li, S. Zhao, Y. Qian, and Wei. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.05370, 2024.
Y.-A. Chung. Y. Zhang, W. Han, C.-C. Chiu, Qin, R. Pang, and Y. Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244- 250. IEEE, 2021.
C. Cieri. Fisher english training speech part transcripts Idc2004t19. Web Download, 2004. LDC2004T19.
7. Du, Q Chen, S. Zhang, K. Hu, H. Lu, Y. Yang, H. Hu, S. Zheng, Y. Gu, 7. Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. 2024a.
arXiv preprint arXiv:2407.0544
Z. Du, Y. Wang, Q Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang, al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv22412.10117. 2024b.
Z. Gao, S. Zhang. I. Mcl ooghhinn,,, and Z. Yan. Paraformer: Fast and accurate parallel transformer for mon-autoregressive end-to-end speech recognition. arXiv preprint urXiv:2206.0833 2022.
10

---

H. He, Z. Shang, C. Wang, X. Li. Y. Gu, H. Hua, L. Liu, C. Yang. J. Li, P. Shi. et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In 2024 IEEE Spoken Language Technology Workshop (SI.T), pages 885-890. IEEE, 2024.
Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang. Y. Liu, Y. Leng, K. Song, S. Tang, al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100, 2024.
R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar. High-fidelig audio compression with improved rvggan. arXiv preprint urXiv:2306.06546 2023.
M. Lajszezak, G. Cámbara, Y. Li. F. Beyhan, A. van Korlaar. F. Yang. A. Joly, A. Curtin-Cortinas, A. Abbas, A. Michalski, et al. Base tts: Lessons from building palrame- text-to-speech model on 100k hours of data. arXiv preprint ar 2ix::4422.08093 2024.
M. Le, A. Vyas, B. Shi, B. Karrer, L.. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, Ma- hadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36, 2024.
S.gg. Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon. Bigvgan: A universal neural vocoder with scage training. arXiv preprint arXiv:2206.04658, 2022.
Lewis, Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. Retrieeal-augmeente generation for nowledge-intensing nlp tasks. Advances in Neural Information Processing Systems, 33:9459-94777 9474, 2020.
H. Li, X. Zhu, L. Xue, Y. Song, Y. Chen, and L. Xie. Spontts: modeling and transferring spontaneous style for tts. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12171-12175. IEEE, 2024a.
W. Li, P. Yang, Y. Zhong, Y. Zhou, Z. Wang. Z. Wu, X. Wu, and H. Meng. Spontaneous style text-to-speech synthesis with controllable spontaneous behaviors based on language models. arXiv preprint wxxv:2407.115009 2024b.
Z. Liu, S. Wang, Inoue, Q. Bai, and H. Li. Autoregressive diffusion transformer for text-to-speech synthesis. arXiv preprint MrXiv:2406 2406.0551,, 2024.
H. Lu, G. Cheng, L. Luo, L. Zhang, Y. Qian, and P. Zhang. Slide: Integrating speech language model with Ilm for spontaneous spoken dialogue generation. arXiv preprint Sxxiv:2501100800 2025.
K. Mitsui, Y. Hono, and K. Sawada. Towards human-like spoken dialogue generation between ai agents from written dialogue. arXiv preprint 10X0022310.01088 2023.
T. A. Nguyen, E. Kharitonov, J. Copet, Y. Adi, W.-N. Hsu, A. Elkahky, P. Tomasello, R. Algayres, B. Sagot, A. Mohamed, al. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics. 1 12500266. 2023.
W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4205, 2023.
M. Poli, S. Massaroli, A. Yamashita, H. Asama, Park, and S. Ermon. Torchdyn: Implicit models and neural numerical methods in pytorch.
S. Schneider, A. Baevski. R. Collobert, and M. Auli. wav2vec: Unsupervised pre-training for speech recognition. Proc. Interspeech 2019, pages 3465-3469, 2019.
B. Schuller, S. Steidl, A. Batliner, F. Burkhardt, L. Devillers, C. MüLler, and S. Narayanan. Par- alinguistics in speech and language-Sstae-off -fate-of-tth-art and the challenge. Computer Speech & Language, 27(1 2114):4939, 2013.
K. Shen, Z. Ju, X. Tan, Y. Liu, Y. Leng, I. He, T. Qin, S. Zhao, and J. Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09116 2023.
11

---

C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li. et al. Neural codec language models are zero-shot text speech synthesizers. arXiv preprint vtXXiv2301.02111 2023.
Y. Wang, H. Zhan, L. Liu, R. Zeng, H. Guo, J. Zheng, Q Zhang, X. Zhang, S. Zhang, and Z. Wu. Maskget: Zero-shot eext-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.00750 2024.
J. Xu. Beyond goldfish memory: Long-term open-domain conversation. arviv preprint arXiv:210700766, 2021.
Y. Yan, X. Tan, B. Li, G. Zhang, Т. Qin, S. Zhao, Y. Shen, W.QQ. Zhang, and T.-Y. Liu. AdaSpeech 3: Adaptive text to speech for spontaneous style. In INTERSPEECH, 2021.
A. Yang, B. Yang, Zhang, B. Hui, B. Zheng, Yu, C., Liu, F. Huang, Wei, et al. Qwen2.
5 technical report. arXiv preprint arXiv:2412.15113 2024.
J. Yu, X. Li, J. Y. Koh, H. Zhang, R. Pang, J. Qin, A. Ku, Y. Xu, J. Baldridge, and Y. Wu. Vector- quantized image modeling with improved vegan. arXiv preprint arXiv:21 10.04627, 2021.
Yu, Y. Luo, H. Chen, R. Gu, and C. Weng. High fidelity speech enhancement with band-split rnn. 2022.
arXiv preprint arXiv:2212.0044
Yu, H. Chen, Y. Bian, X. Li, Y. Luo, J. Tian, M. Liu, J. Jiang, and Wang. Autoprep: An automatic preprocessing framework for in-the-wild speech data. In ICASSP 2024-2024 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 1 1136-1140. IEEE, 2024.
L. Zhang, Y. Qian, L. Zhou, S. Liu, D. Wang, X. Wang, M. Yousefi, Y. Qian, J. Li, L. He, et al. Covomix: Advancing zero-shot speech generation for human-like multi-talker conversations, arXiv preprint arXiv: arXiv:2404.06690 2024.
Y. Zhang, S. Sun, M. Galley, Y.C.. Chen, C. Brockett, X. Gao, J. Gao, J. Liu, and B. Dolan. DIALOGPT: Large-scale generative pre-training for conversational response generation. A. Ce- likyilmaz and T.-H. Wen, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online, July 2020. As- sociation for Computational Linguisties. doi: 1..188633v1/220 acl-demos.30. URL https //aclanthology.orgg 202/ 2020. acl demos..30
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. The abstract and introduction accurately state the paper's s main contribu- tion: proposing solution for the novel task of high-quality zero-shot podcast generation. They also correctly identify the primary challenges addressed- speech and spontane- ity-which define the scope of work. The paper then delivers these claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes. We discuss the limitations of the work in Appendix G
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [NA]
Justification: NA. The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes. As detailed in the paper, we facilitate reproducibilityy by open-sourcing MoonCast. This includes the prompts for script generation and the audio modeling module (inference code and model weights) for speech generation. This disclosure, along with the comprehensive explanation of our evaluation process and any specific configurations reported in the experiments section, allows for the main results to be independently reproduced.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
14

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Yes. As detailed in the paper, we opensource the prompts for script generation and the audio modeling module (inference code and model weights) for speech generation. We also provide the input knowledge sources adopted in the evaluation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
Saagli/c/guidees/oeeeSubbbbbbb hww.:/oubbbmsiioooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
guides/ CodeeSummiee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes. We provide the training and test details in Section 5.1 including data preparation, model details and evaluation details. We also opensource the infer code and model weights of the audio modeling module.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes. We report 95% confidence intervals for the subjective scores. These CIs are calculated using established statistical functions within the scipy package
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes. We report the type and amount of GPU resources Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the ines?
NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes. The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes. We discuss it in Appendix H
16

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.oymmly of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes. We release MoonCast accompanied by clear usage guidelines. These guidelines, which users are instructed to adhere to, are designed to prevent potential misuse of the podcast generation technology, such as the creation misleading or harmful content, and to promote responsible application.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes. We cite the original paper and provide the specific version and URLs of the existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
17

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwpsssipppsseeeee thcode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes. We will provide these details along with the release of MoonCast. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: No. The paper provides a summary of the human evaluation. While we do not plan to release these specific materials at this time, confirm that participants were ethically compensated, and the summary accurately reflects the evaluation conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
18

---

Justification: Yes. The paper states that risks to participants were minimal. Participants were informed about the evaluation task, and the study received ethical approval from our institution.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: Yes. This paper include LLM-pooered script generation module.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy huto::///oorrigeccc//offfrrreere for what should or should not be described.
19

---

A Speech Semantic Codec Details
We adopt the semantic speech tokens [Borsos et al. 2022 Wang et al. 2024 Du et al. 2024a which are discretized from the self-supervised learning (SSL) features due to the superiority of robustness [Wang al. 2024]. Following the MaskGCT [Wang et al.) 20224, , choose the 50-HZ SSL features and adopt the VQ- VAE approach to maintain the loss contained in the discrete semantic codes, thereby enhancing the reconstruction quality.
In detail, we train a VQ-VAE model to learn the discrete speech semantic representation by recon- structing the SSL. features. For the SSI. feature, we adopt the 17th layer of the pretrained W2v-BERT 2.0 [Schneider et al. 2019 Baevski et al. 2020 Chung et al., 202T, Barrault al., 2023], and normalize to mitigate the impact of varying scales across differnt feature dimensions. For the VQ-VAE model, we first encode the SSL feature S by an encoder and obtain E(S). Then we discrete the encoded feature by a VQ-VAE with codebook and obtain the quantized speech feature E(S). Finally, we apply speech decoder reconstruct the SSL. feature S with a reconstruction loss. To enhance the codebook utilization and improve the reconstruction quality, we follow the design of improved VQ-GAN (Yu et al. 2021 and DAC [Kumar al. 2023] project the E(S) into an 8-dimension latent space.
B Data Preparation
Since the raw audio data contain artifacts such as background noise, overlapping speech, and reverberation, we first apply an automated data processing pipeline as described in (Yu et al. [2224] at. [2024). Specifically, to improve speech quality,
band-split RNN speech enhancement model Yu et al. 2022] to suppress background noise. Subsequently, speech diarization is performed using the Pyannotate toolki to segment the audio into distinct speaker segments. Finally, the Paraformer ASR model Gao et al. 2022) from the FunASR toolki (T
is utilized to generate pseudo- transcriptions for each segment. The DNSMOS toolki is also employed to evaluate speech quality. based align the traucitrannsriptiion with the speech audio,
Additionally, to mitigate recognition errors introduced by the ASR system, a forced alignment system is employed using a narrow beam size of 5. Only the speech segments with successful alignment are retained for subsequent processing. For curriculum learning training. DNSMOS score greater than 2.6 to obtain single-speaker, single-turn speech. For long-context, two-speaker, multi-turn training data, select two-speaker data based on our diarization results. Specifically, for conversational sources, we retain speech data involving exactly two speakers, with more than 10 conversational turns, and where the average duration of each turn is less than 30 seconds. This process results in dataset comprising 300,000 hours from Chinese audiobook sources, 15,000 hours from Chinese conversational sources, and 200,000 hours from English conversational sources. Notably, to preserve the contextual information in the long speech data, we did not filter any segments based on DNSMOS scores or alignment results.
all speech segments with
C Comparison with Dialogue Generation Baselines
We evaluate MoonCast against two state-of-the-ar art English dialogue generation baselines: Sesame and Dia A notable limitation of these baselines is their maximum context length (2048 and 3072 tokens, respectively), which necessitates truncating long-range context during inference on podcast- length audio. As shown in Table MoonCast achieves substantially lower WER while maintaining a competitive SIM-O score, highTighting its superior ability to generate intelligible long-form audio.
http:://hhthh.oooyyyaa////aatttt git
http:///gthabbboo///dddlee/ppppu
20

---

Table 4: The performance comparison against dialogue generation baselines. Bold for the best result.
Models
SIM-O (1)
WER (L)
Sesame
0,53
2.71
Dia
0.54
3.10
MoonCast
0.53
1.81
D Ablation Study on Data Scale
To evaluate the effect of data scale, we train model subset containing only 10% of the full dataset. The results, presented in Table reveal substantial drop in performance on English podcast generation across most subjective metrics, while only slightly impacting intelligibility. This underscores the critical role of large-scale data for high- sigh-quality podcast generation.
Table 5: Ablation study on data scale for English podcast generation. Bold for the best result.
Subjective
Objective
Models
Spontaneity (1)
Coherence (f)
Inelligihility (f)
Quality (t)
Similarity (1)
SIM-O (t)
WER (1)
1/10 Data
4.18
4.23
4.58
4.18_0.12
4.12
0.50
1.94
Full Data
10
2050
1.61
4.30
4.25 4.25
0,53
1.81
E
Prompts
We choose 'Gemini 2.0 Pro Experimental 02-05 15 for script generation because of more conver- sational language style, natural dialogue design, and better topic coverage. We open-source LLM prompts to enhance reproducibility, covering brief generation and brief-to-script generation.
F.1 English Prompt For Brief Generation
### Task Description
Please summarize the input document in plain text format according to the following structure. The summary should be creative, comprehensive, and include all interesting, uncommon, and valuable vieupoints and information.
**Text Requirements*#:
1. Directly output the result without any additional information.
2. The summary should be in English. Retain small number of proper nouns, names, and abbreviations in their original form (e.g., Chinese characters).
3. Do not include any mathematical formulas.
Do not alter any proper nouns, names, or abbreviations from the original text. Unless there is common translation, do not translate proper nouns. not attempt to modify meaning of proper nouns.
5. IIItelligentll convert numbers in abbreviations. For example, "a2b" should be interpreted as "a to b," not "a two b"; "a4b" as "a for b," not "a four b"; "v2" may represent "version two" or "second generation." " Provide the original abbreviation and your suggested English translation.
### Title and Author
* Language Requirements**: English, formal written language.
21

---

**Content Requirements**: Provide the title and author document. Briefly summarize the theme of the document and the author's s background. Ensure all important information included without omission and sufficient context is retained.
### Abstract
**annuaage Requirements**: English, formal written language. **Content Requirements**:
1. What this document has done.
2. Whether similar work has been done before.
3. If similar work exists, why this document is still necessary.
4. How this document specifically addresses the topic.
5. How well this document achieves its goals.
**Additional Requirements**: Include an additional paragraph
explain any terms, concepts, or methods that may confuse readers unfamiliar with the field. Ensure proper
explained
consistently with the original text, covering all potential points of confusion, including abbreviations and entity
### Main Themes and Concepts
**Language Requirements**: English, formal written language.
**Content Requirements**: Each theme and concept should be organized according to the 3W principle:
**What**: Clearly define the problem.
**#hhy** Analyze the problem and identify its root causes.
**Hou**: Explain how the document addresses problem.
#Additional Requirements**:
1. Ensure each theme and concept comprehensive and includes all important details. Fully elaborate the "What" and "Why" sections.
2. Avoid technical details such as mathematical formulas in the "How" section. Use language that is easily understood by a general audience.
3. Ensure themes and concepts do not overlap and maintain clear logic.
4. Include an additional paragraph to explain any terms, concepts, methods that may confuse readers unfamiliar with the field. Ensure proper nouns are explained consistently with the original text, covering all potential points of confusion, including abbreviations and entity names.
### Key Citations
Language Requirements**: English, formal written language. **Content Requirements*:: Organize the content according to the following structure:
1. **Argument**: State what needs to be proven.
*#Evidence**: Provide the material used support the argument.
3. **Reasoninge*; Describe the process of using evidence to prove the argument.
Addiitoonl Requirements**:
1. Ensure all evidence and reasoning directly sourced from the original text without fabrication.
2. Ensure citation content is complete and retains sufficient context without simplification. Avoid using mathematical formulas in citations.
22

---

3. Include an additional paragraph to explain any terms, concepts, or methods that may confuse readers unfamiliar with the field. Ensure proper nouns are explained consistently with the original text, covering all potential points of confusion, including abbreviations and entity names.
### Conclusion
*anguage Requirements*:: English, formal written language. **Content Requirements**: Highlight the most important and impactful aspects of the document. Compared to the abstract, this section should provide more detailed insights related to the main themes and concepts. It may also include future directions for improvement, current application scenarios, and existing challenges.
E.2
Chinese Prompt For Brief Generation
### fE h tR EN
A RE
2.
3. 1
4. 1E
02 き
$ 2
5. ** + 73 2
b",
T> 24b for b" 1
* vvrsion - Hb
### म
**
X5 4B
R A &
### T
** #
*
1. T
2 #
3. Ao HI
4. #1 SH T
5. $
2 $
### 1
* #x. #
-+#Why** A
23

---

** -* -
***
1. 3 T - Æ
2. TFw.33448842223...33............
3.
4.
ক $ $
### H
4X. #
F
1. ** 2
2. *+ AF
*+* -1
** ***
1. 38 H
2. TT T
3.
### N1
** $ PX. P
SWN # + MI
HÚ -
E.3
English Prompts for Brief-to-Script Generation.
## 1. Task Overview
Please generate lively English podcast script based on the provided English summary text and your knowledge of the topic. The script should feature dialogue between two speakers who take turns speaking. Output format should be JSON JSON-parsable **list**, Each speaker's turn a **dictionary** containing "speaker" and "text" fields. Example format: "[[("speaker": "1", "text": "xxx"}}]' The "speaker" field indicates the speaker's identity (1 for host, 2 for guest), and the "text" field is the spoken content. Output should start directly with the JSON code block, without any extra information.
## 2. Content and Structure
### (1) Text Content
The summary text contains all important information, which needs to be comprehensively selected and incorporated into the script.
Present information through dialogue between two speakers, maintaining creativity and abstracting away unimportant details. For example, listeners aren' 't concerned with specific test names, but rather the task itself, the results, and the analysis.
24

---

### (2) Structure Design
* **Dening: ** Introduce the topic and briefly describe the discussion content, without mentioning speaker names.
**y Theme * Discuss important themes based on the summary text. Expand on the summary, don't just repeat verbatim. **Closing:** Briefly recap the discussion highlights and offer an outlook on future or technological developments.
## 3, Language Style
### (1) Conversational Style
The text should be as conversational as possible, aiming for a style similar to automatic speech recognition output. Include filler words such as 'um,' 'uh,' 'like,' 'you know,' 'so,' 'right?', and so Response words such 'Yeah,' 'Right,' 'Okay,' and similar. Conversational expressions, repetitions, informal grammar, etc. Use short sentences. Avoid directly copying and pasting structured text from the summary text. Parentheses and other symbols not typically found in speech recognition transcripts should be avoided. Spaces within sentences indicate pauses. Be aware that there might be homophone errors, potentially due accents. Questions should sound very conversational. Pay particular attention to incorporating conversational details, especially questions. For example:
{{ "speaker": "1",
"text": "Welcome back to the podcast, everyone. Today we 're diving into, uh, something that's really changing everything around us, I."
}},
{{ "speaker": "2",
"text": "Yeah, A is, like, everywhere now, isn't it? It's kinda wild to think about."
}},
{{ "speaker": "1",
"text": "Totally. And we re're seeing it in so many areas of daily life. Like, just recommending what to watch, or, you know, suggesting products online."
}},
{{ "speaker": "2",
"text": "Mhm, exactly. And it' not just online stuff, right? Think about smart homes, or even self-driving cars. It's getting pretty advanced."
}},
{( "speaker": "1",
"text": "Right, self self-driving cars are still bit futuristic for most of us, but, uh, even things like voice assistants on our phones, that's A I, isn't it?"
}},
{{ "speaker": "2" "text": "Definitely. Siri, Alexa, Google Assistant, all powered by I. It's become normal, almost don 't even think about it anymore."
}},
{{ "speaker": "1",
"text": "Yeah, it's like, integrated into everything. But is that a good thing, you think? Like, are there downsides to all this A in lives?"
}},
25

---

{{ " "speaker"; "2",
"text": "Well, that's the big question, isn't it? On the one hand, it makes things so much more convenient, saves us time, maybe even makes things safer in some ways."
}},
{{ "speaker": "1",
"text": "Safer how?"
}h,
{ "speaker": "2",
"text": "Uh, well, like in healthcare, for example. I help doctors diagnose diseases earlier, maybe even more accurately. That's huge plus, right?"
}},
{{ "speaker": "1",
"text": "Yeah, that's s a really good point. Medical applications definitely exciting. But what about the concerns, you know? Like job displacement or privacy issues?"
}},
{{ "speaker": "2",
"text": "Right, those are super valid concerns. Job displacement a big one. If can more and more tasks, what happens to human workers? And privacy,"
}),
{{ "speaker": "1",
"text": "And privacy is huge, especially with all the I systems collect. It's a lot to process."
}},
{{ "speaker": "2" "2", "text": "Exactly. So, it's not just sunshine and roses, is it? We need to be mindful of the ethical implications and make sure it's used responsibly."
}},
{{ "speaker": "1", "text": "Definitely. It's powerful tool, but like any tool, it can be used for good or, you know, not so good. It 's up to to guide its development, right?"
}},
{{ "speaker": "2"
"text": "Absolutely. And that's conversation we all need be part of, not just the tech people, but everyone."
}}
### (2) Punctuation
Use English punctuation marks. Avoid using other punctuation marks beyond commas, periods, question marks. Exclamation points are prohibited. Ellipses (' ('...'), parentheses, quotation marks (including
") or dashes are prohibited, otherwise it will considered unqualified. do not use markdown syntax. For
example, #xbold** or #italic* text should be avoided. Use plain text only.
If interrupted by the other person's response, the sentence should end with comma, not period.
## 4. Information Organization and Logic ### (1) Referencing Issues
26

---

Given that listeners won't have access summary text, any references must provide sufficient context for comprehension.
Avoid simply paraphrasing; instead, explain referenced content in your own words.
Explanations technical terms should creative avoid simply stating 'this means what?' You can use examples, metaphors, and so for explanations,
ensure you also clarify the rationale behind the metaphor. Explanations can be provided in response to question from the other speaker, or you can offer explanations proactively. Technical terms that not mentioned don't 't need explanation.
Technical terms that are mentioned don' 't necessarily need inmediate explanation; they
be explained alongside other technical terms. Technical terms in the summary text might differ slightly from the surrounding text; you'll 'Il1l need to provide reasonable explanations based on the context.
### (2) Information Density
Ensure moderate information density, avoiding excessively high or low density. The goal of appropriate information density is enable listeners without prior knowledge to quickly grasp the document's purpose, rationale, and methodology.
To prevent information overload, the script should avoid delving into details like mathematical formulas, test setups, or specific experimental metrics. Instead, it should use simple, generalized language for descriptions.
To avoid excessively low information density, ensure each topic is discussed for at least speaker turns, moving beyond simple keyword listings. Discuss topics from multiple angles whenever possible, going beyond the provided summary text. Given that the summary text is highly generalized, the script should elaborate further details. Feel free to use your knowledge to supplement background information, provide examples, and forth, to enhance listener understanding.
it and discuss
Techniques to increase information density:
1. Incorporate memorable quotes. Add impactful,
attention-grabbiing sentences to the script, either original or quotes from other sources.
2. Boost knowledge content. Judiciously add knowledge points the script to make listeners feel more informed and rewarded. Introduce novel information. Incorporate new concepts spark listener curiosity, particularly information they're unaware of but would find valuable. This crucial.
4. Employ reverse thinking. Include information from diverse angles, challenging listeners' existing perspectives and presenting alternative viewpoints.
script can offer unconventional (yet plausible) descriptions of familiar concepts to create a contrast with listener expectations. This contrast contributes to information density.
Generate contrast and impact.
Techniques to decrease information density:
1. Use short sentences: Concise and easy to understand, making the narrative more compact. Do not have too much information in sentence.
2. Describe details: Vague and abstract information makes it difficult for listeners to build understanding, while more details create a sense of imagery and are easier to read.
27

---

Use more scenario-based descriptions: Scenarios are concrete and visual. Listeners can easily receive the conveyed information and emotionally touched.
4. Talk more about facts: Talking about facts makes it more real, and readers can empathize more, thus lowering information density of the copy.
Tell more stories: Tell your own stories, stories around you, and stories you 've heard. Stories can bring listeners into the scene, making it easier to concentrate on listening.
6. Use more verbs and concrete nouns: Verbs and concrete nouns make it easier for listeners to visualize, while adjectives make complex copy harder to understand.
7. Avoid using mathematical formulas: Mathematical formulas are not conducive public understanding.
## 5. Dialogue Design
### (1) Speaker Roles
The script includes host and guest. Speaker 1 is the host, responsible for opening and closing the show, skilled at using questions to control examples to make knowledge less dry. Speaker 2 is the guest, primarily responsible for introducing the document content, has amazing knowledge reserves in the field, and is good at organizing language structured and easy-to- unddessand way.
pace of the conversation, and using vivid
Both speakers are enthusiastic and cheerful, like to combine personal stories or examples for discussion, and bring a direct experience to listeners. They are happy to discuss digressive stories.
The two speakers actively interact and frequently words such as "um" to indicate agreement with each other. Response words need to be inserted into the dialogue according to the timing. Sentences before being interrupted end with a comma, not a period. Ensure consistent speaker roles. Do not have the host introduce technical details, or have the guest guide the host to discuss topics.
interruption
The host gradually increases their understanding of the field based the guest's answers. However, the host may not understand immediately or completely correctly. The host can express misunderstanding or raise some questions that ordinary people might have. In this case, the guest will further explain in more accessible language, or specifically answer common questions misunderstandings. This kind of interaction is more realistic and easier for listeners to understand than always correct hosts and guests.
### (2) Topic Order Arrangement
The host will arrange the topics according to the summary text and ensure logical connections between topics, such as transitioning from overall to details, from details to overall, from cause to effect, from technology to application, etc.
The host will guide the pace of the conversation and discuss topics in the order of the summary text. Guests should not interfere with topic transitions.
### (3) Knowledge Rate
The knowledge rate in the script needs be reasonable. Do not introduce a large amount of knowledge too quickly in a short period of time. Knowledge
28

---

## Other Requirements ### (1) English Numbers and Foreign Words
The script will used for English podcast content recording. Please ensure most numbers and foreign words are rendered naturally English to facilitate correct pronunciation.
2. Please intelligently determine the correct promunciation according the context. For example, "2021" if expressing a year, should be converted to "two thousand and twenty-one" or "twenty twenty-one". But if expressing a number, it should be "two thousand and twenty-one". For some uncommon English abbreviations, if the pronunciation needs to be read letter by letter according to the context, you must ensure that there is space between each letter, such as "AI" adding space as "A I", to avoid the model misinterpreting it as "A P I"
word. For example, "API" should be rendered
3. Small amount of Chinese is allowed, especially for nouns, if it fits naturally within the conversational English context.
### (2) Script Length
1. Please ensure that the total length of the 'text' values does not exceed 3,000 words and the number of speaker turns kept within 60, otherwise it will be unqualified. Please choose technical details and topic concepts to discuss. depth of discussion on each topic for the sake of word limit, do not be limited to the summary text, and give full play your knowledge.
not shorten
INPUT: (BRIEF)
## Re- Re-emphasize:
Speaker 1 is the host, and Speaker is the guest. Neither speaker has name. The script text only commas, periods, and question marks. Use English punctuation marks. Avoid using other punctuation marks beyond commas, periods, and question marks. Exclamation points are prohibited. Ellipses ('...'), parentheses, quotation marks ") dashes considered unqualified. Please prioritize in-depth discussion for each topic. Don's 't limit yourself to the summary text; instead, your knowledge to expand upon the topics, providing background information and illustrative examples enhance listener understanding.
(including
prohibited, otherwise it will be
Ensure that numbers and foreign words are rendered naturally in English for accurate pronunciation during recording. In technical contexts, English abbreviations sometimes use numerical digits place of words (e.g., "a2b" for "a to b," "a4b" for "a for b"). Please translate these abbreviations based on the context. While the script is primarily in English, small amount of Chinese, especially for nouns, acceptable if integrates naturally into the conversational flow.
appropriate English phrases
OUTPUT:
29

---

E.4
Chinese Prompts for Brief-to-Script Generation.
## 1
KM , サ
XX "
JSON +| 31
A "[{("speaker" "1", "text": "xxx")]]' "
( 2
"
M
## - #
### (-) ス
X >
kn,
### (3) ビ Htt.://AHHHAAEEEEEE.2  .TEAAAEEA
## -. A
T "
12
イ | 1U 1E 12 /
मा F:
{ "speaker": "0" "text": " 72 "
},,
{{ "speaker": "1"
"text": ".
}},
{{ "speaker": "0"
"text": "M.... st. HA A 1T
},,
{{ "speaker": "1"
"text": "M #R 14 B 1??
}},
{{ "speaker": "0" "text": "Xt. "
},,
{{ "speaker": "1", "text": "#B R
}},
{{ "speaker": "0"
"text": " Н :
}},
{ "speaker": "1",
30

---

"text": "text": 32 EEE
:
},,
{{ "speaker": "0"
"text": "M. X 2 MM 7
},,
{{ "speaker": "1"
"text": "1S. xt. ",
}},
{ "speaker": "0",
"text": "1g
}),
{{ "speaker": "1"
"text": "2 E SIRE. #
? "
}),
{{ "speaker": "0",
"text": "Xt, 2
},,
{ "speaker": "1", "text": "Mß?
},,
{( "speaker"; "0"
},,
{ "speaker": "1" "text": "*t 1
}),
{{ "speaker": "0", "text": "*t 3 B T SA
}), "speaker": "1"
{{
"text": SH
A A EE 1 HA *
}},
{{ "speaker": "O" "text": "Ra 19 И -
}},
{{ "speaker": "1" "text": "øt Mi A 2 W
}},
{{ "speaker": "0"
"text": " m 1. 2
},,
{{
"speaker": "1"
"text": "#C "B
}}
31

---

###
;
1 3 (615 77
## M
###
:
R
.)
### #
T 12 M 10
$
M #
1. #)
IE. セП H.
2. :
3. 3. -
4. 1
A.
5. 34 ( A
1. 4): M TT
1 - 6 m
2. # T
M $ #2
3. 7
8 -
4. $
ННН 8 -
1
5. $$ # ; W M
# 11
6. HII
7. A ##ATТ
32

---

#* -
#*# (-) A
SAT K
A
R
A SH 2U the
###
C)
#A F
:
### ST
AA 1
i4 TA SAR EAAEE #
オ ### (-) #
1. BU D , 2. 3| MY
24 to b" "a4b"tt"" for b",
3. 3F
TV I"
### ()) K
1. Tttp::/0hHHHHHHA  o  o oo   .1 0 # R M SE
INPUT:
{BRIEF)
33

---

" b" "aab" for ""
b". # M #К
DUTPUT:
F
Discussions
F.1 Phoneme vs. BPE for Text Representation
Traditional TTS systems tend use phonemes as the text representation enhance intelligibility, but this pronunciation-bast approach strips away semantic information needed for long-form, multi- speaker scenarios, hindering natural speaker transitions, emotion, and prosody. In contrast, we opt for BPE, which preserves semantic content and aligns with the text representation used in LLMs. thereby enabling more straightforward future integration. Empirically, BPE maintains intelligibility while improving prosody and spontaneously generating paralinguistic phenomena like laughter based on context.
F.2 Hallucination Issues
We observe that hallucinations sometimes occur in the generated speech, that the synthesized output may confuse the identity of speakers, leading to incorrect attributions of utterances. These issues stem from the interplay of three main factors: First, the semantic tokens retain some timbre information, enabling reconstructed speech to deviate from the prompt's s timbre. Second, the data pipeline may introduce errors, such as speaker identification errors or diarization errors, especially in distinguishing rapid transitions between speakers. Third, ambiguous text interpretations complicate the generation fprocess. For example, the sentence "Today, we're discussing climate change um and its impact on global biodiversity.' can be interpreted in several ways. It might be understood as single speaker using 'um' as a filler, such as: 'Host: Today, we're discussing climate change, um, and its impact global biodiversity.' Alternatively, it could be interpreted as dialogue between two speakers, such as: 'Host: Today, we' re discussing climate change. Guest: Um. Host: And its impact on global biodiversity." Therefore, the model struggles to determine whether the 'um' is filler word from the same speaker, or a response word from another speaker, even with adequate semantic understanding. Additionally, we find that the trade-off between increasing sampling diversity (i.e.. increasing temperature, top-k and top-p values) to enhance spontaneity and the consequent aggravation of hallucinations restricts the model's 's ability achieve higher levels of spontaneity.
G Limitations and Future Works
Despite our proposed system has achieved great progress, we still have the following limitations:
Language Coverage.
The current system is limited to Chinese and English. Future work should focus on expanding language coverage to support multiple languages to enhance the system's applicability in diverse linguistic contexts.
The system is currently designed for two-person interactions. Extending it to handle multi-person conversations is an important direction for future development to accommodate more complex and dynamic conversational scenarios.
Multi-Speaker:
Data Quality.: The data pipeline currently generates data that may contain errors in ASR and diarization. In addition, current data preparation pipeline filters out the overlapped part. To address
34

---

these challenges, future work should prioritize the use of high-qualing human rimman-annotatdd spontaneous speech for fine-tuning.
H Broader Impacts
Given our model's s ability generate speech with high fidelity to the original speaker's 's voice, there is a risk of improper application, including deceptive voice recognition or mimicking an individual's speech. To mitigate potential abuse, it is crucial to devise reliable method for detecting synthetic speech and implement mechanism for flagging suspected malicious use.
35


Reviews:
[{"review_id": "Frq1HvoLxi", "reviewer": "Reviewer_U3tD", "summary": "The paper proposes a podcast-generation framework trained on a large-scale corpus of conversational speech, enabling spontaneous two-speaker dialogue synthesis. To model spontaneous dialogue, the system uses ASR transcripts, containing natural filler words such as “yeah,” “um,” and “uh\", as a proxy for spontaneous scripts during training. At inference time, a large language model generates spontaneous dialogue scripts from knowledge sources such as URLs, PDFs, or plain text. These scripts serve as input for the audio model.\n\nTo support long-form podcast generation, the model leverages the long-context modeling capability of Transformer to produce text-to-semantic token sequences for audio segments up to 800 seconds. These semantic tokens are converted into mel-spectrograms using a chunk-wise causal flow-matching model, followed by a vocoder to generate waveforms. The text-to-semantic module is trained to generate both speakers' utterances using speaker turn tokens and speaker-specific transcripts. Because long-form spontaneous speech data is limited, the model is trained using a curriculum learning strategy. The full pipeline is trained on approximately 500,000 hours of data, and both subjective and objective evaluations show that it outperforms CosyVoice 2 and a simple concatenation baseline in generating spontaneous dialogues.", "questions": "To what extent is the large-scale training data necessary for generating spontaneous speech from ASR transcripts? Could similar results be achieved with less data if more advanced modeling were used?\n\nSpeaker turn tokens are inserted based entirely on diarization. Is there any empirical evidence showing that these tokens remain robust and effective in the autoregressive text-to-semantic module, especially in cases of noisy diarization or frequent speaker changes?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe model effectively reflects spontaneous characteristics in generated dialogues, producing speech that includes filler words and natural turn-taking.\n\nThe architecture is carefully designed to enable generation of long-form dialogues exceeding 10 minutes, and demo samples sound natural and coherent.\n\nThe use of LLM-based spontaneous script generation makes it possible to turn arbitrary knowledge sources into conversational audio, opening applications similar to systems like NotebookLM.\n\n**Weaknesses**\n\nThe system requires a substantial amount of data and compute resources to train.\n\nThere may be a mismatch between the ASR transcripts used as training targets and the LLM-generated spontaneous scripts used at inference time, which is not addressed in the analysis.\n\nBaseline comparisons are limited to CosyVoice 2 and simple concatenation; comparisons with existing dialogue-generation models such as Sesame or Dia are not included, which could further highlight the model’s advantages.", "was_revised": true, "final_justification": "The authors have addressed my concerns regarding data scale and baseline comparisons. Including this analysis in the appendix would aid understanding. I maintain my score as \"borderline-accept\".", "rebuttal": "First of all, we want to thank the reviewer for your careful reading and providing a lot of constructive comments! Below we address the concerns mentioned in the review. \n\n**Weaknesses-1. The system requires a substantial amount of data and compute resources to train.**\n\nA1.  Thanks for your question. We acknowledge that our model's substantial resource requirements present a challenge. This is a design trade-off motivated by the proven success of scaling in the zero-shot TTS domain, as shown by models like VALL-E and NaturalSpeech 2. To mitigate this barrier for the community, we plan to open-source our pre-trained model. This will provide a resource-efficient pathway for future works to build upon our work by fine-tuning on smaller datasets instead of training from scratch. By taking this step, we hope to make the benefits of our large-scale approach more accessible. While our current model aligns with the scaling paradigm, enhancing TTS performance under intrinsic resource constraints remains a crucial and open research question that we intend to explore in future work.\n\n**Weaknesses-2. There may be a mismatch between the ASR transcripts used as training targets and the LLM-generated spontaneous scripts used at inference time, which is not addressed in the analysis.** \n\nA2. It is an excellent point. We analyze this potential mismatch in Section 5.2.2 (Table 3), where we compare outputs from ASR transcripts (GT Script) with those from LLM-generated text (Spontaneous Script). Specifically,  the overall performance is highly consistent between the two settings. While the ASR transcripts yield better spontaneity,  the LLM-generated scripts offer better intelligibility. This demonstrates that the mismatch does not significantly degrade performance.\n\nTo further bridge this training-inference gap, we incorporate ASR transcripts as demonstrations within the LLM prompt (see Appendix C). This strategy guides the language model to generate outputs that more closely align with the stylistic properties of our training data.\n\n**Weaknesses-3. Baseline comparisons are limited to CosyVoice 2 and simple concatenation; comparisons with existing dialogue-generation models such as Sesame or Dia are not included, which could further highlight the model’s advantages.**\n\nA3. We thank the reviewer for this valuable suggestion. We did not initially include Sesame (released March 13) or Dia (April 22), as they are considered concurrent work under the NeurIPS policy (post March 1). Furthermore, as these systems were released as engineering projects without formal academic publications, a rigorous methodological comparison was challenging at the time. \n\nHowever, we agree that this comparison is beneficial and will conduct a new evaluation following the reviewer's suggestion. A critical challenge in this comparison is that both Sesame and Dia have significant constraints on their context length (2048 and 3072 tokens, respectively), which is far shorter than what is required for generating long-form podcasts. To accommodate these baselines, we have to discard parts of the long-range context during inference.  The results shown in the table below demonstrate that while maintaining competitive Sim-O, our model achieves a substantially lower WER, highlighting its superior ability to generate intelligible long-form audio. \n\n| Model  | Sim-O (↑) | WER (↓)|\n|-|:-:|:-:|\n| Sesame | 0.53 | 2.71  |\n| Dia    | 0.54 | 3.10  |\n| Ours   | 0.53 | 1.81  |\n\n\n**Questions-1. To what extent is the large-scale training data necessary for generating spontaneous speech from ASR transcripts? Could similar results be achieved with less data if more advanced modeling were used?**\n\n\nA4. Thank you for this critical question. Our experiments provide a clear rationale for our focus on data scale. As shown in Table below, We observe that reducing the training set by a factor of ten results in a marked decline in speech spontaneity and coherence, a decline in speaker similarity and speech quality, and a minor decline in intelligibility.\n\n|  | Spontaneity (↑) | Coherence (↑) | Intelligibility (↑) | Quality (↑) | Similarity (↑) | SIM-O (↑) | WER (↓) |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 1/10 Data | 4.18±0.14       | 4.23±0.15     | 4.58±0.13           | 4.18±0.12   | 4.12±0.15      | 0.50      | 1.94    |\n| Full Data | **4.54±0.16**       | **4.50±0.15**     | **4.61±0.12** |**4.30±0.10**   | **4.25±0.18**      | **0.53**      | **1.81**    |\n\n\n\nWhile advanced modeling has yet to offer a full substitute for large-scale data, we believe our plan to open-source the pre-trained model provides a practical way to mitigate this challenge. This will provide the community with a resource-efficient pathway, to fine-tune on smaller, high-quality datasets without the substantial cost of training from scratch.\n\n**Questions-2. Speaker turn tokens are inserted based entirely on diarization. Is there any empirical evidence showing that these tokens remain robust and effective in the autoregressive text-to-semantic module, especially in cases of noisy diarization or frequent speaker changes?**\n\nA5. Thanks for your insightful question. Our reported Word Error Rate (WER) provides direct empirical evidence for this. The WER is calculated on speaker turns that are segmented using these predicted speaker turn tokens. The low WER we achieve therefore serves as strong validation for their robustness, as significant errors in turn prediction would invariably lead to incorrect segmentation and a much higher error rate. Additionally, our data pipeline includes denoising and post-processing stages to mitigate noisy diarization issues."}, {"review_id": "3GKOr3fgHj", "reviewer": "Reviewer_mbG2", "summary": "The paper introduces a long-form, high-quality, multispeaker podcast generation system, which can generate contextually coherent multi-turn long podcasts. They employ a DAC [Kumar et al., 2023] type discrete tokenizer over the 17th layer of pretrained W2vBert 2.0 [Schneider et al., 2019]. Then they use this encoded representation for further training a 2.5B llama model. They employed a data preprocessing pipeline [Yu et al., 2024] to prepare 500k hours of data. They employ a three-stage learning process where the first two stages are trained with a Chinese-only dataset, and the third stage is employed using English and Chinese datasets. The first stage is similar to training of Vall-E [Wang et al., 2023], the second stage is pretraining on chinese audiobooks and the last stage is pretraining on both english and chinese podcast data. The turns during the podcasts are interleaved with speaker ID tags to learn better turn-taking and multi-turn prosody. Finally, a causal speech de-tokenizer similar to CosyVoice 2 [Du et al., 2024b] was employed to transform speech representation to a mel-spectrogram, allowing for generating long audio chunks.", "questions": "- In Section 4.1.1 subheading - Sequence Design:\nIt is a bit unclear with the text where the speaker change token is added, it says\n>Line 165: incorporate a special speaker change token after prompts and each podcast turn.\n\nWhile in the concatenation $\\mathcal{S}$  does not specifies how it is added Line 172-173.  Is it like in the Figure 2? (If yes please update this section)\n\n- The speech tokeniser uses RVQGAN and generates 8 dimensional codes. How do you model these 8 dimensional codes with the LLM? Is it just the first dimension is modelled or all 8?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* First open the weight models for multiturn multispeaker podcast generation \n* High-quality listening examples improvements over concatenative or single-speaker baselines\n* Uses ASR-based weaker labels as a proxy for spontaneous annotated data\n\nWeakenesses:\n- The paper lacks novelty; the data was preprocessed using [Yu et al., 2024] pipeline, the tokenizer was similar to [Kumar et al., 2023]. The spontaneous dataset was generated similarly to SLIDE [Lu et al., 2025]. Multistage training is a common approach to pretraining and fine-tuning on the required dataset, and the speech de-tokenizer is similar to CosyVoice 2 [Du et al., 2024b]. However, the authors do stitch them together very well and scale them to 500k hours of data. \n- The authors make a novel assumption: \"ASR transcripts can act as a partial proxy for speech spontaneity.\". However, it is not formally validated; it would have been nice to quantify this assumption on a small subset for which they have ground truth text transcriptions to see how much of the spontaneity is retained after ASR transcription. This would validate this assumption.\n- Subjective evaluations were performed by 10 raters, that is number is low for such long generations. Further, the author's evaluation of long podcasts is done similarly to short sentence evaluation, while this should not be the case, i.e, one number for a few minutes of audio. It should be performed over time; one way to do this was proposed in [1].\n- Some evaluation details are lacking, like the question asked for subjective evaluation (especially for coherence), how the confidence intervals were calculated for subjective evaluation, as in Table 3, the confidence intervals are around 0.09, which seems to be hard to get with 10 raters.\n\n[1] Park, S.J., Salazar, J., Jansen, A., Kinoshita, K., Ro, Y.M. and Skerry-Ryan, R.J., 2024. Long-Form Speech Generation with Spoken Language Models. _arXiv preprint arXiv:2412.18603_.", "was_revised": true, "final_justification": "There are still questions around novelty and a little over-exaggeration of hyperparameter tuning. However, the added experimentation during the rebuttal period made me change the score, as initially the experiments were weaker and more details around them were required. I still feel that for the initial paper submission, the confidence intervals are hard to get, but the experiments done during the rebuttal seem to be adequate enough.", "rebuttal": "First of all, we want to thank the reviewer for your careful reading and providing a lot of constructive comments! Below we address the concerns mentioned in the review. \n\n**Weaknesses-1. The paper lacks novelty; the data was preprocessed using [Yu et al., 2024] pipeline, the tokenizer was similar to [Kumar et al., 2023]. The spontaneous dataset was generated similarly to SLIDE [Lu et al., 2025]. Multistage training is a common approach to pretraining and fine-tuning on the required dataset, and the speech de-tokenizer is similar to CosyVoice 2 [Du et al., 2024b]. However, the authors do stitch them together very well and scale them to 500k hours of data.**\n\nA1.  We appreciate the reviewer's detailed feedback. While we build upon a strong foundation of prior research, we would like to clarify that our work is not merely an assembly of existing methods. We introduce several critical innovations at each stage to address the unique challenges of generating long-form, multi-speaker podcast speech.\n- Data Processing Pipeline: The pipeline from [Yu et al., 2024] is designed for single-speaker, short-form TTS, where low-quality segments can be filtered. This is not feasible for long-form dialogue, where discarding any segments would destroy conversational continuity. We therefore develop a novel, multi-stage diarization post-processing pipeline to enhance robustness. This includes 1) speaker cluster merging to handle the speaker fragmentation (i.e., assign multiple speaker labels to the same actual speaker). 2)  chunk-based reassignment to handle segments containing multiple speakers, and 3) segment merging to handle  impractical sequence lengths (shorter than 1s or longer than 100s).\n- Semantic Tokenizer: There is a fundamental distinction here. The tokenizer in [Kumar et al., 2023] operates on the raw audio waveform to produce **acoustic** tokens. In contrast, our tokenizer is designed to process self-supervised features to extract **semantic** tokens.\n- Spontaneous Speech Data: In fact, we did not **generate** any spontaneous dataset. Instead, we leverage the **raw** spontaneous speech and its corresponding ASR transcript from our large-scale raw data.\n- Multi-Stage Training: Our training strategy is not a general pretraining-and-finetuning schedule. It is a purpose-built curriculum learning where each stage addresses a distinct learning objective essential for conversational TTS: from foundational zero-shot ability, to long-term context modeling, and finally to the nuances of spontaneous speech.\n- Speech De-tokenizer: Our de-tokenizer is also distinct from that of CosyVoice 2, as it features a novel chunk-wise causal mask method that we develop specifically to accelerate the training of our architecture.\n\nThese targeted innovations were integral to successfully scaling our model and achieving robust performance in the complex domain of multi-speaker conversational speech. While previous TTS research has mainly focused on short, single-speaker tasks, our model is one of the first academic works to tackle long-form, multi-speaker generation, highlighting the novelty of our contribution to this emerging field.\n\n\n**Weaknesses-2. The authors make a novel assumption: \"ASR transcripts can act as a partial proxy for speech spontaneity.\". However, it is not formally validated; it would have been nice to quantify this assumption on a small subset for which they have ground truth text transcriptions to see how much of the spontaneity is retained after ASR transcription. This would validate this assumption.**\n\nA2. Thanks for your excellent insights! To validate our novel assumption, we conduct a quantitative analysis. We select a one-hour subset of podcast audio and perform a manual transcription to obtain the GT transcript. Subsequently, we manually count the occurrences of spontaneous speech elements, including filler words, response words and repetitions & colloquialisms, in both the GT and the corresponding ASR transcripts. The results, detailed in the table below, show that the ASR output successfully retains a significant majority of these elements, providing strong evidence to validate our assumption.\n\n\n||Filler Words|Response Words|Repetitions & Colloquialisms| \n|:---:|:---:|:---:|:---:|\n|GT Transcript|425|191|94|  \n|ASR Transcript |359|164|88|\n\n\n\n**Weaknesses-3. Subjective evaluations were performed by 10 raters, that is number is low for such long generations. Further, the author's evaluation of long podcasts is done similarly to short sentence evaluation, while this should not be the case, i.e, one number for a few minutes of audio. It should be performed over time; one way to do this was proposed in [1].**\n\nA3. Thank you for your valuable feedback. We agree that evaluating long-form generation is a nuanced task. While the over-time method proposed in [1] is insightful, it is primarily designed for single-speaker scenarios. Our initial choice of assigning a single score to the entire podcast was primarily to mitigate the impact of speaker diarization errors during evaluation.\n\nHowever, we recognize the importance of the point you raised. To address this, we conduct a new evaluation with 10 knowledge sources and 20 raters on English podcast generation. In this round, the audio is segmented by speaking turns, and each turn is rated individually. As the results presented below indicate, this more granular, turn-by-turn evaluation aligns with the findings reported in our main paper.\n\n| | Spontaneity(↑) | Coherence(↑) | Intelligibility(↑) | Quality(↑) | Similarity(↑) | SIM-O(↑) | WER(↓) |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Cosyvoice2| 3.75±0.04 | 3.80±0.04 | 4.55±0.03 | 4.35±0.04 | **4.54±0.02** | 0.76 | 2.53 |\n|Concat Baseline| 3.62±0.03 | 3.65±0.04 | 4.12±0.03 | 3.65±0.03 | 4.01±0.03 | **0.78** | 2.36 |\n|Our System| **4.60±0.02** | **4.53±0.03** | **4.61±0.02** | **4.38±0.02** | 4.22±0.04 | 0.56 | **1.69** |\n\n\n**Weaknesses-4. Some evaluation details are lacking, like the question asked for subjective evaluation (especially for coherence), how the confidence intervals were calculated for subjective evaluation, as in Table 3, the confidence intervals are around 0.09, which seems to be hard to get with 10 raters.**\n\nA4. Thanks for your question. We are happy to provide the following clarifications. For the subjective evaluation, raters were asked to assess the generated audio based on the following questions.\n- Spontaneity: Assesses how spontaneous the audio sounds. This focuses on the presence of common spoken elements (e.g., filler words such as 'um,' 'uh,' 'like,' 'you know,' 'so,' 'right?', and response words such as 'Yeah,' 'Right,' 'Okay') as well as a fluid and casual flow. Rated on a 1-5 scale with 0.5-point increments, where 5 is best.\n- Coherence: Evaluates the overall coherence of the audio. This includes the naturalness of transitions between speakers and whether the audio is globally consistent. Rated on a 1-5 scale with 0.5-point increments, where 5 is best.\n- Intelligibility: Measures whether the speech is clearly articulated and consistent with the provided text. Rated on a 1-5 scale with 0.5-point increments, where 5 is best.\n- Quality: Assesses the overall quality of the audio. Rated on a 1-5 scale with 0.5-point increments, where 5 is best.\n- Similarity: Determines if the speaker's voice is consistent with the voice in the provided reference audio. Rated on a 1-5 scale with 0.5-point increments, where 5 is best.\n\nRegarding the confidence intervals in Table 3, they are calculated by pooling together all individual ratings from all raters across every sample in the test set. We hope this clarifies our procedure.\n\n\n**Questions-1. In Section 4.1.1 subheading - Sequence Design: It is a bit unclear with the text where the speaker change token is added, it says \nLine 165: incorporate a special speaker change token after prompts and each podcast turn.\nWhile in the concatenation  does not specifies how it is added Line 172-173. Is it like in the Figure 2? (If yes please update this section)**\n\nA5. We apologize for the ambiguity in the original paper. We will revise the text in that section to explicitly reference the figure, ensuring the description of the concatenation process is clear. We appreciate you bringing this to our attention.\n\n**Questions-2. The speech tokeniser uses RVQGAN and generates 8 dimensional codes. How do you model these 8 dimensional codes with the LLM? Is it just the first dimension is modelled or all 8?**\n\nA6. Thank you for the question. There appears to be a misunderstanding based on the text in our paper, \"The 1024-dimensional SSL feature is projected into an 8-dimensional space for quantization using an 8192-entry codebook.\" In fact,  we do not model 8-dimensional residual VQ codes or a 8-dimensional latent with the LLM. The \"8-dimensional space\" refers to the latent dimensionality of each vector within the codebook, not the output of the quantization process. Such expressions are consistent with previous works [1]. The quantizer itself outputs a single, discrete index from this codebook for each speech frame. Consequently, the LLM models the sequence of these discrete indices, not the 8-dimensional latent vectors. \n\n[1] Kumar, Rithesh, et al. \"High-fidelity audio compression with improved rvqgan.\" Nips 2023."}, {"review_id": "js5CLVyPeP", "reviewer": "Reviewer_DLz7", "summary": "This paper proposes a pipeline for generating podcasts, specifically focusing on conversations between two speakers. The pipeline is designed to generate a podcast script from given content, convert the script into a speech code, and then reconstruct the code into speech. It integrates various components such as zero-shot generation, curriculum learning, and chunk-wise generation to form a cohesive system. The authors present their main experimental results in the main text.", "questions": "The questions I had were addressed in the Weaknesses.", "limitations": "The limitations of the work are well documented in the Appendix.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### **Strengths**\n\n**(S1)** To achieve their objectives, the authors propose and incorporate several components, combining original ideas with existing methods. They aim to demonstrate the effectiveness of their model and contribute to the research community through a demo and a promise to release the code, although it has not yet been made publicly available.\n\n**(S2)** The description of their methodology is written in a clear and accessible manner, and the various figures included throughout the paper effectively support the reader’s understanding.\n\n### **Weaknesses**\n\n**(W1)** To achieve their objectives, the authors incorporate various elements, including the adoption of techniques such as chunk-wise flow matching models, also employed in other recent studies (e.g., CosyVoice 2, 2024.12), and the use of a large language model (Gemini 2.0 Pro) for podcast script generation. They also propose several techniques, including sequence design and curriculum learning. While these proposed components are promising, it is unfortunate that their individual contributions and effectiveness are not clearly demonstrated through experiments, either in the main paper or the Appendix.\n\n- **(W1-1)** What kind of performance degradation occurs if the model is trained directly without the proposed curriculum learning? Demonstrating such differences could help highlight the effectiveness of the components proposed in the paper.\n\n- **(W1-2)** Similarly, I am curious about the advantages of the interleaving approach you adopted compared to the turn-based interleaving method mentioned in line 163, particularly in terms of performance, sample quality, or perceptual (auditory) improvements.", "was_revised": true, "final_justification": "I agree with the overall novelty of the framework proposed by the authors. However, I had pointed out the insufficient proof of effectiveness for several of its constituent elements and requested relevant experiments. The authors have addressed this, and therefore, I am changing my score to positive.", "rebuttal": "First of all, we want to thank the reviewer for your careful reading and providing a lot of constructive comments! Below we address the concerns mentioned in the review. \n\n**Weakness-1. To achieve their objectives, the authors incorporate various elements, including the adoption of techniques such as chunk-wise flow matching models, also employed in other recent studies (e.g., CosyVoice 2, 2024.12), and the use of a large language model (Gemini 2.0 Pro) for podcast script generation. They also propose several techniques, including sequence design and curriculum learning. While these proposed components are promising, it is unfortunate that their individual contributions and effectiveness are not clearly demonstrated through experiments, either in the main paper or the Appendix.**\n\nA1. Thank you for your insightful question. First, we wish to clarify our contributions, as our paper is one of the first academic works to systematically address long-form, multi-speaker podcast generation. Our academic contributions include 1) proposing a robust pipeline for this task with novel script and audio generation modules; 2)identifying a key blind spot in the current TTS field, that is, the overlooked impact of text quality on speech spontaneity; and 3) introducing a chunk-wise causal mask to accelerate our speech detokenizer's training. On the engineering side, we 1) validate the powerful potential of scaling in this new TTS research direction and 2) plan to open-source our entire pipeline, including LLM prompts and audio module, to ensure reproducibility and establish a foundational baseline for this emerging research area.\n\nRegarding component-wise evaluation, we conduct a crucial ablation study in Section 5.2.2 (Table 3), which validates our most central insight about the impact of  spontaneous text script. We acknowledge that we do not perform an exhaustive ablation of every single component, as our overarching strategy was to present a simple enough yet highly effective system to demonstrate the immense potential of this research direction as a whole.\n\nTo further prove our system's holistic advantage, we conduct a new comparison against current SOTA spontaneous conversational TTS models, including Sesame and Dia,  during this rebuttal period. A critical challenge in this comparison is that both Sesame and Dia have significant context length constraints (2048 and 3072 tokens, respectively), forcing them to discard parts of the long-range context during inference for podcast-length audio. As shown in the results below, this highlights our system's superior ability to generate intelligible long-form audio (significantly lower WER) while maintaining competitive speaker similarity.  We leave a more granular exploration of optimal architectures for this new task as a promising direction for future work\n \n| Model  | Sim-O (↑) | WER (↓)|\n|-|:-:|:-:|\n| Sesame | 0.53 | 2.71  |\n| Dia    | 0.54 | 3.10  |\n| Ours   | 0.53 | 1.81  |\n\n\n**Weakness-1_1. What kind of performance degradation occurs if the model is trained directly without the proposed curriculum learning? Demonstrating such differences could help highlight the effectiveness of the components proposed in the paper.**\n\nA2. Thank you for your question. We find that the proposed curriculum learning is essential for stable model convergence. Our preliminary experiments show that if we train the model directly on spontaneous data while skipping the audiobook data stage, the model fails to converge within 2000 steps. This result confirms the necessity of our curriculum learning approach. This initial stage builds an essential foundation, preparing the model for stable convergence on the more challenging spontaneous speech data.\n\n**Weakness-1_2.  Similarly, I am curious about the advantages of the interleaving approach you adopted compared to the turn-based interleaving method mentioned in line 163, particularly in terms of performance, sample quality, or perceptual (auditory) improvements.**\n\nA3. That is an interesting question. We did, in fact, compare these two interleaving approaches on a very small scale during our preliminary experiments. The results, detailed in the table below, show that while objective metrics like SIM-O and WER are comparable, our proposed full-text-to-full-audio interleaving approach yields subjectively superior results, particularly in terms of coherence and spontaneity.\n\n| Method | Spontaneity (↑) | Coherence (↑) | SIM-O (↑) | WER (%) (↓) |\n|-|:-:|:-:|:-:|:-:|\n| Turn-based Interleaving | 4.33 | 4.16 | **0.81** | **1.80** |\n| Our Full-text-to-full-audio Interleaving | **4.58** | **4.50** | 0.80 | 1.83 |\n\nWe hypothesize two primary reasons for this advantage. First, given the strong text-to-speech mapping, the holistic nature of our interleaving approach inherently guides the model to capture **long-range, global dependencies**, as the corresponding text and speech segments are often distant from each other. In contrast, a turn-based method biases the model towards **local, turn-level dependencies**, which can result in less coherent output over an entire dialogue. Second, when generating the speech for any given turn, our interleaving approach allows the causal model to attend to the **complete** text transcript of the entire podcast. A turn-based method, however, provides only a **partial** context, as the model cannot attend to the text of subsequent turns. We believe this complete contextual view is a key factor in the improved coherence."}, {"review_id": "PgoclaoHP1", "reviewer": "Reviewer_biCE", "summary": "This paper introduces a pipeline for podcast generation, comprising two main modules: 1) a script generation module that transforms text sources (e.g., news website URLs) into podcast scripts, and 2) an audio modeling module that converts these scripts into audio podcasts. The authors argue that existing text-to-speech (TTS) systems often struggle with generating long, complex podcasts that include spontaneous speech. To address these challenges, they propose pre-training a long-context audio model to enhance contextual coherence and generate spontaneous speech based on automatic speech recognition (ASR) transcripts. During training, the audio model is conditioned on prompt text, speaker prompts, and podcast scripts to model audio tokens. The training process employs a curriculum learning strategy, beginning with single-speaker, single-turn TTS tasks and progressing to long-context, multi-speaker audio sequence modeling. Experimental results demonstrate that the proposed pipeline outperforms baseline methods, such as CosyVoice2, in human evaluations of intelligibility and speech quality.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The framework is novel in addressing the challenges of generating long, complex podcasts that incorporate spontaneous speech.\n- A long-context TTS model, pre-trained with half a million hours of audio, is released. This model is capable of generating podcast audio of up to approximately 13 minutes.\n\nWeaknesses:\n- The details of the large language model (LLM) used in the script generation module are not thoroughly discussed in the main paper (hidden in a small section in appendix).\n- Several details are missing in the evaluation:\n   - The number of samples in the evaluation set and the distribution across different domains.\n   - The definition and rating guidelines for all subjective metrics.\n   - The evaluation process lacks clarity: How many raters were hired? How many reviews were conducted per evaluation sample?\n- Evaluation comprehensiveness: podcast generation latency and the factualness of generated podcast is not discussed in the paper.", "was_revised": true, "final_justification": "Most of my concerns have been addressed by the author and I've adjust my score accordingly", "rebuttal": "First of all, we want to thank the reviewer for your careful reading and providing a lot of constructive comments! Below we address the concerns mentioned in the review\n\n**Weakness-1. The details of the large language model (LLM) used in the script generation module are not thoroughly discussed in the main paper (hidden in a small section in appendix).**\n\nA1. Thank you for the feedback. We would like to clarify that these details are discussed in detail in Section 4.2 of the main paper. We also open-source the LLM prompts for script generation in Appendix C.\nTo further elaborate, our LLM-powered module operates in three key stages:\n1. Content Analysis: The module first processes user-provided input, such as a web URL or PDF, and employs an LLM to recognize its content.\n2. Briefing Document Generation: We find empirically that generating a script directly from raw content can lead to vague outputs and a significant loss of information. To address this, we introduce a crucial intermediate step: generating a structured briefing document. This document synthesizes the source material into five key components: a title, an abstract, main topics, key citations, and a conclusion. It also includes explanations for technical terms to ensure clarity.\n3. Script Generation: Finally, using this briefing document as a high-quality guide, the LLM generates the final podcast script. We guide this process by specifying the desired structure (e.g., engaging openings and closings), format (a two-speaker JSON), and content and style. Critically, to infuse the text with spontaneity, we explicitly prompt the LLM to incorporate elements like filler words (e.g., ‘um’, ‘uh’), response words (e.g., ‘yeah’, ‘right’), informal grammar, and repetitions. This encourages the output to stylistically match our ASR training data. To reinforce this, the prompt also includes an example ASR transcript as a demonstration, further guiding the LLM's output.\n\n\n**Weakness-2. Several details are missing in the evaluation:**\n- **The number of samples in the evaluation set and the distribution across different domains.**\n- **The definition and rating guidelines for all subjective metrics.**\n- **The evaluation process lacks clarity: How many raters were hired? How many reviews were conducted per evaluation sample?**\n\nA2.  Thank you for the detailed feedback regarding our evaluation methodology. We are happy to provide the following clarifications. \n\nDetails on our evaluation setup are provided in Section 5.1.3. Specifically, our evaluation set consists of four samples derived from diverse knowledge sources: a computer science paper (PDF), an economics paper (PDF), a technology blog (URL), and a news article (URL). For the evaluation presented in the paper, we hire 10 raters. Each generated audio sample from every system is assessed by each rater, resulting in a total of 10 ratings per sample.\nTo address your request for the rating guidelines, here are the specific instructions and questions provided to the evaluators for each subjective metric:\n- Spontaneity: Assesses how spontaneous the audio sounds. This focuses on the presence of common spoken elements (e.g., filler words such as 'um,' 'uh,' 'like,' 'you know,' 'so,' 'right?', and response words such as 'Yeah,' 'Right,' 'Okay') as well as a fluid and casual flow. Rated on a 1-5 scale with 0.5-point increments, where 5 is best.\n- Coherence: Evaluates the overall coherence of the audio. This includes the naturalness of transitions between speakers and whether the audio is globally consistent. Rated on a 1-5 scale with 0.5-point increments, where 5 is best.\n- Intelligibility: Measures whether the speech is clearly articulated and consistent with the provided text. Rated on a 1-5 scale with 0.5-point increments, where 5 is best.\n- Quality: Assesses the overall quality of the audio. Rated on a 1-5 scale with 0.5-point increments, where 5 is best.\n- Similarity: Determines if the speaker's voice is consistent with the voice in the provided reference audio. Rated on a 1-5 scale with 0.5-point increments, where 5 is best.\n\nFurthermore, we conduct a new, more comprehensive subjective evaluation on English podcast generation during the rebuttal period. This new round involves an increased pool of 20 raters and an expanded set of 10 distinct knowledge sources.  In this round, the audio is segmented by speaking turns, and each turn is rated individually. The results are presented below:\n\n\n| | Spontaneity(↑) | Coherence(↑) | Intelligibility(↑) | Quality(↑) | Similarity(↑) | SIM-O(↑) | WER(↓) |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Cosyvoice2| 3.75±0.04 | 3.80±0.04 | 4.55±0.03 | 4.35±0.04 | **4.54±0.02** | 0.76 | 2.53 |\n|Concat Baseline| 3.62±0.03 | 3.65±0.04 | 4.12±0.03 | 3.65±0.03 | 4.01±0.03 | **0.78** | 2.36 |\n|Our System| **4.60±0.02** | **4.53±0.03** | **4.61±0.02** | **4.38±0.02** | 4.22±0.04 | 0.56 | **1.69** |\n\n**Weakness-3. Evaluation comprehensiveness: podcast generation latency and the factualness of generated podcast is not discussed in the paper.**\n\nA3. Thank you for raising these important points. Considering the term \"podcast\" here can refer to either the generated text script or the final synthesized audio, so we will address the latency and factualness for both aspects.\n\nIf \"podcast\" refers to the text script:\n\n- Latency: The script generation latency is primarily determined by the external LLM used. In our implementation with the Gemini API, the process of converting a knowledge source into a full podcast script takes approximately 5 minutes on average.\n- Factualness: The factualness of the generated text is tied to the broader challenge of hallucination in LLMs. While this remains an open research problem without standardized methods for quantitative evaluation, we can report that in our extensive testing, we have observed very few instances of factual inaccuracies in the generated scripts.\n\nIf \"podcast\" refers to the synthesized audio:\n- Latency: For audio synthesis, latency is best measured by the Real-Time Factor (RTF). Our audio modeling module achieves an average RTF of 0.9 on a single Nvidia H20 GPU. In practice, tensor parallelism can be employed to significantly reduce the RTF.\n- Factualness: The \"factualness\" of the audio is its fidelity to the input script, which is measured by Word Error Rate (WER). We would like to clarify that we do report this objective metric in the paper, and the low WER demonstrates that our model faithfully synthesizes the content from the provided script."}]

Author rebuttal:

Meta-review: **Summary**

The paper proposes a podcast-generation framework trained on a large-scale corpus of conversational speech, enabling spontaneous two-speaker dialogue synthesis. 

**Reasons to accept**

This presents the first open-weight  multi-turn multi-speaker TTS model.  The experimental results and examples were convincing to the reviewers.

**Reasons not to accept**

There is some concern over the novelty of the work (mbG2).  While the components have note put together in this way before, many of the technical pieces have clear origins in other works.

**Decision rationale**

While the reviewers did not express outright enthusiasm to accept this paper, the written reviews were consistently positive.  The weaknesses (unaddressed by the discussion period) were relatively minor.  GIven the novelty of application and open-source contribution, I would advocate for this paper being accepted.  There is sufficient innovation (novelty and significance) without substantial enough weaknesses to reject.
Decision: Accept (poster)