CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching
Authors: leying zhang
Yao Qian
Xiaofei Wang
Manthan Thakker
Dongmei Wang
Jianwei Yu
Haibin Wu
Yuxuan Hu
Jinyu Li
Yanmin Qian
sheng zhao
Venue: NeurIPS 2025 poster ()
Keywords: text-to-speech
dialogue generation
flow-matching
Primary area: applications

Abstract: Generating natural-sounding, multi-speaker dialogue is crucial for applications such as podcast creation, virtual agents, and multimedia content generation. However, existing systems struggle to maintain speaker consistency, model overlapping speech, and synthesize coherent conversations efficiently. In this paper, we introduce CoVoMix2, a fully non-autoregressive framework for zero-shot multi-talker dialogue generation. CoVoMix2 directly predicts mel-spectrograms from multi-stream transcriptions using a flow-matching-based generative model, eliminating the reliance on intermediate token representations. To better capture realistic conversational dynamics,  we propose transcription-level speaker disentanglement, sentence-level alignment, and prompt-level random masking strategies.  Our approach achieves state-of-the-art performance, outperforming strong baselines like MoonCast and Sesame in speech quality, speaker consistency, and inference speed. Notably, CoVoMix2 operates without requiring transcriptions for the prompt and supports controllable dialogue generation, including overlapping speech and precise timing control, demonstrating strong generalizability to real-world speech generation scenarios. Audio samples are available at https://www.microsoft.com/en-us/research/project/covomix/covomix2.

Co VoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching
Leying Zhang 122
Yao Qian
Xiaofci Wang"
Manthan Thakker"
Dongmei Wang"
Jianwei Yu²
Haibin Wu?
Yuxuan Hu
Jinyu Li
Yanmin Qian'
Sheng Zhao?
"Shanghai Jiao Tong University, China
Microsoff, USA
Abstract
Generating natural-sounding. multi-speaker dialogue is crucial for applications such as podcast creation, virtual agents, and multimedia content generation. How- ever, existing systems struggle to maintain speaker consistency, model overlapping speech, and synthesize coherent conversations efficiently. In this paper, intro- duce Co VoMix,, a fully non-autoregressive framework for zero-shot multi-talker dialogue generation. VoMMx2 directly predicts mel-spectrograms from multi- stream transcriptions using flow-matching. based generative model, eliminating the reliance intermediate token representations. To better capture realistic conversational dynamics, we propose transcription-level speaker disentanglement, sentence-level alignment, and prompt-level random masking strategies. Our ap- proach achieves state-of-the-arr performance, outperforming strong baselines like MoonCast and Sesame in speech quality, speaker consistency, and inference speed. Notably, Co VoMix2 operates without requiring transcriptions for the prompt and supports controllable dialogue generation, including overlapping speech and pre- cise timing control, demonstrating strong generalizability to real-world speech generation scenarios. Audio samples are available
Introduction
The field of speech synthesis has witnessed remarkable progress in recent years, particularly in shot text-to-speech (TTS) systems that can generate high-quality, natural-sounding speech in voices not seen during training [ These advancements have enabled applications such as personalized virtual assistants, audiobook narration, and interactive voice response systems. However, extending these capabilities multi-talker dialogue generation, where multiple speakers engage in natural, dynamic, multi-turn conversations, remains a significant challenge.
Conventional approach synthesize dialogue by generating multiple monologues sequentially and concatenating them, which often results in unnatural interactions and poor speaker coordinations g 12 Recent efforts have shifted towards generating the entire dialogues in more integrated manner. CoVoMix [1] was the first attempt to employ a multi-stream autoregressive (AR) text-to-semantic model and non-autoregressive (NAR) acoustic model to synthesize mixed melsspectrograms for zero dialogue generation. NotebookLM [13] [4] leverages hierarchical transformer to
produce stream of audio tokens autoregressively, which are decoded back to dialogue waveform.
Work done during internship at Microsoft Azure AI. shnneeeyyiyyiggsrsuu.uu....o
Conrespondence: yaogian @microsoft.comm
http://wwww..eeeccc///oc/////ooa fttsp:://www.micret/ wwwoooootootooommommmmmo.owwc.tt www.://rooooooooooooooooooooooee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Training Stage
Step flow-matching backbone
Step Randomly choose dialogue for training
Predicted
Targat
Loss
Loss
Loss
|||ni]|ton- HilI- ||=
Spk1 's prompt
Spk2's prompt
t=1
Flow-matching Backbone
Spk1 Hil [t] [e 15
Morning! istert ie
dw; = (e(We(m) z)dt
5pk1
Nice to meet meet ee eegi
Step 2: Randomly choose segmens
segments
Noisy Speech
Prompt
Masked
Speech
prompt
Spk2's promot
Spk1 sequence
gjart
la new dialogue pererasion modell
CoVoMix27 What's than?
sequence
[8pk]] [8o
Marning
Do you know CoVoMix2?
Stop 3: Generate wavs through ODE and vocoder
Vocoder
Output
Discarded
Inference Stage
t=
Flow-matching Backbone
Step 1: Provide speaker's prompts
(m) 1i (w,, (m).. z)
Sok1's prompt
Spk2's prompt
Noise
Step 2: Choose desired speed/duration//ilenee of each utterance
Masked
Speech
Spk1 speaks fest Second sentonce's
sptore, aprai (g (n
Define
Spk sequence
[Pa1]]
[5t27]
thank
duration
Spk2's sequence
[See1]
How newareyou?
Figure 1: The overview the proposed CoVoMix2 framework
MoonCast [12] utilizes long- long-context text-to-semantic model to support coherent dialogue generation. These systems typically adopt a two-stage AR+NAR pipeline involving intermediate representations, such semantic audio tokens.
Despite these advances, current systems face several key limitations. Firstly, dependence on AR components and intermediate representations introduces considerable complexity and slow inference speed. Secondly, the lack of fine-grained control over prosodic features, such as speaking rate, utterance duration, overlap, and pauses, often results in unnatural and less coherent dialogue flow. Thirdly, speaker identity inconsistencies, where the wrong voice is occasionally assigned to an utterance, undermine perceived authenticity. Lastly, current methods [15]]] [6] are hindered by their reliance on stereo audio for training and paired audio-text prompts for inference, often necessitating external ASR models and limiting scalability.
To address these challenges, we propose CoVoMix2 (Conversational Voice Mixture Generation), novel framework for zero-shot multi-talker dialogue generation based on fully NAR flow-matching approach. Our main contributions are summarized as follows:
1. We exploit fully non-autoregressive framework for zero-shot multi-talker dialogue gen- eration. It end-to-end modeling from the interleaved character sequence of two speakers to their mixed mel-spectrograms, with intrinsic alignment between characters and mel-spectrogram frames.
2. We propose wanscription--eeal disentanglement strategy in which each speaker's tran- scriptions are provided as separate streams, enabling potential control over the timing of generated overlapping speech.

---

3. Additionally, we introduce sentence-level alignment and prompt-level masking strategies that facilitate accurate speaker identity assignment and eliminate the need for intermediate representations or transcriptions of speaker prompts during inference.
4. Extensive experiments show that Vo outperforms strong baselines, achieving state-of- the-art (SOTA) performance among open-source checkpoints, while requiring significantly less training data and delivering faster inference speed.
2 Related Work
2.1 Flow-matching based Speech Synthesis
flow-maachinng-baeed speech synthesis has emerged as a promising approach in the field of text-to- speech (TTS) systems [17 [8]. It models speech generation as a continuous transformation from
simple distributions to complex speech representations, enabling fast, parallel inference and improved
controllability compared to traditional AR models 6 19
Several recent models [20 have demonstrated the effectiveness flow matching across tasks such as zero-shot TTS, speech inpainting, and voice conversion. Notably, E2-TTS [26] and F5-TTS [27] remove the need for phoneme alignment or explicit duration modeling, resulting in simpler pipelines with high-quality output. These advancements establish flo--maching a promising direction for building scalable and efficient TTS systems.
2.2 Multi-talker Dialogue Generation
Multi-talker dialogue generation aims synthesize entire multi-turn conversations between multiple speakers, where each utterance must be coherent, contextually relevant, and speaker-specific This task significantly more complex than single-speaker synthesis due to the need for speaker turn-taking, identity preservation, and interaction dynamics.
Early approaches, such Co [IT]. introduced a multi-stream AR text-to-semanpe model combined with a NAR acoustic decoder to generate dialogues in zero-shot manner. Other systems like Soundstorm [14] and NotebookLM [13] incorporate both AR and NAR components to model hierarchical audio tokens or contextual embeddings. Further, works such as Chats and SLIEE 15), built upon the dGSLM framework 19]. leverage dual-tower transformer architectures to capture interleaved speaker information. Encoder-decoder models like Dia 1.6B [ and Parakeet [9] directly predict audio codec tokens, which are then decoded into waveforms. MoonCast [12] addresses long-context dialogue generation using a transformer-be prompts at each turn to manage speaker changes.
language model, concatenating acoustic
While these methods have advanced the field by improving naturalness and speaker consistency, most rely on AR decoding or hybrid AR/NAR architectures. These designs introduce several drawbacks: slow inference due step-by-step generation, reliance on complex intermediate representations. lim- ited controllability over conversational dynamics such as overlap, pauses, or timing, and dependence on the paired texi-audio speaker prompts.
2.3 Conversational Speech Synthesis
Conversational Speech Synthesis (CSS) focuses on generating individual utterances that are contextu- ally appropriate within dialogue, typically from single speaker. Unlike full dialogue generation, CSS produces each utterance one-by-one, conditioned on previous dialogue history, rather than
synthesizing an entire conversation simultancously [30 33
For instance, GPTTalker 34 models multimodal dialogue context by converting history into discrete tokens processed by language model to generate expressive, context-aware responses. Sesame [10] adopts two AR transformers [35]: a multimodal transformer backbone processes text and audio tokens, followed by audio decoder transformer that reconstructs high-quality speech. Although Sesame supports dialogue-styl synthesis, it fundamentally generates each speaker's utterance sequentially given the previous dialogue context, rather than modeling simultaneous multi-speaker interaction.
While CSS methods achieve high expressiveness and coherence for individual utterances, they fall short in handling the broader structure and dynamics of real-time, overlapping multi-speaker dialogue.
3

---

Our proposed CoVoMix2 differs fundamentally from both traditional talti-talker dialogue generation systems and CSS approaches. Unlike prior work that relies on AR or hybrid pipelines, or CSS systems that synthesize one speaker's utterance at time, CoVoMix2 enables fully NAR, simultaneous generation of multi-speaker dialogues. It directly predicts speecroorgs from disentangled multi- stream transcriptions, allowing efficient, accurate, and controllable synthesis with better support for real conversational dynamics such as overlapping speech and natural pauses.
3 CoVoMix2
Zero-shot dialogue generation aims to synthesize multi-speaker conversations voices not encoun- tered during training. We propose CoVoMix,, fully NAR zero-shot dialogue generation model that directly generates mel-spectrograms from raw dialogue transcriptions. As shown in Figure [ Vo operates without intermediate representations such as phonemes or audio tokens, enabling efficient and scalable zero-shot dialogue generation.
Let the training dataset be denoted as D {ær, y). where is dialogue waveform containing utter- ances from two speakers, and y = [_/,.... yn] is the corresponding transcription. Each transcription segment Y/ is annotated with the content T; speaker label 8; the start and end time #stant tannd The corresponding -peesppctrogram of is noted as m. To support simultaneous, zero-shot, multi-speaker dialogue generation, we introduce three key design strategies in the following sections, yielding a pair of input text sequences [21, #2), each representing separate speaker stream, and the acoustic prompts Matz the target speakers.
3.1 Transcription-LLevll Speaker Disentanglement
Natural speaker switching and overlapping speech are essential features realistic dialogue genera- tion. Prior work often inserts speaker-change tokens (e.g., [spkchange]) indicate switch between speakers [1]14]. However, single-stream representations entangle multiple speakers' utterances into flat sequence, making it difficult for the model to explicitly capture speaker-specific timing, and overlap. Moreover, this design creates ambiguity in conditioning, particularly when aligning acoustic prompts textual content. In shared text stream, the model must disentangle which portions belong to which speaker and match them to the correct prompt, increasing the risk of identity confusion.
Instead, shown in Figure 2 we propose more structured approach by disentangling the transcript into multiple parallel streams, one per speaker. Each text stream 2, contains two types of content: Active speech segments, defined by their respective time intervals [tstart . (gnd). and Silence intervals, denoted using special token [S]. These streams enable precise temporal control over individual speaker utterances, including overlapping and silence. This design grants ane-grained control over the interaction structure, improving naturalness and flexibility.
Prompt
Spk1
Spk2
Spk2
Spk1
Candidates
Part
12
store
1
13
start
end start
PRd
Z1
Hello!
Silence token [S]
I think it's sunny?
22
[S]
Hello!
[S]
How is the weather today?
w
a
t
a
y
Character sequence
Continuation token [P]
Figure 2: Example of the input data organization
3.2 Sentence-Level Alignment
High-quality alignment at the phoneme level is often unavailable real-world data 11] Likewise, modeling intermediate representations (c.g., codec tokens) is resource-intensive and requires large-

---

scale training data. As an alternative, inspired by recent models 27, we design the sentence-level alignment that is suitable for both monologue and dialogue scenarios.
As indicated in Figure each active speech segment in the speaker stream converted into sequence of characters (including upper and lower cases letters and punctuation). For the rest of the active speech segment, we add a special continuation token [P]] for padding. These sequences, temporally anchored by the corresponding start and end times, serve as the text input for the mel-spectrogram prediction. The model learns the intrinsic alignment between characters and mel-spectrogram frames and synthesizes each utterance within its designated time range without explicit duration prediction.
3.3 Prompt-Level Random Masking
Accurate voice conditioning is critical to avoid speaker confusion. Previous approaches either main- tained parallel conditioning streams [5] used concatenated speaker prompt for continuation [14] or appended speaker prompts at each dialogue turn to guide speech synthesis [12. We introduce prompt-level random masking strategy to ensure robust and diverse speaker conditioning.
As shown in Figure and 2 for each training sample, we first find all the available monologue segments for each speaker as the prompt candidates. We then randomly select prompt segment from the candidates. The chosen prompts concatenated at the very beginning with the masked training sample using separator token to construct the prompt sequence Matz- Moreover, in the text stream 2, transcription of the prompt, distinguishing which segments belong to each speaker.
special tokens [Spk1] and [Spk2] indicators (not text tokens) replace the
Furthermore, to prevent prompt leakage, where the model copies the prompt directly into the output, we exclude the prompt region from the loss computation. This encourages the model to generalize voice characteristies rather than memorize the prompt audio.
3.4 Flow-Matching-Basset Mel Spectrogram Generation
Our model employs Flow Matching (FM), a simulation-free training method for Continuous Nor- malizing Flows (CNFs) T8 36.. It is a class of generative models that learn to transform a simple distribution (e.g., Gaussian noise) into complex data distribution (e.g., mel-spectrograms) through continuous mapping. This mapping is defined by solving an Ordinary Differential Equation (ODE).
Specifically, the model learns the distribution q(m)z, motz) where m is the target mel-spectrogram, is the speaker-aware text streams and Metz is the acousic prompts. At training time, noise sample mo is drawn from a standard Gaussian distribution. The training objective is to minimize the L2 distance between the predicted and true flow, given by Eq where (-) is the model's predicted vector field at time t E [0, 11, w (1 (1 Jmin)t) )t) + tm, and Jmin is hyper-parametee to control the deviation of flow- fow-matching. Notably, the loss not computed on segments that originate from the prompt region, which is excluded using masking function M().
f E [( - (1 Omiin m - (eww Matz: z; 0))[
(1)
We also apply Classifier- Free Guidance (CFG) 37 21 to improve sample quality by interpolating
between conditioned and unconditioned flows. During training, the acoustic prompt Mata and text
sequences 2 are dropped with Puncond-
During inference, the CFG vector field becomes Eq with controlling the strength of guidance. Durations for each utterance are computed based on syllable counts and a predefined speaking rate, allowing the construction of the input text streams z. A mel-spectrogram is then generated by sampling noise mo and solving the ODE defined by the flow field.
(w, Metas 2; 0) = (1 + (w) mate: 2) - aU (w;
(2)
3.5 Training Strategy: Curriculum Learning and Data Mixing
To enable the model with dialogue generation capability efficiently, we adopt a two-stage curriculum learning strategy during training. First, the model is pretrained on high-quality monologue datasets, which helps it learn accurate pronunciation and acoustic modeling. contrast, directly training on multi-speaker data causes degraded output quality, including mispronunciations and unintelligible
5

---

speech. Then, in the second stage, we train the model on multi-speaker dialogue datasets to enhance the dialogue generation capability.
To improve robustness and generalization, built on the large scale of monologue dataset, we design the data mixing strategy, using various sources of data during the second stage training. Specifically, mix data from several sources, including ASR-transcriber podcast dialogues, audiobook-style single- speaker datasets and simulated overlapped dialogues to support overlapping capability. Benefited from this data diversity, CoVoMix2 does not require any human-annotated dialogues to get satisfied and natural results. For further enhancement, we demonstrate in Appendix D.2 that that fine-tuning the model with just 20 minutes of clean, human-annotate dialogue can significantly boost performance.
4 Experimental Setup
4.1 Training and Inference Data Preparation
5559
Segmentation
Right secomm m5820
Diarization
(Yahh 58000m 4000mm
Spesker Number
Podkssst dislogue
Denoiaing
Spick Ryee 100000m 11:000m
Mooimum Dunetons
ASR
Podcast monolague
Podcsst data
Speaker Awarn
Duration
Word-level Transcription
Simulited dialogue
Figure Data processing pipeline
To train VoMMMx curated diverse internal dataset comprising both dialogues and clean monologue data. The core training corpus consists of 3,000 hours of internal English podcast data, which we processed with steps, as shown in Figure 3
First, we applied Microsoft speech enhancement API to remove background noise and music. Second, we implemented automatic speech recognition and speaker diarization. The diarization results were used to assign speaker identities to each utterance with timestamps. Specifically, we used the Deepgram API [38 to obtain word-level transcriptions and speaker labels. Third, following [I] long dialogues were segmented into shorter clips, each involving two or one speakers, to improve data quality and training efficiency. Finally, simulate dialogue segments using the monologue datasets. By pairing utterances from different speakers and introducing controlled overlap or silence ratios, we generate synthetic two-speaker dialogues. Detailed data-processing pipeline is open-sourced |
For the first training stage, we used the LibriHeavy dataset [39, comprising 60k hours of high-quality single-speaker audiobook-style recordings. In the second stage, in addition to the podcast dataset, we simulated dialogue-style data by concatenating utterances from different speakers using both LibriHeavy and LibriTTS (40). To further enhance the model's ability to handle overlapping speech, we generated highly overlapped data from LibriHeavy, varying the overlap ratio from 0% to 100%.
In order to evaluate the model performance, we design a dialogue test set containing 1000 dialogue transcriptions from Dailydialog [41 and the acoustic prompts are from Librispeech--est-cllan [42 We also use samples from this dialogue dataset for subjective evaluation.
4.2 Model Configuration
our experiments, backbone architecture closely followed the configurations in (26]. Specifically, we used Transformer with 24 layers, 16 attention heads, and an embedding dimension of 1024 with U-Net 43 style skip connections. The Jmin is set to 0. We modeled the 100-dimensional log mel-filter bank features, extracted every 10.7 milliseconds from audio samples with 24kHz sampling rate. A BigVGAN-based [4] vocoder was employed to convert the log mel-filter bank features
audio oo-ppooossssign-orrrieeee
https://deepgran. /oteepgraa.comm/

---

into waveforms. In addition, we implemented Classifier- Free Guidance (CFG) [37 with a dropout probability Puncond 20%, randomly removing conditioning during training.
In the first training stage, we train the model on 60k hours LibriHeavy (39] dataset for 200k steps with peak learning rate(Ir) of 7.5e-5. In the second training stage, train it for another 200k steps on the combined podcast, audiobook, and simulated dialogue datasets with peak Ir of 5e-5. The model was optimized using the Adam optimizer. linear-decay learning rate schedule was used in both stages. Each training batch contained two samples, each less than 30 seconds in duration.
Training conducted on 32 NVIDIA Tesla V100 GPUs (32GB) with gradient accumulation set to 4. During inference, we used a guidance strength a of 1.0 and performed sampling with 32 function evaluations (NFE) using an ODE solver.
4.3 Baseline and Evaluation Metrics
We adopt MoonCast [12], latest state-of-the-ard dialogue generation model employing a hybrid AR+NAR architecture, as a representative baseline. While simple audio concatenation based on single-speaker models has been proved to be ineffective in capturing speaker interactions [1 we additionally include Sesame [10] as strong baseline. As representative model for the CSS task, Sesame generates each utterance sequentially, leveraging previously generated audio as contextual input to maintain coherence across dialogue turns. Detailed baseline configuration comparison is provided in Appendix [A]
Although other models are capable of dialogue generation, we exclude Dia Co [1] and Mo [13 from comparisons. Dia tends to produce unnaturally rapid and truncated dialogues, and CoVoMix is trained exclusively on 8kHz audio, leading to low-fidelity outputs that are directly comparable to our high-quality generation setting. NotebookLM is not open-sourced, preventing us from making a reasonable comparison.
To comprehensively assess the recognition accuracy and speaker consistency, we adopt the following objective evaluation metrics: Real-Time Factor (RTF), Word Error Rate (WER), Speaker- Aware Word Error Rate (SA-WER) [45), Speaker-Award Speaker Similarity (SA-SIM) and UTMOS [46]. Specifically, We measure the RTF on a single NVIDIA A 100 machine. We utilize Microsoft Fast Transcription API I as automatic speech recognition and diarization tool transcribe the generated speech, and we calculate the SA-WER for each word and their corresponding speaker identity. The SA-SIM enhances SIM by ensuring that the similarity computed between embeddings attributed to the speaker identity detected by the diarization model. We utilize WavLM-TDNN 47 speaker embeddings.
to extract the
We perform a human evaluation on the generated dialogue examples. We conducted a Comparable Mean Opinion Score (CMOS) experiment assess user preference in terms of speaker turn handling, interactivity, fluency, and coherence. 15 professional linguistic experts provide judges for all subjec- tive evaluations. They provide rating to the second audio, which is randomly selected from pair of audios, in the (-3 to +3) range. Detailed instructions are in Appendix E
Result and Analysis
5.1
Objective and Subjective Evaluation Results
Table 1: Model performance comparison on dialogue data
Model
RTF
WER 4
SA-WER 1
SA-SIM f
UTMOS f
CMOS
MoonCast
1.37
7.08+46.23
20.40:151.09
0.40
2.65=0042
-0.25+0..3
Sesame
2.08
5.62+:5.61
9.65+13.20
0.49
2.70=0.44
-0.39:40.00
CoVoMix2
0.30
5.73+6.68
6.311.9224
0.56
3.10
0.00
Table 1 presents evaluation results comparing CoMo2 against the baseline models MoonCast and Sesame on dialogue test sets. Standard deviations (1-sigma) reported assuming approximate
http:///llarn..mcooooteto comt www.oooo//oooo us http:///cc.ococooooooooooooaooaa sor.i////./ppcceccccooccoccco///
ppeech-to-textHaase htt.
7

---

normality. While WER is positive, some large standard deviations result in negative lower bounds, which are not meaningful in practice but reflect high sample variability.
Across nearly all metrics, CoVoMix2 demonstrates clear superiority in speech quality, speaker consistency, and inference speed. VoMixxx achieves RTF of 0.30, significantly faster than both MoonCast and Sesame, demonstrating its efficiency as a fully NAR model.
In terms of content accuracy and speaker consistency. Co attains the best SA-WER and SA-SIM, highlighting its ability to maintain linguistic accuracy and consistent speaker identity across multi-turn interactions.
In contrast, MoonCast, being language-model badelb often suffers from issues such as speaker confusion, hallucinated content, repetitive outputs, and improper termination. These artifacts lead to unstable generation, reflected in its relatively high WER and SA-WER scores.
Sesame, despite its strong performance standard WER, occasionally exhibits speaker confusion. We observe cases where monologue-style outputs include voice characteristics from the other speaker, even though speaker identities are clearly specified. This suggests that the shared contextual input, containing prompts and prior audio from both speakers, may introduce ambiguity, making it difficult for the model to consistently distinguish between speakers in extended dialogue scenarios.
Furthermore, the lower standard deviations observed across all metries for Co VoMix2 indicate more stable and reliable output quality, further validating the effectiveness of our flow- www-mmacchin--bee based architecture for multi-speaker dialogue synthesis.
Finally, the subjective CMOS comparison between VoMxx and other models demonstrates that model performs better in terms of speaker turn handling, interactivity, fluency, and coherence with the transcription. We also ask judges to give detailed comments, where better pitch and intonation, better rhythm and less distortion are three main advantages of our proposed VoX22
5.2 Speaker Consistency
Mooncast
Sesame
CoVoMix2
Average Speaker Similarity
20.
MoonCast Sesame
m
2G
S4I
242
T1
C/
5.
A37
0.8
Coleltu22
Spesch Turna
645
9.6
0.7
0.6
3.0
3.
+
4:14
602
140
6.0
6.0
6.0
0.5
3-4
Speech Turns
13 Speech Turns
14 15
12
Speech Turns
16 15
11 12
Speech Turns
15 15
16
(a)
(b)
Figure 4: Speaker consistency analysis across dialogue turns. (a) Average speaker similarity between each generated turn and its corresponding prompt. (b) Pairwise speaker similarity between turns from the same speaker within a dialogue. Consistent color indicates stable speaker timbre across turns.
To evaluate speaker consistency across turns, we selected three long dialogue sequence containing 12 speech turns, with each speaker contributing 6 utterances.
Figure a) a)) prompt. While Sesame achieves relatively high similarity in some turns, its performance is incon- sistent. This may be due to its design, which generates each utterance independently, relying a prompt placed only at the beginning. As the dialogue progresses, the prompt information may be forgotten or get confused, especially in longer contexts. MoonCast, by contrast, demonstrates consistently low similarity. Although it incorporates prompts at the beginning of each turn, this additional conditioning does not improve speaker accuracy, indicating limited benefit despite the increased computational overhead.
shows the average speaker similarity between each generated turn and its corresponding
In comparison, CoVoMix2 achieves the highest average similarity with the lowest variance, even though the prompt is provided only once at the beginning of the dialogue. This result demonstrates the robustness in maintaining speaker identity over long conversations.
8

---

Figure further investigates intra antra-dialouue speaker consistency by measuring pairwise speaker similarity across all turns from the same speaker long dialogue. more uniform and consistent color distribution along the rows and columns reflects stronger identity preservation. Both MoonCast and Sesame display noticeable variability, indicating timbre drift or identity shift during generation. In contrast, our model demonstrates consistently high pairwise speaker similarity across all turns, highlighting its effectiveness preserving speaker timbre throughout multi-turn dialogues.
5.3 Overlapping Analysis
Achieving overlap dialogue generation a challenge for current models. Most models rely on data to achieve overlaps [1]] and are constrained by the overlap reconstruction capabilities of codecs [ Figure 5 shows a visual comparison of mel-spectrograams, extracted from overlapping segments generated by NotebookLM, VoVoMix. Vo and real overlapping sample, where the first two samples are extracted from the official demo page.
Overlapping speech is characterized by the simultaneous presence of multiple harmonic structures in the mel-spectrogram, resulting in smooth, continuous spectral patterns. These patterns blend naturally over time, leading to dense, interwoven energy distributions without abrupt boundaries or artificial transitions (48.50). observe that Co VoMix2 generates overlapping speech with higher spectral fidelity, smoother harmonic structure, and more natural timing alignment, closely matching real overlapping dialogue. In contrast, NotebookLM output resembles concatenation of sound segments rather than genuine learned overlapping process, and VoMi'ss speech has low fidelity because of the lower data quality.
(a) NotebookLM
(b) CoVoMix
(c) CoVoMix2
(d) Real overlapping sample
Figure 5: Mel-spectrogram comparison between overlapping samples generated by NotebookL.M, CoVoMix, CoVoMix2 and the real sample.
Ablation Studies and Extension
We conducted extensive ablation studies in Appendix[Dee D validate our design choices across data composition, training stages, and special token representation. Results show that pretraining on monologue data is essential for learning stable pronunciation, while data mixing strategy improves generalization to diverse dialogue patterns. Simulated dialogue data may introduce noise.
Our experiments further demonstrate the robustness of Co to monologue generation, genera- tion with very short prompts, and long dialogue generation. Specifically, it maintains stability even when other baselines fail with prompts under 10 seconds. Despite being trained on data under 30 seconds, CoMMMix is capable of generating stable conversations up to three times longer (approxi- mately 90 seconds). Morcover, demonstrate that brief fine-tuning stage using just 20 minutes of human-annotated data can significantly boost transcription accuracy and speaker consistency.
Our framework enables wide range of practical applications beyond standard dialogue synthesis. Since CoVoMix2 does not require transcriptions for speaker prompts, it supports cross-lingual voice cloning, allowing voices to be transferred across languages. Its temporal control features-inclan fine-grained manipulation of speech overlap, pauses, duration-make it especially suited for applications like podcast creation and video dubbing, where precise alignment with visual content or conversational pacing essential. The ability to generate overlapping speech also enhances realism in multi-speaker speaker scenes, such dramatic dialogues or animated character interactions.

---

7 Conclusion, Limitation, Future Work and Broader Impacts
In this work, introduced CoVoMix2, fully NAR framework for zero-shot multi-talker dialogue generation. By directly predicting mel-spectrsgrams from disentangled multi-stream transcriptions and leveraging Now-matching-bass based method, CoVoMix2 enables efficient, high-quality synthesis of natural, speaker-consistent dialogues, including controlled overlapping speech and fine- fine-grained timing control. Through extensive experiments, we demonstrated that Co VoMixx outperforms existing models in both speech quality and speaker accuracy while achieving significantly faster inference.
Future work In future work, plan extend framework support conversations involving more than two speakers, with improved modeling of naturalistic speech overlap and richer conver- sational dynamics. We also aim to scale up the training even larger and more diverse datasets, enabling broader generalization across domains and languages.
Limitation While our training data covers wide range of conversational scenarios, it is primarily automatically transcribed, which may introduce minor inaccuracies, particularly in handling disflu- encies such as repetitions backchannel words. Additionally, since ASR tools lack support for word-level timestamps in overlapping speech, we rely on simulated dialogue data to train overlap scenarios, which can introduce slight deviations in naturalness and degraded audio quality.
Broader Impacts CoM22ix2 offers a versatile and scalable solution for high-quality, human-like speech generation, with potential applications in assistive technology, media production, language learning, and virtual agents. However, since CoMMxxx could synthesize speech that maintains speaker identity, it may carry potential risks in misuse of the model, such spoofing voice identifica- tion or impersonating specific speaker. To mitigate such risks, is possible to build a detection model to discriminate whether an audio clip synthesized by Co Moxx2.
10

---

References
[1] Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang. Y. Liu, Y. Leng, K. Song, S. Tang et al., "Naturalspeech 3: shoo-shot speech synthesis with factorized codec and diffusion models," arXiv preprint arXiv:2403.03 100, 2024.
[2] Y. LLng. Z. Guo, K. Shen, Z. Ju, X. Tan, E. Liu, Liu, D. Yang, leying zhang, K. Song, L. He, X. Li, sheng zhao, T. Qin, and J. Bian, "PromptTT'S 2: Describing and generating voices with text prompt," in The Twelfth International Conference on Learning Representations, 2024.
[3] M. Lajszezak, G. Cámbara, Y. Li, F. Beyhan, A. Van Korlaar, F. Yang, A. Joly, Á. Martín- Cortinas, A. Abbas, Michalski et al., "Base tts: Lessons from building billion billion-parameter parameter text-to-speech model on 100k hours of data," arXiv preprint arXiv:2402.0889., 2024.
[4] Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang al., "Cosyvoice 2: Scalable streaming speech synthesis with large language models," arXiv preprint 1xXi:24422101117 2024.
[5] Anastassiou, J. Chen, J. Chen, Y. Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gao et al., "Seed-tts: A family of high-quality versatile speech generation models," arXiv preprint arXiv:2406.040 024400 2024.
[6] S. Chen, C. Wang, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, Zhao, and Wei, "Neural codec language models are zero-shot text to speech synthesizers," IEEE Transactions on Audio, Speech and Language Processing, vol. 33, pp. 705-718, 2025.
[7] Y. Lee, I. Yeon, J. Nam, and S. Chung, "Voiceldm: Text-to-speech with environmental context," in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12 566-166611
[8] K. Lee, D. W. Kim, J. Kim, S. Chung, and J. Cho, "Ditto-tts: Diffusion transformers for scalable text-to-speech without domain-specific factors," arXiv preprint arXiv:2406. 2024.
[9] T. A. Nguyen, E. Kharitonov, Copet, Adi, W.-N. Hsu, A. Elkahky, P. Tomasello, R. Algayres, B. Sagot, A. Mohamed et al., "Generative spoken dialogue language modeling," Transactions of the Association for Computational Linguistics, vol. 11, pp. 250 250-266, 2023.
[10] J. Schalkwyk, A. Kumar, D. Lyth, S. Eskimez, Z. Hodari, C. Resnick, R. Sanabria, and R. Jiang, "Crossing the uncanny valley of conversational voice sesame.com comm"" https://www.sssame..um research/crossing ttheeeuccnnn_a_eeeeeeeee_ee_eeee sarlll_llllllllleeeeee Sf voice [Accessed 17-04-2025).
[11] L. Zhang, Y. Qian, L. Zhou, S. Liu, D. Wang, X. Wang, M. Yousefi, Y. Qian, J. Li, L. He et al., "CoVoMix: Advancing zero-shot speech generation for human-like multi ttaker conversations," Advances Neural Information Processing Systems, vol. 37, pp. 100 200299100 317, 2024.
[12] Ju, D. Yang, J. Yu, K. Shen, Y. Leng, Z. Wang. X. Tan, X. Zhou, T. Qin, and X. Li, "MoonCast: High-quality zero-shot podcast generation." arXiv preprint arXiv:2503.14344 2025.
[13] "Pushing the frontiers of audio generation - decpmind.googleee https://deepmmin.ggoee discover/bloo/pusshigg lanee--ronteers--o of
[Accessed 27-04-2025).
[14] Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, and M. Tagliasacchi, "Sound- storm: Efficient parallel audio generation," arXiv preprint arXiv:2305.000: 096366 2023.
[15] H. Lu,, G. Cheng, L.. Luo, L.. Zhang, Y. Qian, and P. Zhang, "SLIDE: model with I1m for spontancous spoken dialogue generation." in ICASSP 2025 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1-5.
Integrating speech language
[16] K. Mitsui, Hono, and K. Sawada, "Towards human-like spoken dialogue generation between ai agents from written dialogue," arXiv preprint 10.00:23 2023.
[17] R. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, "Neural ordinary differential equations," Advances in neural information processing systems, vol. 31, 2018.

---

[18] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, "Flow matching for generative modeling," " in The Eleventh International Conference on Learning Representations, 2023.
[19] B. Han, Zhou, S. Liu, S. Chen, L. Meng. Y. Qian, Y. Liu, S. Zhao, Li, and F. Wei, "Vall-e Robust and efficient zero-shot text-to-speech synthesis via monotonic alignment," arXiv preprint arXiv:2406.07855 07855, 2024.
[20] S. Mehta, R. Tu, J. Beskow, É. Székely, and G. E. Henter, "Matcha-tts: A fast tts architecture with conditional flow matching." " in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 11 341-11 345.
[21] M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar et al., "Voicebox: Text-guided multilingual universal speech generation at scale," Advances in neural information processing systems, vol. 36, pp. 14 0054444434, 2023.
[22] A. Vyas, B. Shi, M. Le, A. Tjandra, Y.-C. Wu, B. Guo, J. Zhang, X. Zhang. R. Adkins, W. Ngan et al., "Audiobox: Unified audio generation with natural language prompts," " arXiv preprint 2023.
arXiv:2312.15871
[23] L. Zhang, W. Zhang, Z. Chen, and Y. Qian, "Advanced zero-shot text-to-speech for background removal and preservation with controllable masked speech prediction," in ICASSP 2025- 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025.
[24] Y. Guo, C. Du, Z. Ma, X. Chen, and K. Yu, "Voiceflow: Efficient text-to-speech with rectified flow matching," in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 11 121-11 125.
[25] Z. Chen, S. Wang, M. Zhang, X. Liu, J. Yamagishi, and Y. Qian, "Disentangling the prosody and semantic information with pre- rre-trained model for in-context learning based zero-shot voice conversion," in 2024 IEEE Spoken Language Technology Workshop (SLT). 2024.
[26] S. E. Eskimez, X. Wang. M. Thakker, C. Li, C.-H. Tsai, Z. Xiao, H. Yang. Zhu, M. Tang, X. Tan et al., "E2 tts: Embarrassingly easy fully non- non-autoregressive zero-shot tts." in 2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, pp. 682-689.
[27] Y. Chen, Z. Niu, Z. Ma, K. Deng, C. Wang. Zhao, K. Yu, and X. Chen, "F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching," arXiv preprint 2024.
[28] "GitHub nari-labs/dia: A TTS model capable of generating ultra-realistic dialogue in one pass. github.com." https://gilubb.com/naaiiiii labs//// [Accessed 27-04-2025).
[29] J. Darefsky, G. Zhu, and Z. Duan, "Parakeet," 2024. [Online]. Available: https:
[30] Y. Hu, R. Liu, G. Gao, and H. Li, "Fetalker: Fine and coarse grained context modeling for expressive conversational speech synthesis," in 2024 IEEE 14th International Symposium Chinese Spoken Language Processing (ISCSLP). IEEE, 2024, pp. 299-303
[31] J. Xue, Y. Deng, F. Wang, Y. Li, Y. Gao, J. Tao, J. Sun, and J. Liang, "M2-ctts: End-to-end multi-scale multi-modal conversational text-to-speech synthesis," in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023,
pp. 1-5.
[32] H. Guo, S. Zhang, F. K. Soong, He, and Xie, "Conversational end-to-end its for voice agents," in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 403 409.
[33] K. Lee, K. Park, and D. Kim, "Dailytalk: Spoken dialogue dataset for conversational text-to- speech," in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1-5.
[34] R. Liu, Y. Hu, Ren, Yin, and Li, "Generative expressive conversational speech synthesis," " in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 4187-4196.
12

---

[35] A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghi- dour, "Moshi: a speech-text foundation model for real-time dialogue," arXiv preprint arXiv:4410.0033, 2024.
[36] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, "Neural ordinary differential equations," Advances in Neural Information Processing Systems, vol. 31, 2018.
[37] J. Ho and T. Salimans, "Classifier-free diffusion guidance," arXiv preprint arXiv:2207.1259.. 2022.
[38] "Enterprise Voice AI: STT. TTS & Agent APIs Deepgram -ewe-dddeepppmoooo hhtps://deeppram com/ [Accessed 17-04-2025].
[39] W. Kang, X. Yang, Z. Yao, F. Kuang, Y. Yang, L. Guo, L. Lin, and D. Povey, "Libriheavy: A 50,000 hours asr corpus with punctuation casing and context," in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10991-10 995.
[40] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu, "LibriTTS: A corpus derived from librispeech for text-to-speech," in Interspeech 2019, 2019, pp. 1526-1530.
[41] Y. Li, H. Su, X. Shen, W. Li, Z. Cao, and S. Niu, "Dailydialog: A manually labelled multi-turn dialogue dataset," arXiv preprint arXiv:/710.039/) 2017.
[42] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, "Librispeech: an asr corpus based on public domain audio books," " 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206-5210.
[43] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," " in Medical image computing and computer-assisted Aitervention-MICA. 2015: 1 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer, 2015, pp. 234-241.
[44] S.-g. Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon, "BigVGAN: A universal neural vocoder with large-scale training," in The Eleventh International Conference Learning Representations.
[45] N. Kanda, Y. Gaur, X. Wang, Z. Meng, Z. Chen, T. Zhou, and T. Yoshioka, "Joint speaker counting, speech recognition, and speaker identification for overlapped speech of any number of speakers," " arXiv preprint arXiv:2006. 200:2000.0090,, 2020.
[46] K. Baba, W. Nakata, Y. Saito, and H. Saruwatari, "The t05 system for the VoiceMOS Challenge 2024: Transfer learning from deep image classifier to naturalness MOS prediction of high- quality synthetic speech," in IEEE Spoken Language Technology Workshop (SLT), 2024.
[47] S. Chen, C. Wang. Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al., "Wavlm: Large- scale supervieee pre-training for full stack speech processing." " IEEE Journal of Selected Topics in Signal Processing, vol. 16. 6. pp. 1505-1518, 2022.
[48] D. Wang and Chen, "Supervised speech separation based on deep learning: An overview," JEEF/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 10, pp. 1702-1726, 2018.
[49] W. Chen, E. C. Van Tung Pham, E. S. Chng, and X. Zhong, "Overlapped speech detection based on spectral and spatial feature fusion." in Interspeech, 2021, pp. 1189-4193
[50] T. Kristjansson and J. Hershey, "High resolution signal reconstruction," in 2003 IEEE Workshop on Automatic Speech Recognition and Understanding (IEEE Cat. No. 03EX721). IEEE, 2003, pp. 291-296.
[51] A. Radford, J. W. Kim, Xu, G. Brockman, C. McLeavey, and I. Sutskever, "Robust speech recognition via large-scale weak supervision," in International conference on machine learning. PMLR, 2023, pp. 28
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
answer: [Yes]
Justification: We introduce CodiFM a novel framework for zero-shot multi-talker dialogue generation based fully NAR flow-matching method. It the first model that can achieve fine-gratn ane--grined control over speech timing, overlaps and pauses.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? answer: [Yes]
14

---

Justification: In the last section, we mentioned our limitation. The training data's 's reliance on automatically transcribed conversations introduces minor inaccuracies disfluencies like repetitions backchannel words, and the absence of word-level timestamps for overlapping speech necessitates simulated dialogue training, leading to slight deviations in naturalness and audio quality.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
answer: [NA]
Justification:
This paper mainly focused on the design of generation pipeline, training
strategy, data preparation, and system implementation. We did not propose new theoretical algorithms or results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
15

---

answer: [Yes]
Justification: We provide detailed introduction system design, data preparation and training configuration. We will make the testing set publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
answer: [No]
Justification: We are not ready to open source the data and code at the current stage. We will consider this plan in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https! for more details.
//nips /nipps.cc/publll http:cccppppccc/tettette.e...clt http://ullggguudeeeeeeee /tt.uuddeeeeddeeuuuuiiiiiioooopa
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
16

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
answer: [Yes]
Justification: We introduce in detail the training and test configuration, including model architecture, optimizer and hyperparameters, etc. also provide ablation studies for these configurations.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
answer: [Yes]
Justification: We calculate the mean and standard deviation of each system across the whole testing set. We report the I-sigma error bars for all results under the assumption of normal distribution.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
17

---

answer: [Yes]
Justification: We provide detailed configuration training and test, including the GPU, memory, total training steps, evaluation tools, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e-g-. preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics h
answer: [Yes]
Justification: This research conducted in the paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
answer: [Yes]
Justification: We discussed broader impacts and potential risks in the last paragraph of Section 7.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
18

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
answer: [NA]
Justification: Currently do not have plans for releasing the model or dataset. If going to release, we will make sure to release a safeguard with it.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
answer: [Yes]
Justification: All the open-source code and evaluation tools that we use are credited in this paper, and the license and terms are properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprssitthaaeee thcode cote..co/aaae com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- ee-pacaaedd both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
answer: [Yes]
Justification: We introduce the dialogue testing set a new asset. This asset is well documented with license, limitations, data resources etc.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
19

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
answer: [NA]
Justification: Our paper neither involves crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
answer: [NA]
Justification: Our paper neither involves crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
answer: [NA]
The core method development in this research does not involve LLMs any important, original, or non-standard components. We only use LLM for writing and editing of this paper.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, ntn-standard components.
Please refer to LLM
for what should or should not be described.
20

---

A Model Comparison
We compare proposed Co VoMix with two baseline models: MoonCast [12 and Sesame 10 The detailed model comparisons are shown in Table where N/A means not available.
Table 2: Model detail comparison
Model Params Decoder Vocoder
Training Data (hour)
Training Resource GPU
Model
Type
Backbone
Dialogue Monologue
MoonCast
AR+NAR
2.5B
0.8B
0,25B
215k
300k
64 A100 80GB
Sesame
AR
IB
0.1B
1000k
N/A 32 VI00 32GB
CoVoMix2
NAR
0.3B
/
0.01B
3k
65k
B
Monologue Performance Comparison
To ensure that the dialogue generation models retain strong performance in monologue settings, we evaluate all systems on audiobook-sto -stlle monologue test set. The transcription of the audio prompts is obtained using Whisper-large-v3 [S 12 As shown in Table 3 our Co trained solely on monologue data, outperforms all baseline models. Notably, after the second- stage training on dialogue data, Co VoMix2 maintains its monologue generation capabilities without much degradation. In contrast, Sesame and MoonCast exhibit degraded performance due to hallucination issues, frequently producing repetitive semantically irrelevant content.
In extreme cases, MoonCast fails terminate the speech generation, resulting in infinite loops. Note that the transcription recognized by Whisper may contain errors, which might be one of the reasons why these language model's hallucination worsens. Our Co VoMMix however, does require the transcription corresponding to the prompt, thus effectively avoiding such issues.
Table 3: Model performance comparison on monologue data.
Model
WER 4
SIM
UTMOS
MoonCast
15.92-+28.50
0.63
2.82##0448
Sesame
12,89
0.72+0..17
2.81
CoVoMix2t
4.30
0.78
2.97+:0.5
CoVoMix2
4.45:14.19
0.66
3.19
CoVoMix21 is only pre-trained on monologue data.
C Extended Dialogue Performance Comparison
C.1 Dialogue Performance Comparison with Prompts of different length
To demonstrate that our model maintains a stable advantage even with very long or very short prompts, we conducted an additional experiment. We selected 20 speech clips, each longer than 20 seconds, from 20 different speakers in the LibriSpeech dataset [42]. We then trimmed these clips to lengths of 3 to 18 seconds. For transcription, used Whisper-large-v3, and for the text component, we used 100 dialogues from the Dailydialog dataset. As shown Figure 6 our model's SA-WER and SA-SIM performance remains consistently strong across all prompt lengths. This further confirms that our model performs reliably well, regardless of how long the input prompt is.
21

---

SA-WER Values Over Time
SA-SIM Values Over Time
30
0.55
25
0.50
SA-WER
20
SA-SIM
0.45
15
10
0.30
Time Intervals (s)
Time Intervals (s)
MoonCast
Sesame
CoVoMix2
Figure 6: SA-WER and SA-SIM comparison with prompt of different lengths
C.2 Dialogue Performance Comparison with Different Synthesized Length
Due to resource constraints, our training data currently consists of segments less than 30 seconds in duration. Nevertheless, our model demonstrates the ability to infer dialogues longer than seconds, and our test set includes numerous dialogues exceeding one minute.
To further investigate long dialogue generation, we manually designed four dialogues with varying durations. As shown in Table 4 we observed that both Co and Mooncast exhibit degraded performance after 90 seconds, while Sesame fails to generate dialogues longer than 120 seconds. Additionally, Sesame showed unstable results even for dialogues under 30 seconds, primarily due to speaker confusion issues.
Table 4: Comparison of SA-WER/SA-SIM with Different Synthesized Length
System
30s
60s
90s
120s
Mooncast
5.81/0.647
4.57/0.576
13.95/0.487
19.75/0.516
Sesame
24.41/0.677
5.22/0.585
3.98/0.263
Failed
CoVoMix2
6.97/0.686
5.88/0.648
14.74/0.618
14.64/0.627
C.3 Dialogue Performance Comparison under Real-world Scenarios
more realistic conversation style speeches prompts. The speaker prompts for this new set are derived from 10 distinct real-life dialogues from the NCSSD dataset-CEN (34). This dataset collected from the internet and features prompts with natural conversational prosody, including various acoustic scenarios such as background noise. The corresponding text content for this test set is sourced from the DailyDialog dataset 41
We designed an additional test
As presented Table 5 our model, CoMoiix,, consistently demonstrates superior performance compared to baseline models even under these challenging real-world conditions.
Table 5: Performance Comparison under Real-world Scenarios
System SA-WER
SA-SIM
UTMOS
Mooncast
26.26
0.276
2.09
Sesame
25.19
0.250
1.80
CoVoMix2
9.32
0.322
2.89
22

---

D
Ablation Studies
To better understand the design choices and training strategies that contribute to the performance of CoVoMix2, we conducted series of ablation studies covering data mixing, training stages, and the handling of silence tokens.
D.1 Data Mixing Strategy
We examine how various combinations of training data affect performance. Table pressnts results on both monologue and dialogue test sets, comparing models trained different subsets of the data. We observe that in addition to dialogue podcast data, incorporating monologue podcast data significantly improves performance. Adding LibriTTS and LibriHeavy audiobook datasets provides further gains, with LibriTTS showing slightly better results due to its cleaner and more accurately aligned transcriptions. Interestingly, simulated dialogue data (created by combining monologues with certail ratio of overlap or silence) does not consistently improve performance and may even slightly degrade it. This is likely due to distribution mismatch artifacts introduced by the simulation process. However, simulated dialogues are still necessary to enable overlapping speech generation capability, as ASR -rancraacerd real-world data rarely provides accurate overlapping timestamps.
Table 6: Ablation Study of data
Training Data
Monologue SIM1 UTMOS
Dialogue SA-WER, SA-SIM UTMOS1
ID Dia Mono
Simu
LH
LT
WER
WERL
1
X
X
x
x
8.58
0.65
3.12
8.11
8.63
0.50
2.97
2
X
x
6.30
0.61
3.18
7.89
8.51
0.50
3.00
3
/
/
/
x 7.69
0.64
3.10
8.72
9.31
0.51
2.99
4
/
x
6.18
0.66
3.24
7.08
7.43
0.55
3.06
5
/
/
V
V
5.37
0.65
3.21
6.00
6.58
0.57
3.10
6
X
V
5.42
0.65
3.27
5.81
6.27
0.56
3.13
D.2
Training Stages
CoVoMix2 is trained using two-stage training pipeline, with an optional third fine-tuning stage that can further improve performance. To assess the benefit of the first and the third stage, we conduct ablation study by eliminating the first stage and fine-tuning the model for 2,000 steps on just 20 minutes of
dialogue data. Although this fine-tuning data is small in scale, it is highly accurate and clean, in contrast the -SRatransrribed training data.
Table shows results on a test set featuring the same two speakers as in the fine-tuning data. Models trained without the pretraining perform very poorly, with WER and SA-WER exceeding 80%, highlighting the critical importance of the first stage. Fine-tuning with just 2,000 steps on a small amount of samann-nnotated dialogue data significantly improves accuracy, especially in WER and SA-WER. This suggests that even limited high-quality data can help correct transcription noise learned from -SR-transcribed training sets.
Table Impact of training stages on two-speaker test set
Pre-train
Fine-tune
WER
SA-WERJ
SA-SIM
UTMOS*
X
82.66
92.40
0.29
3.00
X
5.67
6.21
0.53
3.48
/
3.66
3.81
0.56
3.35
D.3 Silence Token Representation
Our input text streams include not only characters but also two types of special tokens: a continuation token [P] and silence token [S]. We explore three options. 1) using [P] for both continuation and silence. 2) using generic [S] for silence. 3) using speaker-aware silence tokens [S1] and [S2].
23

---

Table 8 summarizes the results. We find that using separate silence token [S] improves dialogue accuracy and controllability over using [P] alone. However, using speaker- aware silence tokens does not provide significant benefits, and in some cases slightly degrades dialogue performance, possibly
due to over-specification.
Table 8: Impact of different silence token representation
Monologue SIM
Dialogue
Silence Token
WER
UTMOS
WER
SA-WERJ
SA-SIMf
UTMOS1
5.26
0.64
3.15
6.45
7.09
0.55
3.08
S
6.83
0.65
3.14
5.22
5.59
0.56
3.02
[S1, S2]
5.37
0.65
3.21
6.00
6.58
0.57
3.10
E
Subjective Evaluation
Table shows the Comparative Mean Opinion Score (CMOS) Evaluation instruction. 15 professional linguistic experts provide judges for this CMOS evaluation. They provide a rating of preference in the (-3 to +3) range, given audios with the same transcription and the speaker prompts.
Table 9: Comparative Subjective Mean Opinion Score (CMOS) Evaluation Instructions
Instruction
This to compare two Al podcast audio. Listen both audios as you are listening to real human podcast and give your preference considering the following aspects.
1. Speaker Attribution Accuracy: Is each utterance spoken by the correct speaker as indicated in the transcription? Does the voice match the intended identity?
2. Speaker Turn Handling: Are speaker changes handled correctly and clearly? Does the transition between speakers align with the dialogue flow?
3. Speaker Consistency: Does each speaker maintain consistent voice, tone, and speaking style throughout the clip?
4. Interactivity and Fluency: Does the conversation sound natural and interactive? Are the
responses well-timed and appropriate, without awkward pauses overlaps?
5. Coherence with the Transcription: Does the spoken content accurately follow the given transcription, including emotion, prosody, and speaking style?
Which one better?
-3: Dialogue is much better
-2: Dialogue is better
-1: Dialogue is slightly better
0: Can't tell which is better
1: Dialogue slightly better
2: Dialogue 2 is better
3: Dialogue is much better
24


Reviews:
[{"review_id": "5AEf1XmgCn", "reviewer": "Reviewer_zySF", "summary": "CodiFM is presented as a fully non-autoregressive, flow-matching model that performs zero-shot dialogue generation. The method adapts the E2TTS recipe to multi-speaker scenarios by first splitting a dialogue transcript into separate text streams for each speaker, padding each sentence so its length matches the target mel duration, and inserting special silence tokens wherever a given speaker is not speaking. During training only one short reference utterance per speaker is provided and its loss region is masked to prevent the model from simply copying the prompt. Training proceeds in two stages: the model is first pretrained as a monologue zero-shot TTS system, and is then fine-tuned on several thousand hours of dialogue audio. Experiments show that, relative to strong baselines such as Mooncast and Sesame, CodiFM retains higher pronunciation accuracy and speaker-similarity scores with lower real-time factor (RTF).", "questions": "1. Can CodiFM learn timing from text-level turns only?\n\nA fair comparison would train your model without ground-truth sentence timings (analogous to how Mooncast/Sesame operate) and evaluate whether it can infer durations.\n\n2. Non-verbal events and long-form generation\n\nMooncast generates laughter, sighs, etc., and scales to lengthy dialogues. Can CodiFM do the same? \n\n3. Prosodic variability\n\nIn multi-sentence dialogues, later utterances may naturally deviate in emotion or prosody from the initial prompt. How does CodiFM balance speaker similarity with such variability, and is SA-SIM the right metric?", "limitations": "The method assumes external knowledge of sentence-level timing for each speaker. This timing information limits usability in real-world dialogue synthesis, where only turn-level transcripts are typically available.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper shows that the E2TTS recipe can be adapted to multi-speaker settings with a few thousand hours of dialogue data.\n2. CodiFM outperforms Mooncast and Sesame on pronunciation accuracy, SA-SIM, and real-time factor.\n3. The authors dissect the effect of data composition, the two-stage training schedule, and the use of silence tokens.\n\n**Weaknesses**\n1. CodiFM requires exact start/end times for every utterance, whereas Mooncast and Sesame learn timing implicitly. The comparison is therefore not apples to apples, and the requirement itself is unrealistic in many settings.\n2. The baselines support long, natural conversations with non-verbal sounds. CodiFM’s ability to handle non-verbal events or very long samples is not demonstrated.\n3. Speaker-similarity metric may overstate quality. SA-SIM compares each generated sentence to its single reference prompt, potentially penalizing desirable prosodic variation over a longer dialogue.", "was_revised": true, "final_justification": "While I initially had concerns regarding the lack of duration modeling in the NAR approach compared to the AR baseline, the demo convincingly demonstrated the necessity of the proposed model for real-world applications such as video dubbing, where dialogue must fit within specific time constraints. Moreover, the model showed no disadvantage against the baseline in challenging cases like long sentences, and the remaining concerns were well addressed. Therefore, I raise my score from borderline reject to borderline accept.", "rebuttal": "**Dear reviewer zySF,**\n\n**We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. We have clarified some unclear writing that caused misunderstandings and added extra evaluations, which will appear in the revised version. The detailed responses are listed below.**\n\n**R1: Importance of introducing duration (Weakness1 and Question1)**\n\nWe acknowledge your query regarding the importance of introducing duration control. Currently, there are no existing Non-Autoregressive (NAR) dialogue generation models, compelling us to compare our work primarily with Autoregressive (AR) models.\n\nA key distinction of our approach, in contrast to conventional NAR models such as VoiceBox and FastSpeech which necessitate highly accurate duration prediction, is our reduced dependency on precise duration, requiring only utterance-level duration information. During training, durations are extracted using ASR and diarization tools. We recognize that these tools may introduce inaccuracies, meaning our extracted durations are not exact ground truth. Crucially, during inference, our model does not rely on ground truth durations. For instance, for a single dialogue, diverse durations can be employed to generate a variety of dialogues.\n\nWe emphasize that the ability to control speaker duration addresses a critical gap in existing AR models. Many real-world scenarios, such as video dubbing or advertisements with strict time limits (e.g., 15-second spots), necessitate precise duration control for each speaker. Furthermore, as illustrated in the first sample on our demo page, our model can generate crucial silences—for example, a waiter instructing someone to wait—which AR-based baseline models are unable to achieve.\n\nIt is important to note that CodiFM does not strictly require users to provide duration information. We offer an optional functionality where duration can be roughly estimated by counting syllables in the text.\n\n**R2: The non-verbal and long dialogue generation capability (Weakness2 and Question2)**\n\nRegarding non-verbal capabilities such as laughter, our model's generation of such behaviors is data-driven, consistent with findings in prior work [9]. While we did not explicitly annotate these behaviors in our dataset, their presence allows our model to generate them, albeit in a non-controllable manner. We observe that Mooncast also exhibits non-controllable laughter generation, sometimes at inappropriate moments, as demonstrated in audio sample 2. We will include additional samples to showcase our model's non-verbal behavior generation capabilities in the revised demo page. We plan to annotate future datasets to enable controllable generation of such functionalities.\n\nDue to resource constraints, our training data currently consists of segments less than 30 seconds in duration. Nevertheless, our model demonstrates the ability to infer dialogues longer than 30 seconds, and our test set includes numerous dialogues exceeding one minute. To further investigate long dialogue generation, we manually designed four dialogues with varying durations. We observed that both CodiFM and Mooncast exhibit degraded performance after 90 seconds, while Sesame fails to generate dialogues longer than 120 seconds. Additionally, Sesame showed unstable results even for dialogues under 30 seconds, primarily due to speaker confusion issues.\n\nTable R2 presents the SA-WER/SA-SIM evaluations for synthesized data of different durations. None of the compared systems support streaming. For long-form generation, AR models typically require significant waiting times. In contrast, our CodiFM can generate each segment in parallel with a Real-Time Factor (RTF) of less than 0.3, significantly reducing waiting times.\n\nWhile Mooncast is specifically designed for long-form generation, it actually utilizes a chunk-wise Flow-matching detokenizer. For each turn in a long dialogue, it calls the Flow-matching detokenizer, requiring the audio of each speaker prompt to be provided every time. Therefore, it can't be considered a purely long-form generation model because the audio detokenizer still generates and concatenates multiple audio segments. Similarly, Sesame generates each utterance separately, relying on previous history, which means it is not a truly long-form generation model and is consequently less impacted by increased duration.\n\n**Table R2: SA-WER/SA-SIM evaluation of different duration of synthesized data**\n| Model    | 30s         | 60s         | 90s         | 120s        |\n| :------- | :---------- | :---------- | :---------- | :---------- |\n| Mooncast | 5.81/0.647  | 4.57/0.576  | 13.95/0.487 | 19.75/0.516 |\n| Sesame   | 24.41/0.677 | 5.22/0.585  | 3.98/0.263  | Failed      |\n| CodiFM   | 6.97/0.686  | 5.88/0.648  | 14.74/0.618 | 14.64/0.627 |\n\n\n\n\n**R3: About SA-SIM metric (Weakness3 and Question 3)**\n\nWe would like to clarify that our proposed SA-SIM metric is designed to measure speaker identity similarity relative to the speaker prompt, rather than prosodic similarity. Even with variations in prosody, the speaker similarity for the same speaker consistently remains high.\n\nWe did, however, account for the potential for prosodic variations in speakers over different time intervals. To address this, we separately calculated speaker consistency, as detailed in Figure 4b of Section 5.2. This was achieved by comparing the similarity of multiple segments from the same speaker within the same conversation. Our results demonstrate that CodiFM can better maintain speaker identity regardless of prosodic changes. Our experiments further revealed that Mooncast and Sesame exhibit lower SA-SIM scores because certain speakers' voices undergo complete identity changes, not merely prosodic shifts.\n\n**R4: About the choice of NAR instead of AR model**\n\nThe core of several questions revolves around the fundamental distinctions between NAR and AR models. While all previous research in dialogue generation has focused on AR models, there is a lack of purely NAR dialogue models. Although AR models have demonstrated effectiveness in dialogue generation, they possess several drawbacks:\n\n1.\tSpeaker Confusion: As evidenced by the SA-SIM results, some sentences may be attributed to an incorrect speaker.\n\n2.\tSlow Inference Speed: AR models typically suffer from slow inference.\n\n3.\tUncontrollable Generation: They lack control over crucial aspects such as duration, speaking speed, and silence.\n\nNAR models, on the other hand, can effectively mitigate these limitations. Their efficacy has been established in monologue generation, and they demonstrate strong potential in Text-to-Speech (TTS) generation tasks. This motivated our investigation into a purely NAR model for dialogue generation.\n\n**Finally, we would like to express our gratitude again for your time and effort in reviewing our paper. Considering this is the first attempt at non-autoregressive multi-talker dialogue generation and that we have added a comparison with previous work, we would appreciate it if you could consider increasing your score. Please do not hesitate to let us know if you have any further concerns or comments. We would be happy to address them.**"}, {"review_id": "1knPwHBLMf", "reviewer": "Reviewer_W9cW", "summary": "This paper systematically tackles the long-overlooked challenge of multi-speaker zero-shot speech dialogue generation: ensuring speaker consistency, controllable speech overlap, and fast inference without relying on intermediate token representations. The authors introduce CodiFM—an end-to-end, fully non-autoregressive framework based on flow matching—and propose three mechanisms to overcome current limitations. To validate the approach, the authors compile a mixed training corpus, and diverse simulated overlapping-speech samples. Extensive evaluations against state-of-the-art baselines show that CodiFM delivers significantly better speaker consistency and overlap naturalness, achieves 4–7× faster inference, and earns the highest human ratings for fluency and interactivity.", "questions": "- Could the authors clarify what fundamental modeling challenges are newly addressed in the multi-speaker dialogue generation context? What makes CodiFM more than a direct combination of known techniques?\n- What are the difficulties in expanding beyond two person conversations or other languages?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The paper presents a technically sound framework, combining transcription-level speaker disentanglement, sentence-level alignment, and prompt-level masking within a fully non-autoregressive (NAR) flow-matching architecture. The proposed training curriculum and data mixing strategies are practical and well-explained.\n- The paper is generally well-written and logically structured, with clear motivation and modular presentation of model components.\n- The work addresses an important and underexplored problem: zero-shot multi-talker dialogue synthesis with overlapping speech and fine-grained timing control.\n- The integration of flow matching with multi-stream transcription input is novel within the domain of zero-shot multi-speaker TTS.\n### Weaknesses\n- Despite the claim that CodiFM works without prompt transcriptions, the advantages of this property are not convincingly quantified or analyzed in ablation studies.\n- The experiment is too simple. The scalability and robustness of the model to more than two speakers or longer, more complex dialogues are not demonstrated.\n- Most technical components (e.g., flow-matching, classifier-free guidance, silence token modeling) are adapted from existing literature. The novelty primarily lies in how these pieces are combined.\n- Typo. There are two Step 3 in Figure 1.", "was_revised": true, "final_justification": "The author addresses the issues I am concerned about. I find the idea of this paper interesting, and I keep my score.", "rebuttal": "**Dear Reviewer W9cW,**\n\n**We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. We have addressed the raised concerns by clarifying certain aspects of our methodology and incorporating additional analyses regarding the influence of prompt transcription and our system's performance in extended and complex scenarios. These revisions will be reflected in the updated version of the paper. Our detailed responses are listed below.**\n\n**R1: Benefits of not using prompt’s transcription (Weakness 1)**\n\nOur decision to not utilize prompt transcription is motivated by two primary factors: \n\nFirst, this design enhances the model's robustness across diverse speaker prompts. Conventional TTS models typically rely on ASR models or user-provided transcriptions. However, when prompts contain noise or exhibit unconventional speaking styles (e.g., whispering or speech with accent), ASR performance can degrade significantly, leading to inaccurate transcriptions (e.g., WER exceeding 50%). As demonstrated in Table R1, an inaccurate prompt transcription can severely compromise the quality of the generated speech in TTS systems. Therefore, our approach is not aimed at improving baseline performance but rather at mitigating performance degradation when confronted with challenging prompts and eliminating dependency on ASR models. \n\nSecond, this approach facilitates in-context learning by enabling the extraction of speaker prompts from the same dialogue samples used during training. Obtaining accurate transcriptions for arbitrary segments of an utterance is challenging. By removing the requirement for prompt transcription, our model can utilize any monologue segment from a training dialogue sample as a speaker prompt during the training phase. \n\n**Table R1: WER (%) evaluation for monologue and dialogue models with different prompt’s transcription**\n| Scenario  | Model      | Correct trans | 50% wrong trans | 100% wrong trans |\n| :-------- | :--------- | :------------ | :-------------- | :--------------- |\n| Monologue | CosyVoice2 | 6.44          | 28.40           | 64.12            |\n|           | F5TTS      | 4.41          | 33.96           | 67.14            |\n|           | MastGCT    | 2.62          | 37.70           | 40.81            |\n| Dialogue  | MoonCast   | 2.83          | 6.16            | 25.37            |\n|           | Sesame     | 2.06          | 2.12            | 8.68             |\n\n\n**R2: Scalability and Robustness of Model (Weakness 2, Question 2)**\n\nOur current experiments are constrained by resource and data limitations, specifically a dataset containing only two speakers. Despite these limitations, our proposed framework is language-independent and can be readily extended to accommodate multiple speakers. Future work will focus on developing a multi-speaker, multilingual system with three or more speakers.\n\nNonetheless, our model exhibits robust performance during inference, even for dialogues three times longer than those in our training data (less than 30s). We conducted evaluations on four manually designed dialogues of varying durations, assessing SA-WER and SA-SIM as presented in Table R3. We found that CodiFM's performance begins to degrade after 90 seconds, a degradation also observed with MoonCast. Sesame fails to generate dialogues longer than 120 seconds. Furthermore, Sesame exhibited unstable results even under 30 seconds due to speaker confusion issues.\n\nIt is important to clarify that MoonCast, although designed for long-form generation, employs a chunk-wise Flow-matching detokenizer. For each turn in a long dialogue, it calls the Flow-matching detokenizer, requiring the audio of each speaker prompt to be provided every time. Therefore, it can't be considered a purely long-form generation model because the audio detokenizer still generates and concatenates multiple audio segments. Similarly, Sesame generates each utterance separately, relying on previous history, which means it is not a truly long-form generation model and is consequently less impacted by increased duration.\n\n\n**Table R2: SA-WER/SA-SIM evaluation of different duration of synthesized data**\n|           | 30s         | 60s         | 90s         | 120s        |\n| :-------- | :---------- | :---------- | :---------- | :---------- |\n| Mooncast  | 5.81/0.647  | 4.57/0.576  | 13.95/0.487 | 19.75/0.516 |\n| Sesame    | 24.41/0.677 | 5.22/0.585  | 3.98/0.263  | Failed      |\n| CodiFM    | 6.97/0.686  | 5.88/0.648  | 14.74/0.618 | 14.64/0.627 |\n\nMoreover, we have evaluated our model under extreme multi-turn dialogue generation scenarios. While our training data consists of segments less than 30 seconds, with 81% having fewer than two speaker changes and only 0.5% exceeding six speaker changes (maximum of 12), CodiFM successfully generated dialogues with 20 or more speaker changes, where each speaker speak only single words, without errors. In contrast, baseline models failed in these challenging conditions due to severe speaker confusion. We will incorporate these challenging samples into the revised demo page.\n\n**R3: About similarity to existing works (Weakness 3 and Question 1)**\n\nTo achieve state-of-the-art (SOTA) performance in dialogue speech synthesis, we have fully leveraged existing SOTA technologies in our model building. While this may give the impression that CodiFM is a mere combination of existing methods, these individual methods, on their own, do not fully address the critical challenges inherent in our specific scenario: *generating a synthesized dialogue with accurate pronunciation by the correct speaker and appropriate turn-taking, including silences and overlaps.*\n\nLet’s clarify our contributions compared to existing work, which can be further illustrated in Table R3: \n\n1.\tOur work enables the generation of entire dialogues with accurate pronunciation by the accurate speaker, effectively resolving the speaker confusion problem prevalent in prior AR-based models.\n\n2.\tWe employ a multi-stream Transcription-Level Speaker Disentanglement strategy to enable the model to precisely understand \"who speaks what and when\".\n\n3.\tWe propose the Prompt-Level Random Masking strategy to enhance in-context learning capabilities in dialogue scenarios. This strategy eliminates the need for prompt transcription, thereby removing the dependency on ASR tools.\n\n**Table R3: System comparison**\n| System    | Arch | Duration  | Monologue | Dialogue | Controllable Speed | Prompt’s transcription | Silence/Overlap |\n| :-------- | :--- | :-------- | :-------- | :------- | :----------------- | :--------------------- | :-------------- |\n| VoiceBox  | NAR  | Phoneme   | Yes       | No       | No                 | Yes                    | No              |\n| E2TTS     | NAR  | Utterance | Yes       | No       | Yes                | Yes                    | No              |\n| MoonCast  | AR   | /         | Yes       | Yes      | No                 | Yes                    | No              |\n| Sesame    | AR   | /         | Yes       | Yes      | No                 | Yes                    | No              |\n| CodiFM    | NAR  | Utterance | Yes       | Yes      | Yes                | No                     | Yes             |\n\n**R4: About the two Step 3 in Figure 1(Weakness 4)**\n\nWe appreciate you bringing this point to our attention. You're right to highlight the potential for confusion with \"Step 3\" in Figure 1. Our intention was to depict the training stage and the inference stage as separate processes, each with three distinct steps. We'll revise Figure 1 to clearly differentiate between the steps of the training process and the steps of the inference process, ensuring there's no ambiguity. \n\n**Finally, we would like to express our gratitude again for your time and effort in reviewing our paper. Considering this is the first attempt at non-autoregressive multi-talker dialogue generation and that we have incorporated extensive comparisons under challenging scenarios with previous works, we would appreciate it if you could consider increasing your score. Please do not hesitate to let us know if you have any further concerns or comments. We would be happy to address them.**"}, {"review_id": "DlV5uGlxey", "reviewer": "Reviewer_qZtn", "summary": "The paper introduces CodiFM, a fully non-autoregressive (NAR) framework for generating high-quality, zero-shot multi-speaker dialogues. Unlike previous approaches that rely on autoregressive decoding or intermediate representations (like audio tokens), CodiFM directly predicts mel-spectrograms from disentangled, multi-stream transcriptions using a flow-matching-based generative model. Key innovations include transcription-level speaker disentanglement, sentence-level alignment, and prompt-level masking, enabling precise control over speaker timing, overlap, and identity. Trained with a two-stage curriculum on large-scale monologue and simulated dialogue data, CodiFM achieves good performance in speech quality, speaker consistency, and inference speed, outperforming strong baselines like MoonCast and Sesame. It also supports fine-grained timing control, overlapping speech, and cross-lingual voice cloning, making it highly suitable for real-world applications such as podcast generation and virtual agents.", "questions": "* How long can the synthesized dialogue be with the current model?\n* The conclusion drawn in Section 5.2 seems somewhat inconsistent with subfigure (a) of Figure 4. If I understand correctly, MoonCast appears to exhibit overall higher similarity than Sesame, and its performance continues to improve with more dialogue turns. Of course, this might be due to the analysis being based on a single data example. Providing average results over a batch of data would make the conclusion clearer and more convincing.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "* Strengths:\n    * The proposed approach is well-motivated, and the experimental results demonstrate its feasibility and effectiveness.\n    * The writing of the paper is clear and easy to follow.\n\n* Weaknesses:\n    * While the paper is technically sound, the level of novelty is somewhat limited, as some of the key design choices — such as using fully non-autoregressive methods for speech synthesis and avoiding explicit duration modeling — have already been established in prior TTS literature.\n    * The generalizability of the proposed method to real-world applications remains unconvincing. The training data used in this work is substantially smaller than that of several baseline methods (the training data processing pipeline does not clearly demonstrate which steps contribute to generating higher-quality data.), and the prompts used during evaluation are drawn from the LibriSpeech test-clean set. Given that these prompts differ significantly from real-world user speech in terms of content, style, and acoustic conditions, it is difficult to assess how well the model would perform when conditioned on more realistic prompts.", "was_revised": true, "final_justification": "The authors have provided comparative experiments under realistic and long-duration scenarios in their rebuttal, which has alleviated some of my concerns. However, I still feel that the novelty of the proposed method is somewhat limited, as it appears to resemble training an E2TTS model on conversational data. Nevertheless, considering that the field of dialogue generation is still at a relatively early stage, I believe this work can be regarded as having a certain level of contribution at this point. This is the reason why I ultimately assigned a score of 4.", "rebuttal": "**Dear Reviewer qZtn,** \n\n**We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. We have carefully considered all your comments and suggestions, which have significantly helped us improve the manuscript. We have conducted additional experiments on realistic scenarios and analyzed the model's performance across different durations, and these results will be incorporated into the revised version of our paper. The detailed responses are listed below.**\n\n**R1: About similarity to existing works (Weakness 1)**\n\nTo achieve state-of-the-art (SOTA) performance in dialogue speech synthesis, we have fully leveraged existing SOTA technologies in our model building. While this may give the impression that CodiFM is a mere combination of existing methods, these individual methods, on their own, do not fully address the critical challenges inherent in our specific scenario: *generating synthesized dialogue with accurate pronunciation by the correct speaker and appropriate turn-taking, including silences and overlaps.*\n\nTo clarify our distinct contributions, we have provided a comparative analysis with existing works in Table R1. Our key contributions are as follows:\n\n1.\tOur work enables the generation of entire dialogues with accurate pronunciation by the accurate speaker, effectively resolving the speaker confusion problem prevalent in prior AR-based models.\n\n2.\tWe employ a novel multi-stream Transcription-Level Speaker Disentanglement strategy to enable the model to precisely understand \"who speaks what and when.\n\n3.\tWe propose the Prompt-Level Random Masking strategy to enhance in-context learning capabilities specifically within dialogue scenarios. This strategy eliminates the need for prompt transcription, thereby removing the dependency on ASR tools and simplifying the generation process.\n\n**Table R1: System comparison**\n| System    | Arch | Duration  | Monologue | Dialogue | Controllable Speed | Prompt’s transcription | Silence/Overlap |\n| :-------- | :--- | :-------- | :-------- | :------- | :----------------- | :--------------------- | :-------------- |\n| VoiceBox  | NAR  | Phoneme   | Yes       | No       | No                 | Yes                    | No              |\n| E2TTS     | NAR  | Utterance | Yes       | No       | Yes                | Yes                    | No              |\n| MoonCast  | AR   | /         | Yes       | Yes      | No                 | Yes                    | No              |\n| Sesame    | AR   | /         | Yes       | Yes      | No                 | Yes                    | No              |\n| CodiFM    | NAR  | Utterance | Yes       | Yes      | Yes                | No                     | Yes             |\n\n**R2: Generalizability to real-world application (Weakness 2)**\n\nWe appreciate your concern regarding the generalizability of using LibriSpeech for evaluating real conversation styles. We agree that assessing performance in real-world scenarios is important.\n\nTo address this, we have designed an additional test set. The speaker prompts for this new set are derived from 10 distinct real-life dialogues from the NCSSD dataset-CEN, as proposed in Reference [31]. This dataset is collected from the internet and features prompts with natural conversational prosody, including various acoustic scenarios such as background noise. The corresponding text content for this test set is sourced from the DailyDialog dataset [37].\n\nAs presented in Table R2, our model, CodiFM, consistently demonstrates superior performance compared to baseline models even under these challenging real-world conditions.\n\n**Table R2: System performance comparison under real-world scenarios**\n|   | SA-WER | SA-SIM | UTMOS |\n|---|--------|--------|-------|\n| MoonCast | 26.26  | 0.276  | 2.09  |\n| Sesame | 25.19  | 0.250  | 1.80  |\n| CodiFM | 9.32   | 0.322  | 2.89  |\n\n\n**R3: About CodiFM’s maximum inference duration (Question 1)**\n\nOur training data is limited to durations under 30 seconds due to resource constraints. However, our model is capable of inferring dialogues longer than 30 seconds, and our existing test set includes numerous dialogues exceeding one minute in length. We conducted evaluations on four manually designed dialogues of varying durations, assessing SA-WER and SA-SIM as presented in Table R3. We found that CodiFM's performance begins to degrade after 90 seconds, and this degradation is also observed with MoonCast. Sesame fails to generate dialogues longer than 120 seconds. Furthermore, Sesame exhibited unstable results even under 30 seconds due to speaker confusion issues.\n\n**Table R3: SA-WER/SA-SIM evaluation of different duration of synthesized data**\n|           | 30s         | 60s         | 90s         | 120s        |\n| :-------- | :---------- | :---------- | :---------- | :---------- |\n| Mooncast  | 5.81/0.647  | 4.57/0.576  | 13.95/0.487 | 19.75/0.516 |\n| Sesame    | 24.41/0.677 | 5.22/0.585  | 3.98/0.263  | Failed      |\n| CodiFM    | 6.97/0.686  | 5.88/0.648  | 14.74/0.618 | 14.64/0.627 |\n\nIt is important to clarify that MoonCast, although designed for long-form generation, employs a chunk-wise Flow-matching detokenizer. For each turn in a long dialogue, it calls the Flow-matching detokenizer, requiring the audio of each speaker prompt to be provided every time. Therefore, it can't be considered a purely long-form generation model because the audio detokenizer still generates and concatenates multiple audio segments. Similarly, Sesame generates each utterance separately, relying on previous history, which means it is not a truly long-form generation model and is consequently less impacted by increased duration.\n\n\n**R4: About the mis-leading visualization of Figure 4a (Question 2)**\n\nThank you for pointing out the potential for misunderstanding in Figure 4. We acknowledge that Figure 4a was indeed generated using statistics from a single dialogue, which may have led to misinterpretations. \n\nAfter averaging across multiple dialogues, our model consistently maintains its leading performance. In contrast, Sesame and MoonCast exhibit unstable generation, leading to inconsistent results across different scenarios. For instance, in Table R2, we observe that MoonCast's speaker similarity on this realistic test set is slightly higher than Sesame's. Given that both models frequently encounter speaker confusion problems, their performance shows high variance, making it challenging to compare them when both are unstable.\n\nWe will update this Figure 4a in the revised manuscript with a new visualization that incorporates multiple speech samples, thereby providing a more representative and clearer illustration. Additionally, we will update these new samples on our revised demo page.\n\n**Finally, we would like to express our gratitude again for your time and effort in reviewing our paper. Considering this is the first attempt at non-autoregressive multi-talker dialogue generation and that we have incorporated extensive comparisons under realistic and long duration scenarios, we would appreciate it if you could consider increasing your score. Please do not hesitate to let us know if you have any further concerns or comments. We would be happy to address them.**"}, {"review_id": "XikmmYlzeF", "reviewer": "Reviewer_49qE", "summary": "The paper introduces CodiFM, a non-autoregressive framework that generates realistic multi-speaker dialogues directly from transcriptions using a flow-matching-based generative model. It employs techniques such as speaker disentanglement, sentence-level alignment, and prompt-level random masking to improve the naturalness of dialogue synthesis. Additionally, it supports controllable speech overlap and precise timing. The performance of CodiFM is compared against strong baselines like MoonCast and Sesame in terms of speech quality, speaker consistency, and inference speed.", "questions": "- Regarding the intermediate representations mentioned at line 111, it's clear that avoiding audio tokens offers efficiency gains, but does it also yield performance improvements? Specifically, as reported in [4], using a timely compact codec as an intermediate representation improved WER compared to directly modeling mel-spectrograms. Conducting explicit performance comparisons would clarify this trade-off, enabling informed decisions in the research community. Have the authors explored or conducted comparative experiments using such intermediate representations and if so, what were the findings?\n- In Section 3.4 (line 173), have the authors empirically compared the performance of simultaneous unconditional conditioning of prompt speech and text versus separate unconditional conditioning? Additionally, approaches like dual classifier-free guidance [5] might be relevant; has this been explored experimentally?\n- In Table 5, why did UTMOS scores decrease after fine-tuning? Could the authors provide the specific audio samples used for evaluating each case to facilitate detailed analysis?\n- Regarding line 195 (\"3,000 hours of English podcast data\"), details about data acquisition are missing. Was this dataset collected via web crawling, and how were copyright concerns addressed? An explicit ethical review discussion on this matter seems necessary.\n- Concerning line 209, setting the overlap rate to 100% may lead to unnatural dialogue scenarios, as complete overlap is rare in natural speech. Could the authors share some training samples exhibiting this full overlap scenario to clarify the nature of such data and its potential influence on model performance?\n- The official Sesame demo on Hugging Face (https://huggingface.co/spaces/sesame/csm-1b) uses significantly longer prompts compared to the prompts provided in the supplementary samples of this paper. Typically, shorter prompts can degrade the quality of the synthesized speech. Have the authors explicitly investigated how the length of prompts affects performance, particularly in a direct comparison with Sesame using longer prompts?\n- The proposed methodology naturally raises interesting considerations when extending to real-user interactive applications. In such a scenario, the following questions become relevant, and the authors' intuitions would be valuable:\n    - How might the proposed non-autoregressive approach be practically extended to interactive scenarios? For instance, introducing autoregressive modeling might seem necessary to handle real-time responsiveness, but could this dilute the efficiency and speed advantages central to CodiFM?\n    - Additionally, while simultaneous generation is highlighted as a benefit (line 103), it could pose challenges for real-user interactions, where incremental responsiveness might be required. Have the authors considered how simultaneous generation might complicate or limit real-world interactive applications, and how could this be effectively addressed?\n\n[4] Lee, K., Kim, D. W., Kim, J., Chung, S., & Cho, J. (2025). DiTTo-TTS: Diffusion transformers for scalable text-to-speech without domain-specific factors. In *The Thirteenth International Conference on Learning Representations*.\n\n[5] Lee, Y., Yeon, I., Nam, J., & Chung, J. S. (2024, April). Voiceldm: Text-to-speech with environmental context. In *ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)* (pp. 12566-12571). IEEE.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Quality**\n\n- The authors reasonably implemented a model and data setup for generating natural-sounding, multi-speaker dialogues and conducted appropriate experiments to support their claims. They also clearly outlined the limitations of their proposed methodology.\n\n**Clarity**\n\n- The paper is generally well-structured and clearly articulates its contributions, making most explanations intuitive and accessible. However, several clarifications are needed.\n- Figure 1 contains substantial information; adding detailed explanations for each element directly to the caption would significantly aid comprehension, as currently, its full meaning becomes clear only after reading the entire paper.\n- In the \"Related Work\" section (line 92), citations for the CSS methodology appear incomplete; references [1, 2] should likely be included.\n- In Figure 2, the meaning of the term \"Prompt Candidate\" (top-left) requires clarification. Specifically, line 151 mentions available monologue segments—is this exactly what \"Prompt Candidate\" represents, and does the model select prompts from these candidates?\n- In Section 3.1,\n    - It's unclear how the text streams corresponding to the speaker sequences are exactly fed into the model. Clarifying explicitly if each speaker’s utterance undergoes separate text encoding (or embedding) followed by concatenation, and clearly differentiating how this process differs from a single stream approach, would better highlight the strengths of the proposed modeling method. Currently, even considering the description around line 176 (\"\\~ the construction of the input text streams z.\"), the precise method remains ambiguous.\n    - Further, since `[Spk1]` and `[Spk2]` tokens appear in both text streams, the structural reason that allows the model to distinguish between individual speakers despite this overlapping information needs clearer articulation.\n    - The approach used for identifying sentence-level alignment within overlapping speech segments should be explicitly described.\n    - It is unclear how the text streams' lengths are aligned precisely with the corresponding audio durations. Specifically, are the `t_start` and `t_end` indices based on audio timestamps, and how exactly are timings synchronized between streams `z1` and `z2`? Clarifying these alignment details is essential to ensure the method's reproducibility.\n- In Section 3.3,\n    - Regarding the phrase in line 149, \"~ to ensure robust and diverse speaker conditioning,\" the authors should explicitly define \"robustness\" and \"diversity.\" Is robustness implying consistent model performance even with unseen speakers, and diversity indicating coverage of a broader range of speaker characteristics?\n    - Concerning the context embedding $m_{ctx}$ mentioned in line 153 and visualized in Figure 1, during training, $m_{ctx}$ appears to be derived from partial real dialogue segments, some of which overlap with $z$. However, during inference, $m_{ctx}$ could come from entirely unrelated speech segments, suggesting potential train-inference mismatch. This could theoretically lead to performance degradation. To mitigate this, have the authors considered strategies such as using randomized segments or shuffling within the selected prompt, potentially eliminating the need for loss masking altogether?\n- In Section 4.1,\n    - At line 197, the phrase \"a market-leading speech enhancement API\" requires a proper citation or explicit naming of the exact API used, as it is currently unspecified.\n    - For the \"simulated dialogue-style data\" described in lines 203 and 207, it is understood that the authors concatenate speech segments from two different speakers (datasets). However, it would help if the authors clarified how they maintain coherent context between these segments. Explicitly addressing how context continuity is ensured or justified in these simulated dialogues is crucial for understanding the reliability and validity of their data augmentation approach.\n- In Section 4.3,\n    - Have you compared your approach with Moshi [3] as a baseline? If not, explicitly stating the rationale for excluding Moshi from baseline comparisons would strengthen the evaluation's completeness.\n    - Regarding the 15 professional linguistic experts employed for CMOS evaluations, please clarify the hiring or recruitment process, such as whether a platform like Amazon MTurk was used or if other methods were applied.\n- In Section 5.1,\n    - At line 275, the statement \"containing prompts and prior audio from both speakers\" appears applicable to both Sesame and CodiFM based on Figure 2. Clarifying explicitly how the input formulation for CodiFM differs from Sesame would better highlight the methodological advantages of the proposed approach.\n    - At line 278, the claimed \"stable and reliable output quality\" contrasts somewhat with the provided supplementary samples, where CodiFM outputs seem slightly less expressive and more flat compared to the baselines. It would be beneficial if the authors discussed this aspect clearly.\n    - Based on personal observation, individual utterances from Sesame sound more natural, but it appears less consistent in maintaining speaker identity compared to CodiFM. Given this:\n        - In Table 7, speaker consistency and fluency are evaluated together. Wouldn't evaluating these two criteria separately yield more accurate insights?\n        - Have the authors measured the actual error rates related to speaker inconsistency for each model? Such measurement is crucial because high inconsistency could over-penalize models that otherwise generate natural-sounding speech.\n        - Additionally, performance differences related to prompt length should be explicitly evaluated or discussed to better contextualize and interpret results.\n- In Section 5.2,\n    - Evaluating performance based on a single selected long dialogue sample may limit the statistical validity of the analysis. Could you please share the samples of the selected dialouge?\n    - Regarding the interpretation at line 293 (\"additional conditioning does not improve speaker accuracy\"), rather than stating no improvement, it seems more appropriate to conclude that additional conditioning reduces variance in speaker similarity scores.\n        - This interpretation is supported by the observation that CodiFM and Sesame exhibit similar trends without per-utterance prompts, while MoonCast significantly deviates in Figure 4 (a).\n        - Given that per-sentence prompts naturally reduce variability in similarity scores, this might adversely impact naturalness in dialogue synthesis. Supplementary audio samples indeed suggest MoonCast maintains a stylistically consistent but somewhat unnatural and disconnected delivery style. Clarifying this aspect would strengthen the analytical insights of the paper.\n- In Section 6,\n    - At line 321, stating that \"simulated dialogue data may introduce noise\" implies ambiguity regarding its usefulness. Clarifying explicitly whether including this data (despite noise) remains beneficial specifically due to the necessity of handling overlapping speech would prevent confusion.\n    - Separately, poorer performance with simulated data might be due to extreme overlap (100% overlap scenarios) or context discontinuity between concatenated segments. Have the authors conducted a detailed analysis on specific problematic samples to identify underlying issues? Providing concrete observations or examples from such analyses would significantly enhance understanding of limitations and potential improvements.\n- In Table 4, the authors should clarify whether WER calculations for single utterances and overlapping segments were performed identically. The noticeable discrepancy—maintained similarity and UTMOS but worsened WER when moving from condition 2 to 3—suggests potential issues specifically in overlapping speech. Explicitly addressing how overlaps were handled in WER calculations would resolve this confusion.\n\n**Significance**\n\n- The analysis of the two-stage training and the impact of data characteristics at each stage is insightful and impactful. Readers looking for substantial real-time factor (RTF) reduction without sacrificing significant quality will likely find the proposed method useful.\n- However, based on sample listening, the improvements are not groundbreaking across all dimensions, including speech accuracy and expressiveness, beyond the primarily emphasized speaker consistency. Additionally, evaluations appear somewhat narrow and potentially biased. Expanding evaluation methodologies and addressing possible biases could significantly enhance the impact of this work.\n\n**Originality**\n\n- The paper clearly differentiates itself from prior work, presenting unique solutions to address identified problems and experimentally validating efficiency improvements.\n- However, the related work survey has gaps, such as missing key citations, which somewhat weaken the comprehensiveness of the literature review. Addressing these omissions would further strengthen the paper's originality and academic thoroughness.\n\n[1] Guo, H., Zhang, S., Soong, F. K., He, L., & Xie, L. (2021, January). Conversational end-to-end tts for voice agents. In *2021 IEEE Spoken Language Technology Workshop (SLT)* (pp. 403-409). IEEE.\n\n[2] Lee, K., Park, K., & Kim, D. (2023, June). Dailytalk: Spoken dialogue dataset for conversational text-to-speech. In *ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)* (pp. 1-5). IEEE.\n\n[3] Défossez, A., Mazaré, L., Orsini, M., Royer, A., Pérez, P., Jégou, H., ... & Zeghidour, N. (2024). Moshi: a speech-text foundation model for real-time dialogue. *arXiv preprint arXiv:2410.00037*.", "was_revised": true, "final_justification": "**After the first round:**\nI've raised my score from **2** to **3**, acknowledging the authors' detailed clarifications and additional experiments:\n\n**Resolved Issues:**\n\n- Methodological details regarding model input construction and conditioning strategies are largely clarified.\n- Concerns around the evaluation procedures and data characteristics were significantly addressed.\n\n**Remaining Issues:**\n\n- The procedure for generating simulated dialogue data remains unclear, especially regarding how context coherence is ensured.\n- The evaluation of baselines (e.g., Sesame) with sufficiently long prompts was not comprehensively addressed, potentially biasing the comparative analysis.\n- Questions related to training-inference mismatch, specifically prompt-context conditioning, remain unresolved.\n\nGiven these critical remaining concerns, especially regarding simulated data creation and prompt-length effects, the submission still falls slightly short of the acceptance threshold. I encourage the authors to thoroughly address these points in future revisions.\n\n**After the last round**:\nBased on the authors’ detailed rebuttal and clarifications, many of my initial concerns regarding methodology, data construction, and evaluation have been satisfactorily addressed. In particular, the explanation about prompt exclusion from loss calculation (masking), the simulated dialogue data format, and the challenges related to overlapping speech reconstruction were clear and convincing.\n\nHowever, some issues remain, including the need for explicit incorporation of prompt length performance analysis in the main paper and addressing a few missing citations. These points, while important, do not undermine the overall contribution.\n\nGiven the substantial progress made in clarifying key points and the promising results demonstrated, I recommend increasing the score to **4**. I encourage the authors to carefully address the remaining minor issues in the final revision to strengthen the paper further.", "rebuttal": "**Dear Reviewer 49qE,** \n\n**Thank you for your comprehensive review and valuable feedback. We use C,O,S,Q to abbreviate Clarity, Originality, Significance and Question for our responses.**\n\n**C1-C3, O2** We will modify the Figure 1 with detailed explanations, and add references [1~5] you pointed out in the corresponding sections.\n\n**C4**During training, we extract a random segment from each dialogue sample to serve as the speaker prompt. This segment must consist of consecutive frames spoken exclusively by a single speaker. We refer to such segments as \"candidates.\"\n\n**C5** Our method processes each speaker's text as a two-stream sequence (z = [$z_1$, $z_2$]), which is first concatenated under the following description, and then encoded.Each stream includes a prompt part and a synthesized part. The prompt part uses [spk1] and [spk2] as indicators (not text tokens) to replace the transcription of the prompt, distinguishing which segments belong to each speaker.  The synthesized part contains the actual dialogue, aligned with mel-spectrogram frames at the frame level. For example, in a 6-second dialogue with 100 frames per second, if spk1 speaks 30 characters from 1-3s and spk2 speaks 20 characters from 4-5s, $z_1$ includes 100-130 real characters and 130-300 continuation tokens [P], while $z_2$ includes 400-420 real characters and 420-500 tokens [P]. Other tokens in z are silence tokens. \n\nFor the real-world dialogue, we utilize the ASR and diarization tool to get the sentence-level timestamp $t_{start}$ and $t_{end}$. These timestamps are not ground truth and may contain errors. We also use simulated LibriTTS data to provide accurate ground-truth timestamps, even for overlapping speech segments. This approach contrasts with single-stream methods by enabling temporal alignment and features like specific silence insertion and overlapping, as shown in our demo.\n\n**C6.1, C6.2** The robustness and diverse means the model is robust given prompts under challenging and noisy acoustic environments. We achieve that by eliminating the dependence of prompt’s transcriptions, while other model face serious performance degradation if the ASR transcription is wrong, demonstrated in Table R1 replied to reviewer W9cW.  Additionally, the random chosen of $m_{ctx}$ enables in-context learning by allowing the model to extract speaker prompts directly from training dialogue samples. This design can even improve speaker similarity because the prompts and generated speech share the same acoustic environment.\n\n**C7.1** We did not mention the name of the speech enhancement API due to the double-blind review policy. We will cite it in the camera-ready paper if the manuscript is accepted.\n\n**C7.2** Our current simulated data did not take contextual coherence into consideration due to its complexity and our goal is to model overlapped speech generation. we recognize the value of contextual coherence and will revisit this consideration in future work.\n\n**C8.1** The open-source version of Moshi does not support multi-speaker dialogue generation. Moshi's dialogue capability is primarily focused on human-user and agent interactions, not the generation of speech for multiple speakers. While Moshi's paper mentions a multi-stream TTS model for dialogue synthesis, there are no publicly available codes or data to enable reproduction of their described model.\n\n**C8.2:** The 15 professional linguistic experts we selected were specifically trained in linguistics and experienced in subjective testing, and they were crowd-sourced for this evaluation. Due to the double-blind review policy, we cannot explicitly name the company platform used but will disclose it upon paper acceptance.\n\n**C9.1:** The main differences between our input format and the baselines lie in two key perspectives (1) CodiFM employs multiple text streams to distinguish \"who speaks when and what,\" whereas Sesame utilizes only a single text stream with only the content and the speaker id, which can more readily lead to speaker confusion. (2) CodiFM places the prompt at the very beginning of the input sequence and generates the entire dialogue in parallel. Sesame, being an AR model, generates each utterance sequentially. While its prompts are also at the beginning, this can cause confusion because the previously generated audio is often more proximate to the segment being synthesized than the initial prompt.\n\n**C9.2:** The primary focus of this paper is on generating synthesized dialogue with accurate pronunciation by the correct speaker and appropriate turn-taking, including silences and overlaps. The \"stable and reliable output quality\" mentioned on Line 275 refers to the enhanced stability of our model in terms of speaker accuracy and content fidelity, minimizing instances where Speaker A utters content intended for Speaker B.  In this work, we did not specifically design or optimize for speech expressiveness. As a zero-shot model, the expressiveness should be guided by the given prompts in addition to speaker timbre.  \n\n**C9.3 and Q6:** Thank you for your valuable suggestion. We conducted an additional fluency test using CMOS scores as per your request, and the results are presented in Table C93-1.\n\n**Table C93-1: Fluency CMOS**\n| | Fluency-CMOS | \n| :--- | :--- | \n|MoonCast|-0.644| \n|Sesame|-0.317| \n|CodiFM| 0.000 | \n\nTo assess speaker change accuracy, we introduced two metrics: Diff-SC (difference between correct and generated speaker changes, ideally zero) and Speaker Change Error Rate (SCER) (proportion of dialogues with incorrect speaker change counts). Table C93-2 proves that our CodiFM outperformed others on both metrics, proving its superior accuracy in handling speaker changes.\n\n**Table C93-2: Evaluation for Speaker Change**\n| Model|Diff-SC|SCER|\n| :--- | :-- | :-- |\n| Mooncast | -0.78 ±2.14 | 47.57% |\n| Sesame   | -0.66±1.87  |21.98% |\n| CodiFM   | -0.16±0.57  | 9.50% |\n\nTable C93-3 demonstrates that prompt length affects Sesame's performance, with shorter prompts leading to poorer results. In contrast, our CodiFM model is robust to varying prompt lengths, showing minimal impact on performance. MoonCast, however, exhibits less stable performance.\n\n**Table C93-3: SA-WER/SA-SIM evaluation of different speaker prompt length**\n| Model |1-3s|3-6s|6-9s|9-12s|12-15s|\n| :----| :--- | :--- | :--- | :---- | :--- |\n|Mooncast|21.526/0.297|34.218/0.215|7.723/0.380|46.02/0.276|3.72/0.480|\n|Sesame|7.425/0.373|5.584/0.364|6.114/0.447|5.889/0.421|4.865/0.446|\n|CodiFM|6.95/0.556 |6.002/0.543|6.002/0.535|6.002/0.543|6.002/0.542|\n\n**C10.1** Thank you for highlighting the potential for misunderstanding in Figure 4. We acknowledge that Figure 4 was initially generated using statistics from a single dialogue, which may have led to misinterpretations. After averaging across multiple dialogues, our model consistently maintains its leading performance. \n\n**C10.2** We will revise our description accordingly.\n\n**C11.1, C11.2 and C12** Adding *simulated dialogue data* can hurt model performance, likely due to noise from concatenating utterances with different acoustic environments. However, simulated LibriTTS data is usable because it it clean. The performance degradation noted in Table 4 is not related to overlapping speech, as the test set does not include overlapping speech and WER was calculated after segmenting sentences with the Deepgram ASR+Diarization API.\n\n**S2** We acknowledge that preferences vary and baseline models may excel in expressiveness due to their larger parameters and datasets. We've addressed concerns about evaluation bias by introducing a new, more diverse real-world test set with varied acoustic conditions and conversational prosody (Table R2 for reviewer qZtn), and we've expanded our analysis to include longer dialogue durations (Table R3 for reviewer qZtn).\n\n**Q1** The utilization of timely compact codec as an intermediate representation does not always yield performance improvements for dialogue generation task. [9] have demonstrated that using semantic tokens as intermediates could cause instances of words being omitted or duplicated in synthesized dialogues with the Hubert semantic tokens. \n\n**Q2** We dropped both the text and audio conditions for $p_{uncond}$. In the remaining cases, we dropped the audio condition with $p_{dropx}$. We will revise the paper accordingly.\n\n**Q3** Fine-tuning with dialogue data, led to a slight, but not significant, decrease in UTMOS scores, dropping from 3.48±0.19 (pre-train only) to 3.35±0.17 (pre-train and fine-tune). It is because the dialogue data is not clean even after speech enhancement.\n\n**Q4** The 3,000-hour dataset used in this study is internal and consists of two-person conversations. Due to confidentiality constraints, we are unable to disclose specific details regarding data acquisition. However, the dataset has been reviewed by our internal legal department to ensure compliance with copyright and ethical standards. \n\n**Q5** Thank you for pointing that out. Indeed, full overlap data is uncommon in real life. Currently, we’ve used the LibriTTS dataset to simulate overlapping data (0%–100%), similar to speech separation methods. In the future, we’ll integrate real-world overlap statistics to improve our simulations.\n\n**Q7** Thank you for your interest in extending CodiFM to real-user interactive applications. While this lies beyond the scope of our current study, which focuses on use cases such as video dubbing and two-host podcast generation. We would like to share a few of our initial thoughts. A flow-matching model could potentially support streaming by developing a chunk-aware, causal flow-matching variant [1]. Additionally, techniques such as consistency distillation or leveraging mean flows could enable one-step inference. Therefore, it also has the potential to real-user interactions.\n\n[1] Z. Du, et al, CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models"}]

Author rebuttal:

Meta-review: The submission introduces CodiFM, a fully non-autoregressive framework for zero-shot multi-speaker dialogue generation using flow matching. The paper tackles the problem of speaker consistency, overlapping speech, and efficient inference, while avoiding reliance on intermediate token representations. The rebuttal provided additional experiments showing robustness across prompt lengths, realistic noisy conditions, and extended dialogue durations. Major concerns of reviewers were addressed. Ethical concerns regarding dataset licensing and human evaluation standards were also addressed.

Overall, strengths outweigh weaknesses, and I recommend to accept this paper.
Decision: Accept (poster)