CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations
Authors: leying zhang
Yao Qian
Long Zhou
Shujie LIU
Dongmei Wang
Xiaofei Wang
Midia Yousefi
Yanmin Qian
Jinyu Li
Lei He
sheng zhao
Michael Zeng
Venue: NeurIPS 2024 poster (2024)
Keywords: text-to-speech
dialogue generation
zero-shot
Primary area: speech_and_audio
TL;DR: We introduce CoVoMix: Conversational Voice Mixture Generation, a novel model for zero-shot, human-like, multi-speaker, multi-round dialogue speech generation

Abstract: Recent advancements in zero-shot text-to-speech (TTS) modeling have led to significant strides in generating high-fidelity and diverse speech. However, dialogue generation, along with achieving human-like naturalness in speech, continues to be a challenge. In this paper, we introduce CoVoMix: Conversational Voice Mixture Generation, a novel model for zero-shot, human-like, multi-speaker, multi-round dialogue speech generation. CoVoMix first converts dialogue text into multiple streams of discrete tokens, with each token stream representing semantic information for individual talkers. These token streams are then fed into a flow-matching based acoustic model to generate mixed mel-spectrograms. Finally, the speech waveforms are produced using a HiFi-GAN model. Furthermore, we devise a comprehensive set of metrics for measuring the effectiveness of dialogue modeling and generation. Our experimental results show that CoVoMix can generate dialogues that are not only human-like in their naturalness and coherence but also involve multiple talkers engaging in multiple rounds of conversation. This is exemplified by instances generated in a single channel where one speaker's utterance is seamlessly mixed with another's interjections or laughter, indicating the latter's role as an attentive listener. Audio samples are enclosed in the supplementary.

Co VoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations
Leying Zhang 1.2+
Yao Qian
Long Zhou²
Shujie Liu
Dongmei Wang"
Xiaofei Wang"
Midia Yousefi²
Yanmin Qian'
Jinyu LE
Lei He
Sheng Zhao²
Michael Zeng"
Shanggaai
Jiao
Tong University, China
Microsoft, USA
Abstract
Recent advancements in zero-shot text-to-speech (TTS) modeling have led sig- nificant strides in generating high-fidelity and diverse speech. However, dialogue generation, along with achieving human-like naturalness speech, continues to be a challenge. In this paper, we introduce CoVoMix: Conversational Voice Mixture Generation, novel model for zero-shot, human-liks like, multi-speaker, multi-round dialogue speech generation. CoVox first converts dialogue text into multiple streams of discrete tokens, with each token stream representing semantic informa- tion for individual talkers. These token streams are then fed into flow-matching based acoustic model generate mixed specsppctooos.ss. Finally, the speech waveforms are produced using HiFi-GAN model. Furthermore, we devise com- prehensive set of metrics for measuring the effectiveness of dialogue modeling and generation. Our experimental results show that Co CoVoxix can generate dialogues that are not only human-like their naturalness and coherence but also involve multiple talkers engaging in multiple rounds of conversation. This exemplified by instances generated mixed with another's interjections or laughter, indicating the latter's 's role attentive listener. Audio samples are available at https://aka.
single channel where one speaker's utterance seamlessly
Text-To To-Semanii Model
Semantic token sequences of each speakee
Acoustic Model
Text Encoodr
Semantic Decoder
Vector Field Estimator
Voooder
Tokenizer
Prompt speaker1
Mixed Mel-speetrogram
Mono-channel
Multi raund
Dialogue
Which animal you like?
Prompt
B: Emm, like elephonts.
speaker2
A: Wow! Me coo!
Figure I: The overview of CoVoMix framework, which consists of multi-stream text-to-semantic model, conditional flow-m fow-matching based acoustic model for mixed mel-spectrogram generation, and HiFi-GAN based vocoder for waveform production.
Wook done during an internship at Microsoft Azure Al. zhangleying@ssjuu.du..mo t "Coresponddncc: yaoqian @ micrrosoft.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Zero-shot Text-to-speech (TTS) technology aims to create human-like natural speech with voice characteristics prompted by the context. Recent deep learning advancements have significantly improved synthesized speech quality, especially in formal reading scenarios [13] However, TTS systems still struggle with rendering spontaneous-stylee speech and managing seamless transitions during conversations corvesations-ccommon common occurrences in everyday human communication 4 S On the one hand, spontaneous-style speech encompasses phenomena like filled pauses, interjections, repairs, repetitions, and laughter, which lend human-realistie naturalness spoken language [6]. On the other hand, natural conversations, speakers intuitively time their speech, determining when to speak and when to yield, resulting in scamless transitions with appropriate overlaps or moments of silence 87. Overlapping speech, defined as more than one person is speaking, can easily exceed 20%, informal gathering conversational speech [9]. Considering these conversational features, summarize three main challenges in generating spontaneous dialogues.
First, the scarcity of high-quality, spontaneous conversational datasets, along with the difficulty in segmenting paralinguistic behaviors, continues to be significant obstacle the field. Spontaneous behavior and non-verbal expressions such as laughter, receive insufficient attention in speech synthesis. Existing high-quality datasets for spontaneous and conversational speech are relatively small and involve limited number of speakers [10]. Identifying and segmenting these paralinguistic features, as highlighted by studies 12 poses difficulties. Models that require pre-alignment necessitate sophisticated manual annotation. Otherwise, the low-quality data can adversely impact performance, particularly in TTS tasks [13].
Second, research on turn-taking mechanisms in multi-speaker dialogues is less explored. In such dialogues, the forthcoming speaker anticipates the end of the current speaker's turn by analyzing structural and contextual cues, and then begins their speech seamlessly at the anticipated transition point 14. Speakers tend to adapt the pause length to match other participants [15. Overlapping speech occurs when one speaker starts talking before another finishes, which can be a sign of enthusiasm or an attempt to take turns [16
Third, the consistency in multi-round dialogues is not guaranteed in conventional methods. Simply concatenating each utterance form dialogue may result in inconsistent speaker characteristics, particularly when the same speaker engages in multi-round dialogue. In addition, the context of the preceding utterance plays an important role in the control of pauses and prosody, and thus influences the naturalness of generated dialogues [19].
To effectively generate human-likk dialogue, we propose Co named Conversational Voice Mixture Generation, for multi-talker dialogue generation, shown in Figure [ The main contributions of the paper can be summarized follows:
1. To the best of our knowledge, it is the first attempt at zero-shot, human-like, multi-talker conversational mixed speech generation. We propose 1) a simultaneous multi-stream semantic token prediction, with each stream representing an individual talker, from dialogue text; and 2) multi-talker flow-matching based acoustic model for generating a mixed mono mel-spectrogram given multiple contexts. It is capable of generating single-channel multi-round dialogue containing multiple speakers concurrently, enabling simultancous timbre cloning of multiple speakers in zero-shot scenarios.
2. We design variety of evaluation metrics for dialogue generation, and demonstrate that the CoVoMix model is proficient at generating both human-like dialogues and monologues, exhibiting natural speaker turn-taking, realistic vocal burst-like laughter, consistent speech in terms of speaker similarity throughout multiple rounds of dialogue.
3. We employ the Fisher dataset 20 for this study, which was curated for robust speech recognition. Our approach includes comprehensive strategy for processing this dataset, including both training and evaluation for monologue and dialogue speech. The data processing script, along with the model training and inference codes are publicly available

---

2 Related Work
2.1 Zero-shot Text-to-Speech
The goal of Zero-shot TTS is to synthesize speech in target voice which was unseen during training. given only the target transcript and a short reference of the target voice 21 24
Zero-shot TTS
systems are generally divided into two categories: (i) Diffusion-based Shoo TTS
3 35 and
(ii) Neural Codec-based Zero-shot TTS 2 36 39
Diffusion-bassed Zero-shot TTS models handle the problem in a non-auto regressive manner and have shown excellent performance in audio generation tasks [28 40 41. Many previous works, such as [25 32, 42), use log Mel spectrograms as intermediate features and generate speech waveforms using high-quality vocoders. For instance, Viff 44 employs a VAE-GAN autoencoder [44] to encode the Mel-Spectrogram into a latent space, jointly modeling phoneme duration and Mcl- spectrogram. FastSpeech 45] generates mel-spectrograms alignment between phoneme sequence and generated spectrogram with an explicit length regulator and duration predictor model. Flow matching training is method that is closely related to Diffusion models offering simpler trajectories and requiring fewer function evaluations during inference [46]
parallel for faster inference, managing
47 34). Flow Matching (FM) [46] is a simulation-fres approach for training continuous normalizing flows (CNFs) at scale based on regressing vector fields of fixed conditional probability paths. The relationship between the vector field and the flow & is defined via an ordinary differential equation (ODE) dæ, (y) ( do y (46). Benefiting from flow matching models, sext-to-specch models, such as VoiceFlow 488. MatchaTTS [4] and Voicebox
[49] can generate high-quality
speech efficiently.
On the other side, Neural Modecc-based methods formulate the TTS problem as a token-based language modeling task (2 [5)] VALL-E (2) zero-shot text-to-speech model, is a text conditioned language model trained on EnCodec tokens [0 SPEAR-TTS [51] similar AudioLM [52] which carries out the speech generation process in two steps: first, it maps the text into discrete semantic tokens, then, in the second step, converts the semantic tokens into acoustic tokens. In the recently proposed BASE-TTS [53, the authors propose to model the joint distribution of text tokens and discrete speech representations referred to as speechcodecs followed by a convolution- these speechcodes into waveforms in an incremental, streamable manner. NaturalSpeech3 [54] combines codees and diffusion modeling, achieving significant improvements in speech quality, prosody, intelligibility, and scalability with IB IB-parameter model trained 200K hours of data. It decomposes speech waveforms into content, prosody, timbre, and acoustic details, reconstructing speech from these disentangled representations using factorized diffusion model.
decoder which converts
2.2 Dialogue Generation
dGSLM 8 represents the pioneering textless model for generating naturalistic spoken dialogues. It utilizes dual-tower transformer with cross-attention its architectural backbone and leverages HuBERT [53] semantic token sequence as its input for the speech continuation task. This model generates two-channel spoken dialogue regressessss,,,,, without reliance text labels. However, its textless nature constrains its ability to direct the content of the speech it produces, occasionally leading to less intelligible outputs.
CHATS (56). while based on the same architectural principles as dGSLM, is designed to convert written dialogues into spoken conversations. It is capable of generating speech for both speaker and listener sides, conditioning on speaker ID, phoneme sequence, and context from the speaker's s side, without requiring transcriptions for spontaneous behaviors or laughter. However, it does not support the capabilities of zero-shot voice cloning, relying solely on speaker ID for retaining speaker characteristics.
SoundStorm (36), on the other side, an iterative generative method that converts semantic tokens into acoustic audio tokens. It can perform zero-shot monologue and dialogue synthesis. Yet, the synthesized dialogue generated in a sequential manner and thus sounds less realistic, lacking any spontancous behaviors or instances of overlapping speech.
3

---

3 CoVoMix
Zero-shot speech generation is a task where a model synthesizes speech in a target voice that was not present in its training data. This task requires only transcript of what to be spoken and speech prompt-a brief sample recording of the target voice. It is generally achieved by in-context learning with a dataset of transcribed speech {z. y) where y and denote speech utterances and their transcriptions, respectively. Zero-shot multi-talker conversational speech synthesis is designed to generate the voices of multiple speakers simultancously, based on their transcriptions and prompts. Our approach differs from the traditional method in which each voice is synthesized individually, and then concatenated to form a dialogue. Our goal in this work is capture the dynamic nature of real conversations, where participants may speak over each other or respond spontaneously with interjections such as laughter.
Our proposed CoMix, shown in Figure D consists of a multi-stream text-to-semantic model, an acoustic model and a vocoder. The text-to-semantic model first generates multi-stream semantic token sequences for each speaker, given the dialogue transcription. Then the acoustic model transforms these semantic sequences into a mixed mel-spectrogram. A vanilla HiFi-GAN vocoder [57] finally synthesizes mono-channel
dialogue from the mel-spectrogram. We utilize a conversa- tional dataset D = {z. y) for training, where y = [y': yy) represents a stereo dialogue featuring two speakers, and corresponds to the text transcription annotated with speaker tags.
3.1 Multi-stream Text-to-Semante Model
The multi-stream text-to-semantee model is architecture. It takes in a text token sequence generated by BERT text tokenizer (58), augmented with special tokens denoting speaker transitions and interjections. The output comprises multi- stream semantic token sequence. For this study, we focus on a dual-stream setup for a dialogue between two speakers. We employ a pre-trained HuBERT model g speech tokenizer to extract the clustered discrete HuBERT hidden units as semantic token sequences and process two channels of waveform, separately. If the dialogues are captured in a single-channel recording, is necessary to perform speaker separation to produce a dual-channel waveform in our approach. The process of semantic token extraction operates at the frame level, with a time shift of 20 milliseconds, resulting in the presence of duplicated tokens within the sequence. We train this model on a paired speech-texx dataset with cross-entropy loss, as
model based on encoder-decoder
(0) Liss ΣΣ log (/
where 8; is the ith semantic token and e denotes the eth speaker. In order to predict two-stream semantic token sequences, we adopt strategy wherein we divide the semantic embedding into two distinct segments (splitting it into two halves along the feature dimension) in the final linear layer of the decoder. Each segment corresponds This approach enables the model to capture contextual information not only from each individual speaker but also from their interaction. The dynamic exchange between speakers significantly shapes the semantic content, especially in scenarios involving multi-round conversations.
different speaker participating in the conversation.
3.2 Acoustic Model
The acoustic model is a flow-matchin based transformer encoder, which generates mixed mel- spectrogram, given multi-stre stream semantic token sequences and multi-speaker prompts.
At each timestamp E [0. 1). lookup table first embeds the semantic token sequence = [s s into Semb = [sLmb 2mm) for two speakers. We extract the corresponding mixed mel-spectrogram m and individual mel-spectrogram [m² m²] for each speaker of dialogue y. We randomly choose mask. The masked part m m @ mask is to be predicted, while the seen part maiz [m @ (1 mask), m² o mask)] is considered as prompt.
wttp://wwweccoooooooooooooo/oooc fisher fisher (MIT Li- cense)

---

At each flow step i, we sample w = (1 (1 min )t) - tm, where miin is a papee--arreeter to control deviation and my is sampled from (m, Then, the sample w flow step t, the acoustic prompt Metz; and semantic embedding sequences Semb are concatenated frame-by-frame to obtain an input matrix Winput. Conditional Flow Matching (CFM) [46] is a per-example training objective, which provides equivalent gradients and does not require explicit knowledge of the intractable target vector field. Therefore, we train the acoustic model to learn the mixed mel-spectrogram with objective as in 2 where V/ (w, metar Sembi 0) is the transformer output with flow step t.
CCFM = p))(())))))) mask @ ((m (1 min )mo) e' (w metar sembi 0))
(2)
During inference, to sample mixed mel-spectrogram m from learned distribution P1 (m |8, metx) ), we sample gaussian noise my from pa N (m I) use an ODE solver evaluate the flow Φ1 (ma) given do: (mo)) = we mata. Sembi 8) and (o
m70-
We also classifier-free guidance, method to trade off mode coverage and sample fidelity 49 59], in the training for flow-matching model. During training, the acoustic prompt matz and semantic sequences Semb dropped with Puncond- During inference, vector field (w, Meta. Semb; 8) shown in Equation 3 to replace (2 meta: Sembi f), where a is a hyperparameter controlling the strength of guidance.
the modified
ūy (w. mate: Sembi 0) = (1 + a)va(w, metz: Semb: 0) (w;; 0)
(3)
4 Experimental Setup
4.1 Data Preparation
The dataset used in this work is Fisher dataset [20). which is telephone conversation dataset with 2,000h English conversations about various topies. Each dialogue was recorded in two channels with an 8kHz sample rate and an average duration of 10 minutes. We randomly divide the Fisher dataset into train/valid/test sets with 97/1/2 split. Each set has different speakers.
The data preparation is different for monologue and dialogue. For monologue, following Nemo [60 script 2 minimum duration requirement, which is set to 10 seconds by default. We prepare the corresponding transcripts, extract the mel-spectrogram and semantic token sequence for each sample. Spontaneous behavior such as laughter is labeled by [laughter] token in the transcription. For dialogue, we slice long dialogues into shorter, kereo-channel dialogues containing at least two utterances from distinct speakers. We ensure that the first and last sentences of each processed dialogue do not overlap with other dialogues, thus avoiding any extraneous content in the transcriptions. Motivated by serialized output training in speech recognition task [61 62), we organize the multi-round dialogue transcript chronologically by the start time of each utterance. Two neighboring same-speaker utterances concatenated directly, while different speakers' utterances are separated by [spkchange] token, without explicit processing for overlap labelling. A dialogue transcription preparation example is shown in Figure 2 The HuBERT speech tokenizeris is employed to extract the semantic tokens for each channel. Additionally, we mix audio with two channels and extract the mel-spectrogram from the mixed waveform. The detailed algorithm for dialogue data preparation is described in Appendix E
we slice long dialogues into smaller mono-channel samples and concatenate them meet the
good momnin
it's been long time saw you.
good morning
yeah
I'll be in touch
good morning good morning it's been long time since saw you yeah i'll be in touch
Figure 2: Dialogue transcription preparation. To better demonstrate our method, we use and emoji to represent (spkchange] and [laughter] tokens.
httpp//iii..iialaioooooooeooeeee fisher _dtappddtypyyyy (Apache License 2.0)
5

---

4.2 Model Configurations
We develop two text-to-sene to-semantic models, named CoSingle and CoMix, and two acoustic models, named VoSingle and VoMix. CoSingle and VoSingle are trained exclusively on monologue data, VoMix is trained on dialogue data, and CoMix is trained on a combination of monologue and dialogue data. In addition, the vanilla HiFi-GAN vocoder is trained on monologue data.
The text-to-semantic model is transformer-based model with rotary embedding 63 The encoder has 4 layers and the decoder has 4 layers. We set the dimension of text encoder and CoSingle decoder 512, and set CoMix decoder 1024, In order to process multi-stream for multiple talkers, CoMix applies multiple heads for generating semantic token sequences. The acoustic model is based on transformer encoder with rotary embeddings [63] and adaptive RMSNorm (44 for time conditioning, which has 8 layers and hidden dimension of 1024. VoMix and VoSingle have the same architecture except for the first input linear layer. More details of model architecture demonstrated in Appendix A
To demonstrate the performance of methods, the baseline that compare with is flow- matching speech synthesis model with phoneme representation, similar to VoiceBox [49 G The baseline contains models: the acoustic model and the duration model. The acoustic model of the baseline is the same as VoSingle model, but generates mel-spectrogram from the phoneme sequence. The duration model of baseline is to predict the duration of each phoneme, which also trained with flow matching objective and has the same architecture with 2 layers and hidden size of 1024.
We train all models from seratch and perform inference on the best performed model on validation set. We use NVIDIA TESLA V100 32GB GPUs for training. The text-to- -ee-tmt--emniccc model is trained for 10 epochs with batch size 48. The acoustic model and duration model are trained for 100 epochs with batch size 64. We adopt Adam optimizer with le-4 learning rate. The probability of dropping condition during training is Puncond 0.3, and the strength of guidance is a 0.7 during inference.
4.3 System Configuration and Evaluation Setting
We built two systems: Co Vo and CoVoMix, and evaluated them on both monologue and dialogue testing sets. Co VoSiillle contains CoSingle and VoSingle models. CoVoMix system contains CoMix and VoMix. For monologue generation, CoVoSingle and CoVoMix systems directly feed the output of text-to-semantic model into the acoustic model. The acoustic prompt is extracted from another utterance of the target speaker. For dialogue generation, Co VoSiile generate each utterance of the dialogue and concatenate these waveforms according to the order of transcript. CoVoMix receives dialogue transcription as input and synthesizes mono-channel dialogue directly. The acoustic prompts are extracted from another dialogue of target speakers.
4.4 Evaluation Metrics
Objective Metrics: We use cosine speaker similarity (SIM), word error rate (WER), Mel cepstral distortion (MCD) and NISQA to evaluate generation results [65]. SIM measures the cosine similarity between speaker embeddings of generated utterance and the acoustic prompt, extracted from WaVLM-TDNN (66). We use a market-leading Speech Recognition API for WER calculation, which measures the correctness and intelligibility. We use an improved MCD metric that adopts the Dynamic Time Warping (DTW) algorithm to find the minimum MCD between two speeches [67]. measures the speech quality and naturalness of the synthesized speech.
NISQA 65
We perform a human evaluation on the generated monologue and dialogue examples. For monologue, we measure naturalness using comparative mean option score (CMOS). For dialogue, we use CMOS to measure naturalness and how seamlessly the conversation flows. We the similarity mean option score (SMOS) between the synthesized and prompt speech to measure the speaker similarity for both monologue and dialogue. 14 professional linguistic experts provide judges for all subjective evaluations. They provide rating to the second audio, which is randomly selected from a pair of audios, in the (-3 to +3) range. The instructions of subjective evaluations are provided in Appendix F
Subjective Metrics:
hhttss//gituub.co//uucdddiii " thtpps//githu..coo///o/ http://gghhtttbcoomcccee////o/np (MIT License) (MIT License)
htaps///ghhbbbommmggaamdmmttttss (MIT License)

---

Dialogue Metrics: We assess the naturalness of the generated dialogue speech through three metries: 1) Turn-taker Turn-taking Statistics: By employing a pre-trained speaker diarization model [68] 69
we measure the duration of inter- and intra-speaker silences, overlapped speech, and active speech. 2) Pura-linguistic Behaviors: Our evaluation focuses on laughter in this study. Employing a laughter we identify instances of laughter and calculate both the total count and average duration of these events. and 3) Speech Consistency: To evaluate consistency, we generate ten dialogues, each containing more than five utterances from the target speaker. We then select five three-second segments at random from the target speaker and compare the cosine similarity of speaker embeddings among these segments.
detection tool 70 10
5 Result and Analysis
5.1 Objective and Subjective Metrics
Table 1 shows objective and subjective evaluation results for monologue and dialogue generation across various systems.
We observe that the systems leveraging our proposed methods, i..., CoVoSingle and CoVoMix, achieve higher speaker similarity. lower WER and MCD than baseline on monologue evaluation baseline model requires accurate phoneme-level alignment. however, it is challenging to perform accurate foreed-aligmment using conventional alignment tool [7] especially for speech with spontaneous behavior and noisy background. These inaccuracies in alignment can lead to significant performance degradation. By substituting phoneme representation with semantic token sequences, our approach eliminates the dependency on phoneme-level alignment, thereby enhancing model performance.
set. The phoneme-bass
The dialogue results show that, unlike monologue, the ground truth and Co exhibit high WER due overlapping speech segments. The transcriptions are chronologically sorted, leading to mis- matches between transcription and speech in overlapped parts. Furthermore, automatic recognizing overlapped speech while maintaining low WER remains a challenging task to date. Co VoSingle, which generates utterances separately and combines them, avoids this issuc, resulting in lower WER.
In terms of speech quality, observe that the proposed systems surpass the ground truth both monologue and dialogue sets. The dow-matching based acoustic model is able to eliminate background noise, and therefore produces cleaner audio than real data. CoMixix can generate overlapped speech, which may result in a slightly lower NISQA, comparing with CoVoSingle.
Table 1:
Objective and subjective evaluation results for monologue and dialogue generation across various systems. The symbol "4" is used to indicate that the system performance is significantly different (p<0.01) from CoVoSingle system terms of CMOS and SMOS scores.
Eval Set
System
SIM
WER J
MCD
NISQA
CMOS t
SMOS t
GroundTruth
0.59
6.10
3.03
Baseline
0.42
15.85
9.45
2.93
-1.60
-1.18;
Monologue
CoVoSingle
0.49
9.99
6.15
3.04
0.00
0.00
CoVoMix
0.49
8.95
6.04
3,01
0.83f
0.11
GroundTruth
14.91
2.73
Dialogue
CoVoSingle
11.77
6.91
2.90
0.00
0.00
CoVoMix
19.84
6.82
2.87
0.81
0.60f
The subjective evaluations consistently support the findings of the objective metrics. As shown in Table [] Co VoSingle significantly outperforms baseline in terms of both CMOS and SMOS scores for monologue testing set. Furthermore, across both monologue and dialogue testing sets, Co VoMix demonstrates significantly better performance over CoSniie
https://iitttllcommmyymmc/////mt (MIT License)
wttpps:/gituu...mo/pppppt/////ee http://ghhhthooooolllllll/nnnhee (MIT License)
htnpss//ihhilooomoooooossssstoos -owcaal---o-ro--aiae Aligner (MIT License)

---

We have not found good way to measure the objective similarity metric for the dialogue testing due the necessity of speaker diarization, since the potential errors in speaker diarization could impact the fairness of the comparison. Therefore, for the dialogue SMOS evaluation, testing dialogues were manually segmented into multiple single-speaker utterances to avoid speaker diarization errors.
5.2
Dialogue Metrics
5.2.1
Turn-taking Statisties
define four turn-taking activities in a dialogue: intra speaker pause (silence between active speech of the same speaker), 2) inter speaker silence (silence between active speech of different speaker), 3) overlapped segments, and 4) active speech of each speaker [3 g].
Figure 3 shows the distribution of various turn-taking activities. The degree of similarity to the ground truth reflects the model's ability simulate turn-taking dialogue. While CoVolingle can synthesize high-quality monologue, it exhibits subpar performance in dialogue turn taking events, particularly in managing intra-speaker pause, inter-speaker silence, and overlap control. Simply concatenating monologue at utterance level results in low variance in inter- and intra- speaker silence distribution, leading dialogue which sounds robotic and lacks the natural flow of conversation [8] In contrast, Co demonstrates high similarity to the ground truth in these turn-taking events, yielding more human-realistic dialogues.
Duration of Intra-speaker Pause
Duration of Inter-speaker Silence
GroundTruth
Truth
CoVoMix
CoVoMix
CoVoSingle
CoVoSingle
0.00
0.25
0.50
0.75 1.00
1.25
1.50
1.75 2.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75 2.00
Seconds
Seconds
Duration of Overlapped segments
Duration of Activate Speech
Ground Truth
Ground Truth
CoVoMix
CoVoMix
CoVoSingle
CoVoSingle
0.25
0.75
1.00
1.75
Seconds
Seconds
Figure 3: Distribution of durations of turn- turn-taking events across models. The blue line and the green line represent the median and mean of each event. The more similar to groundtruth. the better.
5.2.2 Para-linguistic Behaviors
We computed the frequency and duration of spontaneous laughter behaviors across the conversation test set and compared these metrics across models to check their closeness to the ground truth. As illustrated in Figure [4] it shows that all proposed models can generate laughter with frequency close to the ground truth, demonstrating precise control over these human- behaviors. Moreover, Co Voxix can produce dialogues with an average laughter duration that is closer the ground truth, whereas Co VoSigle tends to synthesize shorter instances of laughter.
5.2.3 Speech Consistency
We calculate the speaker similarity between any two pairs of different utterances in long conversation. Figure presennss heatmap the cosine similarity between different segments, contrasting utterance- level concatenation methods like Co Vo with non-concatenation approaches like CoVoMix. lighter shade indicates lower speaker similarity. The figure's 's color inconsistencies reveal that utterance-level concatenation can indeed lead to dissimilar speaker characteristics, particularly for

---

Number of laughters
Duration of each laughter
146
2.11
2.06
136
138
Duration (Seconds)
20-
1.82
220
Number
200
14-
23-
CoVoSingle CoVoMix GroundTruth
CoVoSingle CoVoMix GroundTruth
Figure 4: Comparison of number and duration of laughter among models
non-adjacent utterances. Generating the entire dialogue without concatenation results in significantly improved consistency of speaker similarity across various utterances.
Utterance Level Concatenation
No Concatenation
1.0
0.8
0.6
0.4
0.2
Figure 5: Speech consistency of CoVoSingle and CoVoMix for dialogue generation
Ablation Studies and Extension
To enhance the effectiveness of text-to-semantic modeling, we conducted ablation studies focusing on data augmentation and model size. In addition to real dialogue data, we incorporated simulated dialogues and monologue sentences into training data, Results show the benefits of such augmentation. as evidenced by improved model prediction accuracy, i.c., reduced WER, in both monologue and dialogue generation tasks. Furthermore, we explored the impact of output channel configurations for the acoustic model by comparing single-channel mixed speech output with dual-channel outputs, where each channel contained speech from an individual speaker. Experimental results show that dual- channel outputs underperformed in WER, and outperformed in NISQA compared with single-channel outputs. Please refer to Appendix for the detailed results of all ablation studies.
Our acoustic model can generate specific speakers' voices, given semantic token sequences and target speakers' prompts. So is straightforward be extended a voice conversion task, which modifies the speech of a source speaker and makes their speech sound like that of another target speaker without changing the content information. Instead of predicting semantic tokens from given text, we extract the semantic tokens from the speech of the source speaker. VoSingle performs voice conversion of dialogue by processing each channel individually and then mix them up, while VoMix model achieves voice conversion simultaneously. We notice that in addition to achieving high speaker similarity, these systems can also achieve high spectral similarity, indicating the strong zero-shot voice conversion capability. Moreover, VoMix performs better than VoSingle in both monologue and dialogue sets. The detailed results are shown in Appendix and the corresponding demo provided in https://aka.ms//ooomiis ms/covomix.

---

7 Conclusion, Limitation, Future Work and Broader Impacts
We introduce the CoVoMix system for human-like monologue and dialogue generation. The system is composed of an auto-regressive text-to-senee -e-mantte model and flow-matching based acoustic model, with semantic token sequence an intermediate representation. A 2k-hour conversational telephone speech dataset is leveraged in training these two models of CoVoMix. Through both objective and subjective evaluations, Co VoMix not only achieves high naturalness and zero-shot speaker similarity in both monologue and dialogue generations but also demonstrates its proficiency in the fluency of dialogue turn-taking and spontaneous behavior generation.
Limitation and Future work We have observed instances of words being omitted or duplicated occasionally synthesized speech. This is primarily attributed to the text-to-semantic model being an regressveeeev model without forced duration. Additionally, the dataset utilized for this study is sampled at 8 kHz with background noise, factors that contribute to the degradation of speech quality. In future work, we aim to enhance the text-to- -eetatt-ee model by scaling it up or initializing it with pre-trained model, and employing super-resolution methods improve the training data fidelity.
Broader Impacts A high-quality and human-like speech generation model like CoVoMix can enable many applications that improve the quality of our life. However, since CoVoMix could synthesize speech that maintains speaker identity, may carry potential risks in misuse of the model, such as spoofing voice identification impersonating a specific speaker. To mitigate such risks, is possible to build a detection model to discriminate whether an audio clip was synthesized by CoMMM.x.
10

---

References
[1] X. Tan, T. Qin, F. Soong, and T.-Y. Liu, "A survey on neural speech synthesis," arXiv preprint arXiv:2106.15500 2021.
[2] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang. Li et al., "Neural codec language models are zero-shot text to speech synthesizers." arXiv preprint arXiv:2301.02111. 2023.
[3] K. Shen, Z. Ju, X. Tan, Y. Liu, Y. Leng, L. He, I. Qin, S. Zhao, and J. Bian, "NaturalSpeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers, arXiv preprint arXiv:2304.09116 2023.
[4] W. Li, Lei, Q. Huang. Y. Zhou, Z. Wu, Kang, and H. Meng, "Towards Spontaneous Style Modeling with Semi-supervised Pre-training for Conversational Text-to-Specch Synthesis," in Proc. INTERSPEECH, 2023, pp. 3377-3381.
[5] S. C. Levinson, "On the human" interaction engine"." in Roots of human sociality. Routledge, 2020, pp. 39-69.
[6] W. Ward, "Understanding spontaneous speech," in Speech and Natural Language: Proceedings of Workshop Held at Philadelphia, Pennsylvania, February 21-23, 1989, 1989.
[7] E. A. SCHEGLOFF, "Overlapping talk and the organization of turn-taking for conversation," Language in Society, vol. 29, no. 1. p. 1-63, 2000.
[8] T. A. Nguyen, E. Kharitonov, J. Copet, Adi, W.-N. Hsu, A. Elkahky, P. Tomasello, R. Algayres, B. Sagot, A. Mohamed et al., "Generative spoken dialogue language modeling," Transactions of the Association for Computational Linguistics, vol. 11, pp. 250-266, 2023.
[9] T. Neumann, K. Kinoshita, M. Delcroix, S. Araki, T. Nakatani, and R. Maeb-Umbach "All-neural online source separation, counting, and diarization for meeting analysis," in Proc. ICASSP, 2019, pp. 91-95.
[10] K. Lee, K. Park, and D. Kim, "Dailytalk: Spoken dialogue dataset for conversational text-to- speech," in Proc. ICASSP. IEEE, 2023, pp. 1-5,
[11] T. Nagata and Mori, "Defining laughter context for laughter synthesis with spontaneous speech corpus," IEEE Transactions Affective Computing, vol. 11, no. 3, pp. 553-559, 2018.
[12] N. Tits, K. F. Haddad, and Dutoit, "Laughter Synthesis: Combining Seq2seq Modeling with Transfer Learning," in Proc. INTERSPEEHH, 2020, pp. 3401-3405.
[13] Yu, H. Chen, Y. Bian, Li, Y. Luo, Tian, M. Liu, J. Jiang, and Wang, "Autoprep: An au- tomatic preprocessing framework for in-the-wild speech data," arXiv preprint 1rXiv.2309.13905 2023.
[14] H. Sacks, E. A. Schegloff, and G. Jefferson, "A simplest systematics for the organization of turn taking for conversation," in Studies in the organization of conversational interaction. Elsevier, 1978, pp. 7-55.
[15] M. Heldner and Edlund, "Pauses, gaps and overlaps in conversations," Journal of Phonetics, vol. 38, no. 4, pp. 555-568, 2010.
[16] N. Dethlefs, H. Hastie, H. Cuayáhuitl, Y. Yu. V. Rieser, and O. Lemon, "Information density and overlap in spoken dialogue," Computer speech & language, vol. 37, pp. 82-97, 2016.
[17] L. Zhang, Z. Chen, and Y. Qian, "Enroll-awasd attentive statisties pooling for target speaker verification." " Proc. INTERSPEECH, pp. 311-315, 2022.
[18] L. Zhang, Y. Qian, L. Yu, H. Wang, H. Yang, S. Liu, I.. Zhou, and Y. Qian, "DDTSE: Dis- criminative diffusion model for target speech extraction," IEEE Spoken Language Technology Workshop, 2024.
11

---

[19] M. Heldner, J. Edlund, and J. B. Hirschberg, "Pitch similarity in the vicinity of backchannels," 2010.
[20] C. Cieri, D. Miller, and K. Walker, "The fisher corpus: A resource for the next generations of speech-to-text." in LREC, vol. 4, 2004, pp. 69-71.
[21] Taylor, Text-to-speech synthesis.
Cambridge university press, 2009.
[22] Z. Mu, X. Yang, and Y. Dong, "Review of end-to-end speech synthesis technology based on deep learning," " arXiv preprint NXXiv:2004.099955 2021.
[23] E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. Gölge, and M. A. Ponti, "Yourtts: Towards and zero-shot voice conversion for everyone," in International Conference on Machine Learning. PMLR, 2022, pp. 2709-277 2720.
zero-shot multi-speaker
[24] Y. Leng, Z. Guo, K. Shen, Z. Ju, X. Tan, E. Liu, Y. Liu, Yang. K. Song, He al., "PrompITTS 2: Describing and generating voices with text prompt," in The Twelfth International Conference on Learning Representations.
[25] M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar et al., "Voicebox: Text-guided multilingual universal speech generation at scale," Ad Advances in neural information processing systems, vol. 36, 2024.
[26] R. Huang, M. W. Lam, J. Wang, D. Su, D. Yu, Y. Ren, and Z. Zhao, "Fastdiff: A fast conditional diffusion model for high-quality speech synthesis," arXiv preprint wrXiv22204009344 2022.
[27] R. Huang. Z. Zhao, H. Liu, J. Liu, C. Cui, and Y. Ren, "Prodiff:: Progressive fast diffusion model for high-quality text-to-speech." " in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 2595-2605.
[28] M. Jeong, H. Kim, J. Cheon, B. J. Choi, and N. S. Kim, "Diff-tts: A denoising diffusion model for text-to-speech," arXiv preprint arXiv:2104.0144 2021.
[29] M. Kang, D. Min, and S. Hwang, "Any-speaker adaptive text-to-speech synthesis with diffusion models," arXiv preprint S..Kv:2221.093385 vol. 2, 2022.
[30] H. Kim, S. Kim, and S. Yoon, "Guided-ts:: A diffusion model for text-to-speech via classifier guidance," " in International Conference on Machine Learning. PMLR, 2022, pp. 11 119-11 133.
[31] Z. Kong, W. Ping. J. Huang, K. Zhao, and B. Catanzaro, "Diffwave: A versatile diffusion model for audio synthesis," arXiv preprint arXiv:2009.09761, 2020.
[32] V. Popov, I. Vovk, V. Gogoryan, T. Sadekova, and M. Kudinov, "Grad-tts: A diffusion proba- bilistic model for text-to-speech," International Conference on Machine Learning. PMLR, 2021, pp. 8599-8608.
[33] C. Miao, S. Liang, M. Chen, Ma, S. Wang, and Xiao, "Flow-tts: non-autoregressive network for text to speech based on flow," in Proc. ICASSP. IEEF, 2020, pp. 7209-7213.
[34] Kim, S. Kim, J. Kong, and S. Yoon, "Glow-tts: A generative flow for text-to-speech via monotonic alignment search," Advances in Neural Information Processing Systems, vol. 33, pp. 8067-8077, 2020.
[35] S. Kim, K. Shih, J. F. Santos, F. Bakhturina, M. Desta, R. Valle, S. Yoon, B. Catanzaro el al., "P-flow: A fast and data-efficient zero-shot through speech prompting." Advances in Neural Information Processing Systems, vol. 36, 2024.
[36] Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, and M. Tagliasacchi, "Sound- storm: Efficient parallel audio generation." " arXiv preprint arXiv:2305. 09636, 2023.
[37] Z. Zhang, I.. Zhou, C. Wang, S. Chen, Y. Wu, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li et al., "Speak foreign languages with your own voice: Cross-lingual neural codec language modeling." arXiv preprint 23X::22003.3322, 2023.
12

---

[38] T. Wang, L. Zhou, Z. Zhang, Y. Wu, S. Liu, Y. Gaur, Z. Chen, J. Li, and F. Wei, "Viola: Unified codec language models for speech recognition. synthesis, and translation." arXiv preprint (.Kiv:2305.61107 2023.
[39] N. Zeghidour, A. Luebs, A. Omran, Skoglund, and M. Tagliasacchi, "Soundstream: An end-to-end neural audio codec," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 495-507, 2021.
[40] A. Kulkarmi, V. Colotte, and D. Jouvet, "Analysis of expressivity transfer in non-autoregressivv end-to-end multispeaker tts systems," in Proc. INTERSPEECH, 2022.
[41] S. Mehta, R. Tu, Beskow, É. Szêkely, and G. Henter, "Matcha-tts: A fast architecture with conditional flow matching," arXiv preprint arXiv:2309.03199, 2023.
[42] S. Liu, D. Su, and D. Yu, "Diffgan-us: High-fidelity and efficient text-to-speech with denoising diffusion gans," arXiv preprint arXiv:2201.11972, 2022.
[43] Z. Liu, Y. Guo, and K. Yu, "Diffvoice: Text-to-speech with latent diffusion," in Proc. ICASSP. IEEE, 2023, pp. 1-5.
[44] A. Larsen, S. Sønderby, H. Larochelle, and O. Winther, "Autoencoding beyond pixels using learned similarity metric. december 31, 2015," 2023.
[45] Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, 7. Zhao, and T.-Y. Liu, "Fastspeech: Fast, robust and controllable text to speech," 2019.
[46] Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le, "Flow matching for generative modeling," arXiv preprint 1rX0::2210.02777, 2022.
[47] S. Mchta, R. Tu, J. Beskow, É. Szěkely, and G. E. Henter, "Matcha-tts: A fast tts architecture with conditional flow matching," in Proc. ICASSP. IEEE, 2024, pp. 11 341-11 345.
[48] Y. Guo, C. Du, Z. Ma, X. Chen, and K. Yu, "Voiceflow: Efficient text-to-speech with rectified flow matching," arXiv preprint arXh:2230.05002, 2023.
[49] M. Lc, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar et al., "Voicebox: Text-guided multilingual universal speech generation at scale," arXiv preprint arXiv:2306. 1.006.16666 15687, 2023.
[50] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, "High fidelity neural audio compression," arXiv preprint arXiv:22 13438, 2022.
[51] E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour, "Speak, read and prompt: High-fidelity text-to-speech with minimal supervision," arXiv preprint arXiv:2302.03352 2023.
[52] Z. Borsos, R. Marinier, D. Vincent, Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi al., "AudioLM: language modeling approach to audio generation," IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[53] M. Lajszezak, G. Cámbara, Y. Li, F. Beyhan, A. van Korlaar, F. Yang, A. Joly, Á. Martín- Cortinas, A. Abbas, A. Michalski et al., "Base uts: Lessons from building paalame-prrrmeter text-to-speech model on 100k hours of data," arXiv preprint arXiv:2402. 002.40002.0.... 2024.
[54] Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang. Y. Liu, Y. Leng. K. Song, S. Tang et al., "NaturalSpeech 3: Zero-shot speech synthesis with factorized codec and diffusion models." arXiv preprini arXiv:2403.03100, 2024.
[55] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, "Hu- BERT: Self-supervised speech representation learning by masked prediction of hidden units." IFFF/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451-3460,
2021.
13

---

[56] K. Mitsui, Y. Hono, and K. Sawada, "Towards human-liks spoken dialogue generation between agents from written dialogue," arXiv preprint arXiv:2310.01088, 2023.
[57] Kong, J. Kim, and J. Bae, "Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis," Advances in Neural Information Processing Systems, vol. 33, pp. 17 072-17 033, 2020.
[58] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: transformers for language understanding." arXiv preprint arXiv: 2018.
Pre-training of deep bidirectional
[59] Ho and T. Salimans, "Classifier-free diffusion guidance," arXiv preprint arXiv:2207115998 2022.
[60] O. Kuchaiev, J. Li, H. Nguyen, O. Hrinchuk, R. Leary, B. Ginsburg, S. Kriman, S. Beliaev, Lavrukhin, J. Cook et al., "Nemo: a toolkit for building ai applications using neural modules," arXiv preprint 190Xvv99990097777 2019.
[61] N. Kanda, Y. Gaur, X. Wang, Z. Meng, and T. Yoshioka, "Serialized output training for end-to- end overlapped speech recognition," arXiv preprint arXiv:2003.12687, 2020.
[62] C. Li, Y. Qian, Z. Chen, N. Kanda, D. Wang, T. Yoshioka, Y. Qian, and M. Zeng, "Adapting Multi-Lingual ASR Models for Handling Multiple Talkers," in Proc. INTERSPEECI, 2023, pp. 1314-1318.
[63] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, "Roformer: Enhanced transformer with rotary position embedding." Neurocomputing, vol. 568, p. 127063, 2024.
[64] B. Zhang and R. Sennrich, "Root mean square layer normalization," Advances in Neural Information Processing Systems, vol. 32, 2019.
[65] G. Mittag and S. Möller, "Deep learning based assessment of synthetic speech naturalness." arXiv preprint arXiv:2104.11611 2021.
[66] S. Chen, C. Wang. Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao al., "Wavlm: Large-scale self-supervised pre-training for full stack speech processing," " IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505-1518, 2022.
[67] E. Battenberg, R. Skerry-Ryan S. Mariooryad, D. Stanton, D. Kao, M. Shannon, and T. Bagby, "Location-relatee relative attention mechanisms for robust long-form speech synthesis," in Proc. ICASSP. IEEE, 2020, pp. 6194 6198.
[68] A. Plaquet and H. Bredin, "Powerset multi-class cross entropy loss for neural speaker diarization." arXiv preprint arXiv:2310. 3305,, 2023.
[69] H. Bredin, "pyannote. audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe." in Proc. INTERSPEECH. ISCA, 2023, pp. 1983-1987.
[70] J. Gillick, W. Deng, K. Ryokai, and D. Bamman, "Robust laughter detection in noisy environ- ments." in Proc. INTERSPEECH, 2021, pp. 2481-2485.
[71] M. McAuliffe, M. Socolof, S. Mihue, M. Wagner, and M. Sonderegger, "Montreal forced aligner: Trainable text-speech alignment using kaldi." in Proc. INTERSPEECH, 2017, pp. 498-502.
14

---

A Model Architecture
@ggure/coopann the generation pipeline among conventional method and our proposed CoVoSingle and CoVo.Mi methods. Figure ((a) shows the conventional monologue generation process with phoneme representation. (b) shows our proposed CoVoSingle approach for monologue generation. (c) demonstrates the concatenation method for conventional and Co CoVolingl models to generate dialogue. (d) shows the architecture of our proposed CoVoMix model for monologue and dialogue generation.
Hil am Arry | Hi am Sam (Nice you
Hi am Amy
Hi am Any
Hi am Amy OR
Text- Te-tma--eeaaiccc CoSingle
rext-To-Semanti
CoMix
Phonome
Sequiince
Sequ.end
Acoustic Model
Acoustic
Semantic
VoSingle
Sequences
Pramot Spk11
Spid
Acoustic
VolVix
Promes
Prompt 9pk22
Vocoder
Vocoder
Mixdd Meil eelespeerroormm
Vocoder
(a) Conventional Monologue Generation (b) CoVoSingle Monalogue Generation
He am Sent
System
How are you
Systerm
and you
(d) CoVoMix Monologue and Dialogue Generation
(c) Conventional and CoVoSingle Dialogue Generation
Figure 6: Comparison of generation pipeline among conventional method and our proposed Co CoV ingle and CoVoMix methods.
Figure 7 shows the architecture of eext-to-semantic model. We propose two types of text-to -esttt--eemanic model: CoSingle and CoMix. CoSingle model has single-stream decoder, while CoMix applies multi- stream decoder generate multiple semantic token sequences for different speakers, introduced in Section
am Amy
CoSingle Semantic Decoder
Text Tokenizer
Tokenizer
Sementic loken Secuence
Text Encodor
(4) CoSingle Text-to-Semmanicc Se--ntmatti Model
CoMix Somantic Decodor
Speech
Text Tokenizer
Tokonizer
Amy I Hi am
Sam | Nice meet
poxt-to-Semantic
Figure 7: -ow-mantiemiiccc model
Figure 3 shows the architecture of acoustic model, a flow now-matching based transformer encoder. We propose three types acoustic model: VoSingle, VoMix and VoMix-stereo. VoSingle is a single stream transformer encoder to generate single talker mel-spectrogram. VoMix and VoMix-stereo have the same architecture except for the last linear layers, which generate mono channel mixed mel-spectrogram and multiple single talker mel- mel-sppctrograms respectively.
15

---

mi Pi (m)
Transformer
Linegr
(1) VoSingle
Encoder
Vector Pred Estimaio
Linear
Ordinary Differential Equation
Flow Φ
Transformer
2) VoMix
Encoder
dd,(m) (e(((m))ss)
Linear
Transformer
(3) VoMix
Encoder
(Sterea)
Linear
[m]s) (mm(0,
(a) Matching
(b) Architecture
Figure 8: The acoustic model
B Additional Experiments
We perform detailed ablation experiments for model combination, model size and training data.
B.1 Ablation Study Model Combination
Table illustrates our proposed systems in monologue and dialogue evaluation set, which are a combination of text-to-semantic model and acoustic model. We abbreviate -e-tt--eemantc as T2S and stereo as S.
For monologue generation, Vo and CoVoMix systems directly feed the output of text-to- semantic model into the acoustic model. However, CoVoSinx model the second semantic sequence as all silence tokens. The acoustic prompt is extracted from another utterance of the target speaker.
We first observe that when using the same text-to-semantic model, different acoustic models influence WER, indicating that the pronunciation and rhyme also affect the intelligibility of speech. For example, CoVoSinx achieves better speech intelligibility than Co VoSiill due the use of VoMix. The speaker similarity performance for monologue generation is the similar across models.
For dialogue generation, notice that stereo systems show comparable WER than systems. Moreover, we observe that stereo systems show higher NISQA speech quality, indicating that predicting each channel separately causes less distortion in speech quality than predicting mixed mel-spectrogram.
Table 2: Objective evaluation on monologue and dialogue for mono and stereo acoustic model
Monologue WER
Dialogue
System
T2S
Acousic
SIM
NISQA
WER &
NISQA 1
GroundTruth
/
0.59
6.10
3.03
14.91
2.73
Co CoVoSinx
VoSingle
0.49
9.99
3.01
11.76
2.90
CoSingle
VoMix
0.49
8.78
3.12
12.27
2.97
Vo
VoMix-S
/
12.95
3.19
CoMix CoVoMix-S
VoMix VoMix-S
0.49
8.95
3.01
19.84
2.87
CoMix
/
20.35
3.00
16

---

B.2 Ablation Study on Discrete Semantic Representation and Model Size
Figure compares the speaker similarity oracle phoneme, predicted phoneme using duration predictor, oracle semantic token, and predicted semantic token using text-to-semantic model under the same architecture of acoustic model.
First, we observe that larger acoustic model improves the speaker similarity of the generated speech the model layer deepens. Second, the similarity using semantic token sequences higher than using phoneme and even exceeds oracle phoneme representations. This demonstrates the advantages of using semantic token sequences, which not only avoids forced-alignment and improves the word error rate, but also improves the model's ss speaker modeling capabilities. Third, for both phoneme and semantic token, the predicted representations are not as good as oracle representations. The duplicated semantic token sequence is more difficult to predict, leading to bigger gap between oracle and prediction, indicating further improvement space for the performance and accuracy of text-to-semantic model in the future.
Speaker similarity with acoustic model of different sizes
0,550
Pred Phoneme
Oracle Phoneme
0.525
Pred Semantic token
Oracle Semantic token
0.500
Speaker Similarity
0.475
0.450
0.400
0.375
0.350
Number Transformer Layers
Figure 9: Speaker similarity across acoustic model of different size
B.3 Ablation Study on Data Augmentation for Text-To- To-Semantic Model
When the total amount of data is deterministic, the diversity of data, such data duration and the dialogue content, becomes important. shble the performance text-to-semantic model with different data augmentation method on monologue and dialogue generation in terms of correctness. In this study, the acoustic model used only contains transformer layers instead 8 layers mentioned in 4.2 for faster inference. We incorporate diverse training dataset including both short and long sentences for the text-to-semantic model to accurately generate dialogues of varying lengths. The long monologue has minimum duration of 10 seconds, whereas the short monologue has minimum duration of second. Additionally, we enhance data variety by simulating dialogues from monologues through concatenation, which also improved the prediction accuracy of semantic token sequences. As result of utilizing monologue data of different lengths and synthetic dialogue data, the eext-to-semantic model demonstrated the best performance.
Table Ablation study of data augmentation methods
Training Data Short Monologue Long Monologue
WER J Dialogue
Real Dialogue Simu Dialogue
Monologue
28.50
22.30
11.99
21.82
11.22
20.79
10.54
20.87
17

---

C Extension to Voice Conversion
In addition to zero shot speech synthesis, our methods can also achieve voice conversion for single person monologues and multi talker conversations. VoSingle performs voice conversion of dialogue by processing each channel individually and then mix them up, while VoMix model achieves voice conversion simultaneously.
Table 4 demonstrates the objective results monologue and dialogue scenario. We notice that in addition to achieving high speaker similarity, these systems also achieve high spectral similarity, indicating the strong zero-shot voice conversion capability of our proposed system. Moreover, VoMix performs better than VoSingle in both monologue and dialogue sets.
Table 4: Objective evaluation on voice conversion for monologue and dialogue generation. The symbol "f" is used to indicate that the system performance is significantly different (p<0.01) from VoSingle system
Type
System
SIM t
MCD
VoSingle
0.47
6.47
Monologue
VoMix
0.49
6.46
VoSingle VoMix
6.70
Dialogue
/
6.59t
D Experiment Statistical Significance
In order to determine statistical significance of the main experiment in Table 1 we first use mumpy.mean and numpy.std in python to calculate the mean and the standard deviation of objective metrics across the test set Table Moreover, we use z-test to determine if the differences are statistically significant. We notice that results statistically significant for WER, MCD and NISQA in both monologue and dialogue evaluation sets. Similar to subjective evaluation result, the speaker similarity performance of the Co VoSingle and CoVoMix systems relatively close and do not show significant differences. Besides, the WER has a large deviation because the text-to-semantic model might synthesize speech with omitted duplicated words, mentioned in the limitation part in Section
Table Objective evaluation results for monologue and dialogue generation across various systems. The symbol "f" is used to indicate that the system performance is significantly different (p<0.01) from CoVoSingle system
Eval Set
System
SIM
WER
MCD
NISQA f
CoVoSingle CoVoMix
0.49+0.17
9.99+9.02
6.15
3,0410.39
Monologue
0.49+0.18
8.95+8.681
6.0442.03
3.01#0.44
CoVoSingle CoVoMix
11.77+9.00
6.91
2.9010.28
Dialogue
19.84+19.83
6.82+2.12
2.87+0.374
To investigate the effect randomness, we assess the same model using three different random seeds. For each seed, we evaluate the model performance and compute the mean and standard deviation across different seeds. The results are shown in Table [ Our findings indicate that the standard deviation among the different random seeds relatively small, suggesting that the system exhibits stability in the presence of randomness.
E
Data Preparation
Algorithm Saaioammillluuaa lillustrates the dialogue data preparation pipeline, introduced in Section4.1 parameter maxDuration is set 40 seconds by default.
The hyper-
18

---

Table 6: Objective evaluation results for monologue and dialogue generation across various systems with different random seeds
Eval Set
System
SIM t
WER +
MCD +
NISQA t
CoVoSingle CoVoMix
0.484+0.005
10.206+0.166
6.147=0.017
3.035=0.014
Monologue
0.488+0.005
9.378+0.349
6.058+0.015
3.008+0.002
CoVoSingle
11.903+0:262
6.916#0.031
2.902+0.002
Dialogue
CoVoMiix
19.542+0.587
6.829+0.006
2.870#0.008
Algorithm Dialogue Data Preparation
Require:
Dialogue recordings corresponding transcriptions maxDuration.
Segment dialogues into utterances per speaker: y y with transcripts x x and identity 2.
2: Sort æA ,B. y 8 by start times into sequences X, Y, Z.
3: Initialize cache = D. spkeache D. Dialogue []. OutputuTanssrriiiin D.
4: for each (Inew: Unew, knew) in zip(X, Y. 2) do
5: if' cache empty then
6: Add (Inew: Vnews knew) to cache; add Tnew to spkeache. (eacheccceee)1) AND |set(spcacche) then 8: Compile dialogue from cache.
7: else if StartTime (ynew))))
9: Compile transcription from cache by start time with speaker change symbol.
10:
Reset cache and spkeache.
11:
Add compiled dialogue to Output transcription to OutputT ceauutrrrasrripp
12:
else if EndTTime (cachel-11) StartTime(cache(() maxDuration then
13: Reset cache and spkcache.
14: else
15:
Continue populating cache and spkcache.
end if
17:
end for
18:
return Dillog OutpulTronscripn
Subjective Evaluation Instruction
Figure 10 and Figure 11 shows the CMOS subjective evaluation template for monologue and dialogue
generation respectively. Figure 12 show the SMOS subjective evaluation template.
Monologue Naturalness Test
In each of the following case, you will hear reference voice and voices. Play all voices finst, then select which one more natural.
Reference Transcription::
absolutely juey communication from the beginning would them
Reference
Voice 1:
Whice 2:
similar
the
option according
their naturalness, This includes evaluating how ciosely the speech resembles that of natural, human speaker in temss of fluency, the rhythm, the intonation,
the overall listening experience.
Carefully: You may iisten to each audio as many times as necessory:
lynore External Factors: Please do not consider other factors such the the speech background nokse distortions, Only focus on the speech.
botter?
mush
Vase bechter
Voite
Alghdy berter
Carl wiach bitller
aluhty
Figure 10: CMOS evaluation template for monologue generation
19

---

Dialogue Naturalness Test
following case, you hear dialogue cialogues. all dialogues
select one natural.
Transcription: ah think I'm going to sea x men tomorrow night oh ooh that was woagood jit was good hhard it was pretty good yeah yeah
cialogue:
Dislingue
Dinlngun 2:
Focus Naturalness: Please evaluate how similar the dialogue to the reference conversation between two human speakers each audio file and select option according to their naturalness
evaluating how closely dialogue resembbes conversation fluency, rhythm, the interation,, and the overall listening experience. Additionally pay special attention to how natural the speaker changes occur within the dialogue Consider the other, appropriateness
seamlesih
flows from
realistk
conversational experience.
Listen Carefully:: You listen each audio file as times necessary. quality speeth background
Ignore naturainess of the speech.
Only
is
Cnpanaer mush botler
better
Can Tellnnin batter
Daaingue alghes batter
Figure 11: CMOS evaluation template for dialogue generation
Speaker Similarity test
In each of the following case, you hear a reference voice and voices, Play all voices first, then select which one has more similar speaker characteristics to the reference.
Referince
Voice 1:
Voice 2:
Similarity: Please characteristics of the voices, such as their overall resonance (the voice quality). pitch thigher or lower), power (amplitude or volume), and the overall
similar speaker sounds compared
focus should be solely
that these characteristics you.
listen Carefully: You may listen to each audio file as many times necessary.
expressive characteristics (emphasis, intonation, rhythm), understand the words, any background noise distortions, Only focus on the audio samples are from the same person as In the reference audio.
External
Which one better?
Vaice stuahey better
Calli wa betler
Vnice la alghtty better
Tracce la better
Vakce mosh better
Figure 12: SMOS evaluation template
20

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We introduce CoVoMix, a novel model for zero-shot, human-like, multi- speaker, multi-round dialogue speech generation. Our experimental results show that CoVoMix can generate dialogues that are not only human-like in their naturalness and coherence but also involve multiple talkers engaging multiple rounds of conversation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the last section, we mentioned our limitation. We have observed instances of words being omitted duplicated occasionally in synthesized speech and the use of low-quality dataset may degrade the quality of generated speech.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
This paper mainly focused on the design of generation pipeline, model archi- tecture, data preparation, and system implementation. We did not propose new theoretical algorithms or results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We provide detailed introduction system design, data preparation and train- ing configuration. We use public available dataset, provide all codes in the Supplementary and will make them publicly available.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
22

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We public available dataset. We provide detailed codes of training, inference and data preparation in the Supplementary. These codes will be publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ wttpps.cc/ppblic////
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We introduce in detail the training and test configuration, including model architecture, optimizer and hyperparameters, etc. We also provide ablation studies for these configurations. Corresponding codes are in Supplementary and will be publicly available.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We demonstrate the experiment statistical significance in Appendix. We calculate the mean and standard deviation of cach system with different random seeds.
23

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed configuration training and test, including the GPU, memory, training epochs, evaluation tools, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'tttt make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://e//..eaeees.ccce///l//eea
Answer: [Yes]
Justification: Ethics.
This research conducted in the paper conforms with the NeurlPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?

---

Answer: [Yes]
Justification:
We discussed broader impacts and potential risks the last paragraph of
Section 7.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Currently we do not have plans for releasing the model or dataset. If we are going to release, we will make sure release safeguard with it.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the open-source code and evaluation tools that credited in this paper, and the license and terms are properly respected.
Guidelines:
The answer NA means that the paper does not existing assets.
25

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce the implementation code new asset. This asset is well documented with training, license, limitations, etc.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper neither involves crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper neither involves crowdsourcing research with human subjects.
26

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "J4kkoY0Tz0", "reviewer": "Reviewer_XEst", "summary": "This paper proposed CoVoMix, a zero-short TTS model for multi-speaker conversations. CoVoMix consists of a multi-stream text-to-units model, a flow-matching acoustic model for mixed-speaker spectrogram generation and HiFiGAN vocoder for waveform generation. The major contribution of CoVoMix is that it is one of the first attempts to generate natural multi-talker conversations in zero-shot manner and according to experimental results and demos, CoVoMix is able to synthesize natural pause, overlap and laughter in the conversation.", "questions": "1. Clarify line 149-152.\n2. The author mentions that \"the potential errors in speaker diarization could impact the fairness of the comparison\", is there any examples or numbers that can be shared so we better understand the situation? also, why is it hard for human evaluation?\n3. In speech consistency, the CoVoMix shows better consistency of different utterances than CoVoSingle. But CoVoSingle synthesizes each sentence given same target speaker prompt right? How can this be? Or is this speaker similarity measuring characteristics beyond speaker?", "limitations": "1. CoVoMix is only applied to two-talker scenario instead of more.\n2. CoVoMix lacks the option for personalizing turn-taking. The turn-taking phenomenon itself varies from person to person and even the same person in different moods.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. CoVoMix is able to zero-shot generate natural two-speaker dialogue from text without additional input on spontaneous behaviors like laugh. Objective and subjective evaluations further substantiate this.\n2. This line of work is a crucial step towards natural human-machine dialogue. There are prior TTS works like CHATS, which also support natural turn taking, and CoVoMix further extends it to support zero-shot, so that user can designate the speaker's voice.", "weaknesses": "1. Unclear writing in the method section. Line 149-152, author says \"we divide the semantic embedding into two distinct segments in the final linear layer of the decoder\", does it mean dividing each embedding into two or dividing the embedding sequence into two halves?\n2. Insufficient baseline comparison in the experiments. CoVoMix is only compared to CoVoSingle and a baseline sharing similar architecture. Though many of the prior works are not public, there are open-sourced TTS models like SoundStorm, and public but not open-sourced models like GPT-4o. More baseline comparison will provided readers a better sense of CoVoMix's performance.\n3. The authors use speaker diarization for turn-taking statistics, but not for speaker similarity in dialogue, so we basically don't know whether it can recover target speaker's voice for dialogue generation.\n4. Speech consistency is only evaluated based on speaker similarity instead of flow of speech, whether laughter is appropriate and etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. We have clarified some unclear writing that caused misunderstandings and added a baseline system for comparison, as well as a speaker similarity subjective test for dialogue, which will appear in the revised version. The detailed responses are listed below.**\n\n**R1: About unclear presentation of Line 149-152 (Weakness 1, Question 1)**\n\nLines 149-152 indicate that we need to divide the semantic embedding into two halves. For instance, if there is a semantic embedding of shape [B,T,D], we divide it into [B,T,:D/2] and [B,T,D/2:] to obtain these two embeddings. We have revised the paper to clarify this.\n\n**R2: About insufficient baseline comparisons (Weakness 2)**\n\nWe agree that providing more baseline comparisons would give a better understanding of CoVoMix’s performance. We have also spent a significant amount of time surveying this. However, to the best of our knowledge, there are no suitable baselines available for comparison. For instance, Soundstorm models and codes are not officially open-sourced. The publicly available implementation was only evaluated using LibriSpeech. Therefore, there is no Soundstorm implementation for dialogue synthesis, which requires a special design for speaker turn changes. The original paper also lacks sufficient details on this aspect. Furthermore, GPT-4o was not accessible during our research period.\n\nWe have tried to reproduce the dialogue model of Soundstorm based on our understanding of the paper, but the results were not as reported, so we did not put the results into the paper. To address your concerns, we continue implementing a SoundStorm-style baseline using the Fisher dataset. We utilized EnCodec for acoustic tokens (since Soundstream is not publicly available) and HuBERT for semantic tokens (the same in CoVoMix). Soundstorm lacks a mechanism to handle two-channel speech for training, and consequently, it does not model overlapping speech from two speakers effectively. Therefore, we had to reprepare the training and test datasets to ensure the speech was mono-channel with no overlapping parts. To isolate other effects, we used oracle semantic tokens for the model comparison shown in Table R1.\n\nTable R1: *Objective evaluation results in non-overlapped dialogue test set across models*\n|Model|SIM|MCD|NISQA|\n|---|---|---|---|\n|GroundTruth |0.59|/|2.85|\n|GroundTruth-EnCodec|0.53|2.39 |2.56|\n|CoVoSingle|0.47|4.41|2.88|\n|SoundStorm|0.25|5.60|2.49|\n|CoVoMix|0.46|4.98|2.88|\n\n**R3: About speaker similarity in dialogues (Weakness 3)**\n\nTo address your concerns, we use oracle diarization results for similarity evaluation shown in Table R1. We also added a subjective evaluation for speaker similarity in a dialogue as in Table R3. Ten dialogues randomly selected were manually segmented into multiple single-speaker utterances to avoid speaker diarization errors. (For turn-taking statistics, we had to leverage an automatic diarization system due to the large size of whole testing set). We had 15 linguistic experts evaluate the speaker similarity of these utterances compared to the prompt speaker. \n\nTable R3: *Speaker similarity across models for dialogue generation*\n|Model|SMOS|\n|---|---|\n|CoVoSingle|0.00|\n|CoVoMix|0.60|\n\n**R4: About speech consistency and flow of speech (Weakness 4)**\n\nWe agree that speech consistency should also consider the flow of speech and appropriate laughter, in addition to speaker similarity, which is predominantly used in related papers. In our dialogue naturalness subjective test, we have provided with the following specific guidelines (please refer to Fig.12 in appendix):\n\n*...evaluating how closely the dialogue resembles a natural conversation in terms of fluency, the rhythm, the intonation ... Consider how seamlessly the conversation flows from one speaker to the other, the appropriateness of pauses, and how these transitions contribute to a realistic conversational experience*\n\nConsequently, the flow of speech and appropriate laughter have already been taken into account in the subjective test scores.\n\nMoreover, we added Table R4 to show the distribution of F0, demonstrating that CoVoMix produces speech more similar to the Ground Truth in multi-turn dialogues.\n\nTableR4: *F0 distribution across models for dialogue generation*\n|Model|F0|\n|---|---|\n|GroundTruth|253±80|\n|CoVoSingle|229±87|\n|CoVoMix|255±59|\n\n**R5: About the potential errors in speaker diarization (Question 2)**\n\nWe use an open-source diarization tool, pyannote. It achieves an 11.9% Diarization Error Rate in its original domain (AMI dataset), but it may be less accurate on the Fisher Dataset due to domain mismatch.\n\nThe challenge for human evaluation lies in the co-appearance of multiple talkers in a dialogue. An interfering speaker can influence the speaker perception during human evaluation. Moreover, although humans can easily distinguish between two speakers of different genders, more than 60% of the utterances in Fisher dataset are of the same gender. To address this issue, we manually segmented the testing utterances into several single-speaker segments and had linguistic experts evaluate the speaker similarity. The results are shown in Table R3.\n\n**R6: About speech consistency of CoVoSingle (Question 3)**\n\nSpeech consistency involves comparing the speaker similarity among different segments within the same dialogue. Although CoVoSingle demonstrates high zero-shot capability, it is less likely to generate multiple identical utterances with the same prompt due to the sampling of the flow-matching model. In contrast, CoVoMix addresses this consistency issue by generating the entire dialogue at once, rather than through multiple generations and concatenation.\n\n**Finally, we would like to express our gratitude once again for your time and effort in reviewing our paper. We would greatly appreciate it if you could consider increasing your score.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kvQS08C5sl", "reviewer": "Reviewer_yp1s", "summary": "This study proposes a personalized speech synthesis model capable of generating monologues or dialogues. The study achieves this goal through the development of a text-to-semantic token model and the conversion of semantic tokens into mel-spectrograms. By utilizing the Fisher dataset, which contains natural speech characteristics, the proposed model is able to produce naturalistic utterances that include paralingustic components such as laughter and coughing.", "questions": "1. I am curious if the HuBERT tokens used in this study contain any speaker information at all. Specifically, when performing personalization in the acoustic model, could any speaker information potentially present in the HuBERT tokens degrade the personalization performance?\n\n2. To model laughter, it seems that the semantic tokens must include information about laughter. I am curious whether the self-supervised model that incorporates such laughter information in the semantic tokens is limited to the Fisher dataset, or if the speech tokenizer is designed to work robustly with other datasets as well.\n\n3. Unofficial Implementations of Soundstorm have been made publicly available on the web. For example, https://github.com/ZhangXInFD/soundstorm-speechtokenizer offers an acoustic model capable of personalized speech synthesis using semantic tokens, similar to your approach. I'd like to ask if you have considered comparing your model with Soundstorm using publicly available implementations.", "limitations": "They addressed these aspects in the conclusion, limitation, future work and broader impacts section.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Dialogue generation, which involves converting given conversations into speech, has not been sufficiently explored. This study implements a dialogue generation model that not only addresses this gap but also incorporates personalization features.\n\n2. Various metrics that facilitate evaluation have been devised.\n\n3. The included demo provides a convenient way to distinguish the strengths of their model, and the various figures in the paper help facilitate understanding of the text.", "weaknesses": "1. The task in this study is similar to existing dialogue-to-speech tasks, with the addition of personalization features. However, the method of adding personalization does not appear to be novel or specifically tailored for dialogue. Instead, it seems to be a combination of existing approaches.\n- The text-to-semantic token approach has already been widely adopted in many previous studies, particularly with recent advancements in personalized speech synthesis. The use of an autoregressive method, as employed in this study, has been extensively researched. The authors' contribution appears to extend this to dialogue data. However, the approach of using an autoregressive text-to-semantic prediction model for dialogue has already been utilized in Soundstorm.\n- The acoustic model handles the personalization aspect and shows a structure almost identical to Voicebox. Similarly, apart from an increase in channels compared to the original Voicebox, there seem to be no additional distinguishing features.\n\n2. The paper mentions that Soundstorm generates in a sequential manner, but it should be noted that Soundstorm also uses an autoregressive model for text-to-semantic conversion and a non-autoregressive model based on MaskGIT for the acoustic model, similar to the approach in your model. Therefore, the statement in the paper that \"generated in a sequential manner and thus sounds less realistic\" could equally apply to CoVoMix. Concerns are raised that issues such as spontaneous behaviors might stem from the differences in the data used for training rather than the proposed method itself. Given that Soundstorm is one of the few spoken dialogue generation models that support personalization, it seems necessary to compare models trained on the same data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. We added a table to highlight the unique aspects of our model in comparison with previous works. Additionally, we implemented the Soundstorm-style baseline for dialogue synthesis to compare with our CoVoMix model. The detailed responses are listed below.**\n\n**R1: About similarity to existing works (Weakness 1)**\n\nTo achieve state-of-the-art (SOTA) performance in dialogue speech synthesis, we have fully leveraged existing SOTA technologies in our model building. This is why CoVoMix appears to be a combination of existing methods. However, these methods alone could not completely address the critical issues in our scenario: *human dialogues are naturally characterized by turn-taking behaviors, such as overlaps and hesitations and non-verbal behaviors, such as laughter and coughing*\n\nLet’s clarify our contributions compared to existing work, which can be further illustrated in Table R1:\n\n1. Our work generates the entire dialogue at once. In contrast, previous studies generate each utterance separately and then concatenate them, or generate each utterance sequentially without considering overlapping speech during speaker turn changes, such as in Soundstorm.\n2. We employ simultaneous multi-stream semantic token prediction from dialogue text, with each stream representing an individual talker. We use a multi-talker flow-matching-based acoustic model to generate a mixed mono mel-spectrogram given multiple contexts.\n3. We propose a comprehensive strategy for processing ASR datasets, including both training and evaluation for monologue and dialogue speech generation.\n\nTable R1: *System comparison*\n| System| Semantic | Acoustic | Data | Monologue| Dialogue| Non-verbal Behaviors | Turn-taking |Overlapping speech\n|---|---|---|---|---|---|---|---|---|\n| VoiceBox | Phoneme | Mel-spectrogram (flow-matching) | High Quality monologue | Yes | No | No | No  | No |\n| SoundStorm | Semantic Tokens | Codec tokens (NAR) | High Quality monologue and dialogue (internal) | Yes | Yes | No | yes  | No |\n| CoVoMix | multi-stream Semantic Tokens | Mel-spectrogram (multi-stream flow-matching) | ASR dialogue | Yes | Yes | Yes | Yes  | Yes|\n\n**R2: About comparison with SoundStorm (Weakness 2, Question3)**\n\nThe main difference between Soundstorm and CoVoMix lies in the generation process. Although Soundstorm is a non-autoregressive model, its dialogue generation pipeline relies on an autoregressive text-to-semantic model, which generates speaker A and B's content in a sequential ABABAB way and thus fails to produce overlapping content. In contrast, CoVoMix generates multiple streams of semantic tokens (including silence tokens) in parallel, with each sequence corresponding to one speaker. Since these streams are temporally aligned, multiple streams may speak simultaneously, resulting in a more natural dialogue with such overlapped speech.\n\nRegarding your concerns that spontaneous behaviors might arise from differences in the training data, this is one of the critical issues we have addressed. We have proposed a comprehensive strategy for processing ASR datasets, including both training and evaluation for monologue and dialogue generation. \n\nSince Soundstorm is not officially open-sourced, we have considered comparing CoVoMix with publicly available implementations of Soundstorm. However, these implementations were only evaluated on LibriSpeech, and thus there is no implementation for dialogue synthesis, which requires special design for speaker turn changes. Instead, we implemented a Soundstorm-style baseline using the Fisher dataset based on our understanding of the paper. We utilized the EnCodec model for acoustic tokens (since Soundstream is not publicly available) and HuBERT for semantic tokens (the same in CoVoMix). Since Soundstorm is unable to handle two-channel or overlapping speech for training, we had to reprepare the training and test datasets to ensure the speech was mono-channel with no overlapping. To isolate other effects, we used oracle semantic tokens for the model comparison shown in Table R2.\n\nTable R2: *Objective evaluation results in non-overlapped dialogue test set across models*\n|Model|SIM|MCD|NISQA|\n|---|---|---|---|\n|GroundTruth |0.59|/|2.85|\n|GroundTruth-EnCodec|0.53|2.39 |2.56|\n|CoVoSingle|0.47|4.41|2.88|\n|SoundStorm|0.25|5.60|2.49|\n|CoVoMix|0.46|4.98|2.88|\n\n**R3: About speaker information in HuBERT tokens (Question 1)**\n\nWe did notice that HuBERT tokens contain speaker information. As shown in Appendix B.2, there is a larger speaker similarity gap between models using predicted and oracle semantic tokens than phoneme sequences. We have attempted to solve this problem using various methods, such as extracting semantic tokens from voice-converted utterances to remove the original speaker identity during the training stage. However, the performance has not met our expectations and is therefore not reported in the paper. We will continue investigating this issue in the future.\n\n**R4: Robustness of Laughter in Speech Tokenizer (Question 2)**\n\nWe believe that Speech Tokenizer HuBERT trained on the Fisher dataset contains tokens representing laughter. The laughter can be generated by giving a tag in the text or automatically generated by using contextual info. In addition, we observed that CoVoMix shows a certain generalization capability across different domains. For example, the transcripts used for generating demo videos are derived from DailyDialog (a dialogue dataset) with manually annotated positions of laughter. \n\nHowever,  according to References [8], HuBERT trained on Fisher shows degraded performance when generalized to LibriSpeech due to domain mismatch.  \n\n**Finally, we hope we have addressed all your concerns. We would appreciate it if you could consider increasing your score.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dnwzuCmH3w", "reviewer": "Reviewer_LEoq", "summary": "The authors introduce the CoVoMix framework for human-like monologue and dialogue generation. They point out the shortcomings of the previous multispeaker dialogue systems that they are less explored, and there are lack of high-quality, spontaneous conventional datasets. The proposed CoVoMix achieves high naturalness and zero-shot speaker similarity in both monologue and dialogue generations with its proficiency in the fluency of dialogue turn-taking and spontaneous behavior generation.", "questions": "Please refer to the Weakness section.", "limitations": "The authors have adequately addressed the limitations in that they observed instances of words being omitted or duplicated occasionally in synthesized speech.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper overall is easy to follow. \n2. It is the first attempt to propose zero-shot, human-like, multi-talker conversational mixed speech generation with a simultaneous multi-stream semantic token prediction and a multi-talker flow-matching-based acoustic model.\n3. The paper introduces dialogue evaluation metrics: Turn-taking Statistics, Para-linguistic Behaviors, and Speech Consistency.\n4. The example demo video extensively shows the generated conversation with naturalness and intelligibility.", "weaknesses": "1. While the paper shows the objective and subjective evaluation results for monologue and dialogue generation, it does not explicitly show the performance comparisons between the previous literature. The authors may compare the proposed model with previous dialogue papers.\n2. The authors address that they employ the Fisher dataset which is curated for robust speech recognition. In order to generalize the proposed model, I would suggest the author to utilize one or two more datasets to verify the effectiveness of the proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely appreciate your efforts in reviewing our paper and providing valuable and constructive feedback. We have implemented the Soundstorm model for dialogue synthesis as previous work to compare with our CoVoMix model, which will be included in the revised version. The detailed responses are listed below**\n\n**R1: About comparison with previous work (Weakness 1)**\n\nWe completely agree with your comments regarding the need to compare our proposed model with previous dialogue papers. We have also dedicated a significant amount of time to surveying this area. However, to the best of our knowledge, there are no suitable baselines available for comparison. Our work generates the entire dialogue at once, incorporating natural turn-taking behaviors such as overlap. In contrast, previous studies generate each utterance separately and then concatenate them, or generate each utterance sequentially without considering overlapping speech during speaker turn changes. The latter type of model, such as Soundstorm, is not officially open-sourced. Public implementations, however, were only evaluated on LibriSpeech, and there is no implementation for dialogue synthesis, which requires a special design for speaker turn changes.\n\nTo address your concerns, we implemented a SoundStorm-style baseline using the Fisher dataset based on our understanding of the paper. We utilized the EnCodec model for acoustic tokens (since Soundstream is not publicly available) and HuBERT for semantic tokens (the same in CoVoMix). Soundstorm lacks a mechanism to handle two-channel speech for training and overlapping speech from two speakers. Therefore, we had to reprepare the training and test datasets to ensure the speech was mono-channel with no overlapping parts. To isolate other effects, we used oracle semantic tokens for the model comparison with results shown in Table R1.\n\nTable R1: *Objective evaluation results in non-overlapped dialogue test set across models*\n|Model|SIM|MCD|NISQA|\n|---|---|---|---|\n|GroundTruth |0.59|/|2.85|\n|GroundTruth-EnCodec|0.53|2.39 |2.56|\n|CoVoSingle|0.47|4.41|2.88|\n|SoundStorm|0.25|5.60|2.49|\n|CoVoMix|0.46|4.98|2.88|\n\n**R2: About extra dataset (Weakness 2)**\n\nWe are eager to utilize one or two more datasets to verify the effectiveness of the proposed model. However, to the best of our knowledge, there are few multi-speaker conversational datasets available for training. One of our novel contributions is the proposal of a data-processing pipeline to leverage speech recognition datasets, such as the Fisher Dataset. We propose a comprehensive strategy for processing ASR datasets, including both training and evaluation for monologue and dialogue speech generation.\n\nIt is beneficial to utilize more datasets for training to improve model generalization. According to our observations, CoVoMix demonstrates a certain generalization capability across different domains. For example, the transcripts used for generating example demo videos are derived from DailyDialog[1], a dialogue dataset. We will consider simulating conversational datasets for training purposes in future work. \n\nRef: [1] Li, Yanran, et al. \"Dailydialog: A manually labelled multi-turn dialogue dataset.\" arXiv preprint arXiv:1710.03957 (2017).\n\n**Finally, we would like to express our gratitude again for your time and effort in reviewing our paper. Considering this is the first attempt at multi-talker conversational mixed speech generation and that we have added a comparison with previous work, we would appreciate it if you could consider increasing your score. Please do not hesitate to let us know if you have any further concerns or comments. We would be happy to address them.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Thank you for your efforts in reviewing our paper. We greatly appreciate your acknowledgment of our contributions, including our first attempt at zero-shot, human-like, multi-talker conversational mixed speech generation, the various metrics to facilitate evaluation, and the good demos and figures for demonstration.

Regarding the main concern about the comparison with previous work, such as Soundstorm, we have dedicated a significant amount of time to surveying this area. However, to the best of our knowledge, there are no suitable baselines available for comparison. Although there are publicly available implementations of Soundstorm, these implementations were only evaluated on LibriSpeech and thus do not include dialogue synthesis, which requires special design for speaker turn changes. We have implemented a Soundstorm-style baseline using the Fisher dataset based on our understanding of the paper. We did not use it for comparison in the paper because the results were not as expected. We will continue to improve it and add the results to address your concerns.

Additionally, we will include new results of the speaker similarity subjective test for dialogue generation in the revised version.

Please check the details in the responses to the individual reviewers.

Thanks again!

Authors

Meta-review: CoVoMix represents a novel approach to synthesizing multi-talker (though specifically two-speaker) conversations.  Zero–shot speech generation has received a substantial amount of attention. The novelty in this approach is the use of a multi-stream model to be able to natively synthesize overlapping speech (a necessary component of natural human-human conversation as backchanneling and smooth turn taking involve overlapping speech).

The major weakness of this work is that the comparison to other similar single-stream approaches is limited.  This is a very fair criticism, and one that the authors should take seriously.  The discussion around the overlapping speech in SoundStorm demos suggest that the model (benefiting from a lot of data) is, in fact, able to generate overlapping speech using a single stream model.  Since the semantic tokens used in SoundStorm represent overlapping speech, it is not obvious that this is not controllable, only that the semantic token prediction needs access to both speaker’s content. Of course, this version of SoundStorm is not publicly available making this investigation particularly challenging.
Decision: Accept (poster)