Generative Expressive Conversational Speech Synthesis

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 13:14:32

Abstract: Conversational Speech Synthesis (CSS) aims to express a target utterance with the proper speaking style in a user-agent conversation setting. Existing CSS methods employ effective multi-modal context modeling techniques to achieve empathy understanding and expression. However, they often need to design complex network architectures and meticulously optimize the modules within them. In addition, due to the limitations of small-scale datasets containing scripted recording styles, they often fail to simulate real natural conversational styles. To address the above issues, we propose a novel generative expressive CSS system, termed  **GPT-Talker**. We transform the multimodal information of the multi-turn dialogue history into discrete token sequences and seamlessly integrate them to form a comprehensive user-agent dialogue context. Leveraging the power of GPT, we predict the token sequence, that includes both semantic and style knowledge, of response for the agent. After that, the expressive conversational speech is synthesized by the conversation-enriched VITS to deliver feedback to the user. Furthermore, we propose a large-scale Natural CSS Dataset called **NCSSD**, that includes both naturally recorded conversational speech in improvised styles and dialogues extracted from TV shows. It encompasses both Chinese and English languages, with a total duration of 236 hours. We conducted comprehensive experiments on the reliability of the NCSSD and the effectiveness of our GPT-Talker. Both subjective and objective evaluations demonstrate that our model outperforms other state-of-the-art CSS systems significantly in terms of naturalness and expressiveness. _The Code, Dataset, and Pre-trained Model are available at: https://github.com/AI-S2-Lab/GPT-Talker._

Main paper: 
 introduction: 
 Conversational Speech Synthesis (CSS) (or Conversational Text-to-Speech (TTS), CTTS) aims to generate speech with proper style in the user-agent conversation scenario. In such scenarios, the user usually initiates a dialogue, then the agent and the user take turns to speak. During the interaction, the agent is expected to understand the user's needs and provide assistance and emotional support. Currently, with the increasing popularity of smart devices, there is a growing demand for human-machine interaction in various application scenarios such as smartphone assistants [47] , smart home control [21] , intelligent vehicle systems [15] , and virtual reality / augmented reality interactions [13] .Many attempts on CSS have been proposed to enhance the naturalness and expressiveness of synthesized conversational speech from the perspective of context modeling. Guo et al, [17] proposed a GRU-based context encoder to extract global prosodic information for the agent from the dialogue context. FCTalker [19] further incorporates the word-and sentence-level context knowledge, that represents the fine-and coarse-grained context, to enhance the context understanding ability of the agent. However, these works only consider the textual information, ignoring the audio and multi-modal dependencies in the conversation. To this end, researchers contributed to the study of multi-modal context modeling [10, 35, 39, 40] . However, they are increasingly inclined towards designing complex network architectures and optimizing the modules within them meticulously. For example, multi-scale multimodal CTTS (M 2 -CTTS) system [53] includes a textual context module and an acoustic context module with both coarse-grained and fine-grained modeling. Li et al. [28, 29] and Liu et al. [34] proposed the graph neural network based context learning schemes.With the advancement of the Large Language Model (LLM), several studies aim to construct spoken language models that extend language models for the speech domain [23] . We have noticed that the Generative Pre-training Transformer (GPT) possesses concise and powerful context modeling capabilities [14] and has demonstrated impressive performance in capturing fine-and coarsegrained dependencies in tasks such as dialogue generation [58] and understanding [32, 33, 56] . Despite the successes, all the above works are not applicable to CSS, where multi-modal context is used as input to generate response speech output. We note that the context modeling of GPT aligns well with the requirements of CSS, yet it has been largely overlooked. Therefore, how to leverage GPT to build concise yet powerful context understanding solutions for user-agent interaction, will be the focus of this work.In addition, the existing spoken dialogue dataset is challenging to meet the requirements for training GPT-based generative expressive CSS models in terms of both scale and quality. Specifically, M 2 -CTTS [53] , EmoSit-TTS [35] and ECSS [34] , etc. all utilized a small-scale DailyTalk dataset [27] , which contains 2541 dialogues audio in about 20 hours in total. More importantly, it only consists of recordings in a reading style, lacking sufficient expressiveness. Some works [10, 29] have attempted to use datasets such as IEMO-CAP [4] and ECC [28] . However, these datasets are designed for purposes like conversational emotion recognition [4, 41] and conversation education, and they may contain background noise issues that can affect the synthesis results. Therefore, it is necessary to construct a large-scale and high-quality expressive CSS dataset to support GPT-based CSS models in achieving a real natural conversational style.In this paper, we propose a novel generative expressive CSS system, termed GPT-Talker. We transform the multimodal information of the dialogue history into discrete token sequences and seamlessly integrate them to form a comprehensive user-agent dialogue context. Leveraging the power of GPT, we predict the token sequence, that includes both semantic and style knowledge, of response for the agent. After that, the expressive conversational speech is synthesized by the conversation-enriched VITS to deliver feedback to the user. Furthermore, we propose a large-scale Natural CSS Dataset called NCSSD, that includes both naturally recorded conversational speech in improvised styles and dialogues extracted from TV shows. It encompasses both Chinese and English languages, with a total duration of 236 hours. The NCSSD dataset and related annotation will be public freely. We conducted comprehensive experiments on the reliability of the NCSSD and the effectiveness of our GPT-Talker. Both subjective and objective evaluations demonstrate that our model outperforms other state-of-the-art CSS systems significantly in terms of naturalness and expressiveness. In summary, our main contributions are as follows: 1) To the best of our knowledge, we are one of the earliest to introduce GPT into conversational speech synthesis and build a concise and powerful context modeling scheme for the user-agent conversation. 2) We have proposed a new large-scale Natural CSS dataset, termed NCSSD, that can support our GPT-Talker, even future GPT-style CSS model to achieve a real natural conversational speech style. 3) We have conducted comprehensive validations of the model's effectiveness and the dataset's reliability. Our model significantly outperforms baseline models in terms of naturalness and expressiveness. 
related work 2.1 conversational language model: 
 Text and audio are two important modalities for human communications. Text-based LLMs have demonstrated remarkable achievements across various domains, including conversational chatbots [2] , code generation [6] , creative writing [46] , and machine translation [36] .Inspired by the aforementioned works, many studies have recently explored conversational language modeling to address a variety of tasks involving speech and text [48, 49] , such as automatic speech recognition [12] , spoken question answering [1] , and speech-to-text translation [51] , etc. For instance, SpeechGPT [57] exhibited cross-modal conversational capabilities by employing discrete unit representations to convert continuous speech signals, and integrating LLMs with unit vocoder. This enables the model to effectively process multimodal input and generate corresponding output. dGSLM [38] proposed a dual-tower spoken LLM on discrete speech units to model two-channel spoken dialogue, but the generated spoken sentences lack semantic meaning. USDM [24] proposed a generalized speech-text pretraining scheme that leverages the chain-of-reasoning capabilities of LLMs to generate coherent spoken responses based on conversational speech. In human conversation, while the dialogue primarily relies on the lexical aspect, the speaking styles convey rich information beyond text, and can even alter the semantics of the spoken sentences [5] . To this end, E-Chat [52] proposed a novel LLM-based spoken dialogue system, that leverages an emotion embedding extracted by a speech encoder, enabling it to respond according to different emotional contexts. ParalinGPT [31] takes the conversational context of text, speech embeddings, and paralinguistic attributes as input prompts for LLM to improve current and response sentiment prediction, as well as response text generation, in natural human-human speech dialogue. Furthermore, Spoken-LLM [32] proposed to fuse the LLM and a self-supervised speech emotion representation model to help the LLM to predict response speaking style and text, enabling the subsequent expressive TTS model to generate natural and diverse speech responses.Our work performs significantly differently from the abovementioned studies. Specifically, our work focuses on the task of CSS, where we integrate multi-turn multi-modal dialogue context into a unified sequence and predict the semantic and stylistic representations of the speech to be synthesized along with the current utterance. We then utilize this representation to generate the final expressive conversational speech. However, the aforementioned works, such as dGSLM, SpeechGPT, and USDM, overlooked effective mechanisms for modeling multi-turn dialogue history, while Echat, ParalinGPT and Spoken-LLM etc. only generate text responses based on dialogue history rather than speech representations. Although MQTTS [7] and Pheme [3] also claim to be conversational speech generation systems, they do not model the dialogue context and only focus on the naturalness of the synthesized speech. Our GPT-Talker is similar to the recent style transfer TTS model, GPT-SoVITS [16] , but an obvious difference is that GPT-SoVITS does not pay attention to the dialogue context information.Table 1 provides a summary of the existing relevant datasets to our knowledge. Lines 6-11 present information about some conversational speech datasets specifically designed for CSS task. It can be observed that these datasets are relatively small in scale. For example, ECC [28] gathered 66 sets of public videos from YouTube's English Conversation channel, amassing 24 hours of content. These dialogues involve two, three, or multiple participants. DailyTalk [27] is derived from DailyDialog dataset [30] . It showcases 2,541 high-quality English conversations between a male and female, spanning a total of 20 hours. The Japanese corpus for empathic conversations STUDIES [44] and CALLS [43] covers two scenarios, that are communication between teachers and students in school and customer service via telephone, featuring 8.2 and 6.5 hours of speech, respectively. RyanSpeech [54] is a high-quality male TTS corpus in the conversational domain, that contains 9.84 hours audio samples. Note that CALLS and RyanSpeech claim to be conversational datasets, but they only contain one speaker and are not suitable for CSS task. Guo et al. [17] recorded an internal dataset (we call "ASLP-CH" here) comprising 3 hours of conversations between 2 Chinese women, who assumed the roles of a customer and a customer service representative, respectively. In summary, the scale of these datasets is insufficient to train a high-quality CSS model based on the GPT model. This necessitates the creation of a large-scale, freely available CSS dataset. Additionally, our dataset has advantages in terms of language diversity and data source diversity compared to existing datasets. Specifically, it includes bilingual data in both Chinese and English, as well as subsets of recorded data and collected data. Furthermore, we have developed an automated processing pipeline for collected data, greatly improving the efficiency of creating natural CSS datasets. We hope that this initiative can contribute to the development of the community. Furthermore, in Table 1 , we have listed some of the latest dialogue speech datasets designed for tasks such as dialogue emotion understanding. It can be seen that our data still has significant advantages in terms of language diversity, data source diversity, and dataset scale compared to these datasets. 
gpt-talker: methodlogy 3.1 task definition: 
 In user-agent spoken conversation, the user and the agent take turns speaking. The user speaks first, and the agent understands the user's semantics to provide a spoken response. As time progresses, multi-turn user-agent dialogue history accumulates and forms. The task setting in our CSS task, therefore is to generate the speech of agent's response according to the conversation context, where the text of the response is given. Specifically, assume that the 𝑁 -turn conversation context includes the multi-modal dialogue history H and the current utterance C, whereH = {(𝑈 𝑡 1 , 𝑈 𝑎 1 , 𝑆 1 ), (𝑈 𝑡 2 , 𝑈 𝑎 2 , 𝑆 2 )..., (𝑈 𝑡 𝑁 -1 , 𝑈 𝑎 𝑁 -1 , 𝑆 𝑁 -1 )} and C = (𝑈 𝑡 𝑁 , 𝑆 𝑁 ).(𝑡 and 𝑎 means the text and audio modalities respectively, 𝑆 is the speaker identify label.) The goal of CSS is to ensure that the synthesized speech 𝑈 𝑆 𝑁 is suitable for the whole dialogue situation. Therefore, how to model the multi-turn multi-modal conversational context and provide a proper speaking style for the agent is the focus of the task.To this end, our GPT-Talker consists of two key components, that are GPT-based context modeling and Expressive conversational speech synthesis. Specifically, GPT-based context modeling proposes a novel Conversation GPT (ConGPT) to model the multi-turn multimodal conversational context by treating the discrete token sequence of context as the condition prompt, and predict proper semantic and style expression for the agent. Expressive conversational speech synthesis proposes the Conversational VITS (ConVITS) to enrich the VITS with the agent's semantics, style, and timbre to generate expressive conversational speech based on the known response content.As shown on the left side of Fig. 1 , The ConGPT encompasses 1) Multi-turn Multi-modal Context Tokenization and 2) ConGPTbased Semantic and Style Inference. The former module converts the multi-modal context into a unified discrete sequence, and then constructs a discrete representation of the multi-turn conversation context. Based on this representation, The latter module infers the semantics and style of the response speech.Tokenization. Unlike traditional CSS works that adopt complex graph neural networks [29, 34] or cascade pipelines [35] to understand the context, our ConGPT seeks to understand the multi-modal context within a unified discrete sequence directly.Textual Tokenization: Similar with VALL-E [48] , we convert the text data of dialogue history H and current utterance C, including 𝑈 𝑡 1→𝑁 -1 and 𝑈 𝑡 𝑁 , into the discrete phoneme sequences. As shown in Fig. 1 python module 1 for English and the opencpop-strict dictionary 2 for Chinese.Acoustic Tokenization: To model natural speech conversations, the discrete speech representation must contain not only the semantics of the utterance but also expressiveness features such as speaking style, which are crucial for CSS task. Previous research [24] has demonstrated that HuBERT tokens contain rich semantic information as well as strong traces of paralinguistic features that can be used to accurate classify speech emotions. Therefore, we follow [24] and employ HuBERT [18] to acquire high-dimensional speech representations. Inspired by [26] , we do not employ unit deduplication on HuBERT units, since it maintains a consistent length ratio with the speech, removing the need to prepare a separate duration predictor. Specifically, we adopt speech units as acoustic context units that are derived from k-means clustering of the HuBERT's intermediate speech representation. We follow GPT-SoVITS and then adopt a Vector Quantization (VQ) layer to convert the acoustic context units to the learnable token sequence, thus better matching the expressive speech generation task. In this way, the audio data of dialogue context 𝑈 𝑎 1→𝑁 -1 are represented by 𝑇 𝑎 1→𝑁 -1 . Unified Context Serialization: After acquiring discrete tokens for text and audio modalities of context, we emulate the real-time dialogue flow and combine the textual and acoustic tokens alternately into a multi-modal conversational format as {𝑇 𝑡 1 , 𝑇 𝑎 1 , 𝑇 𝑡 2 , 𝑇 𝑎 2 , ..., 𝑇 𝑡 𝑁 }. We adopt an alternating approach between the text and speech modalities instead of serializing the conversation history by first combining the text modality and then concatenating the speech modality. We believe that the alternating approach can better capture the cross-modal contextual dependencies in the conversation history.Inference. To inference the Semantic and Style information of response speech according to the serialized multi-modal user-agent context. the ConGPT takes the entire serialized conversation context {𝑇 𝑡 1 , 𝑇 𝑎 1 ..., 𝑇 𝑡 𝑁 -1 , 𝑇 𝑎 𝑁 -1 ,𝑇 𝑡 𝑁 } as input, then predicts the semantic and style knowledge 𝑇 𝑎 𝑁 . The process is formulated as follows:EQUATION1 https://pypi.org/project/g2p-en/ 2 https://wenet.org.cn/opencpop/ where 𝑇 𝑎 𝑁 also follows the same tokenization method as 𝑈 𝑎 1→𝑁 -1to discretize 𝑈 𝑎 𝑁 in advance, serving as the output of ConGPT. The sub-tokens in the discrete token sequence 𝑇 𝑎 𝑁 are decoded using an autoregressive approach until the End of Sequence (EOS) label is decoded. Please note that, similar to [33] , we do not assign explicit speaker identities to the token sequence of the serialized context.After unified context modeling, the Conversational VITS (ConVITS) is proposed to provide proper expressive speech for the agent's response. Note that ConVITS takes three information sources of the agent, including the agent's utterance 𝑈 𝑡 𝑁 for content rendering, the semantic and style representations of the agent inferred by the ConGPT for semantic and style rendering, and the additional reference speaker for timbre rendering. As shown in the right panel of Fig. 1 , it consists of four key complements, which are the context encoder, token encoder, timbre encoder and the VITS synthesizer. The cross-attention mechanism is used to integrate the content, semantic and style information of agent.Rendering. First, we utilize the text encoder 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑡 (•), that consists of 6 layers of transformer encoder, to process the response content 𝑈 𝑡 𝑁 , extracting its inherent textual information 𝑓 𝑐 to ensure the intelligible of synthesized speech. After that, the predicted context-aware semantic and style tokens 𝑇 𝑎 𝑁 are converted into a high-level style representation 𝑓 𝑠 using the token encoder 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑠 (•), which consists of 3 layers of transformer encoder.Finally, to achieve unified content, semantic and style rendering for expressive CSS, textual information 𝑓 𝑐 and high-level style representation 𝑓 𝑠 are integrated into a final agent's style embedding 𝑓 via the cross-attention layer. We use a multi-head attention module in the cross-attention layer, which accommodates different input lengths. Here, we treat 𝑓 𝑠 as the query and 𝑓 𝑐 as both the key and value.Rendering. In order to achieve flexible and diverse expressive styles, we do not rely on fixed speaker IDs for speaker control. Instead, we employ a timbre encoder with a reference encoder to perform timbre transfer on any reference speech, allowing for flexible speaker timbre rendering based on content, semantics, and style rendering. As shown in the Fig. 1 , the timbre encoder 𝐸𝑛𝑐𝑜𝑑𝑒𝑟 𝑡𝑖𝑚 (•), consists of 6 convolutional layers, a GRU layer, and a linear layer, takes a reference speech 𝑈 𝑎 𝑎𝑔𝑒𝑛𝑡 for a specific speaker and extracts the timbre embedding 𝑓 𝑡𝑖𝑚 .During training, the speech 𝑈 𝑎 𝑁 -2 from the agent in the last turn is chosen as the reference speech 𝑈 𝑎 𝑎𝑔𝑒𝑛𝑡 . During the inference stage, we can assign a reference with any speakers to achieve zero-shot timbre rendering performance.The speech generation stage extends the vanilla VITS [25] into conversational settings by leveraging the above conversational expressiveness rendering. It's important to highlight that the ConVITS architecture excludes the Stochastic Duration Predictor, since the semantic and style tokens predicted by ConGPT already encompass duration information, there's no necessity to predict it separately.The final agent's style embedding 𝑓 is processed by a projection layer to derive the mean 𝜇 and variance 𝜃 . The agent's timbre embedding 𝑓 𝑡𝑖𝑚 is encoded by the posterior encoder to produce the latent normal posterior distribution variable 𝑧, which is then decoded by a Flow-based Decoder to generate the normalized stream 𝑓 𝜃 (𝑧). The HiFi-GAN generator then upsamples the latent variables 𝑧 to the speech waveform 𝑈 𝑎 𝑁 .We propose a Three-Stage training strategy to ensure the performance of our GPT-Talker: 1) In the first stage, we focused on the modeling capabilities of ConGPT and ConVITS in single-sentence speech scenarios. we trained ConGPT and ConVITS with singlesentence speech datasets including LibriTTS [55] , LJSpeech [20] , AISHELL-3 [45] , etc., that the total duration is about 2.5k hours. In this way, ConGPT can predict the speech tokens based on the text token sequence. The ConVITS performs stable speech generation after inputting the provided text. 2) In the second stage, we continue to train the ConGPT using the collection subset of NCSSD. This enabled it to accurately predict the semantics and stylistic knowledge of the current sentence, leveraging the provided dialogue context and multimodal information. 3) In the third stage, we further enhance the naturalness and expressiveness of the synthesized speech by fine-tuning both ConGPT and ConVITS using the recording subset of NCSSD.Concerning the computation of the model's total loss. The GPT-Talker's loss 𝐿 𝑡𝑜𝑡𝑎𝑙 is composed of two elements: 𝐿 𝐶𝑜𝑛𝐺𝑃𝑇 and 𝐿 𝐶𝑜𝑛𝑉 𝐼𝑇 𝑆 . 𝐿 𝐶𝑜𝑛𝐺𝑃𝑇 calculates the cross-entropy loss between the predicted acoustic units and the real units. 𝐿 𝐶𝑜𝑛𝑉 𝐼𝑇 𝑆 includes mel reconstruction loss 𝐿 𝑟𝑒𝑐𝑜𝑛 , KL divergence loss 𝐿 𝑣𝑎𝑒 from VAE, featurematching loss 𝐿 𝑓 𝑚 , adversarial training loss 𝐿 𝑎𝑑𝑣 , and 𝐿 𝑣𝑞 from Vector Quantization. Specifically, 𝐿 𝑣𝑞 computes the commitment loss between the quantized vector and the input discrete token. 
ncssd construction: 
 As mentioned before, we also propose a large-scale natural conversational speech corpus, NCSSD, to support the GPT-based CSS training. As illustrated in Table 1 , the dataset includes two subsets, that are the collection part and the recording part, covering a total duration of over 236 hours. Our dataset is available with the CC-BY-SA 4.0 license. Unlike the traditional data production method with human participation [8, 41, 59] , note that we propose an automatic data construction pipeline for the collection subset and a ChatGPT-assisted workflow for the recording subset. We will report the data collection process 3 and the data statistics results.The automatic data construction pipeline for the collection subset consists of 1) Video Selection, 2) Dialogue Scene Extraction, 3) Dialogue Segment Extraction, and 4) Dialogue Script Recognition.In order to build a large-scale, diversified, and natural conversational dataset, we collect videos from different TV series, which can simulate spontaneous conversation behavior in the real-world environment [41, 59] . We collect 79, and 34 TV shows for English and Chinese respectively. The detailed list of all TV series are introduced in Appendix. Unlike other data creation methods that require manual selection of TV shows based on certain rules [59] , we automatically filter out eligible dialogues and corresponding timestamps using Dialogue Scene Extraction, Dialogue Segment Extraction, and Dialogue Script Recognition modules.A complete TV show consists of multiple dialogue scenes (may include two or more speakers) that are interconnected but independent from each other. To extract these dialogue scenes, we employ Voice Activity Detection (VAD) technology, which uses silent segments in the audio information of the entire TV show to identify dialogue scenes. Subsequently, the extracted dialogue scene audio is further processed to obtain clean dialogue speech. Specifically, we first employ a pre-trained VAD model, silerovad 4 , to identify the timestamps of non-silent voice chunks in the video, since the silero-vad was trained on huge corpora that include over 100 languages and it performs well on audios from different domains with various background noise and quality levels. We set the silent segments threshold to 4 seconds to get the VAD results, since 4 seconds of silence often indicates the start of a new dialogue, as proven by extensive preliminary data testing. Then we segment the complete audio file of a video into various discontinuous audio clips that represent the various dialogue scenes. To ensure that each dialogue scene includes multi-turn dialogues with appropriate length, we further discard the audio clips where "the ratio of silent to non-silent segments exceeds 30%" or the "total duration is less than 15 seconds".Subsequently, we then perform background music separation with Demucs5 to discard the background music and other distracting information. To further refine the vocal component's quality, we thoroughly assess the signal-to-noise ratios (SNR) of both vocal and background noise, retaining only vocal chunks with an SNR above 4, and adopt speech enhancement model, sepformer 6 , to obtain the clean audio for all dialogue scenes. 4.1.3 Dialogue Segment Extraction. Dialogue segment extraction aims to extract the conversational speech containing only twoperson interaction from the previously obtained dialogue scene speech.To achieve this, we utilized a speaker recognition interface 7based on ByteDance to analyze the speech information from all dialogue scenes. This interface provides us with numerical speaker labels and corresponding timestamp information. Subsequently, we extract dialogue segments for two-person conversations based on the speaker labels and timestamps. Our extraction criteria required the dialogue segment to have more than four utterances, with each speaker contributing at least two utterances.4.1.4 Dialogue Script Recognition. Through the above steps, we obtained the dialogue-level audio and visual information of the two-person conversation. In order to obtain the dialogue script, we employ the Alibaba automatic speech recognition engine 8to recognize the speech. Finally, the audio, scripts and the visual information of all dialogues are combined tighter as the collection suset of NCSSD.Unlike the collection subset, the construction of the recording subset involves designing dialogue scripts and inviting volunteers to record their voices and upper body image signals. The script serves as a prompt for the dialogue content rather than a strict guideline, and volunteers are allowed to spontaneously expand on the dialogue during the recording process. The resulting dialogue speech, the upper body image and the script are then recorded to obtain the final recorded data.Generation. The first step in preparing recorded speech data is to create dialogue scripts. Manual preparation of scripts can be time-consuming, labor-intensive, and may lack diversity in terms of dialogue topics. Therefore, we leverage the powerful text generation capabilities of ChatGPT by using a prompt-based approach to generate diverse dialogue scripts that meet our expectations. Specifically, we employ GPT-3.5 Turbo version of ChatGPT to generate the script of a two-person conversation. As shown in Fig. ?? , we design a two-step prompt template to prompt the ChatGPT to output large-scale dialogue scripts to reflect spontaneous conversation behavior in the real world. In the first step, we aim to prompt ChatGPT to generate various topic words of human conversation ensuring the richness of communication content. In the second step, we select a specific dialogue topic and set the emotional state of the starting speaker. We prompt ChatGPT to generate multi-turn dialogues with a range of 4 to 15 turns. Additionally, we request GPT to add emotion and intent labels 9 to the generated scripts to enhance their interpretability and provide more reference information for subsequent speech recordings. The detailed prompt templates are shown in Appendix.Recording. Based on the previously obtained scripts, we invited volunteers to record the dialogue speech. We also captured the upper body images of the participants during the recording. Importantly, during the recording process, volunteers are encouraged to freely add dialogue content based on the emotional and intent information provided in the script, in order to create spontaneous and natural dialogue speech. Specifically, we employed 27 young volunteers with English as their second language 10 to participate in the recording sessions. Their compensation is based on the number of dialogues recorded. The recording venues varied, including classrooms, meeting rooms, seminar halls, and more, providing a diverse range of settings. We allow the volunteers to engage in spontaneous dialogues guided by the provided script, emotion label, and intent label. This approach ensured that the final recorded voice sounded natural and authentic.Re-identification. Due to the presence of spontaneous utterances by the voice actors during the recording, we use a speech recognition interface 8 to re-transcribe the recorded speech and obtain the recognized text as the final dialogue script. Together with the recorded speech and upper body images, this forms our final recording subset.Table 2 presents the overall statistics of the NCSSD dataset. It contains a total of 19,456 dialogues and 193,450 sentences from 113 TV shows, ensuring the scale and diversity of the data. The dataset spans approximately 236 hours, featuring the longest Chinese conversation at 308.21 seconds and the longest English conversation at 177.51 seconds. Every conversation exceeds 4 seconds, satisfying the minimum duration needed for dialogue tasks. On average, each conversation contains at least 8 turns, facilitating the training of models for extended dialogue sequences. With over 776 speakers, the dataset encompasses a diverse range of speaking styles and habits. For additional statistics of the NCSSD, please refer to Appendix. 10 Due to budget limitations, we did not invite native English speakers for the recording. Although the language proficiency of our participants may not be their native language, they possess fluent English listening, speaking, reading, and writing skills. We believe they are competent enough to carry out our data recording tasks.  
experiments: 
 In the experiment part, we will first assess the quality of NCSSD in a single-sentence speech synthesis scenario. Subsequently, we validate the GPT-Talker by comparing it with various state-of-theart baselines of CSS task. We will introduce the baselines, and evaluation metrics in the following subsections and report the results in the next section. Note that the experimental setup can be viewed in the Appendix.To validate the reliability of NCSSD, we evaluate it on the singlesentence speech synthesis task and select the advanced end-to-end TTS model, VITS [25] , as the baseline. Note that VITS was also chosen as the backbone network by many CSS models [9, 11] , and we believe it is appropriate to serve as the baseline.To evaluate the validity of GPT-Talker in terms of context understanding and modeling, we develop three advanced CSS systems that represent various context learning methods with the following three categories: (1) GRU-based Context Learning: The conversational context-aware TTS (we call "CCATTS" here) model proposed by [17] employs a GRU-based network to model the sentence-level dependency among the dialogue context; (2) Multi-scale Context Learning: FCTalker [19] is an representative work that consider both the sentence-level and word-level contextual within the dialogue context; (3) Heterogeneous Graph-based Context Learning: ECSS [34] is an advanced expressive and emotional CSS model that adopts heterogeneous graph to model the complex relation among the multi-modal context.We conduct comprehensive evaluations using both objective and subjective metrics.Subjective Metrics. The subjective evaluation includes three components: (1) ABX preference test, where listeners had to decide which of the transformed speeches, A or B (produced by two different methods), sounded closer to ground truth speech or if they had no preference. (2) Dialogue-level Mean Opinion Score in terms of Naturalness (N-DMOS), which requires participants to primarily assess its naturalness under the dialogue context. (3) Dialogue-level Mean Opinion Score in terms of Emotion (E-DMOS), which primarily focuses on assessing the emotional expression conveyed through the speech and determining its alignment with the ongoing emotional tone of the dialogue context.Objective Metrics. (1) Dynamic Time Warping Distance DTWD: we follow [42] and adopt average dynamic time warping (DTW) distance [37] of the pitch distribution for the ground-truth speech and synthesized speech to evaluate expressiveness-related performance. (2) Speaker Similarity (SSIM), we follow [22] and utilize the speaker verification model, that fine-tuned with WavLM11 , to extract the speaker embedding for the ground-truth speech and the synthesized speech. After that, we calculate the cosine similarity score between two embeddings as the final speaker similarity metric. The similarity score ranges from [-1, 1], where a higher value indicates a greater similarity. 
results and discussions: 
 We report the results of the following aspects, including "Reliability verification of NCSSD", "Validity verification of GPT-Talker" and "Analysis of Three-Stage Training". Please refer to the Appendix to check more results.We train VITS using our NCSSD in single-sentence scenarios, and the ABX preference test results of synthesizing Chinese and English speech are shown in Fig. 2 .For each language, we combine the collection and recording subsets to train the VITS model. We randomly select 50 sentences from both the Chinese and English datasets for synthesis and conducte an ABX preference test with 30 volunteers to mark the speech that has higher naturalness between the speech synthesized by the VITS model and the ground truth (GT) speech. The results of Fig. 2 indicate that the performance of VITS is very close to that of the GT. The proportion of "No Preference" in Chinese speech reaches as high as 33.35%, while in English speech, it is approximately 34.35%. This demonstrates that our NCSSD can support advanced TTS model in synthesizing high-quality speech.Undoubtedly, the performance of GT speech is expected to be better. However, since our NCSSD is designed to mimic real-world human conversations, effectively incorporating contextual modeling methods is an effective approach to further enhance the performance of TTS models. Therefore, in the next section, we further validate the effectiveness of the proposed GPT-Talker in context learning.In this section, we will compare GPT-Talker with three advanced CSS baselines. Since these baselines have conducted experiments using the English dataset DailyTalk in their respective studies, we will validate our model using the English portion of the NCSSD dataset and the DailyTalk dataset 12 . Similar to the previous section, the English portion will be trained by combining the collection subset and the recording subset.For subjective evaluation, due to budget constraints, we don't invite native English speakers as volunteers. Instead, we invite 30 Chinese students who are proficient in English listening, speaking, reading, and writing as volunteers. We randomly select 50 sentences from the test set to synthesize speech for each system to be tested. Then, we ask the volunteers to rate the speech using N-DMOS 3 .From the DTWD results in the table, it is evident that the speech synthesized by GPT-Talker outperforms the baselines, indicating that GPT-Talker is more capable of capturing pitch-related expressiveness similar to the GT. Furthermore, the N-DMOS and E-DMOS results show that our GPT-Talker significantly outperforms all the baselines. For example, in the DailyTalk dataset, GPT-Talker achieves N-DMOS and E-DMOS scores of 3.890 and 3.908, respectively, while the baselines score below 3.6. Similar trends are observed in the results for the NCSSD dataset. Moreover, GPT-Talker exhibits closer resemblance to GT compared to the baselines. These results demonstrate that GPT-Talker, leveraging ConGPT, effectively models the impact of context on the semantics and style of the current sentence, while ConVITS successfully renders semantic and stylistic aspects in CSS, resulting in highly natural and expressive synthesized speech. Lastly, by observing the SSIM metric results, it can be seen that GPT-Talker achieves the highest speaker similarity compared to all the baselines, proving that our ConVITS excels in timbre rendering on top of semantic and stylistic rendering. With the dialogue context modeling ability of ConGPT, ConVITS achieves highly expressive conversational speech synthesis.This section validates the three-stage strategy, introduced in Section 3.4, used to train GPT-Talker. We design the following four training strategies for validation: 1) One-Stage (𝑤/ CL&RC) means we directly use both the collection and recording subsets of NCSSD to train the GPT-Talker; 2) Two-Stage (𝑤/ CL&RC) means we follow the same method as described in section 3.4 for pre-training. Afterward, we merge the two subsets together and perform finetuning; 3) Two-Stage (𝑤/ CL). This method is similar to the second one, with the difference that during fine-tuning, only the collection subset is selected; 4) Two-Stage (𝑤/ RC): This method is similar to the second one, with the difference that during fine-tuning, only the recording subset is selected. We conduct subjective and objective experiments using the same configuration as in the previous section and report all results in Table 4 . We observe that the multi-stage approach outperforms the single-stage approach in all metrics, suggesting that using a pre-training strategy enables the model to learn basic speech generation capabilities, and fine-tuning on the conversational data helps improve dialogue context understanding and expression abilities. Additionally, the "Two-Stage (𝑤/ CL&RC)" and our three-stage approaches outperform previous two-stage methods, demonstrating that fine-tuning on the entire conversational dataset allows the model to learn dialogue expression capabilities with the support of a large volume of data. Comparing the "Two-Stage (𝑤/ CL&RC)" approach with our three-stage approach, although there is a slight lag in one metric for both the Chinese and English datasets, our three-stage approach exhibits significant advantages in the remaining six metrics. This confirms that our three-stage method gradually guides the model to learn richer dialogue expression information, leading to better conversational speech generation capabilities. 
conclusion: 
 In this work, we propose a novel GPT-based conversational speech synthesis (CSS) model, termed GPT-Talker, for user-agent interaction. It consists of ConGPT and ConVITS to model the semantic and style expression in the unified user-agent discrete dialogue context sequence and infer the expressive speech for the agent. We also propose the largest-scale conversational speech synthesis dataset to date, termed NCSSD, which stands out in terms of language and data source diversity. This dataset can support the training of GPT-Talker and even future GPT-style CSS models, providing a valuable resource for advancing CSS technology. The comprehensive experiments are conducted on the reliability of the NCSSD and the effectiveness of our GPT-Talker. We encourage the research community to use the NCSSD for spoken dialogue modeling. 
 
 Appendix: For a more intuitive flowchart, please refer to the Appendix.https://github.com/snakers4/silero-vadhttps://github.com/facebookresearch/demucshttps://huggingface.co/speechbrain/sepformer-dns4-16k-enhancementhttps://www.volcengine.com/product/voice-techhttps://ai.aliyun.com/nls/The emotion categories are labeled using a 7-category scheme[59], that are Neutral, Happy, Surprise, Fear, Angry, Disgust, and Sad. The intention labels are labeled using a 9-category scheme[50], including Question, Agree, Acknowledge, Sympathize, Encourage, Console, Suggest, Wish, and Neutral.https://huggingface.co/microsoft/wavlm-base-plus-svPlease note that directly training the GPT-Talker model on the DailyTalk dataset leads to poor performance. Therefore, the experimental results in this section are obtained by fine-tuning all systems based on the pre-training described in section 3.4. 


Decision: This paper proposes a novel generative expressive Conversational Speech Synthesis system, GPT-Talker. The system creates dialogue token sequences from multimodal multi-turn dialogue history utterances, that are then posteriorly provided to an LLM as dialogue context. Then, the LLM is used to generate responses that cover both the style and semantics. Finally, generations are synthesized with a conversational VITS.

The paper got three positive (Borderline Accept) reviews, indicating that while there are some limitations with the submitted work, the approach is relevant for the community.

As key strengths, reviewers highlight the general soundness of the framework (mFmg, prwU, 6QF4), the NCSSD dataset contribution (mFmg, prwU, 6QF4), and the performance of the proposed approach (mFmg, 6QF4).

However, some limitations were also pointed out, such as:
- The lack of dialogue-specific tokens limits the expressiveness of the model (mFmg)
- An assessment of Acoustic Tokenizers is missing (prwU)
- An emotion and expressivity variations assessment, on the NCSSD dataset, is not fully clarified (prwU)
- The alternating concatenation effectiveness is only partially confirmed (6QF4)

The author's rebuttal only partially addressed reviewers' initial concerns, with all reviewers having maintained their initial scores.

Striking the balance between the strengths and limitations of the paper, I suggest this work to be accepted as Poster, if there is space. 

Some of the pointed limitations hint at the relevance and research potential of the proposed work, and could be addressed in future work, such as a) latency optimization, b) expanding the framework to other languages, and c) assessment of adaptability to other factors (e.g. different speaking styles).

Reviews: review: paper_summary: paper_summary: The paper introduces GPT-Talker, a novel generative expressive Conversational Speech Synthesis system. GPT-Talker uses GPT to transform multimodal dialogue history into discrete token sequences, integrating them to predict responses that include both semantic and style knowledge. The system then synthesizes expressive speech using conversation-enriched VITS. Additionally, the authors present a new large-scale dataset, NCSSD, featuring 236 hours of naturally recorded conversational speech in both Chinese and English. The dataset includes dialogues from TV shows and improvised styles. Experiments demonstrate that GPT-Talker outperforms existing CSS systems in naturalness and expressiveness. The code, dataset, and pre-trained model would be available online.
strength_weakness: strength_weakness: Although there aren't any novel solid technical contributions in terms of new deep learning models, the overall framework looks interesting. The paper's primary innovation lies in transforming multimodal information from dialogue history into discrete token sequences, which are then integrated to form a comprehensive representation of the user-agent dialogue context.

The experiments conducted demonstrate that GPT-Talker significantly outperforms existing state-of-the-art CSS systems. User studies assess the naturalness and expressiveness of the synthesized speech, and the paper also employs objective metrics for evaluation. The reliability of the NCSSD dataset is validated by demonstrating its impact on the performance of the GPT-Talker system.

The contribution to the field, particularly with the publicly available dataset is laudable.
limitations: limitations: The usage of timbre rendering modules is well-established within the community, as is the capability of the VITS architecture for generating natural speech. It would be valuable to see results on optimizing GPT for conversational context, potentially involving pre-training on general text followed by fine-tuning on dialogue data. The paper does not demonstrate the ability to learn dialogue-specific tokens. Introducing dialogue-specific tokens or embeddings could help the model distinguish between different conversational roles (e.g., speaker vs. listener) and types of dialogue acts (e.g., questions, responses, affirmations).

The processing pipeline, including multimodal tokenization, context integration, and expressive speech synthesis, might introduce latency in real-time applications. Furthermore, the performance of the GPT-Talker system relies heavily on large-scale datasets like NCSSD, which are resource-intensive to create and may not be readily available for all languages or domains.

The current implementation and dataset focus on Chinese and English. Extending support to additional languages and dialects would require significant effort in collecting and annotating large-scale conversational datasets. Each language might also necessitate specific adjustments to handle unique linguistic features and conversational norms. Additionally, the system may not fully capture cultural nuances that influence conversational styles and expressions, potentially affecting the naturalness and appropriateness of synthesized speech in diverse cultural contexts.

While the paper focuses on naturalness and expressiveness in its evaluations, other crucial factors such as robustness to noisy inputs, variability in speaking styles, and adaptability to different user preferences are not thoroughly explored. A comprehensive evaluation across a broader set of metrics is necessary to fully understand the system's strengths and weaknesses. The evaluations are conducted in controlled settings, which may not accurately reflect real-world deployment scenarios. Factors such as background noise, diverse user accents, and varying conversational dynamics can impact the system’s performance in practical applications.
paper_summary: paper_summary: This paper advances the task of conversational speech synthesis in two aspects. First, the authors proposed to apply GPT-style language modeling to consider the conversational context and generate the corresponding semantics and style. Second, the authors curated a new conversational speech synthesis dataset called NCSSD, which is by far the largest dataset for conversational speech synthesis.
strength_weakness: strength_weakness: I am convinced that GPT-style language models are indeed good fits for the task of conversational speech synthesis given their outstanding ability to consider long-term context. Open-sourcing NCSSD is also no doubt a huge contribution to the community.
limitations: limitations: I have several concerns regarding the methodology.

First, I am concerned about how much style information resides in the HuBERT indices. For instance, previous works like AudioLM use audio codecs like SoundStream, which contain much more variation and information, and it is much easier to imagine that such codecs contain the necessary variation. On the other hand, HuBERT indices are known to contain mostly linguistic information, thus this model choice is a bit strange to me.

Second, the novelty of the conversational VITS is not significant, as it does not differ much from conventional VITS. In addition, I am confused about the role of the "Agent's Semantic and Style" and "Reference Speaker for Agent". Is the goal to use some representation (here, HuBERT indices) to represent semantics and style, and use the latter to represent the speaker? Couldn't it be possible that the latter also contains style? Finally, why use VITS to map the tokens to speech, instead of using VALL-E-like models? 

I also find the "Reliability verification of NCSSD" experiment in Sec. 6.1 insufficient. "Training a single-utterance VITS with NCSSD and showing that the output is of comparable quality to the ground truth" seems somewhat unrelated to "NCSSD is a reliable dataset for CSS". Specifically, I question whether there are enough emotion and expressivity variations in NCSSD. I suggest authors to provide more analyses as those done in the STUDIES and CALLS papers.

My final concerns are about the multi-stage training strategy. First, in Sec. 6.3, the authors stated: "We observe that the multi-stage approach outperforms the single-stage approach in all metrics, suggesting that using a pre-training strategy enables the model to learn basic speech generation capabilities," however, I wonder about the possibility that the model simply benefits from being trained on more data. In addition, in Table 4, the "Two-Stage (𝑤/ CL & RC)" and "Three-Stage (Ours)" have their N-DMOS scores' confidence interval overlap, which means there are no statistical differences in performance. It is therefore difficult to persuade me that training on CL and then RC is indeed better. In fact, I am not really convinced in the first place -- why do the authors propose to pre-train on CL and then fine-tune on RC? Do the authors believe that RC is better than CL? If so, how do we verify it?

There are also some unclear parts about the methodology:
(1) Are the ConGPT and Conversational VITS models trained in an end-to-end manner?
(2) In the Conversational VITS model, how are $U^t_N$ and $T^a_N$ integrated? Specifically, what happens if they are of different lengths?
paper_summary: paper_summary: Existing CSS methods generate speech by designing complex network structures and meticulously optimizing the modules within them. Moreover, due to the small amount of data in existing datasets that contain scripted recording styles, it is difficult to simulate real, natural conversational styles. To address these challenges, this work proposes a novel generative expressive CSS system, named GPT-talker. Additionally, this work introduces a large-scale bilingual CSS dataset, NCSSD, for training GPT-talker, and achieving state-of-the-art results.
strength_weakness: strength_weakness: 1. By tokenizing textual and acoustic input into GPT, generating discrete speech tokens, and then using the ConVITS to produce the final speech, the method is intuitive.
2. A natural conversational speech corpus containing 236 hours is proposed, which is beneficial for future research in this field.
3. After training on the proposed dataset with GPT-talker, the model outperforms baseline models in terms of naturalness and expressiveness.
limitations: limitations: 1. Textual tokens and acoustic tokens belong to two different feature spaces, and a purely GPT structure may not align these two features well.
2. Why does using an alternating approach between the text and speech modalities better capture the contextual dependencies of a conversation? GPT relies on all previous information to predict the next token, and the alternating approach does not seem to change the input information.