Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation

Authors: heeseung  kim
soonshin  seo
kyeongseok  jeong
ohsung  kwon
soyoon  kim
jungwhan  kim
jaehong  lee
eunwoo  song
myungwoo  oh
jung-woo  ha
sungroh  yoon
min  kang
  yoo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 02:24:34

Abstract: Recent work shows promising results in expanding the capabilities of large language models (LLM) to directly understand and synthesize speech. However, an LLM-based strategy for modeling spoken dialogs remains elusive, calling for further investigation. This paper introduces an extensive speech-text LLM framework, the Unified Spoken Dialog Model (USDM), designed to generate coherent spoken responses with naturally occurring prosodic features relevant to the given input speech without relying on explicit automatic speech recognition (ASR) or text-to-speech (TTS) systems. We have verified the inclusion of prosody in speech tokens that predominantly contain semantic information and have used this foundation to construct a prosody-infused speech-text model. Additionally, we propose a generalized speech-text pretraining scheme that enhances the capture of cross-modal semantics. To construct USDM, we fine-tune our speech-text model on spoken dialog data using a multi-step spoken dialog template that stimulates the chain-of-reasoning capabilities exhibited by the underlying LLM. Automatic and human evaluations on the DailyTalk dataset demonstrate that our approach effectively generates natural-sounding spoken responses, surpassing previous and cascaded baselines. Our code and checkpoints are available at https://github.com/naver-ai/usdm.

Main paper: 
 introduction: 
 Large language models (LLMs) have gained significant traction thanks to emergent capabilities [1] [2] [3] [4] [5] , such as few-shot in-context learning, complex reasoning [6, 7] , and instruction-following [8] . These remarkable discoveries led to chat-enabled LLMs and generative personal assistants [9] . However, text-based agents are limited in usability due to their medium of interaction. Ideally, speech-enabled LLMs would recognize the user's emotional state or subtle nuance and generate spoken responses with prosody most appropriate to the user's context. Although automatic speech recognition (ASR) and text-to-speech (TTS) systems can be easily employed, the linguistic discrepancy between speech and text causes dialog inefficiencies and result in sub-optimal user experience [10, 11] . As such, systematically integrating the speech modality into LLMs can unlock speech interactivity while retaining LLMs' powerful capabilities. Figure 1 : Overview of our spoken dialog modeling approach (Left). All possible self-supervised learning objectives from our speech-text pretraining scheme. (Right)Recent advances have also spurred the idea of large foundational models (LFM) for other modalities (e.g., vision, speech, etc.) [12] , unifying LLMs with the sensory spaces. In the vision domain, numerous works have explored interfacing pretrained language models with the visual modality [13] [14] [15] [16] . More recently, Yu et al. [16] proposed a vision language model (VLM) based on a pretrained LLM that directly generates discrete vision tokens, which can be decoded into high-fidelity images.Such work shows that autoregressive language models can model the vision modality.In the speech domain, while earlier work focused on text-less speech modeling [17] [18] [19] , recent work has either taken inspiration from the LLM architecture to achieve speech synthesis [20] or incorporate pretrained language models into speech-understanding tasks [18, [21] [22] [23] , where the model is limited to text outputs. More recent work explores the possibility of empowering pretrained LLMs to autoregressively generate discrete speech tokens for speech translation [24] and speechinstruction following tasks [25] [26] [27] . Despite these successes, more work is needed to understand whether LLMs are capable of generating speech, understanding, and incorporating paralinguistics that are appropriate and natural for the social context, especially in spoken dialog settings.We introduce the Unified Spoken Dialog Model (USDM), a novel LLM-based approach for modeling spoken dialogs in an end-to-end fashion. We propose a novel speech-text pretraining scheme that promotes learning cross-modal distributional semantics, which is vital for imbuing LLMs with the ability to generate coherent speeches in spoken dialog modeling. In particular, based on the observation that any subsample of either speech or text in corresponding speech-text pairs form two types of relationships with the other modality (right side of Figure 1 ), we formulate a large number of combinations of training objectives that theoretically benefits all speech-text tasks, including spoken dialog modeling. We then fine-tune our pretrained speech-text model with spoken dialog data, breaking the speech-to-speech modeling problem into intermediary steps that are easier for the underlying pretrained LLM to handle (Figure 1 ). To enhance the effects of our speech-text pretraining and spoken dialog modeling, in addition, we adopt the prosody-infused speech tokenization scheme based on the discovery that the speech token, previously used to convey semantic information, also contains prosody information.We demonstrate that USDM outperforms baselines in spoken dialog modeling for the DailyTalk dataset. We further validate the effectiveness of our pretraining and fine-tuning schemes through comprehensive ablation studies. Along with various analyses, we highlight the capabilities of USDM with diverse samples on our demo page. 2 Our contributions are as follows.• We propose a unified pretraining strategy for modeling the comprehensive relationship between the speech and text modalities that is especially effective for downstream speechto-speech spoken dialog generation.• We present an extensive spoken dialog modeling framework detailing the discrete speech tokenization scheme utilizing a pair of a prosody-infusing encoder and a decoder. Additionally, we propose an LLM-based modeling strategy for generating natural-sounding and semantically coherent dialog responses in an end-to-end fashion.• Our work establishes the foundation for speech-enabled chat-based LLMs, showcasing a prototype that not only maintains the LLM's ability to generate dialog responses but also enhances LLM with speech-interaction capabilities. 
related work: 
 Discrete Speech Representations. To construct spoken language models (SLM), various discrete speech representations have been utilized in previous works [17, 20, [28] [29] [30] . These representations are primarily categorized into two types: tokens based on speech self-supervised representations [17, 30] and neural audio codecs [20] .A discrete token based on speech self-supervised representation [17, 31] is obtained by k-means clustering of the intermediate representation from a speech self-supervised model. These tokens, often called acoustic units, are typically encoded with a frequency range of 25Hz to 50Hz. The amount of speech information within the compressed discrete speech token is determined by the number of clusters, denoted as k. With a relatively small value for k, many works have preserved the semantic information in the tokens and utilized these to construct SLMs [32, 33] .Neural audio codecs, another type of discrete token, capture both semantic and paralinguistic information of speech [34] [35] [36] [37] [38] . A speech encoder and decoder are trained using an autoencoder architecture with residual vector quantizer for the encoder output [39] . This representation includes most of the perceptual information of audio and is widely used for audio synthesis [20, [40] [41] [42] .Spoken Language and Dialog Models. Many studies have recently explored spoken language modeling to address a variety of tasks involving speech and text [23, [43] [44] [45] . Various works tackle tasks such as automatic speech recognition [27, [46] [47] [48] [49] , spoken question answering [50] [51] [52] , and speech-to-text translation [27, 48, 53] , which process speech as input and output text. Conversely, there are also emerging works focused on tasks like speech synthesis [20, 26, 28, 54, 55] , where text is used as input to generate speech output. Early SLMs that process speech as input and output are trained solely based on speech data without language models [17, 19] . With the advancement of LLMs, several studies aim to construct SLMs that extend language models to handle both speech input and output. These studies are primarily proposed for speech modality pretraining [30, 56, 57] , or introduced in specific tasks such as speech-to-speech translation [24, [58] [59] [60] and spoken conversation modeling [25, 61] .Recently, several works have been proposed for spoken dialog modeling with speech input and output [19, 25, 62] . Nguyen et al. [19] develop a decoder-only transformer model trained from scratch, designed for modeling conversations between two speakers. In contrast, Lin et al. [62] adopt a cascaded approach for spoken dialog modeling that consists of separate ASR, an LLM-based emotion-aware text dialog model, and emotional TTS components. Zhang et al. [25] build SLMs on top of a pretrained LLM with objective functions designed for ASR and TTS tasks.Among the previous works, end-to-end pipelines [19, 25] that focus solely on speech-only training or leverage simple cross-modal objectives for speech-text pretraining fail to fully utilize the capabilities of pretrained language models. Additionally, cascaded models [62] , which use separate ASR and TTS for spoken dialog, need explicit labels to incorporate paralinguistic features. This label dependency makes data collection challenging and limits the models to representing label-definable non-verbal cues. Furthermore, the error propagation inherent in the cascaded pipeline [63] increases their susceptibility to compounded errors.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of our work in the second paragraph of the conclusion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
our approach: 
 In this section, we describe the components that enable coherent and prosodic spoken dialog modeling, distinguishing our research from previous works. We first explain the discrete speech representation used for spoken dialog modeling in Section 3.1, demonstrating its suitability for prosody modeling.We then propose a unified speech-text pretraining scheme that extends the capabilities of the pretrained LLM into the domain of spoken language modeling in Section 3.2. Finally, in Section 3.3 and 3.4, we introduce USDM, a spoken dialog model fine-tuned with a multi-step spoken dialog template, and the speech decoder that restores the output speech token to a raw waveform.To model natural speech conversations, the speech representation must contain not only the content of the speech but also paralinguistic features such as emotions, which are crucial for conversation. We adopt acoustic units as speech tokens that are derived from k-means clustering of a self-supervised model's intermediate speech representation, which is known to predominantly capture content and pronunciation [31, 59] . The information captured by an acoustic unit token varies depending on the number of clusters; the greater the number of clusters, the more encoded information. Hence, among publicly available schemes, we consider the acoustic unit tokenization scheme with the largest vocabulary size, k = 10, 000 [59] . We then analyze whether the tokens derived from this scheme contain non-verbal information.The acoustic unit extractor used in SeamlessM4T [59] , first resamples the speech to a sampling rate of 16kHz and then feeds it to the XLS-R [64] , obtaining 50Hz intermediate continuous representations. Unit sequences are extracted by clustering these representations into 10,000 clusters, determining the vocabulary size of the speech tokens. Using this unit extractor, we conduct two experiments to investigate features captured in the unit sequence besides the semantic content. First, we perform unit emotion recognition tasks with speech emotion recognition data to ascertain whether the unit sequences contain paralinguistic information. Next, we train a separate unit-to-speech reconstruction model and use this model to compare the original and reconstructed speech, investigating the information encoded in the unit sequences.For the unit emotion recognition task, we train 3-layer transformer-based emotion classifiers using acoustic units on CREMA-D [65] , which is a speech emotion recognition dataset with six emotion categories. If the units lack paralinguistic information, the classification accuracy would approximate the probability of random guessing, which is 16.6%. However, we observe that the classification accuracy is 60.8%, indicating that the acoustic units contain emotional cues.To further investigate the paralinguistic information contained in the units, we train a separate unit-tospeech reconstruction model using 54,000 hours of speech data. The reconstruction model is trained using the architecture of Voicebox, one of the zero-shot stochastic TTS models [66] . Unlike Voicebox, which takes text and reference speech as inputs for adaptation, our model is trained to generate speech solely from a unit sequence without any reference speech. The comparison between the original audio and the speech reconstructed from the extracted units shows that while the timbre and absolute pitch of the reconstructed speech differ, the pitch variation has a similar trend, as shown in Figure 2 . This implies a crucial role in conveying non-verbal characteristics such as emotions, which closely match the original audio. Additionally, we have uploaded samples of several ground truth audios and corresponding reconstructed audios on our demo page.Through these two experiments, we confirm that the acoustic units, commonly known to primarily encode semantic information, also contain a significant amount of paralinguistic information, such as emotions and pitch variations. We adopt this speech tokenization scheme for our speech-text pretraining and spoken dialog fine-tuning to help capture non-verbal cues in spoken conversations. More detailed descriptions of these experiments are provided in Appendix A.3.1 and A.3.3.In this section, we introduce a unified speech-text pretraining scheme that extends pretrained LLM to speech-text cross-modality. Our overall speech-text pretraining scheme is in Figure 3 . For pretraining the speech-text model, we utilize Mistral-7B [67] as a pretrained LLM. To its existing vocabulary, we add 10,000 unit tokens and 2 special tokens, which will be described later, reinitializing only the embedding weights of these new tokens. We pretrain the speech-text model with approximately 87,000 hours of English ASR data. Each <speech, text> pair is used to create an interleaved speech-text sample I j = i 1,j , i 2,j , ..., i ||Ij ||,j to model various cross-modal relationships.Here, i k,j can be either an acoustic unit token, a text token, or a special token, and we construct the sequence I j with a proposed per-sample speech-text interleaving method in the following paragraph.Given the dataset D = {I 1 , I 2 , ..., I ||D|| }, the objective of our pretraining scheme is as follows:EQUATIONwhere θ refers to the parameters of LLM and the embedding weights of the newly added tokens.When constructing a spoken language model by extending pretrained LLM, relying on a specific task, such as ASR [25] , TTS [25] , uni-modal [30] and cross-modal continuation task [44, 56] , despite its large amount of dataset, may limit the model's capabilities to only those predefined relationships.To build a comprehensive speech-text model capable of both receiving and generating speech, we reinterpret the cross-modal relationship in terms of continuation or correspondence as shown in Figure 1 . Our proposed method, which focuses on this redefined relationship, is capable of generating diverse speech-text interleaved sequences, ensuring the model can handle complex speech-text interactions.We first extract word-level alignments of the speech and its transcript using the Montreal Forced Aligner [68] . These alignments yield speech time intervals for each word, which we then convert into index intervals of unit sequences at a resolution of 50Hz.Pair-wise Segmentation and Segment-wise Main Modality Random Selection Using the intervals, we divide each unit and text pair into N segments. Subsequently, from each of these segments, we randomly sample data of only one modality, either unit or text. A large value of N may lead to each segment containing short acoustic units and text sequences, which poses challenges in modeling unimodal text and unimodal unit sequences. To mitigate this issue, we dynamically set the value of N based on the speech duration, N = ⌊S/10⌋ + 1, where S is the speech length measured in seconds.Sub Modality Random Insertion and Special Token Insertion This segmentation and selection process allows us to generate a unified cross-modal interleaved sequence with continuation relationships.For correspondence relationship modeling, data from the non-selected modality in each segment is inserted with a 50% probability after the pre-selected modality data. Additionally, to indicate the relationship between speech and text tokens, we introduce two special tokens: 
 and 
. The former indicates that a token of the corresponding remaining modality will follow, while the latter indicates that a token of the subsequent position will follow. These tokens are added to the sequence only where the modality of the data changes.Through this procedure, we can obtain interleaved speech-text sequences {I j } j=1,...,||D|| . These sequences enable our speech-text model to perform not only unimodal modeling but also comprehensive cross-modal modeling. These interleaved sequences are utilized in Eq. 1 for the pretraining.We construct the Unified Spoken Dialog Model (USDM) by fine-tuning our speech-text model with spoken dialog data, with an overview presented as shown in Figure 1 . The basic template for spoken dialog fine-tuning involves directly modeling the response speech tokens from the input speech tokens. However, we adopt a template designed to fully exploit the capabilities of the pretrained LLM.Inspired by the step-by-step reasoning mechanism employed by LLMs [69] , existing works [24, 25, 57 ] use text to bridge the speech. Similarly, instead of directly modeling output speech from input speech, our model transcribes the speech, generates the response text, and produces the corresponding speech in an end-to-end pipeline. The insertion of the text-related tasks between speech inputs and outputs allows the model to benefit from the pretraining LLM and chained reasoning over the intermediary modality [6] . Since each stage in the pipeline attends to all input and output tokens generated in prior stages, our approach is more robust against transcription errors and better at generating contextually relevant spoken responses than if it were carried out in independent modules (i.e., the cascaded approach), which we will discuss further in Section 4.3.The supervised fine-tuning template we use is shown in Figure 5 in the Appendix. We calculate the loss using Eq. 1 only for the input transcript, answer text, and answer unit part, as highlighted in Figure 5 .We train the unit-to-speech model using the Voicebox architecture [66] to reconstruct speech from units. Voicebox is a zero-shot TTS model that takes text and reference speech as inputs to perform personalized TTS. Unlike the reconstruction model used in Section 3.1, we leverage not only unit sequences but also reference speech to adapt and perform zero-shot unit-to-speech reconstruction.Our model utilizes the reference speech and the paralinguistic features contained in the units to generate prosodic spoken responses of the target speaker. For clarity, we refer to this unit-to-speech model as unit-Voicebox. More details of our decoder are in Appendix A.3.3.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We have included the link to our code in the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
training details and baselines: 
 We compare USDM to 3 baselines, From Scratch, Cascaded, and SpeechGPT [25] , on DailyTalk [70] . DailyTalk comprises 20 hours of spoken dialog data with a sampling rate of 22,050 Hz, involving one male and one female, and we describe further details in Appendix A.4.1. We also present the models used for each component of USDM and the baselines in Table 4 in the Appendix.USDM. For speech-to-unit module, we adopt the official checkpoint of XLS-R [64] and a quantizer with k = 10, 000 [33] , trained on 436K hours of multilingual speech data. As a speech decoder, we follow the architecture and hyperparameters of Le et al. [66] and train the unit-Voicebox on the English subset of Multilingual LibriSpeech [71] and GigaSpeech [72] for 10 epochs using 64 NVIDIA A100-40GB GPUs, with a batch size of 256. We use the Adam optimizer [73] with a learning rate of 10 -4 . We utilize the official checkpoint of BigVGAN [74] as our vocoder.Our proposed unified speech-text pretraining is conducted using 512 NVIDIA A100-40GB GPUs, with a global batch size of 1,024 for 8,000 iterations. For pretraining, we utilize approximately 87,000 hours of English ASR data; the English subset of Multilingual LibriSpeech [71] , People's Speech [75] , GigaSpeech [72] , Common Voice 15.0 [76] , and the English subset of Voxpopuli [77] . The data used for pretraining is packed to a maximum sequence length of 8,192. For spoken dialog modeling, we fine-tune a speech-text model with a global batch size of 64 for 5 epochs. We use linear learning rate scheduling with a peak learning rate of 2 • 10 -5 for both pretraining and fine-tuning.From Scratch. The From Scratch model is nearly identical to the USDM but excludes speech-text pretraining. Specifically, we fine-tune the pretrained Mistral-7B model directly on spoken dialog data, with the hyperparameters identical to those of the USDM.Cascaded. We include a Cascaded model, which employs separate ASR and TTS models, as a baseline for comparison. For the ASR model, we use the official checkpoint of whisper-large-v3 [78] , which is trained on 5M hours of speech data. For the speech synthesis model, we train Voicebox with text input using the same hyperparameters and datasets as unit-Voicebox. As the LLM, we utilize the transcript of the spoken dialog dataset to create text dialog data and fine-tune the Mistral-7B on this data using the same hyperparameters as the USDM.SpeechGPT. For SpeechGPT [25] , we use the official implementations and checkpoints for the speech-to-unit module, spoken language model, and speech decoder module. Specifically, we finetune SpeechGPT-7B-cm, a pretrained speech-text model, with DailyTalk for a fair comparison.We conduct various evaluations on the spoken responses generated for the given spoken dialogs.When generating samples for evaluation, we adopt a sampling scheme with top_k = 40, top_p = 0.7, and temperature = 0.3, except for SpeechGPT, where we use their own strategy. For Voicebox and unit-Voicebox, we utilize the speech from the previous turn as the reference speech. While SpeechGPT generates audio at 16kHz, other models synthesize speech at 22,050Hz. For a fair comparison, all audio samples are resampled to 16kHz and volume normalized to -27dB for evaluation.To compare the overall preference of our model and the baselines, we conduct a human preference test via Amazon Mechanical Turk. Given a randomly selected 50 spoken dialogs from the test split of a dataset, we instruct the evaluators to compare the spoken response of our model and the baseline, considering the comprehensive aspects such as naturalness, prosody, and semantic coherence. To evaluate the prosody and the naturalness, we additionally measure the 5-scale prosody mean opinion score (P-MOS) and the 5-scale mean opinion score (MOS) through Amazon Mechanical Turk. As explained in Section 3.3, our model first generates the text to be spoken before generating a spoken response, which allows us to fix the content of the generated speech by predetermining the text. Unlike the aforementioned human preference test, to focus solely on the prosody and naturalness, respectively, we provide the ground truth response text to the model to ensure consistency in the content of the output speech, thereby preventing difficulties in evaluations that may arise from variations in content. Instructions and detailed descriptions of our evaluations are in Appendix A.4.2.Furthermore, to evaluate the content appropriateness of spoken responses, we first generate the spoken responses of all models given the spoken dialogs for the entire test set. We then pass these samples through the ASR model, whisper-large-v3, to calculate METEOR and ROUGE-L scores, which are widely used in various NLP tasks such as text summarization and are commonly employed to measure performance in dialog modeling [79, 80] . We also conduct a GPT-4-based [1] preference test [81] between the transcribed texts of our model and all baselines.We also measure the Word Error Rate (WER) for the speech-to-text part and text-to-speech part of each model. For the speech-to-text part (STT WER), we calculate the WER across the entire test set. We use the outputs from the whisper-large-v3 model for the Cascaded baseline, while the remaining end-to-end pipelines are assessed using the intermediate transcribed text of each model.For the text-to-speech part (TTS WER), similar to our MOS and P-MOS evaluations, we calculate the WER using generated samples given the randomly selected 50 spoken dialogs and the corresponding ground truth written-form response. We generate each spoken response 5 times and report the average WER. For measuring TTS WER, we utilize the whisper-large-v3 model as the ASR model.The results are presented in compared to the baselines (p-value < 0.05 from the Wilcoxon signed-rank test). For the semantic aspect, our USDM outperforms the baselines in both quantitative evaluations and the GPT-4-based preference test (p-value < 0.05). We also observe that our model surpasses the baselines in the P-MOS evaluations (p-value < 0.05), which measure the prosody naturalness of the speech given spoken dialog. Notably, the USDM shows superior prosody compared to the Cascaded model. These results demonstrate that our model effectively incorporates prosody information in the spoken language model and is capable of generating spoken responses with content well-aligned to input speech.We also confirm that cross-modal pretraining is essential to leverage the capabilities of LLM. We observe that the From Scratch model, which directly models spoken dialog without pretraining, tends to overlook the bridging text and generates a spoken response that does not correspond to the pre-generated written response, thus negatively impacting its performance. This results in worse TTS and STT WERs and adversely affects the P-MOS and MOS, which are based on the prosody and naturalness of the spoken response given the transcript. This result indicates the difficulty of transferring the capabilities of text models to spoken dialog modeling without cross-modal pretraining.In this section, we compare the effects of correspondence and continuation modeling, which are crucial to our pretraining method. We consider three additional pretraining schemes. Setup 1 uses an interleaved unit-text sequence without a correspondence relationship, relying solely on continuation, similar to previous works [44, 56] . Setup 2 utilizes data that maintains only a correspondence relationship, as seen in Zhang et al. [25] . Setup 3 is similar to our fine-tuning approach, interleaving speech with its transcript and subsequent text and speech, as proposed in Nachmani et al. [57] . All setups are trained in the same way as our speech-text pretraining, with details in Section A.5.2.We evaluate these pretrained models on sequence modeling and spoken dialog modeling tasks. Performance is first assessed by measuring perplexity (PPL) of various speech-text sequences from the test-clean subset of the LibriSpeech dataset [82] . We construct six types of interleaved sequences: unimodal sequences for both unit (1) and text (2), sequences with unit followed by their corresponding text (3) and vice versa (4) to evaluate correspondence relationships, and sequences generated by dividing the unit and text in half, combining the first half's unit with the remaining half's text (5) , and the first half's text with the remaining unit (6) for assessing continuation. We then calculate the average PPL for all combinations by taking the logarithm of each subsequent modality's PPL within each sequence type, averaging these logarithmic values, and then applying We present the average PPL of each modality in Table 3 and the PPL for each combination in Table 9 in the Appendix. Our model demonstrates superior average PPL across both modalities. Focusing solely on either correspondence or continuation relationships, or on specific templates tends to make the model specialize in certain objectives but hinders its ability to model diverse relationships effectively. Our proposed unified speech-text pretraining scheme performs uniformly well without being overly focused on specific relationships. We also show in Table 3 that our speech-text model is beneficial to spoken dialog modeling, as evidenced by the WER, METEOR, and ROUGE-L scores. Particularly, Setup 1, which lacks correspondence relationship pretraining, exhibits significantly higher STT and TTS WERs, resulting in deteriorated semantic performance in spoken responses.As explained in Section 3.3, USDM first models the input and output text as a bridge when given a speech input before generating the spoken response. To demonstrate this approach, we train a spoken dialog model that models the speech output directly from the speech input (S1 → S2). We evaluate the generated response speech through METEOR and ROUGE-L scores with the same samples described in Section 4.1.2, and the results are shown in Table 3 . We find that intermediate text modeling in spoken dialog generation helps generate appropriate spoken responses. This confirms that the process of generating text before speech leverages the capabilities of the pretrained model effectively.As seen in Table 2 , the Cascaded model exhibits a lower ASR WER compared to USDM. This is due to the separate ASR model used in the Cascaded model, whisper-large-v3, which has been trained on approximately 5 million hours of speech data. However, in terms of the semantics of the spoken response, our model outperforms the Cascaded model.Similar to previous works that show the advantages of end-to-end pipelines over cascaded approaches in several tasks [83] [84] [85] [86] , USDM leverages input speech to generate more semantically coherent answers. To empirically verify that our generated text responses utilize information from both the preceding transcript and the input speech, we plot the attention maps for each layer, as illustrated in Figure 4 . We calculate the probability that each token in the generated response attends to each token in the input unit sequence and corresponding transcript by averaging the probabilities across all heads of the attention modules for each layer. Subsequently, we aggregate these probabilities for input unit tokens to compute a cumulative probability for the speech input, and similarly for text tokens relative to the transcript. As shown in Figure 4 , our generated tokens attend not only to the transcribed text but also to the speech input, indicating that our model benefits from the speech input.In Figure 4 , we also observe that the generated responses notably attend to the transcribed text. To analyze the impact of more accurate transcription on model performance, we substitute the modelgenerated input transcript with the ground truth transcript in the middle of the inference of USDM and measure the METEOR and ROUGE-L scores for the generated spoken responses. The measured scores are 13.6 and 16.2, respectively, surpassing our model's previous results of 13.1 and 15.7 in Table 2 . This confirms that enhancing the accuracy of unit-to-text conversion in USDM also improves the semantic coherence of the spoken responses.  
conclusion: 
 In this work, we presented USDM, a model synthesizing spoken dialog responses enriched with natural prosody. We proposed a novel speech-text pretraining scheme that models the comprehensive relationship between speech and text, which proves beneficial for spoken dialog modeling. Our approach is complemented by leveraging an acoustic unit tokenization scheme that preserves prosodic information, coupled with a supporting pair of an encoder and a decoder. We showed that USDM outperforms the baselines regarding content, prosody, and naturalness as a spoken response for the DailyTalk dataset. Additionally, we demonstrated that our pretraining and fine-tuning scheme benefits the USDM in modeling spoken dialog through ablation studies. Various samples for diverse scenarios in our demo page also showcased the capabilities of USDM. We believe that USDM has laid the groundwork for extending the conversational capabilities of LLMs to the voice domains. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report 95% confidence intervals for P-MOS and MOS, and p-values of the Wilcoxon signed-rank test for human evaluation and GPT-4-based test.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] , Justification: We discuss the positive and negative impacts of our work in Section A.2 Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not include codes and checkpoints of our model in this submission. We will consider safeguards when we release our models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We list the license of the datasets and the link of the existing checkpoints and libraries we used in Section A.4.1.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A.1 Audio SamplesWe have included various audio samples from our experiments on our demo page. 3 Furthermore, to demonstrate the applicability and potential of our model, we have added several samples of USDM fine-tuned on the Expresso dataset [87] , which contains emotionally rich spoken dialog data, and the Fisher dataset [88] , a telephony conversation dataset between two speakers recorded at 8,000Hz, totaling approximately 1,960 hours from 11,971 speakers.The Expresso dataset comprises 41 hours of emotionally expressive speech data from 4 speakers. Of this, 11 hours consist of simple reading styles, while 30 hours are improvised dialogs between two speakers. We use this dialog data to train our USDM, noting that these improvised dialogs lack corresponding transcripts. Therefore, we create transcripts using whisper-large-v3 and utilize these to train the USDM. We observe that the trained model often failed in the ASR and TTS parts due to numerous inaccuracies in the transcripts generated by the ASR model, thus not using this data in the main experiments of our paper. Instead, we provide selected samples from the model trained on this data on our demo page.In addition, we extend the spoken dialog templates used in training USDM to multi-turn scenarios to show the capability of USDM for modeling multi-turn dialogs. We explore this possibility by fine-tuning our speech-text model with Fisher [88] . We split the train and test sets with no overlapping speakers to show the possibility of unseen speakers' spoken dialog modeling. We have included samples of USDM's generated responses for multi-turn dialogs with unseen speakers of Fisher on our demo page. These samples demonstrate the potential of USDM for multi-turn spoken dialog modeling.This paper proposes a spoken dialog model designed to generate spoken responses to the speech inputs. Similar to the field of natural language processing, the dialog model might exhibit biases in its outputs, which stem from the training dataset. Such biases could unintentionally lead to the generation of synthesized voices that are biased. Furthermore, there are ethical considerations regarding the possible misuse of high-quality speech synthesis models, such as in voice phishing scams.Despite these concerns, research into spoken dialog models can yield several positive effects. Unlike text, voice interactions are capable of conveying non-verbal information, allowing us to build conversational agents that consider users' emotions, which are challenging to capture with text-based dialog models. Furthermore, by introducing spoken language as an additional means of communication, we offer an alternative to text-based chatbots for individuals facing difficulties in reading and writing. Similar to text-based chatbots, which have rapidly evolved and brought convenience to daily life, we expect that, with careful consideration of ethical issues, research and development in spoken dialog models will significantly benefit everyday life and a wide range of industries.In Section 3.1, we demonstrate through two experiments that the acoustic units extracted from XLS-R [64] contain paralinguistic features. The first experiment utilize CREMA-D [65] , to perform a unit-to-emotion recognition task, to check whether the unit sequence contains information beyond content. Additionally, we train a unit-to-speech reconstruction module and use it to compare the original audio with the audio reconstructed from the extracted units, showing that the unit contains information about pitch variations, as shown in Figure 2 and several samples on our demo page. Further details about the trained unit-to-speech reconstruction module are provided in Section A.3.3.For emotion recognition, we train a 3-layer transformer-based emotion classifier using cross-entropy loss. We utilize the CREMA-D dataset, consisting of 7,442 audios from 91 actors, categorized into six emotions: Anger, Disgust, Fear, Happy, Neutral, and Sad. We split the data into training, validation,Below is a conversation between the user and the agent. Each turn includes the user's text, along with the agent's response text.Below is a conversation between the user and the agent. Each turn includes the user's speech and its corresponding transcript, along with the agent's response text and the corresponding speech ### User speech token 
 text token ### Agent text token 
 speech token USDM - XLS-R Mistral-7B unit-Voicebox + BigVGAN - From Scratch - XLS-R Mistral-7B unit-Voicebox + BigVGAN - SpeechGPT - mHuBERT Llama-7B unit-HiFi-GAN - Cascaded whisper-large-v3 - Mistral-7B - Voicebox + BigVGANand testing sets in a ratio of 70%, 15%, and 15%, respectively, ensuring an equal number of samples for each emotion in both the validation and test sets.Figure 5 illustrates the template of single-turn spoken dialog and text dialog we used for fine-tuning. We perform loss calculations only on the highlighted part. USDM refers to our model, and From Scratch and Cascaded are the baselines we used as described in Section 4.1.1.Voicebox [66] is a Flow-Matching-based zero-shot TTS model [89] that generates a mel-spectrogram from input text and reference speech. During training, Le et al. [66] utilize the Montreal Forced Aligner (MFA) [68] to extract the alignment between the phoneme sequence and the mel-spectrogram. This alignment is utilized to calculate the duration for each phoneme, allowing Le et al. [66] to expand the phoneme sequence to match the length of the mel-spectrogram. Voicebox is trained to produce a mel-spectrogram from an expanded phoneme sequence of equivalent length. Le et al. [66] additionally train a duration predictor to predict the duration of each phoneme in the sequence, which is used for inference.We train three variants of the Voicebox for different purposes, all using the English subset of the Multilingual LibriSpeech [71] and GigaSpeech [72] , totaling 54k hours of ASR data. We perform inference using a total of 50 timesteps with a classifier-free guidance scale of 1. We train the model to produce mel-spectrograms of approximately 86Hz, with the same configuration as the official 22,050Hz checkpoint of BigVGAN [74] . The output mel-spectrograms of all Voicebox models are converted into 22,050Hz speech through BigVGAN. This section will detail each of these 3 cases.Variant for Analyzing Non-Verbal Cues in Acoustic Unit As described in Section 3.1, we train a unit-to-speech reconstruction model to investigate the non-verbal information contained within the unit. This model differs from Voicebox in that it utilizes unit sequence instead of text input and does not use reference speech during training and inference. We do not train a separate duration predictor; instead, we upsample the 50Hz unit sequence to 86Hz to match the length of the mel-spectrogram. Note that this variant is used only for analyzing non-verbal information.Below is a dialog between Speaker 1 and Speaker 2, and two possible responses from Speaker 2. Choose more contextually appropriate response between them. {previous dialog} ### Output A: {response_1} ### Output B: {response_2} Which one is better, Output A or Output B? [Additional Consideration] -Give a penalty to unnecessary repetitions.-There would be cases that reach maximum sequence length. Do not deduct score for these cases.-Give a penalty for unengaging and simplistic response.-Only write a single char as your answer, 'A' for Output A or 'B' for Output B.-Do not add any explanation. Decision: As mentioned in Section 4.1.2, we conduct various human evaluations and GPT-4-based assessments [1] . First, we employ Amazon Mechanical Turk to perform human preference tests and evaluate prosody and naturalness through P-MOS and MOS. In the human preference test, as detailed in the main paper, we present evaluators with previous spoken dialog and spoken input along with two candidate spoken responses. Evaluators are asked to choose which response is more appropriate, considering comprehensive aspects such as content, prosody, and sound quality. We provided evaluators with the instruction, "Given the spoken dialog of two speakers, which response is more suitable? Please consider comprehensive aspects such as content, speech quality, and prosody." All comparative experiments are evaluated by 150 evaluators, respectively, and the total cost for these evaluations is approximately $200.We also conduct a qualitative evaluation by measuring the 5-scale mean opinion score (MOS) and prosody mean opinion score (P-MOS), both ranging from 1 to 5 points. For the P-MOS, we provide evaluators with the input spoken dialog and the corresponding ground truth text response. They are then asked to listen to the speech matching the ground truth text response and evaluate the prosody, considering both the spoken dialog and the response text. Additionally, we measure the MOS to judge audio quality and naturalness. In this scenario, evaluators are given only the response text and its corresponding spoken response without any preceding spoken dialog and are asked to rate the audio quality and naturalness. The instructions provided for the P-MOS and MOS tests are: "How natural is the prosody in this recording? Please focus on the prosody in the context of the spoken conversation flow and the given text response, and ignore other aspects such as speaker ID and sound quality." and "How natural (i.e., human-sounding) is this recording? Please focus on the audio quality and the naturalness of pronunciation.", respectively. 198 evaluators participate in the P-MOS measurement, and another 176 participate in the MOS measurement. For all 5-scale evaluations, we provide examples of speech rated as 1, 3, and 5 points as a reference to guide evaluators. We spend a total of approximately $250 on these evaluations.To evaluate the semantic quality of the audio generated by each model, we utilize transcripts obtained by passing the generated audio through a separate ASR model, whisper-large-v3. These transcripts are then evaluated using GPT-4 [1] . As introduced by Zheng et al. [81] , we provide the GPT-4 model with evaluation instructions, previous dialog, and two candidate responses each from USDM and a baseline model for comparison, asking it to choose the preferred response. The template used for this evaluation is shown in Figure 6 . Among all available APIs, we use gpt-4-0125-preview for evaluation. To avoid bias due to the order of response candidates, we assess the responses from the two models in both their original and reversed orders. Preference is primarily judged based on content appropriateness, but penalties are assigned for unengaging responses such as simple short answers and fillers. If the results differ between the two evaluations, it is marked as a Tie, if both prefer Output A, then A, and if both prefer Output B, then B, with results in Table 2 .All p-values in the main text are obtained through the Wilcoxon signed-rank test. For the preference test, a score of 1 is assigned if a model is preferred, 0 for a tie, and -1 if not chosen. We then conduct a test using these scores and report the respective p-values. Table 8 : Six types of speech-text interleaved sequences used to evaluate the performance of the pretrained model, along with the templates used for measuring PPL. For sequences with a continuation relationship, the speech and text data are split in half, combining one modality from the first half (e.g., speech1 token or text1 token) with the remaining modality from the second half (e.g., text2 token or speech2 token).Unconditional Text text tokenCorrespondence -Unit-to-Text speech token 
 text token In Section 4.1.2, we assess the semantic quality by using the transcribed text of the spoken response generated by each model, using the whisper-large-v3, and measure METEOR and ROUGE-L scores.Our models and baselines first generate a text response either within the end-to-end pipeline or through a separate model. We also measure METEOR and ROUGE-L scores for these intermediate text responses on the test sets of DailyTalk, and the results are presented in Table 7 .Errors occur during the generation of the spoken response using intermediate text and the transcription of that spoken response for evaluation, leading to a performance gap between the results measured using the intermediate text response and the transcript of the spoken response. Despite the error gap, we confirm that USDM outperforms baselines in terms of the semantics of the intermediate response.Notably, a higher TTS WER increases the gap between the results based on the intermediate text response and the semantic performance of the final spoken response.In this section, we provide further explanation of the ablation studies on the pretraining schemes discussed in Section 4.2.1. We design interleaved sequences excluding each key relationship, continuation and correspondence, to demonstrate the necessity of each relationship within our proposed speech-text pretraining scheme. Additionally, we follow the cross-modal pretraining scheme proposed in Spectron [57] , which we name Setup 3.We create interleaved speech-text sequences composed solely of continuation relationships and use these for speech-text pretraining. The interleaved sequences used for Setup 1 can be obtained by skipping the last step of the 3-step data preparation process described in Section 3.2. This approach is similar to previous works such as BLSP [44] and SpiRit-LM [56] . Setup 2 We construct cross-modal sequences exclusively with correspondence relationships. The interleaved sequences with this relationship are typically formatted as "speech token 
 text token" and "text token 
 speech token" sequences, similar to SpeechGPT [25] .Setup 3 We also compare a scheme that utilizes one fixed template for pretraining. Following Spectron [57] we pretrain using an interleaved sequence where the input speech is transcribed into text, followed by predicting the subsequent response text and synthesizing the corresponding speech. Assuming the speech and the corresponding text are split into two parts (speech1, text1, speech2, text2), we perform cross-modal pretraining using the interleaved sequence "speech1 token 
 text token 
 speech2 token", where the text token is obtained by concatenating text1 and text2.Each model is trained with the same data and hyperparameters as our pretraining. All models have the same vocabulary size. We measure the PPL of various combinations of speech-text sequences created using the LibriSpeech test-clean subset. We create 6 types of interleaved sequences and the templates of these sequences are listed in Table 8 . For measuring the PPL of text tokens, we normalize the probability by excluding the probability of units and use this normalized probability to measure the PPL. Similarly, for speech modality PPL, we compute the logits and probabilities for only the unit tokens, which have a vocabulary size of 10,000, and then calculate the PPL. To evaluate pretraining schemes where only one of the special tokens 
 or 
 is used, we insert the special token used during pretraining at the boundary between the two modalities, regardless of the combination being evaluated.The PPL for each combination is listed in Table 8 . Setup 1 and Setup 2, which model only one relationship, fail to model the other and exhibit high PPL values. Additionally, Setup 3, which uses a specific fixed template of interleaved sequences for pretraining, shows superior performance in interleaved sequences with a correspondence relationship but is unable to model a continuation relationship. In contrast, our speech-text model, which universally models various relationships, demonstrates consistently powerful performance regardless of the sequence type.Our model adopts an end-to-end pipeline with intermediate text, where the input speech is first transcribed, followed by generating the response text. Consequently, the model simultaneously learns unit-to-text, text response generation, and text-to-unit, with each task potentially reaching its optimal point at different epochs. To observe the training dynamics of each task, we train the USDM on the DailyTalk dataset for 5 epochs and monitor TTS WER, STT WER, METEOR, and ROUGE-L at each epoch. As shown on the left side of Figure 7 , while STT WER remains consistent over each epoch, TTS WER, METEOR, and ROUGE-L scores improve, suggesting that the dialog modeling task and text-to-unit tasks are more challenging compared to the unit-to-text task.Additionally, we build two spoken dialog models using Low-Rank Adaptor (LoRA) [90] for finetuning the pretrained speech-text model on the DailyTalk dataset, and observe similar tendencies. We fine-tune the model using a higher learning rate of 10 -4 , comparable to USDM, with LoRA ranks of 8 and 256. Consistently, as the right side of Figure 7 illustrates, increasing the number of fine-tuning parameters improves semantic performance in dialog and unit synthesis performance but deteriorates the performance in the unit-to-text task. Considering these observations and the analysis in Section 4.3 that demonstrates the importance of unit-to-text performance in USDM, we plan to explore strategies to mitigate overfitting in the unit-to-text task by varying the loss weight for each task within the pipeline or by applying the curriculum learning approach in future work.We plot the attention maps of the generated text response to the speech input and its model-generated transcript for 6 selected layers in Figure 4 . Given that our spoken dialog model consists of 32 layers, we additionally include attention maps of another sample for all layers in Figure 8 . Justification: We write the abstract and introduction reflecting our paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Demo: https://unifiedsdm.github.io/Despite these advantages, our model has several limitations and areas for improvement. Firstly, the exploration of datasets and models used for pretraining is limited. Further investigation is necessary to determine which data are crucial for our pretraining scheme and to explore whether our pretraining scheme could be effective with other LLMs beyond Mistral-7B. Secondly, building a spoken dialog model capable of directly generating spoken responses from input spoken dialog without the need for cross-modal chaining can be a promising direction. Next, the current pretraining scheme is based on tens of thousands of hours of English data, and it has limitations when applied to other languages with relatively smaller amounts of speech data compared to English. We plan to expand our model to a variety of languages in addition to English. Lastly, we also plan to investigate whether our pretraining approach is beneficial for other speech-text tasks beyond spoken dialog modeling.Our demo is available at https://unifiedsdm.github.io/. 


Decision: This work builds a speech-text LLM focused on speech-in/out generation of spoken dialogue turns. Due to its use of speech units with validated paralinguistic attributes, the model exhibits subjective prosodic appropriateness similar to the ground truth. Via a suite of cross-modal correspondence and continuation tasks and expanded training data, the model also performs well on response semantic appropriateness versus past works’ training schemes. I and the reviewers find the system a solid step towards speech-interactive LLMs. Given the nascence of the task, minor questions around eval fairness and granularity arose but were addressed with finer evaluations or already-existing ablations. **I recommend Acceptance**.

As both reviewers and authors acknowledge, the paralinguistic modeling is more incidental / coarsely evaluated (though the rebuttal to R2 improves this, as well as examples of emotional TTS/control if provided as R1, R3 request). **I suggest the authors change the title** to emphasize that paralinguistic coherence is more emergent than deliberate (e.g., “Integrating Paralinguistics in” —> “Paralinguistics-Aware”), and optionally to better reflect other aspects like the pre-training objective or the focus on multi-turn/holistic dialogue evals versus prior work.

Reviews: review: paper_summary: paper_summary: This paper introduces an extensive speech-text LLM framework, the Unified Spoken Dialog Model (USDM), designed to generate coherent spoken responses with naturally occurring prosodic features relevant to the given input speech without relying on explicit automatic speech recognition (ASR) or text-to-speech (TTS) systems.
strength_weakness: strength_weakness: 1) Unfair comparison. Given that this paper focuses on paralinguistics, based on the findings in section 3.1, it is natural that SpeechGPT's 1k acoustic tokens would perform worse than USDM. 

2) With such a reasonable pre-training process, it is regrettable that this work has not managed to extend the speech-text associations learned by the model to different tasks (such as [1]). The paralinguistics is important for human speech understanding beyond spoken dialog. 

3) Some listening examples of zero-shot TTS (especially emotional TTS) are recommended to demonstrate that USDM can better mimic the emotion and prosody information from reference speech.

[1] Nguyen, Tu Anh, et al. "Spirit-lm: Interleaved spoken and written language model." arXiv preprint arXiv:2402.05755 (2024).
questions: questions: 1) How is the instruction-following capacity of USDM? Since this issue is unrelated to paralinguistics, it's ok to skip this question.

2) Is it possible for USDM to perform speech-to-speech directly with more data or training schedule? Such modality changing results in low efficiency. If can, which kind of strategy is required can be based on USDM?

3) Do authors have open-source plan for pre-trained model? I would like to increase the rating since such an interleaved speech-text foundation model can benefit to both speech and multimodal communities.
limitations: limitations: See weakness
paper_summary: paper_summary: This paper introduces a method for modeling spoken dialog which relies on a speech-text LLM, pretrained with a combination or text and discrete speech tokens which capture semantics as well as prosody. The pretraining regime attempts to get the LLM to capture two types of relations between text and speech tokens: continuation and correspondence.
The approach is evaluated on the DailyTalk dataset using both automatic metrics and human preference judgements, and is shown to outperform a cascaded approach relying on ASR and TTS, as well as other baselines.
strength_weakness: strength_weakness: The main weakness that the method relies on a massive amount (~10 years) of transcribed English speech. This makes it limited in its applicability to just a handful of very resource rich languages.

The evaluation relies on a single dataset in a single language. 

The title places the focus of the work on integrating paralinguistic information, but in the actual paper this is only one, and not the most salient, part of the framework. The prosody is aspect is evaluated via the P-MOS score, but not analyzed in depth.
questions: questions: Do you have any insight into which specific aspects of prosody this approach captures, as compared to the baselines?
limitations: limitations: Several limitations are clearly articulated. The reliance on large amounts of transcribed speech for training is not, however.
paper_summary: paper_summary: The authors present a new large language model (LLM) framework called the Unified Spoken Dialog Model (USDM) that can directly understand and generate spoken dialog responses with natural prosody. This is achieved by incorporating prosodic information into the speech tokens and using a multi-step spoken dialog template for fine-tuning. Both automatic and human evaluations on the DailyTalk dataset show that USDM outperforms previous models in generating natural-sounding spoken responses.
strength_weakness: strength_weakness: Comparative Analysis: Extend the comparative analysis to include a wider range of previous methods in addition to SpeechGPT. While SpeechGPT is a strong benchmark, a more comprehensive comparison with other relevant approaches would provide a clearer understanding of the USDM's relative strengths and weaknesses.

Dataset Diversity: Expand the evaluation to include multiple datasets beyond DailyTalk. Evaluating the model on a diverse range of datasets would offer a more robust assessment of its generalization capabilities and performance across different scenarios.

Emotional Control: Explore and discuss the potential for controlling the emotional expression of the generated responses in the proposed USDM model. Addressing this aspect would provide insights into the model's ability to adapt to different emotional contexts and potentially open avenues for future research in this direction.
questions: questions: --
limitations: limitations: --

Comments: title: Official Comment by Reviewer yvFi (made visible to authors)
Improve My Rating
comment: I'd like to acknowledge the authors' response and thank them for providing the additional analysis. As to my assessment ultimately it remains unchanged as it was largely positive already.
"Dear Authors,

Thank you for addressing my comments and providing detailed responses to each of the points raised. I appreciate the effort you’ve put into refining your paper. Below are my thoughts on the updated content:

**[W1] Comparative Analysis:**

The addition of AnyGPT [1] to your comparative analysis is a valuable improvement.

**[W2] Dataset Diversity:**

Your explanation regarding the challenges of incorporating datasets like Fisher and the choice to focus on the DailyTalk dataset is understandable. The inclusion of the MultiDialog [2] dataset in future evaluations is a commendable step towards assessing your model’s generalizability and performance across diverse scenarios. I look forward to seeing the results from these additional datasets as they become available.

**[W3] Emotional Control:**

The modifications made to enable emotion control in the USDM model are promising.

Overall, your responses have addressed the concerns raised comprehensively."
I appreciate the author's response, which has alleviated most of my concerns. Therefore, I decide to raise my rating to 6 as I mentioned.