Venue: NAACL 2025

# Behavior-SD: Behaviorally Aware Spoken Dialogue Generation

with Large Language Models

Sehun Lee Kangwook Kim Gunhee Kim

Seoul National University

shlee@vision.snu.ac.kr full324@snu.ac.kr gunhee@snu.ac.kr [https://yhytotol2.github.io/Behavior-SD](https://yhytotol2.github.io/Behavior-SD)

Equal contribution.

###### Abstract

Spoken dialogue involves behaviors like turn-taking, interruptions, filler words, and backchannels, which make interactions more natural and engaging but are often overlooked in language models. These models struggle to explicitly model these behavioral traits, resulting in a less natural and personalized communication style that aligns with user needs. To address this challenge, we make two key contributions. First, we introduce **Behavior-SD**, a large-scale dataset containing over 100K spoken dialogues (2,164 hours) annotated with various conversational behaviors, synthesized via LLMs to model diverse full-duplex interactions. Second, we propose **BeDLM**, the first dialogue model capable of generating natural conversations conditioned on specific behavioral and narrative contexts, supporting simultaneous contributions from both speakers. Through human evaluations and behavior-adherence metrics, we demonstrate that BeDLM outperforms baseline models in generating natural, coherent, and behaviorally rich dialogues. Our work opens new possibilities for developing behaviorally-aware dialogue systems that more closely mimic human conversational dynamics, enhancing user engagement and communication effectiveness.

## 1 Introduction

Large language models (LLMs) have significantly improved their ability to generate coherent text; however, producing human-like spoken dialogues remains a considerable challenge. Spoken conversations are inherently dynamic, involving elements like prosody, emotion Osman (2022), turn-taking, and interruptions Nguyen et al. (2023). Capturing these nuances is crucial for generating life-like dialogue, yet current TTS systems Guo et al. (2023); Du et al. (2024) and spoken language models Zhang et al. (2023); Li et al. (2024) struggle to replicate this complexity fully.

A particularly challenging aspect of real-world conversations is managing _full-duplex_ communication, where speakers frequently overlap or interrupt each other. Full-duplex conversations, unlike traditional systems that enforce strict turn-taking, enable more fluid interactions, which can be beneficial in applications such as customer service, education, or voice assistant applications Si et al. (2023); Google (2024); OpenAI (2024). Additionally, conversational dynamics are influenced by personal traits, social contexts, and factors like politeness Bevacqua et al. (2012); Yamamoto et al. (2018), making this task even more complex.

Despite the prevalence of these dynamics in human conversations, they are often underexplored in dialogue generation. While existing spoken dialogue datasets Cieri et al. (2004); Godfrey et al. (1992); Reece et al. (2023) include conversational behaviors such as interruptions and backchannels, these behaviors are not labeled at the speaker level, limiting the ability to model nuanced interaction patterns. Recent datasets Lee et al. (2023); Lin et al. (2024) focus on sequential turn-taking, overlooking critical aspects of natural conversations like overlapping speech and interruptions.

To address these gaps, we present **Behavior-SD**, a large-scale dataset of over 100K spoken dialogues (SD), totaling 2,164 hours, and covering a wide range of conversational behaviors and social situations. Unlike existing datasets constrained by narrow contexts Cieri et al. (2004); Godfrey et al. (1992); Reece et al. (2023), Behavior-SD explicitly models key conversational traits such as verbosity, backchannels, and interruptions, allowing for more realistic modeling of human-like dialogues. This dataset serves as a foundation for improving both the scale and behavioral diversity of dialogue generation. Specifically, it captures four key speaker behavior traits, as outlined in Table 1.



Moreover, we introduce **Be**havior-conditioned spoken **D**ialogue **L**anguage **M**odel (**BeDLM**), the first model capable of generating full-duplex spoken dialogues by incorporating behavior traits. By integrating these traits through control tokens, BeDLM produces more dynamic and lifelike conversations, adapting naturally to speaker personalities and narrative context. This novel approach enables customization of dialogue behaviors in a way that previous models, which typically focus on static turn-taking, cannot achieve.

In our experiments, BeDLM demonstrates improvements in dialogue naturalness and adherence to conditions compared to state-of-the-art models. The dataset, model, and code will be made publicly available to encourage further research in this area.

## 2 Related Works

Automatic spoken dialogue generation.The automatic synthesis of dialogues using LLMs has made significant strides, but generating spoken dialogue remains challenging due to features like paralinguistic cues, interruptions, and unclear turn boundaries in speech Abdullin et al. (2023); Kim et al. (2023); Lin et al. (2024). Approaches such as dGSLM Nguyen et al. (2023) have tackled overlapping speech generation using dual-lower transformers without text inputs, while CHATS Mitsui et al. (2023) builds on this by generating natural spoken dialogues from text transcriptions. However, these methods lack mechanisms to condition overall conversational behavior throughout a dialogue.

Conditional speech generation.Recent models like Prompt TTS Guo et al. (2023), Audiobox Vyas et al. (2023), and CosyVoice Du et al. (2024) have advanced text-to-speech (TTS) by incorporating natural language instructions to vary prosodic features like accent, emotion, pitch, and speed. However, these models focus on isolated utterances and lack the ability to maintain consistency across full conversations, including conversational behaviors such as turn-taking and backchanneling. As a result, they struggle to replicate natural dialogue flow.

Spoken dialogue behaviors.Research has shown that conversational behaviors like backchannels and interruptions play a crucial role in communication, signaling listener engagement and influencing dialogue flow Reece et al. (2023); Bavelas et al. (2000). These behaviors vary based on context and individual traits Blomsma et al. (2024). Traditional dialogue systems have attempted to account for such nuances by manually inserting fillers or backchannels, but they lack the flexibility to adapt dynamically to the conversation De Sevin et al. (2010). Recent efforts focus on equipping LLMs to handle interruptions more naturally, enhancing interaction fluidity Ma et al. (2024); Zhang et al. (2024); Wang et al. (2024), highlighting the importance of integrating these behaviors for more natural dialogue.

Spoken language models.Speak language models (SLMs) have evolved from textless approaches like GSLM Lakhotia et al. (2021), which convert speech into discrete units and training similarly to text-based language models. Recent advancements enhance SLMs by directly integrating speech representations into pre-trained LLMs. For instance, SpeechGPT Zhang et al. (2023) incorporates speech tokens into LLMs through instruction tuning, enabling more natural and context-aware audio interactions. AudioGPT Huang et al. (2024), LLaMA-Omni Fang et al. (2024), and QwenAudio Chu et al. (2023) integrate multimodal inputs, including text and various audio types, to support multimodal interactions without relying on speech transcription.

## 3 Behavior-driven Data Generation

Our framework for dataset generation operates in four stages: (1) generating dialogues with behavior-specific traits, (2) inserting backchannels at contextually appropriate moments, (3) captioning the speech style of each utterance for TTS integration,

\begin{table}
\begin{tabular}{c l p{286.9pt}} \hline \multicolumn{2}{l}{Behavior types} & Definition \\ \hline (V) & Verbosity & The length of utterances during a speaker’s turn. \\ (F) & Filler words & The occurrence of filler words (_e.g._, “like, ”um”, “you know”) during a speaker’s turn. \\ (B) & Backchannels & The frequency of backchannel responses (_e.g._, "yeah, ”uh-huh”) during the other speaker’s turn. \\ (I) & Interruptions & The frequency of interrupting the other speaker’s turn to begin speaking. \\ \hline \end{tabular}
\end{table}
Table 1: Definition of behavior types for full-duplex interactions in conversational dialogues. Each behavior type is classified into three levels: none (0), moderate (1), and frequent (2).

 

and (4) converting text dialogues into spoken form. Figure 1 overviews the framework. All detailed prompts for LLMs can be found in Appendix A.

### Rich Text Dialogue Generation via LLMs

**Sample narrative and conversational behaviors.** To generate diverse social spoken dialogues, we sample two key elements: narratives and conversational behaviors. We sample narratives from the SODA dataset Kim et al. (2023), since it covers a broad spectrum of social interactions. Conversational behaviors include four traits, each categorized into three levels (0, 1, 2), as shown in Table 1. For each conversation, two speakers are sampled with separate behavioral traits to ensure a variety of interaction styles. We further categorize interruptions into seven types: agreement, disagreement, floor taking, tangentialization, clarification, assistance, and topic change, as follows Goldberg (1990). Based on the sampled traits of speakers, specific categories of interruptions are applied, ensuring realistic conversational dynamics.

Dialogue generation via LLMs.With narrative and speaker-specific conversational behaviors, we guide GPT-4o to generate complete multi-turn spoken conversations of 8-12 turns. To enhance the naturalness of the dialogue, we enable the LLM to depict _laughter_, using notations such as [laughter] or <laughter>yeah<laughter/> (indicating a laugh while saying _"yeah"_) to represent it. If an interruption occurs during the conversation, the corresponding utterance is marked as Interrupt for the speech synthesizing process.

Backchannel insertion.Backchannels typically occur at natural pause points in dialogue, often referred to as backchannel opportunity points (BOPs). Previous approaches Kawahara (2019) manually insert backchannels from a predefined set of responses (_e.g_, "hmm," "yeah") with no consideration of conversational context. In contrast, we use LLMs to automatically detect BOPs and generate diverse contextually appropriate backchannels based on predefined conversational behaviors, as illustrated in Figure 1.

Specifically, we first instruct LLMs to segment utterances at natural pause points to identify BOPs. Next, we select a subset of these BOPs based on the speaker's backchannel behavior level. At these BOPs, we insert "speaker (backchannel): [MASK]", allowing LLMs to generate contextually appropriate backchannel responses by filling in the [MASK]. We use GPT-4o-mini for BOP detection and GPT-4o for generating backchannel responses.

Speech style captioning for TTS.We use GPT-4o-mini to generate speaking styles, including pitch, speaking rate, and emotion for each sentence. The generated styles are then input into the conditional TTS model, as described in SS3.2.

### Spoken Dialogue Generation from Text

Conditional TTS model.To convert rich text dialogues into spoken form, we utilize a CosyVoice-Instruct Du et al. (2024), conditional text-to-speech (TTS) model that incorporates style att

Figure 1: Overall data generation pipeline. (a) Narrative and conversational behaviors are sampled. The conversation is then simulated using LLMs, incorporating backchannel opportunity points (BOPs) detection, backchannel insertion, and speech style captioning to model diverse full-duplex interactions (§§3.1). (b) The generated text dialogues are converted into spoken form by CosyVoice and timing-driven utterance placement (§3.2).

 

tributes specific to each sentence. We randomly sample 26 female and 26 male speakers. These speakers are then stored in a speaker bank, from which we randomly sample to construct the dataset. Further details about the speaker bank are provided in Appendix B.1. The final output is rendered as two-channel audio, where each channel corresponds to a single speaker in the dialogue.

Consistent utterance continuation.Each utterance is synthesized independently, so maintaining speaker consistency and smooth transitions is critical. To achieve this, we provide the previous utterance of the current speaker as additional input to the TTS model. CosyVoice-Base supports speech prompting, where generation starts from a provided speech token. To ensure continuity, we augment the input with the current utterance's text \(\tau_{i}\), style instruction \(e_{i}\), the speaker's initial text and speech \((\tau_{0},s_{0})\), and the previous utterance's text and speech \((\tau_{i-1},s_{i-1})\), forming the sequence: \([e_{i},\tau_{0},\tau_{i-1},\tau_{i},s_{0},s_{i-1},<\texttt{EOP>}]\). This structure enables temporal coherence and speaker consistency. Importantly, the input does not include the previous utterance of the other speaker. CosyVoice-Instruct, however, uses only style instructions without prompting. Our approach combines both, improving consistency and coherence across utterances.

Timing-driven utterance placement.The previously generated utterances are categorized as Backchannel, Interrupt, or None. When merging them, their timing is adjusted to mimic the natural flow of conversation. These placements are based on Gaussian distributions derived from statistics in a real-world spoken dialogue corpus Reece et al. (2023). Specifically, none-type utterances are appended with an inter-utterance gap sampled from \(\mathcal{N}(0.4s,0.2s)\), backchannels follow after a brief delay sampled from \(\mathcal{N}(0.2s,0.02s)\), and interrupts overlap the preceding speech by a duration drawn from \(\mathcal{N}(0.45s,0.05s)\).

## 4 The Behavior-SD Dataset

Using the pipeline in the previous section, we contribute Behavior-SD (Behavior-driven Spoken Dialogues) as a large-scale, full-duplex dataset that captures a diverse range of social interactions and conversational behaviors. A sample from our dataset is shown in Figure 3.

Table 2 presents the statistics and human evaluation results of various spoken dialogue datasets. We compare our Behavior-SD with several existing datasets: human-recorded conversations from CANDOR Reece et al. (2023), Switchboard Godfrey et al. (1992), Fisher Cieri et al. (2004), and MELD Poria et al. (2019), human-read text conversations from DailyTalk Lee et al. (2023), and text dialogues converted to speech using TTS from

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline

- \\ Switchboard & ✓ & & & recorded & 2.400 & 260 &
- \\ CANDOR & ✓ & & ✓ & recorded & 1,650 & 850 & 3.73 ± 0.08 & 3.58 ± 0.07 & 3.54 ± 0.07 \\ MEI.D & ✓ & & ✓ & recorded & 1,400 & 12 & 3.79 ± 0.07 & 3.68 ± 0.07 & 3.61 ± 0.08 \\ DailyTalk & & & ✓ & recorded & 2.541 & 20 & 3.89 ± 0.07 & 3.77 ± 0.07 & 3.82 ± 0.07 \\ Style/Talk & & & ✓ & TTS-converted & 2,364 & 7 & 3.88 ± 0.07 & 3.77 ± 0.07 & 3.78 ± 0.07 \\ \hline
**Behavior-SD** & ✓ & ✓ & ✓ & **TTS-converted** & **108,174** & **2,164** & **3.94 ± 0.07** & **3.78 ± 0.07** & **3.87 ± 0.06** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics and human evaluation results of various spoken dialogue datasets. Citations are provided in the main text due to space constraints (§4).

Figure 2: Overall statistics and distribution analysis of the dataset.

 

StyleTalk Lin et al. (2024). We run human evaluation on Amazon Mechanical Turk (MTurk), and ask participants to rate each 30-second audio samples on three aspects: dialogue naturalness, emotion appropriateness, and sound quality, using a 1-5 scale. Dialogue naturalness measures whether the conversation flow feels like a real-life interaction. Emotion appropriateness assesses whether the emotions in the dialogue align with the context. Sound quality evaluates how clear the audio is and the extent to which it is free of noise.

As shown in Table 2, Behavior-SD is the only publicly available dataset that includes speaker behavior labels, setting it apart from other datasets. Moreover, Behavior-SD contains over 100K dialogues and 2K hours of audio, far surpassing the sizes of the other datasets. The human evaluation demonstrates that Behavior-SD is preferred to other datasets in all three aspects. These findings indicate that Behavior-SD, with its larger scale and the inclusion of behavior labels, offers a distinct advantage over existing datasets. Moreover, the higher human evaluation scores of Behavior-SD in naturalness, emotion appropriateness, and sound quality further demonstrate its suitability for the research of behavior-rich spoken dialogues.

Figure 2 presents the detailed statistics of our dataset, showing the distribution of key conversational features across different behavior levels. It illustrates how varying behavior categories impact the length and structure of conversations. For example, the duration of utterances differs notably between behavior levels, indicating different conversational patterns. Additionally, features such as filler words, backchannels, and interruptions show distinct distribution trends across behavior levels.

## 5 BeDLM: Behavior-conditioned Dialogue Language Models

This section introduces BeDLM, a spoken dialogue language model designed to generate simultaneous two-channel speech conditioned on conversational behaviors and a narrative. Our BeDLM addresses several key challenges: (1) enabling LLMs to comprehend spoken dialogue forms beyond mere textual representations and (2) equipping LLMs to interpret conversational behaviors and effectively incorporate them into dialogue generation.

To leverage the rich capabilities of LLMs, we utilize a pre-trained LLM instead of training a transformer-based language model from scratch. This allows us to benefit that LLMs already has extensive knowledge of language patterns and dialogue structures, thereby allowing BeDLM to generate more nuanced and contextually appropriate outputs. In our work, we use Llama3.2-1B Dubey et al. (2024); Meta (2024) as a base model.

### Speech Representations

Following previous approaches on SLMs Zhang et al. (2023); Hassid et al. (2024); Nguyen et al. (2023), we utilize HuBERT Hsu et al. (2021) to discretize speech signals into 500 unique units, which serve as a compact representation of speech. These units are incorporated into the LLM as additional tokens in its vocabulary, labeled \(\langle{\rm S}0\rangle\), \(\langle{\rm S}1\rangle\), ..., \(\langle{\rm S}499\rangle\). To reconstruct audio from these discrete speech representations, we employ a two-stage approach: (1) HuBERT2Mel, which converts HuaBERT representations into mel spectrograms, and (2) HiFi-GAN Kong et al. (2020), which synthesizes raw waveforms from the generated mel spectrograms.

### Streamlined Spoken Dialogue Representations

To model full-duplex spoken dialogue, handling gaps and overlapping speech (_e.g_, interruptions, backchannels) is crucial. Although dGSLM employs a dual-tower model to predict two channels

Figure 3: A rich text dialogue sample of Behavior-SD.

 

simultaneously, this approach inherently restricts the model's ability to fully leverage the power of LLMs. Furthermore, alternating predictions between channels (_e.g_, \([A][B]|[A][B]|[A][B]\)) is inefficient, as it doubles the number of tokens required to represent the same duration of speech.

We introduce a novel duplex dialogue representation that uses control tokens to mark overlapping speech and backchannels. The tokens \(\langle{\rm A}\rangle\) and \(\langle{\rm B}\rangle\) indicate the beginning of each speaker's turn. \(\langle{\rm GAP}\rangle\) and \(\langle{\rm OVERLAP}\rangle\) represent silences and overlaps, with each token covering 40ms. For instance, the sequence \(\langle{\rm GAP}\rangle\langle{\rm GAP}\rangle\langle{\rm GAP}\rangle\langle{ \rm A}\rangle\) denotes a 120ms silence before Speaker A begins, whereas \(\langle{\rm OVERLAP}\rangle\langle{\rm OVERLAP}\rangle\langle{\rm B}\rangle\) indicates an 80ms period of overlapping speech during Speaker B's turn. Backchannels are delineated by enclosing them between the \(\langle{\rm BC\_S}\rangle\) and \(\langle{\rm BC\_E}\rangle\) tokens, enabling precise control over conversational dynamics. More details are in Figure 6.

This method efficiently models two-channel dialogue into single-channel stream while preserving LLM performance, converting the tokenized sequences into two-channel audio via a vocoder for natural speech generation.

### Conditional Spoken Dialogue Generation

To enable LLMs to generate spoken dialogues that align with a given narrative and various conversational behaviors, we train them using our Behavior-SD dataset. However, generating spoken dialogues directly from a narrative is challenging, so we first conduct pre-training by generating text-based dialogues from the conditions. This process is done through supervised fine-tuning using the prompt specified in Figure 7.

Next, as illustrated in Figure 4, we fine-tune the LLM to directly generate streamlined spoken dialogue representations when provided with a prompt that includes the narrative and behavior conditions.

## 6 Experiments

In this section, we provide the details of model training and baselines for spoken dialogue generation.

### Model Training

Finetuning HuBERT and HiFi-GAN.To enhance the expressiveness of BeDLM, we fine-tune the HuBERT Hsu et al. (2021), HuBERT2Mel, and HiFi-GAN Kong et al. (2020) models using the Behavior-SD corpus. Specifically, the k-means quantizer of the HuBERT model is adapted to capture key acoustic features. The HuBERT2Mel module is then trained to convert the HuBERT representations into mel spectrograms. Finally, HiFi-GAN is fine-tuned to synthesize raw waveforms from the generated mel spectrograms. Training details for each module are presented in Appendix C.

BeDLM training.We first create a supervised-finetuning dataset from our Behavior-SD dataset

Figure 4: Overview of the proposed conditional spoken-dialogue generation pipeline. Narrative and behavior conditions serve as prompts for BeDLM to generate streamlined units, which are then converted into a two-channel unit sequence using additional control tokens (_e.g_, \(\langle{\rm A}\rangle\), \(\langle{\rm B}\rangle\), \(\langle{\rm GAP}\rangle\)). Each channel’s unit sequence is individually synthesized into audio by HiFi-GAN, and the resulting audio signals are combined to form stereo audio.

 

for the pre-training, we fine-tune the pre-trained Llama3.2-1B model over 15k iterations, utilizing 4 A40 GPUs. Training was performed with a total batch size of \(32\times 2048\) (65.5k tokens) and a learning rate of \(5\times 10^{-5}\).

In the spoken dialogue generation training stage, we use audio files from the behavior-SD dataset. Each two-channel audio is converted into streamlined units by incorporating control tokens based on backchannels, overlaps, and gaps. These streamlined units are paired with the corresponding narrative and behavior conditions to build a supervised fine-tuning dataset. In our experiments, we trained the model on 94K dialogues (1.87K hours) using 8 A40 GPUs with a total batch size of \(32\times 4096\) (130K tokens). Training spanned 18k iterations with a learning rate of \(5\times 10^{-5}\).

### Baselines

To evaluate BeDLM's ability to adhere to behavior conditions and generate natural spoken dialogues, we compared it against several baseline models.

GT and Resynthesized audios.We randomly select 500 narrative and behavioral instances from the Behavior-SD test set, using their corresponding audio as the ground truth. Additionally, we resynthesize audio using HuBERT units derived from these ground-truth audios as a baseline for comparison.

dgslm.We train dGSLM Nguyen et al. (2023) on the Behavior-SD dataset. As dGSLM is an unconditional model for generating spoken dialogues, we add each speaker's behavior condition tokens at the start of the speech units to guide generation. However, narrative conditions are not included in dGSLM's inputs.

Cascaded LLM + TTS.We implement a baseline using Llama3-70B Dubey et al. (2024) and GPT-4o (2024-08-06) to generate text dialogues based on narrative and behavior conditions, which are then converted to speech using CosyVoice-SFT Du et al. (2024). Cascaded models often struggle to produce parsable text for spoken dialogue generation, complicating direct comparisons with BeDLM. To ensure fairness, we exclude conditions that cascaded models could not handle across all baselines and proposed models, focusing only

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & \multicolumn{3}{c}{LM settings} & \multicolumn{3}{c}{Adherence Score} \\ \cline{2-7} Models & SLM (or LM) & Output Sequence & N(\(\uparrow\)) & V(\(\downarrow\)) & F(\(\downarrow\)) & B(\(\downarrow\)) & I(\(\downarrow\)) \\ \hline Ground-Truth & - & - & 4.48 & 0.00 & 0.00 & 0.00 & 0.00 \\ Resynthesized & - & - & 4.48 & 0.12 & 0.20 & 0.12 & 0.10 \\ \hline Cascaded & GPT-4o & Text dialogue & 4.58 & 0.40 & 0.18 & 1.73 & 0.91 \\ Cascaded & Llama3-70B & Text dialogue & 4.09 & 0.71 & 0.47 & 1.18 & 0.68 \\ \hline dGSLM & DLM & 2-channel units & - & 0.90 & 0.64 & 3.25 & 2.62 \\ \hline BeDLM & Llama3.2-1B & Alternating units & 1.23 & 1.44 & 1.09 & 2.22 & 1.21 \\ BeDLM _(w/o pretrain)_ & Llama3.2-1B & Streamlined units & 2.80 & 0.26 & **0.10** & 0.87 & 0.64 \\
**BeDLM (ours)** & Llama3.2-1B & Streamlined units & **3.11** & **0.25** & 0.15 & **0.87** & **0.58** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison of adherence score across various models. Narrative adherence scores (N) ranging from 1-5, as rated by GPT-4o. Behavioral adherence scores are provided for various conversational behaviors: verbosity (V), filler word usage (F), backchannel usage (B), and interruption usage (I).

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{2}{c}{LM settings} & \multicolumn{3}{c}{Mean Opinion Scores (\(\uparrow\))} \\ \cline{2-5}

- & **4.14 ± 0.08** & **4.02 ± 0.08** & **4.12 ± 0.08** \\ Resynthesized &
- & 4.09 ± 0.08 & 4.02 ± 0.08 & 4.04 ± 0.08 \\ \hline Cascaded & GPT-4o & Text dialogue & **4.12 ± 0.08** & 3.97 ± 0.08 & **4.09 ± 0.08** \\ Cascaded & Llama3-70B & Text dialogue & 4.08 ± 0.09 & 3.97 ± 0.09 & **4.10 ± 0.09** \\ \hline dGSLM & DLM & 2-channel units & 3.96 ± 0.09 & 3.88 ± 0.09 & 3.92 ± 0.09 \\ \hline BeDLM & Llama3.2-1B & Alternating units & 3.90 ± 0.09 & 3.86 ± 0.09 & 3.85 ± 0.10 \\ BeDLM _(w/o pretrain)_ & Llama3.2-1B & Streamlined units & 4.00 ± 0.08 & 3.89 ± 0.08 & 3.99 ± 0.09 \\ BeDLM (ours) & Llama3.2-1B & Streamlined units & **4.09 ± 0.09** & **4.04 ± 0.08** & 4.05 ± 0.09 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of Naturalness, Meaningfulness, and Sound Quality across various models.
 

on conditions where all models could generate viable outputs. This approach ensures a balanced and fair comparison. The text generation prompt for cascaded models is detailed in Appendix D.

## 7 Results

### Human Opinion Scores

For each spoken dialogue, three human evaluators assess the following aspects: dialogue naturalness, meaningfulness, and sound quality. Each sample is rated on a 5-point Likert scale, ranging from 1 to 5. Detailed instructions provided to the evaluators can be found in Appendix E.

Table 3 shows human evaluation scores, with BeDLM using streamlined units and pre-training achieving the best results, closely matching ground truth in naturalness and meaningfulness. Its sound quality is comparable to that of HuBERT and HiFiGAN resynthesis. Cascaded models perform well in naturalness and sound quality but struggle with speaker confusion and misplaced backchannels, lowering meaningfulness. dGSLM consistently scores the lowest. BeDLM variants without pre-training or using alternating units show weaker performance, highlighting the importance of pre-training for natural, meaningful dialogue generation.

### Adherence Scores

To evaluate the model's adherence to narrative conditions, we use GPT-4o to score the alignment between the provided narrative and WhisperX (Bain et al., 2023) transcriptions on a 5-point scale (1=Bad, 5=Excellent), based on 100 samples. The evaluation prompts are shown in Figure 13.

For behavioral adherence, we measure key features (verbosity, filler words, backchannels, interruptions) for model-generated and ground-truth dialogues across 500 samples. We calculate the Wasserstein distance between the distributions of each behavior level and average the normalized distances to arrive at the final adherence score.

Table 4 compares the dialogue generation models based on narrative and behavioral alignment. Our model, BeDLM, excels in behavioral coherence, outperforming cascaded models and dGSLM in managing verbosity, backchannels, and interruptions. Although BeDLM's narrative adherence (N) score is lower than cascaded models using larger LLMs like GPT-4o and Llama3-7OB, it maintains a smoother conversational flow. BeDLM without pre-training shows lower narrative adherence but similar behavioral alignment, while the alternating unit variant underperforms, highlighting the importance of streamlined units and pre-training.

### Speaker Consistency.

To evaluate the effectiveness of our speaker conditioning method in maintaining speaker identity throughout a dialogue, we conducted experiments on 1,000 sampled utterances using randomly selected prompt speech. Table 5 presents the mean cosine similarity computed using the WavLM-Base+ model (Chen et al., 2021). The cosine similarity between (\(s_{0}\), \(s_{i}\)) reflects the consistency of the speaker's identity across the dialogue, while the cosine similarity between (\(s_{i-1}\), \(s_{i}\)) indicates the smoothness of transitions between consecutive utterances. Including the first utterance \(s_{0}\) as prompt speech is essential for both speaker consistency and smooth transitions, whereas incorporating the current speaker's previous utterance, \(s_{i-1}\) does not have a significant impact. However, relying solely on the initial prompt \(s_{0}\) can lead to perceptible discontinuities, often due to breathing sounds or moments of silence. To address this, our proposed method, which concatenates the initial and previous utterances (\(s_{0},s_{i-1}\)), yields the best performance in maintaining speaker consistency and smooth transition throughout the dialogue.

### Qualitative results

Figure 5 shows two example waveforms from full-duplex dialogues generated by our BeDLM. Both follow the same narrative but differ in behavior conditions. In panel (a), the first speaker avoids using backchannels, whereas in panel (b), backchannels occur frequently. This clear contrast in waveforms demonstrates our model's ability to capture varying backchannel conditions in conversation. Furthermore, our model effectively reflects the second speaker's behavior conditions, such as interruptions and verbosity.

\begin{table}
\begin{tabular}{c c c} \hline \hline  & \multicolumn{2}{c}{**Cosine Similarity (\(\uparrow\))**} \\ \cline{2-3}
* & \(0.680\pm 0.09\) & \(0.687\pm 0.10\) \\ \(s_{i-1}\) & \(0.779\pm 0.15\) & \(0.850\pm 0.12\) \\ \(s_{0}\) & \(0.889\pm 0.09\) & \(0.867\pm 0.10\) \\ \hline \(s_{0},s_{i-1}\) & \(0.885\pm 0.09\) & \(0.872\pm 0.10\) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Cosine similarity for different prompts
 



## 8 Conclusion

We present a scalable framework for generating spoken dialogues with a diverse combination of conversational behaviors using LLMs and TTS models. Our contribution includes the Behavior-SD dataset, comprising over 100K spoken dialogues with behavior annotations, and the BeDLM, generating spoken dialogues that outperform baseline models in naturalness, diversity, and behavioral coherence. Future work could explore expanding multi-speaker dialogues, improving behavior control granularity, and integrating sophisticated real-time interaction capabilities to enhance practical applications in various spoken dialogue systems.

## 9 Limitations and Potential Risks

Limitation.One limitation of our system is the occasional mispronunciation, limited control over complex emotions, and limited non-lexical vocalizations other than laughter. To assess the pronunciation accuracy of Behavior-SD, we measure word error rates using the automatic speech recognition model, as detailed in Appendix B.3. Additionally, while the scale of our model and dataset are smaller than LLMs, they still effectively capture the semantics for generating coherent dialogues. However, this may impact the depth of nuance.

Potential Risks.The dataset is synthesized using LLMs and TTS models, which are known to inherit biases from their training data. As a result, Behavior-SD may not fully capture the nuances of diverse cultural, social, or linguistic variations in the real world, potentially leading to biased or unnatural dialogues. To address this issue, we designed our dataset generation process to incorporate diverse narratives from knowledge graphs and various conversational styles, ensuring broad representation. Additionally, we applied a Max-Min sampling algorithm to select a diverse and inclusive range of voices, as detailed in Appendix B.1. However, we acknowledge that real-world conversational complexity is vast, and continuous efforts are required to improve inclusivity and fairness.

Furthermore, while BeDLM generates humanlike speech, it raises ethical concerns, particularly regarding deepfake audio, misinformation, and voice impersonation. To address this, we emphasize that our work is intended for research purposes and support the development of watermarking or detection mechanisms to safeguard against misuse in deceptive applications.

## Acknowledgements

We want to thank the reviewers for their valuable feedback. This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-II20156, Fundamental research on continual meta-learning for quality enhancement of casual videos and their 3D metaverse transformation), the IITP grant funded by the Korea government (MSIT) (No. RS-2019-II191082, SW StarLab), the SNU-Global Excellence Research Center establishment project, the IITP (Institute of Information & Communications Technology Planning & Evaluation)-ITRC (Information Technology Research Center) grant funded by the Korea government (Ministry of Science and ICT) (IITP-2025-RS-2024-00437633), and the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (RS-2023-00274280).

## References

* Abdullin et al. [2023] Yelaman Abdullin, Diego Molla-Aliod, Bahadorreza Ofoghi, John Yearwood, and Qingyang Li. 2023. Synthetic dialogue dataset generation using llm

Figure 5: Waveforms of spoken dialogues generated by BeDLM with the same narrative but different behavior conditions. (a) shows no backchannels, while (b) includes frequent backchannels, reflecting the conditioned behaviors in each case.



agents. In _The 2023 Conference on Empirical Methods in Natural Language Processing_, page 181.
* Bain et al. (2023) Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. Whisperx: Time-accurate speech transcription of long-form audio. _INTERSPEECH 2023_.
* Bavelas et al. (2000) Janet B Bavelas, Linda Coates, and Trudy Johnson. 2000. Listeners as co-narators. _Journal of personality and social psychology_.
* Bevacqua et al. (2012) Elisabetta Bevacqua, Etienne de Sevin, Sylwia Julia Hyniewska, and Catherine Pelachaud. 2012. A listener model: introducing personality traits. _Journal on Multimodal User Interfaces_, page 12.
* Blomsma et al. (2024) Peter Blomsma, Julija Vaitonye, Gabriel Skantze, and Marc Swerts. 2024. Backchannel behavior is idiosyncratic. _Language and Cognition_.
* Chen et al. (2021) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, and Furu Wei. 2021. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. _arXiv preprint arXiv:2110.13900_.
* Chu et al. (2023) Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audiolanguage models. _arXiv preprint arXiv:2311.07919_.
* Cieri et al. (2004) Christopher Cieri, David Miller, and Kevin Walker. 2004. The fisher corpus: A resource for the next generations of speech-to-text. In _LREC_.
* De Sevin et al. (2010) Etienne De Sevin, Sylwia Julia Hyniewska, and Catherine Pelachaud. 2010. Influence of personality traits on backchannel selection. In _Intelligent Virtual Agents: 10th International Conference, IVA 2010, Philadelphia, PA, USA, September 20-22, 2010. Proceedings 10_, pages 187-193. Springer.
* Du et al. (2024) Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. 2024. Cosyo voice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. _arXiv preprint arXiv:2407.05407_.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Fang et al. (2024) Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. 2024. Llama-omni: Seamless speech interaction with large language models. _arXiv preprint arXiv:2409.06666_.
* Godfrey et al. (1992) John J Godfrey, Edward C Holliman, and Jane McDaniel. 1992. Switchboard: Telephone speech corpus for research and development. In _Acoustics, speech, and signal processing, ieee international conference on_.
* Goldberg (1990) Julia A. Goldberg. 1990. Interrupting the discourse on interruptions: An analysis in terms of relationally neutral, power- and rapport-oriented acts. _Journal of Pragmatics_, 14(6):883-903.
* Google (2024) Google. 2024. Notebook LM. [https://blog.google/technology/ai/notebookLM-audio-overviews/](https://blog.google/technology/ai/notebookLM-audio-overviews/).
* Guo et al. (2023) Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan. 2023. Prompttts: Controllable text-to-speech with text descriptions. In _ICASSP_, pages 1-5. IEEE.
* Hassid et al. (2024) Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeye, Emmanuel Dupoux, et al. 2024. Textually pretrained speech language models. _Advances in Neural Information Processing Systems_, 36.
* Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kwshal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_, 29:3451-3460.
* Huang et al. (2024) Rongjie Huang, Mingze Li, Dongchao Yang, Jiaotong Shi, Xunankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. 2024. Audiopgt: Understanding and generating speech, music, sound, and talking head. In _Proceedings of the AAAI Conference on Artificial Intelligence_, pages 23802-23804.
* Kawahara (2019) Tatsuya Kawahara. 2019. Spoken dialogue system for a human-like conversational robot erica. In _9th International Workshop on Spoken Dialogue System Technology_.
* Kim et al. (2023) Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Maliine Alkhani, Gunhee Kim, Maarten Sap, and Yejin Choi. 2023. Soda: Million-scale dialogue distillation with social commonsense contextualization. In _EMNLP_.
* Kong et al. (2020a) Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020a. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. _NeurIPS_, 33:17022-17033.
* Kong et al. (2020b) Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020b. Hifi-gan: High-fidelity generative adversarial networks for text-to-speech synthesis. [https://github.com/jik876/hifi-gan](https://github.com/jik876/hifi-gan). Accessed: 2025-02-07.
* Lakhotia et al. (2020) Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman 

Mohamed, et al. 2021. On generative spoken language modeling from raw audio. In ACL, Vol. 9, pp. 1336-1354. Cited by: SS1.
* K. Lee, Kyumin Park, and D. Kim (2023)Dailytalk: spoken dialogue dataset for conversational text-to-speech. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. External Links: Document Cited by: SS1.
* Y. Aaron Li, X. Jiang, J. Darefsky, G. Zhu, and N. Mesgarani (2024)Styletalker: finetuning audio language model and style-based text-to-speech model for fast spoken dialogue generation. In First Conference on Language Modeling, Cited by: SS1.
* G. Lin, C. Chiang, and H. Lee (2024)Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. In ACL, Cited by: SS1.
* Z. Ma, Y. Song, C. Du, J. Cong, Z. Chen, Y. Wang, Y. Wang, and X. Chen (2024)Language model can listen while speaking. arXiv preprint arXiv:2408.02622. Cited by: SS1.
* Meta (2024)Llama 3.2. [https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_2/](https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_2/). Cited by: SS1.
* Kentaro Mitsui, Y. Hono, and K. Sawada (2023)Towards human-like spoken dialogue generation between ai agents from written dialogue. arXiv preprint arXiv:2310.01088. Cited by: SS1.
* T. A. Nguyen, E. Kharitonov, J. Copet, Y. Adii, W. Hsu, A. Elkahky, P. Tomasello, R. Algayres, B. Sagot, A. Mohamed, et al. (2023)Generative spoken dialogue language modeling. In ACL, Cited by: SS1.
* O. Olman (2022)Emo-tts:parallel transformer-based text-to-speech model with emotional awareness. In 2022 5th International Conference on Computing and Informatics (ICCI), pp. 169-174. Cited by: SS1.
* S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea (2019)MELD: a multimodal multi-party dataset for emotion recognition in conversations. In ACL, Florence, Italy, pp. 527-536. Cited by: SS1.
* A. Radford, J. Wood Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever (2023)Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, Online 

Rich Text Dialogue Generation Pipeline

In this section, we offer further details outlined in Section 3.1 on generating rich text dialogues, including the prompts used for generation and the filtering processes involved.

### Prompts for Rich Text Dialogue Generation

Dialogue generation.We use Figure 8 as the prompt for GPT-4o. At this stage, speaker names and narratives sampled from SODA are incorporated. From here, the process varies depending on the behavioral traits assigned to each speaker. Specifically, if both speakers are conditioned not to interrupt, we manually exclude any constraints related to interruption from the prompt. Additionally, each speaker's behavior is mapped to text based on its level, as shown in Table 6, and incorporated into the prompt. The dialogue generated by GPT-4o based on this prompt is then parsed and proceeds to the next stage.

Backchannel insertion.As described in section 3.1, to insert backchannels, we first identify backchannel opportunity points (BOPs). To do this, we use the prompt in Figure 9 with GPT-4o-mini to find the natural pause points within each utterance. Then, based on the backchannel insertion level, we subsample the BOPs by selecting either 0-30% (Level 1) or 30-60% (Level 2) of the total.

To facilitate this, we insert placeholders such as "speaker! (backchannel): [MASK]" or "speaker? (backchannel): [MASK2]" at the subsampled BOPs to guide the LLM in generating contextually appropriate backchannels at the specified positions. After adding masked utterances for each BOP into the original dialogue, we use the prompt from Figure 10 for GPT-4o.

Speech style captioning.To achieve more natural and emotionally rich speech synthesis, it is crucial to capture the speech style. Since elements like pitch, speed, and emotion can vary even within a single utterance, we do not obtain the speech style description at the utterance level. Instead, we utilize NLTK's sentence tokenizer1 to obtain speaking styles on a sentence-wise basis. For this, we use the prompt in Figure 11 to caption the pitch, speaking style, and emotion for each sentence.

Footnote 1: [https://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.sent_tokenize](https://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.sent_tokenize)

### Interruption Scenario Banks

To ensure that the LLM covers diverse interruption scenarios and that Behavior-SD is behaviorally rich, we provide examples of interruption scenarios in the prompt during dialogue generation. Specifically, we randomly select 1-3 samples from an interruption scenario bank. The scenario bank is composed of examples from seven categories of interruption Goldberg (1990): disagreement, floor taking, topic change, tangentialization, agreement, assistance, and clarification. Figure 14 shows interruption scenarios examples.

## Appendix B TTS Pipeline

This section expands on the conversion of rich text dialogues into spoken dialogues in Section 3.2. We present methodologies for enhancing speaker consistency, improving the pronunciation of vocalization backchannels, and evaluating the pronunciation accuracy of synthesized speech.

### Speaker Bank

When a speaking style is provided as an instruction, TTS models generate random voices corresponding to different speakers. However, this method does not ensure consistent speaker identity throughout a dialogue. To address this limitation, we propose selecting the prompt speech for the TTS model from a pre-sampled "speaker bank" to ensure speaker consistency across multiple utterances. For each synthesized utterance, our method ensures that subsequent utterances are generated using the same speaker's voice.

To ensure a diverse and representative selection of speaker embedding, we applied a Max-Min sampling algorithm, which maximizes the minimum pairwise distance between embeddings to enhance speaker distinction. We applied a Max-Min sampling algorithm on a speaker bank of 30k samples to ensure diversity in speaker selection, ultimately choosing 26 male and 26 female speakers for the final set.

### Enhancing Vocalization Backchannels through Voice Cloning

To mitigate issues related to suboptimal pronunciation of vocalization backchannels, such as _"hmm"_ and _"mhm,"_ in the CosyVoice TTS system, we utilized ElevenLabs' voice cloning2 and text-to 

speech3 technologies. Voice cloning was applied to the 52 speakers in our dataset, and the backchannels were ranked by frequency. We selected the 100 most common backchannels for synthesis, with each one synthesized up to 100 times. These synthesized backchannels were then integrated into the dialogue synthesis process to enhance the naturalness of the system's output.

Footnote 3: [https://elevenlabs.io/text-to-speech](https://elevenlabs.io/text-to-speech)

### Word Error Rate Analysis of TTS Output

We evaluate the pronunciation accuracy of Behavior-SD using Whisper-Large V3 (Radford et al., 2023) for automatic speech recognition (ASR). Specifically, we compute the word error rate (WER) by comparing the ASR-transcribed text with the ground-truth reference. On the test split of Behavior-SD, the model achieves a WER of 3.55%, whereas the original text-to-speech system, CosyVoice Du et al. (2024), attained a lower WER of 2.89% on the LibriSpeech test split. The relatively higher WER of Behavior-SD can be primarily attributed to distributional differences in the text, including the insertion of filler words and the misrecognition of proper names by the ASR model. These results indicate that the synthesized speech maintains high pronunciation accuracy, with errors largely stemming from conversational artifacts and occasional TTS mispronunciations. Further improvements in the pronunciation accuracy of Behavior-SD could be achieved by adopting a TTS model with better pronunciation accuracy in place of CosyVoice.

## Appendix C BeDLM Details

### Streamlined Spoken Dialogue Representations

Figure 6 presents various representations of full-duplex spoken dialogues. Our streamlined units effectively capture diverse aspects of full-duplex

\begin{table}
\begin{tabular}{l l p{170.7pt}} \hline \hline
**Behavior type** & **Level** & **Text** \\ \hline  & 0 & \{speaker\} speak very briefly, sharing only essential content without any extra details \\ Verbosity & 1 & \{speaker\} speak at a moderate length, providing a balanced amount of content—enough to convey your point clearly but without overexplaining \\ \cline{2-3}  & 2 & \{speaker\} speak in detail, often providing lengthy explanations or descriptions whenever possible \\ \hline  & 0 & \{speaker\} never use filler words \\ Filler words & 1 & \{speaker\} moderately use filler words \\ \cline{2-3}  & 2 & \{speaker\} frequently use filler words \\ \hline  & 0 & \{speaker\} never interrupt \\ Interruptions & 1 & \{speaker\} moderately interrupt \\ \cline{2-3}  & 2 & \{speaker\} frequently interrupt \\ \hline \hline \end{tabular}
\end{table}
Table 6: Mapping of speaker behaviors to text based on levels.

Figure 6: Examples of spoken dialogue representations: (a) two-channel units, (b) streamlined units, and (c) alternating units with a chunk size of 4. (d) These representations can be easily converted to stereo audio using a vocoder.

 

interactions, including turn-taking, overlapping speech, gaps, and backchannels.

### Inference

During the inference stage of BeDLM, we utilize the same prompt detailed in Figure 7. We then autoregressively generate the next streamlined unit, using a temperature of 0.8, top-\(k\) sampling with \(k\) set to 60, and top-\(p\) sampling with \(p\) set to 0.8.

### HuBERT

Due to computational constraints, we train only the k-means clustering component of HuBERT while adopting the dense model parameters from TWIST Hassid et al. (2024). This process enhances HuBERT's ability to encode speaker attributes in the Behavior-SD corpus, particularly in capturing laughter. To achieve robust clustering, the k-means model is trained for 25,600 iterations.

### HuBERT2Mel

_HuBERT2Mel_ converts HuBERT token sequences, in combination with a speaker identity embedding, into mel spectrograms that correspond to the target speaker's voice. These spectrograms are then used to fine-tune HiFi-GAN, improving both the quality of waveform generation and training efficiency compared to unit-based HiFi-GAN approaches.

The model predicts mel spectrograms derived from 22,050 Hz speech, utilizing a hop size of 256, a window size of 1024, and an FFT size of 1024, with frequency limits of 0 Hz (fmin) and 8,000 Hz (fmax). HuBERT2Mel processes token and speaker embeddings through a combination of transformer and convolutional layers incorporating gated linear units (GLU). The model then applies upsampling to align its output length with the mel spectrogram's temporal resolution. Finally, mel spectrogram predictions are generated via fully connected layers with residual connections.

Training is conducted for 60,000 steps using an L1 loss function, optimizing the model toward ground-truth mel spectrograms.

### HiFi-GAN

We fine-tune a pre-trained HiFi-GAN model Kong et al. (2020) using pairs of original speech waveforms and their corresponding mel spectrograms, which are generated by first converting the speech into HuBERT tokens and then processing them through HuBERT2Mel. For initialization, we adopt the Universal-V1 checkpoints from the official implementation Kong et al. (2020). Fine-tuning is performed on a 1/100 subset of the Behavioral-SD training split for 100,000 steps, refining the model to enhance the perceptual quality of BeDLM-generated speech.

During inference, HiFi-GAN processes mel spectrograms generated by HuBERT2Mel and synthesizes raw audio at a sampling rate of 22,050 Hz, ensuring high-quality speech reconstruction.

## Appendix D Cascaded Model Description

The text dialogue generation prompts used for the cascaded models are illustrated in Figure 12.

## Appendix E Human Evaluation Guidelines

The evaluators are instructed as follows:

* **Dialogue Naturalness:** Are backchannels and laughter appropriately included to create a human-like interaction? Is there a seamless transition between the speaker and listener at the right moments? Does the conversation flow smoothly and not awkward?
* **Meaningfulness:** Does the dialogue have meaningful content, and is it possible to understand what is being said?
* **Sound Quality:** Is the sound clear and easy to hear, free from noise or other distractions?

These instructions are adapted from Mitsui et al. (2023).

 



## Appendix A

Figure 8: A prompt for dialogue generation.

Figure 7: A prompt designs narrative and conversational behaviors for conditional generation. During the pretraining phase, the corresponding text dialogues are used as the target for the given condition.



* [16] A. K. S. Goyal, J. M. Lafferty, and D. R. Smith, "A new approach to the generation of a single-channel state in a quantum computer", _Journal of Machine Learning Research_**30**(2):475-498, 2009.
* [17] P. B. Anderson, C. E. Shannon, and I. W. Stewart, "Quantum information processing with multiple channels," _IEEE Transactions on Information Theory_**5**(1):105-118, 2002.
* [18] T. H. Cover, N. J. Cover, and R. H. Thomas, "Elements of information theory", _Annals of Mathematical Statistics_**19**(1):1-10, 1994.

[MISSING_PAGE_POST]

 



## 10.2 Thesis

Figure 10: A prompt for backchannel insertion.



* [25] M. C. Bader, J. A. Harvey, and R. H. S. Heath, "A new approach to the design of a digital system for digital systems," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1-30, 2006.
* [26] D. P. Kingma and I. Ba, "Adam: A method for stochastic optimization," in _Advances in neural information processing systems_, volume 489 of _Lecture Notes in Statistics_, pages 77-96, 2006.
* [27] G. K. Hinton, "Deep learning with applications to image recognition," _IEEE Transactions on Image Processing_, vol. 12, no. 1, pp. 1288-1300, 2006.
* [28] L. Bottou, Y. Bengio, and T. Salakhutdinov, "Dropout normalization by reducing the dimensionality of features," _arXiv preprint arXiv:1412.6090_, 2014.

[MISSING_PAGE_POST]

 



## Appendix A

Figure 12: A prompt for cascaded model.

Figure 13: A prompt for GPT-4o to evaluate the narrative adherence score.





**Tangentialization**

The listener grabs the turn and sums up the information received from the current speaker to prevent listening to more unwanted information.

**Speaker A: The quarterly report indicates that we have a 15% increase in sales, which is due to the new marketing strategy we implemented. Additionally, we have [interrupted]**

**Speaker B (interrupt): So, sales are up 15% thanks to marketing. Got it. What's the next step? Speaker A: I was going to explain the other factors, but we can move on.**

Figure 14: Samples from the interruption scenario bank.