Let’s Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation (2024)

Abstract: In this paper, we introduce a novel Face-toFace spoken dialogue model. It processes audio-visual speech from user input and generates audio-visual speech as the response, marking the initial step towards creating an avatar chatbot system without relying on intermediate text. To this end, we newly introduce MultiDialog, the first large-scale multimodal (i.e., audio and visual) spoken dialogue corpus containing 340 hours of approximately 9,000 dialogues, recorded based on the open domain dialogue dataset, TopicalChat. The MultiDialog contains parallel audio-visual recordings of conversation partners acting according to the given script with emotion annotations, which we expect to open up research opportunities in multimodal synthesis. Our Face-to-Face spoken dialogue model incorporates a textually pretrained large language model and adapts it into the audio-visual spoken dialogue domain by incorporating speech-text joint pretraining. Through extensive experiments, we validate the effectiveness of our model in facilitating a face-to-face conversation. Demo and data are available at https://multidialog.github.io and https://huggingface.co/datasets/ IVLLab/MultiDialog, respectively.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16334–16348 August 11-16, 2024 ©2024 Association for Computational Linguistics
In this paper, we introduce a novel Face-toFace spoken dialogue model. It processes audio-visual speech from user input and generates audio-visual speech as the response, marking the initial step towards creating an avatar chatbot system without relying on intermediate text. To this end, we newly introduce MultiDialog, the first large-scale multimodal (i.e., audio and visual) spoken dialogue corpus containing 340 hours of approximately 9,000 dialogues, recorded based on the open domain dialogue dataset, TopicalChat. The MultiDialog contains parallel audio-visual recordings of conversation partners acting according to the given script with emotion annotations, which we expect to open up research opportunities in multimodal synthesis. Our Face-to-Face spoken dialogue model incorporates a textually pretrained large language model and adapts it into the audio-visual spoken dialogue domain by incorporating speech-text joint pretraining. Through extensive experiments, we validate the effectiveness of our model in facilitating a face-to-face conversation. Demo and data are available at https://multidialog.github.io and https://huggingface.co/datasets/ IVLLab/MultiDialog, respectively.

heading: 1 Introduction
text: Spoken Dialogue System (SDS), often referred to as a conversational agent, engages in natural speech conversations with humans by recognizing speech from user input and providing contextually appropriate and accurate responses with speech. With
∗Equal contribution. †Corresponding author. This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2022R1A2C2005529) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-000124, Development of Artificial Intelligence Technology for Self-Improving Competency-Aware Learning Capabilities).
spoken language as the primary interface, it has numerous applications for human-computer interactions such as customer service and voice assistants.
However, when people communicate face-toface, we utilize not only audio but also visual information of the conversing partner to process spoken words and non-verbal cues (i.e., facial expressions, gestures, and emotions) (Petridis et al., 2018; Hong et al., 2023). This multimodal information enhances understanding of the speech content and the speaker’s intent. Furthermore, having a visual counterpart to audio can simulate a real face-toface conversation experience, making the user feel more connected and engaged.
In this paper, we explore an audio-visual spoken dialogue system to facilitate direct face-to-face conversation for the first time. Central to the development of dialogue systems is the large amount of high-quality dialogue data. Current dialogue systems are predominantly text-based, driven by the abundance of text dialogue datasets (Lowe et al., 2015; Li et al., 2017; Zhang et al., 2018; Rashkin et al., 2018; Budzianowski et al., 2018; Zhou et al., 2018; Reddy et al., 2019; Lambert et al.; Ding et al., 2023; Köpf et al., 2023). Recently, several audio dialogue datasets have been released (Lee et al., 2023; Si et al., 2023; Nguyen et al., 2023a) which augment existing text dialogue data (Li et al., 2017; Budzianowski et al., 2018) with speech. However, those with visual components remain limited in scale, comprising less than 15 hours in total (Busso et al., 2008; Poria et al., 2018). Addressing this data gap, we introduce MultiDialog, the first large-scale audio-visual spoken dialogue corpus. It consists of 340 hours of audio-visual recordings of approximately 9,000 dialogues, derived from open-domain text dialogue dataset, TopicalChat (Gopalakrishnan et al., 2023) which is an extensive multi-turn dialogue corpus collected from real conversations covering 9 broad topics. The proposed MultiDialog consists of emotion annotations for each utterance
16334
and simultaneous recordings of both the listener and the speaker, presenting opportunities for diverse research; from face-to-face dialogue system to talking face synthesis (Park et al., 2022; Zhang et al., 2023b), listener’s face synthesis (Song et al., 2023; Zhou et al., 2023), and emotion-conditioned face synthesis (Goyal et al., 2023).
Based on the MultiDialog dataset, we propose the first audio-visual spoken dialogue model that can directly process audio-visual speech as user input and generate audio-visual speech as the output response. Motivated by the recent success of the direct spoken dialogue model using discretized speech tokens (Nguyen et al., 2023b; Zhang et al., 2023a), we introduce audio-visual (AV) speech tokens extracted by quantizing audio-visual speech features from a self-supervised model (Shi et al., 2021). Utilizing the AV speech tokens as pseudo texts, we integrate AV speech into a pretrained large-language model (LLM) (Zhang et al., 2022) through joint speech-text pretraining. The response is also returned in AV speech tokens, which are synthesized into a talking face video as the output for direct interaction with the system.
Our contributions are in three folds: (1) We introduce the first direct Face-to-Face dialogue model which processes multimodal speech from user input and generates multimodal speech as the output response, facilitating a face-to-face conversation system. (2) To build a face-to-face dialogue system, we propose the first large-scale multimodal (i.e., audio, visual, and text) dialogue corpus, MultiDialog consisting of 340 hours of approximately 9,000 audio-visual conversation streams. (3) We demonstrate that joint speech-text pretraining leveraging a pre-trained large language model improves upon direct initialization in retaining knowledge of the original large language model.

heading: 2 Related Work

heading: 2.1 Spoken Dialogue Dataset
text: In recent years, the development of speech dialogue datasets has played a pivotal role in understanding human behavior and building spoken dialogue systems that emulate real-life conversations. Early speech datasets focus on analyzing human behavior such as emotion and intent in speech, establishing the foundation for spoken dialogue systems. IEMOCAP (Busso et al., 2008) and MELD (Poria et al., 2018), comprising audio and video recordings of dialogues, are designed to study emotional dynamics in conversations. In addition to understanding emotions, DSTC2 (Henderson et al., 2014) presents telephone-based speech dialogues for dialogue state tracking to predict user’s goals. Building upon datasets that study human behavior in speech, recent spoken dialogue datasets were built to model realistic dialogue systems. Expresso (Nguyen et al., 2023a) introduces speech dialogues spanning 26 expressive styles for natural speech synthesis. DailyTalk (Lee et al., 2023) and SpokenWOZ (Si et al., 2023) datasets introduce speechtext conversations for spoken dialogues. While existing works have contributed to advancing spoken conversation systems, dialogue datasets are limited in scale and solely consist of audio and text, thereby constraining the development of audio-visual spoken dialogue systems incorporating visual cues. To address these limitations, we expand the spoken dialogue in scale and to the visual modality, and introduce MultiDialog, a large-scale multimodal spoken dialogue dataset. A summary of existing multimodal dialogue datasets and MultiDialog is shown in Table 1.

heading: 2.2 Spoken Dialogue Models
text: Audio Language Model, driven by transformerbased architecture, has made remarkable strides in speech processing. By treating continuous speech as a discrete set of representations, speech can
be effectively modeled as text, allowing the application of Natural Language Processing (NLP) techniques. While it has made notable progress in speech synthesis (Lakhotia et al., 2021; Borsos et al., 2023; Wang et al., 2023a; Hassid et al., 2023; Nachmani et al., 2023), speech translation (Barrault et al., 2023; Dong et al., 2023; Rubenstein et al., 2023), and speech recognition (Wang et al., 2023b), spoken dialogue system is a relatively unexplored field of research due to the scarcity of spoken dialogue datasets. Several works made an effort to tackle data issues by leveraging the power of large language models (LLMs). SpeechGPT (Zhang et al., 2023a) first converts speech into discrete speech tokens, and then designs a three-stage training pipeline on paired speech data, speech instruction data, and chain-of-modality instruction data. AudioGPT (Huang et al., 2023) instructs LLMs to generate commands for controlling external tools before inputting them into the LLMs. dGSLM (Nguyen et al., 2023b) models two-channel conversations to produce natural turn-taking conversations.
There are Multimodal Large Language Models (MM-LLM) (Wu et al., 2023; Gong et al., 2023) capable of processing both visual input and output. However, they are visual grounding dialogue systems that use visual information as supplementary for tasks such as image captioning and image editing. In contrast, we aim to build an audio-visual spoken dialogue system (i.e., facial movement related to the speech) to enhance the understanding of speech content and enrich the communication experience, emulating a real face-to-face conversation.

heading: 3 MultiDialog Dataset

heading: 3.1 Preparation
text: To obtain audio-visual recordings of dialogues, we gathered 12 fluent English speakers, with varying gender, age, and nationality. The participants, aged 20 to 30, came from six different
countries, with six female and six male actors, as shown in Appendix A.2. We derived dialogue scripts from the open-domain dialogue dataset, TopicalChat (Gopalakrishnan et al., 2023) which is a rich knowledge-grounded dataset collected from real human-human conversations. It spans eight broad topics including fashion, politics, books, sports, general entertainment, music, science & technology, and movies. It is annotated for eight emotions: Disgusted, Angry, Fearful, Happy, Sad, Surprised, Neutral, and Curious to dive deeper. The conversation partners don’t have explicitly defined roles as ‘speaker’ or ‘listener’ so they interact naturally similar to how people engage in real-world conversations. Due to the topical variety, emotion annotation, and representation of natural human conversations, we chose TopicalChat as the foundation for constructing the multimodal dialogue dataset.

heading: 3.2 Recording
text: Data was recorded in a professional recording studio with a green screen and minimal background noise, shown in Appendix A.4. During a recording session, two conversation partners sat side-by-side and were recorded with a separate camera and a microphone. The camera position was adjusted according to the individual’s height to capture the upper body, starting from the shoulders. The participants were asked to act according to a given script conveying the desired emotion annotation for each utterance. We specifically provided detailed instructions for visual and audio cues based on the Facial Action Coding System (Ekman and Friesen, 1978) and tone (Gangamohan et al., 2016) for each emotion as follows:
• Neutral: normal resting face, emotionless, speak still with natural information. • Happy: lip corner puller, cheek raiser, lips parts, speak cheerfully in a higher tone. • Sad: drooping upper eyelids, slight pulling down of lips corners, speak in a sad, lower tone.
• Fearful: eyebrows raised and pulled together, eye pulled open, speak in a soft and low tone. • Surprise: eyebrows raised, eyes wide open, mouth open wider, speak excitedly with high tone. • Disgusted: eyebrows lowered and pulled together, nose wrinkled, cheek raised, upper lip raised, speak in a normal tone with disgusted intonation. • Anger: eyebrows lowered and pulled together, eyes glare, speak powerfully with high tone.
For recordings, we combined the emotion labels ‘Neutral’ and ‘Curious to dive deeper’ into a single label ‘Neutral’ due to the lack of visually apparent difference between the two. In addition to the instructions, we displayed sample images on the screen so that the actors could mimic the facial expressions corresponding to the emotion. Moreover, when the turn passes to another participant, they naturally react while listening. Participants were instructed to press a button to proceed to the next utterance, which recorded the start and end times of each turn for post-processing. The audio streams were recorded in a mono WAV format at 48kHz and the video streams in full HD at 30fps.

heading: 3.3 Post-Processing
text: To refine the data, we had an annotator go through the audio-visual recordings to check if there were any misalignments between the audio and visual streams. We asked the annotator to manually adjust the misalignments by sliding the start time. Additionally, we filtered out recordings that were missing either audio or visual streams. Then, we segmented the recordings into conversations and turns based on the recorded timesteps of each turn. As a result, the post-processed MultiDialog dataset consists of approximately 340 hours of audio-visual videos of 9,000 dialogues between 6 pairs of conversation partners. The final statistics of our dataset are shown in Table 2. Furthermore, we release a gold emotion dialogue subset selected based on rigorous annotation evaluation. Please refer to the Appendix A.3.1 for more details.

heading: 4 Audio-Visual Spoken Dialogue System
text: Based on the proposed MultiDialog dataset, we introduce an audio-visual spoken dialogue system that directly understands the audio-visual of the user’s face video and generates appropriate responses with audio-visual face video. It consists of
three main parts: 1) Encoding audio-visual speech into discrete representations, namely audio-visual (AV) speech tokens. 2) Conducting multimodal spoken dialogue language modeling using the AV speech tokens as pseudo texts. 3) Projecting the output AV speech tokens into the audio and visual space for direct face-to-face dialogue.

heading: 4.1 Audio-Visual Speech Encoding
text: By integrating both audio and visual modalities, we can improve the dialogue system’s understanding of the speech content. This is because speech not only comprises auditory signals but also visual cues from the movements of the speaker’s mouth. This visual information complements auditory signals, particularly in noisy environments, resulting in more robust performance (Afouras et al., 2018).
To this end, we adopt a unified approach to model both the audio and visual of talking face input into audio-visual speech tokens. Inspired by the recent success of utilizing discrete speech tokens extracted from self-supervised speech models (Schneider et al., 2019; Baevski et al., 2020; Hsu et al., 2021; Chung et al., 2021; Babu et al., 2021) in speech processing (Lakhotia et al., 2021; Lee et al., 2021; Maiti et al., 2023; Kim et al., 2023), we tokenize the audio and visual streams into audiovisual speech tokens (a.k.a. AV speech tokens). Specifically, we employ one of the multimodal speech models, AV-HuBERT (Shi et al., 2021), a state-of-the-art self-supervised framework for understanding speech by both seeing and hearing. It is trained on raw audio-visual face videos to predict discrete clusters from speech (Hassid et al., 2023). The audio-visual speech features are extracted and quantized into discrete tokens as in (Lakhotia et al., 2021; Popuri et al., 2022; Kim et al., 2024). By combining the visual cues and the auditory information, the audio-visual speech tokens extract both linguistic and phonetic information. Then, we treat the AV speech tokens as pseudo text to train our Audio-Visual Spoken Dialogue LM.

heading: 4.2 Audio-Visual Spoken Dialogue Language Modeling
text: As shown in Fig. 1, our audio-visual spoken dialogue language model is trained with the AV speech tokens on our MultiDialog dataset. Previous work (Hassid et al., 2023) showed that initializing a speech language model with a textually pretrained language model (LLM) leads to better performance and faster convergence. Accordingly, we use a pre-
trained LLM, OPT-1.3B (Zhang et al., 2022) to initialize our model and combine the vocabulary of AV speech tokens with the original text vocabulary, as in (Zhang et al., 2023a; Nachmani et al., 2023; Maiti et al., 2023). This allows us to jointly model the probability of both AV speech tokens and text tokens t, where the loss can be represented as,
L = − N∑
i=1
log p(ti | t1, ..., ti−1), (1)
which is the negative log-likelihood of predicting the next token in the sequence of length N tokens.
Motivated by the joint speech-text training used in speech processing tasks such as speech translation, audio speech recognition, and textto-speech synthesis (Cheng et al., 2023; Maiti et al., 2023; Dong et al., 2023; Wang et al., 2023b), we newly introduce a joint speech-text pre-training scheme tailored for spoken dialogue language modeling. In our setting, each dialogue D = [T ai1 , T user 1 , T ai 2 , T user 2 , . . . , T ai k , T user k ] consists of k rounds of turns T between two speakers which we randomly designate as the AI and the User. The goal of this pre-training is to effectively transform the text-based LLM into the AV speech token-based LLM, enabling it to produce relevant AV speech responses from the AI side given a con-
versation context. It proceeds in the following two stages:
The first stage is instructing the LLM to interpret and generate AV speech tokens. We segment the dialogue into turns T and prepare paired AV speech tokens TAV and text tokens TText. We then concatenate the pair with their respective modality prefix tokens, <speech> and <text>, to indicate the beginning of AV speech and text tokens. Adding the reversed order of concatenation, we construct both audio-visual speech recognition (AVSR) and text-to-speech generation (TTS) training objectives as shown in Fig. 2(a) and (b), where the loss functions can be respectively represented as:
LAVSR = N∑
i=1
− log p( T iAV | T<iAV , TText) (2)
LTTS = N∑
i=1
− log p(T iText | T<iText, TAV). (3)
We omitted the prefix tokens for conciseness. Only the embedding layer and the projection layer are trained in the first stage, which guides the LLM to understand and generate AV speech tokens while fully retaining the given LLM knowledge needed for dialogue generation.
The second stage is jointly learning the text and
(a) AV Speech to Text Token (AVSR) • <Speech> 7 495 123 495 7 21 495 21 495 ... 118 <Text> Are you a football fan?
(b) Text to AV Speech Token (TTS) • <Text> I know. I think because the league prefers offense over defense nowadays. <Speech> 7 381 17 338 21 123 57 123 329 ... 11
(c) Mixed Text and AV Speech Token Dialogue • <User> <Text> do you like drama? <AI> <Speech> 7 278 123 7 21 278 123 21 7 ... 212 <User> <Text> Yeah. Do you like Jim Carrey? <AI> <Speech> 7 278 123 278 123 278 161 7 406 ... 2 <User> <Text> Yes as they do not project positivity <AI> <Text> Yes. Do you listen to radio dramas?
(d) AV Speech Token Dialogue • <User> <Text> 7 123 381 123 7 402 437 21 413 ... 38 <AI> <Speech> 7 278 123 7 21 278 123 21 7 ... 212 <User> <Text> 7 445 123 7 123 329 57 437 161 ... 2 <AI> <Speech> 7 278 123 278 123 278 161 7 406 ... 2 <User> <Text> 7 17 7 437 23 437 7 437 161 ... 225 <AI> <Text> 7 278 123 278 123 21 7 437 380 ... 385
Figure 2: Constructed data based on the MultiDialog dataset used for training the audio-visual speech dialogue model. (a-c) are joint pretraining of the audiovisual speech and text tokens and (d) is used to finetune the model.
AV speech token-based dialogue. We select either one of the speakers as the AI which the model aims to predict and indicate the start of the response with additional speaker prefix tokens, <User> and <AI>. The speaker prefix token is followed by a modality prefix token, <Speech> and <Text>, to indicate whether the utterance is in AV speech or text. The loss function for dialogue language modeling is:
Ldilaog = K∑
k=1
Nk∑
n=1
− log p(T ai,nk | T ai,<n k , T<k),
(4)
where K is the total number of rounds, Nk is the number of tokens in the k-th round, T ai,nk is the n-th token from the AI in the k-th round, T ai,<nk denotes all previous tokens from the AI within the same round k, and T<k is all prior tokens in previous rounds. Note that we dropped the prefix tokens in the equation for brevity. During the pretraining, we utilize a balanced mix of the AV speech tokens and text which allows the model to utilize both token knowledge to generate dialogue response as in Fig. 2(c). Then, we later finetune on pure AV speech token-based dialogue as in Fig. 2(d) for realtime face-to-face interaction. This progressive shift helps the model to gradually adapt to AV speech tokens without compromising the quality of dialogue generation of the text-based LLM.
Evaluation Prompt • <User> Hi, how are you doing today? <AI> • <User> Hi, how are you doing today? <AI> I am good, thanks. I'm wondering if you are a football fan like me? I know a lot of people are sad the season is over, especially since fantasy football is such a hit. <User> I like football, but I don't get much chance to watch. usually I try to catch the SB. <AI> • <User> Hi, how are you doing today? <AI> I am good, thanks. I'm wondering if you are a football fan like me? I know a lot of people are sad the season is over, especially since fantasy football is such a hit. <User> I like football, but I don't get much chance to watch. usually I try to catch the SB. <AI> What is the SB? I know if fantasy football it's all about the picks and people plan for weeks. <User> Super Bowl. Sorry, should have made that clear. I've never played fantasy sports. What's the goal? How do you win? What do you win? <AI>
Figure 3: Evaluation prompt of multimodal dialogue language modeling. It is written in text for illustration but the actual prompt is given as audio and visual.

heading: 4.3 Audio-Visual Generation
text: The generated AV speech tokens are projected to audio and visual to generate the response as a talking face video. As shown in Fig. 1, the audio-visual generator consists of a length predictor, a tokenbased speech decoder, and a token-based face decoder. Since our language model is trained with duplicate reduced AV speech tokens, we train a length predictor to first restore them back to their original length. The token-based speech decoder and token-based face decoder are adapted from an off-the-shelf audio generator (Kong et al., 2020) and a talking face generator (Prajwal et al., 2020) respectively, where we train them to process AV speech tokens as the input instead of raw audio. Additionally, we incorporate speaker identity information by extracting the speaker embedding (Jia et al., 2018) from a target identity sample audio. Also, the target identity’s face and pose prior are utilized as in (Prajwal et al., 2020), to enable the generation of talking face video with desired identity.

heading: 5 Experimental Setup

heading: 5.1 Evaluation Metrics
text: We evaluate the semantic quality and the generation quality of both audio and video. For the semantic quality, we first generate transcriptions from the synthesized audio-visual output using an off-theshelf ASR model (Shi et al., 2021), and employ standard metrics used for text-based dialogue generation: log-perplexity (PPL), BLEU, METEOR, F1, D-1, and D-2. The log-perplexity is calculated using Dialo-GPT model (Zhang et al., 2019) and it is calculated for each utterance and averaged across
the test set. To measure the generation quality of video, we adopt metrics used for TFG. This includes Fréchet Inception Distance (FID) (Heusel et al., 2017) to measure visual quality, and LSEC and LSE-D (Prajwal et al., 2020) to measure the audio-visual synchronization. To evaluate the acoustic quality, we compute speaker similarity (SIM) between the given target sample and generated speech using the WavLM-Base model for speaker verification (Chen et al., 2021). Please refer to the appendices for a detailed explanation of each metric.

heading: 5.2 Implementation Details
text: To encode AV speech tokens, we crop the video into the mouth region of size 96×96 using a face detector (Deng et al., 2020) and a facial landmark detector (Bulat and Tzimiropoulos, 2017), and resample the audio to 16kHz. We take English-trained AVHuBERT (Shi et al., 2021) and finetune it to predict corresponding target clusters from HuBERT tokenizer (Hassid et al., 2023) which operates at 25Hz with 500 clusters. We train it for 100k steps on 6 A6000 GPUs with a maximum token length of 2,000.
We initialize the model with a pre-trained language model, OPT-1.3B (Zhang et al., 2022). We first pretrain the input embedding layer and the projection layer on AVSR and TTS objectives for 200K steps. Then, we continue training the entire model on a mixture of text and AV speech token dialogue for 5K steps, followed by finetuning for additional 3K steps on AV speech token dialogue only. We use a max token length of 700 on 4 A6000 GPUs.
The audio-visual generator is trained using ground truth AV speech tokens. The token-based speech decoder and length predictor are jointly trained for 450K steps with a batch size of 32. For training the AV token-based face decoder, we employ the reprogramming strategy in (Choi et al., 2023) and train an adapter layer consisting of two layers of transformer encoder to bridge between the AV speech tokens and the corresponding audio features of the TFG model (Prajwal et al., 2020). This allows to leverage the face generation capabilities of the pretrained TFG model without further finetuning the generator and can be applied to any other TFG models. It is trained for 250K steps with a batch size of 256. We additionally incorporate a face enhancer (Wang et al., 2021b) to upsample the generated face video into high resolution.

heading: 5.3 Baselines
text: Since there is no previous method that can directly perform audio-visual spoken dialogue synthesis, we compare with the recently proposed spoken dialogue systems, Speech-GPT (Zhang et al., 2023a) and d-GSLM (Nguyen et al., 2023b). They support only audio speech at both input and output. Additionally, we build a cascade system by integrating a series of off-the-shelf pre-trained models: AVSR (Anwar et al., 2023), LM (Tang et al., 2022), TTS (Casanova et al., 2022), and TFG (Prajwal et al., 2020). Please note the objective of the comparisons with the cascaded method is not to achieve state-ofthe-art performance, but rather to assess the extent to which the performance of the proposed system can be attained through the direct strategy. For a fair comparison, we finetune SpeechGPT and dGSLM on our MultiDialog dataset and we use a dialogue language model (Tang et al., 2022) trained on TopicalChat as the LM of the cascade system.

heading: 6 Results

heading: 6.1 Semantic Evaluation
text: To accurately assess the semantic quality of the generated response, we employ the evaluation strategy used for text-based dialogue language models. We conduct evaluations on the test set of MultiDialog, where the model is prompted to sequentially generate a response for each turn in the conversations. Sample evaluation prompts are illustrated in Figure 3. The generated response is then transcribed into text and compared against the ground truth response to evaluate its semantic quality. As shown in Table 3, compared with the state-of-theart spoken dialogue systems, SpeechGPT (Zhang et al., 2023a) and d-GSLM (Nguyen et al., 2023b), our proposed method performs the best in BLEU, D-1, and D-2 which demonstrates that our method can generate contextually coherent and diverse response. SpeechGPT has the highest PPL because it is trained on an extensive amount of speech data and PEFT-finetuned (Hu et al., 2021) on the MultiDialog, which allows it to generate more fluent speech but fails to match with the reference response as indicated by the lower BLEU score. Also, it requires generating text transcription of the input to generate the response in text first. Notably, our proposed method stands as the first approach to directly recognize and generate response in both audio and visual speech video, without requiring intermediate text generation.
Method FID ↑ LSE-C ↑ LSE-D ↓ SIM ↑ • Cascade System
AVSR + LM + TTS + TFG 30.581 7.041 7.640 0.433

heading: 6.2 Ablation on the Pretraining Scheme
text: We analyze the pretraining scheme used for our audio-visual spoken dialogue model in the lower section of Table 3. The results demonstrate that initializing the model with a textually pretrained LLM yields improved semantic quality, which is further enhanced by AVSR/TTS pretraining. Simply training the embedding layer and projection layer to predict corresponding AV speech tokens and text tokens improves the response. When further incorporating mixed text-AV speech token pretraining, we observe an overall enhancement in semantic quality, validating the effectiveness of gradually adapting the AV speech tokens to the LLM. Yet, there is a slight decrease in the PPL score, which we attribute to the model’s increased complexity and adaptability to multimodal inputs.

heading: 6.3 Audio and Visual Evaluation
text: We evaluate the audio and visual generation quality in Table 4. In terms of speaker voice similarity
(SIM), our proposed method not only outperforms the cascaded system but also surpasses spoken dialogue systems. This demonstrates the effectiveness of our AV token-based speech decoder, enriched with speaker embedding, to retain the speaker information from the reference video. When assessing visual quality, we compared it with the cascaded system that uses the same TFG model (Prajwal et al., 2020) as ours. While our FID score is comparable, our approach exhibits superior audiovisual synchronization, due to the utilization of discretized audio-visual tokens, which provide clearer alignment between the audio and visual components than raw audio.
In Figure 5, we show the generated audio-visual response between the two partners along with transcriptions generated with ASR (Shi et al., 2021). Given a conversation context, our model generates the next response that is contextually coherent and adequate. For example, in Figure 5 (a), it answers the question asked by the user in the previous turn and responds accordingly about the chatting topic, NFL. Also, it successfully synthesizes the speech-relevant movements of the reference face to generate a seamless talking face video. Please refer to the demo for more demonstrations.

heading: 6.4 Robustness to Acoustic Noise
text: In Table 5, we analyze the effectiveness of incorporating additional visual modality into the dialogue system. Following (Shi et al., 2021), we corrupt the input speech with random noise of varying SNR levels (-5, 0, 5, and clean). Compared with audio-
<AI>: I did not know that, that is really cool!
(a)
<User>: Do you watch the NFL? <User>: Me too. Did you know women can play in the NFL too? <AI>: I sure do. How about yourself?
(b)
<User>: Boy, I miss football season. I love ESPN’s Monday night football offerings. What about you?
<User>: Ah, yes. … Did you know ABC is the only one of the 4 major broadcast networks without a prime-time NFL game? <AI>: I don't really watch it though sometimes I inadvertently end up watching Thursday Night Football. <AI>: That is right. I find it interesting that football can burst click for 75 million dollars back in the end.
(c)
<User>: I was unaware that the top bowlers in the 60s made as much as top football stars.
<User>: Yeah, crazy. ESPN won an Emmy for the creation of the superimposed yellow line that represents the 1st down marker. <AI>: Yeah, most football players had regular jobs when they weren't playing so I guess it's not unrealistic for top bowlers to make more than them during the 1960s.
<AI>: I never heard of those, but I can't imagine that.
only input, audio-visual input enhances the robustness of the system as indicated by less degradation of the performance under noise. This is because the visual modality which is not affected by acoustic noise can complement the missing information in the audio modality to better recognize the speech content and output response. It further demonstrates that our system is applicable for real-world use in unstable speech input scenarios.

heading: 7 Conclusion and Limitation
text: We introduce a novel face-to-face spoken dialogue model that directly processes audio-visual speech from the user input and generates audio-visual speech response. This is the first step toward creating a talking face avatar chatbot system, without intermediate text in the generation process. In addition, we release MultiDialog, the largest multi-
modal dialogue dataset to date with tri-modality (i.e., audio, visual, and text) spoken dialogue data. As it is an extensive dataset that captures real human-human conversation covering broad topics, we believe it brings diverse research opportunities for multimodal synthesis, ranging from talking face synthesis to multimodal dialogue language modeling.
One limitation of our work is that, although the dataset includes emotion labels for each utterance, we have not utilized these labels yet. We plan to address this in future research by integrating emotion recognition from users’ facial expressions to generate more emotion-aware responses, both in speech content and nuances of generation. Also, since our data provides parallel recordings of the speaker and the listener, we can simultaneously model the generation of both faces for more spontaneous and natural conversation.

heading: A MultiDialog Dataset
text: A.1 Dataset Statistics
Table 2 shows detailed statistics of MultiDialog. MultiDialog consists of 9,920 human-human conversations, 106,624 turns, 218,248 utterances, totalling to approximately 340 hours of audiovisual dialogue data. A single dialogue contains multiple turns, where each turn includes two utterances. An utterance is an instance of speech by one person followed by silence or another person speaking. In our dataset, a conversation averaged 11.0 turns, 21.9 utterances, 140.2 seconds in length. 12 speakers were paired to record an average of 826.7 dialogues per person.
A.2 Participant Information
Prior to recording our dataset, we received an IRB approval to collect facial video, speech, and text data to build human multimodal dialogue technology. We recruited students at a university who were fluent in English and could fulfill the designated portion of the dialogues. A recruitment notice included general information about TopicalChat, the dataset to be recorded, wage and responsibilities of the participants, and potential effects and contributions of building a multimodal dialogue dataset. After receiving 25 applications, interviews were conducted on all applicants. During the interview, we notified that we will be collecting audiovisual data of the participant during recording sessions, which will be released to the research field in the future. We also collected participant information such as race, sex, nationality and age, agreement to release audiovisual data, and assessed the English fluency and ability to read and act out a given dialogue script with emotions. Two interviewees in charge of the dataset collection selected actors by ranking each participant on a scale of 1 to 5 on each criterion and considering the diversity of participant demographics. Thus, six female and six male actors from six different countries, and age varying from 20 to 30 were selected. Details on participant information are outlined in Table 6.
After all participants were selected, we held an orientation to guide participants on the recording procedure. For a single recording session of three hours, two participants were scheduled to film 50 to 60 conversations in TopicalChat. The number of conversations to film in a session was calculated based on a trial recording session, in which two speakers filmed approximately 60 conversations

heading: Id Gender Age Nationality # dialogues Acc.
text: in a three-hour period, including breaks. Participants learned how to navigate through the dialogue display program to start and end recording conversations, and proceed to the next utterance. The display program showed the conversation script along with the corresponding emotion for each utterance, and the remaining number of conversations to film in the current session. We notified each participant to attach a microphone about 15 to 20 cm from their mouth and adjust the camera to the shoulder level before recording. Lastly, we collected consent forms for providing personal information for compensation and informed consent forms for human subject research participants.
A.3 Annotation Evaluation
We conducted a comprehensive user study involving 25 participants, where we randomly sampled 70 utterances from the dataset and participants predicted the emotions conveyed within each utterance to verify the quality of emotions.
Table 6 includes the accuracy of each actor in conveying the intended emotion in the utterance. Given that real-life conversations often involve subtle and layered emotional expressions, the dataset was designed to mirror this intricacy. Based on previous research (Hu et al., 2018; Wang et al., 2021a) on subtle emotion recognition, the results from our user study underscore the effectiveness of the actors in portraying these subtle emotions. To enhance the quality of the emotion annotations to be used in future research, we filter out recordings from actors that exhibit low prediction scores and release a subset of MultiDialog.
Table 7 is the confusion matrix between emotion categories estimated from the user study, focusing
on results from actors who achieved above 40% emotion accuracy. The result closely aligns with the human innate ability to recognize emotion from audio-visual (Busso et al., 2008), underlining the effectiveness of MultiDialog in conveying emotion within utterances. Certain emotions, such as fearful and sad, exhibited lower accuracy rates, which we attribute to the inherent complexity and subtlety of these emotions in natural conversations (Poria et al., 2018).
A.3.1 Gold Emotion Dialogue Subset We provide a gold emotion dialogue subset in the MultiDialog dataset, a more reliable resource for studying emotional dynamics in conversations. Previous research (Hu et al., 2018; Wang et al., 2021a) indicates that the accuracy rates for recognizing subtle emotions are slightly under 40%. Thus, we classify dialogues from actors that exhibit emotion accuracy above 40% as gold emotion dialogue. We release the gold emotion annotations of actor IDs along with the dataset in https://huggingface. co/datasets/IVLLab/MultiDialog.
A.4 Recording Setup
Fig. 5 shows the studio setup for recording sessions.

heading: B Evaluation Metrics
text: BLEU (Post, 2018) evaluates the fluency and adequacy of generated responses based on n-gram overlap. A higher BLEU score indicates a more natural and engaging dialogue model. PPL (Bengio et al., 2000) measures how well a language model predicts the generated response. A lower perplexity indicates that the model is more confident and accurate in predicting the next word, suggesting higher quality in generating coherent and contextually relevant responses. DISTINCT-n (Li et al., 2016) evaluates the diversity of generated response by calculating the
percentage of unique n-grams in the set of responses. Specifically, D-1 measures the percentage of unique unigrams in the generated text, while D-2 measures the percentage of unique bigrams. METEOR (Banerjee and Lavie, 2005) (Metric for Evaluation of Translation with Explicit Ordering) evaluates the quality of generated response by computing the alignment-based precision and recall between the generated output and the ground truth, considering synonyms and paraphrases. F1 (Banerjee and Lavie, 2005) combines the accuracy of the generated response (precision) and the coverage of the relevant response (recall). It provides a balanced measure of how well the model performs in generating relevant and accurate responses.

heading: Future Work
text: To enhance the quality of the emotion annotations to be used in future research, we filter out recordings from actors that exhibit low prediction scores and release a subset of MultiDialog. Table 7 is the confusion matrix between emotion categories estimated from the user study, focusing
on results from actors who achieved above 40% emotion accuracy. The result closely aligns with the human innate ability to recognize emotion from audio-visual (Busso et al. , 2008), underlining the effectiveness of MultiDialog in conveying emotion within utterances. Certain emotions, such as fearful and sad, exhibited lower accuracy rates, which we attribute to the inherent complexity and subtlety of these emotions in natural conversations (Poria et al. , 2018). A. 3. 1 Gold Emotion Dialogue Subset We provide a gold emotion dialogue subset in the MultiDialog dataset, a more reliable resource for studying emotional dynamics in conversations. Previous research (Hu et al. , 2018; Wang et al. , 2021a) indicates that the accuracy rates for recognizing subtle emotions are slightly under 40%. Thus, we classify dialogues from actors that exhibit emotion accuracy above 40% as gold emotion dialogue. We release the gold emotion annotations of actor IDs along with the dataset in https://huggingface. co/datasets/IVLLab/MultiDialog. A. 4 Recording Setup
Fig. 5 shows the studio setup for recording sessions