VoxDialogue: Can Spoken Dialogue Systems Understand Information Beyond Words?
Authors: Xize Cheng
Ruofan Hu
Xiaoda Yang
Jingyu Lu
Dongjie Fu
Zehan Wang
Shengpeng Ji
Rongjie Huang
Boyang Zhang
Tao Jin
Zhou Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: spoken dialogue system
paralinguistic information
benchmark
Primary area: datasets and benchmarks
TL;DR: A benchmark to evaluate whether a spoken dialogue system can effectively understand information beyond words.

Abstract: With the rapid advancement of large models, voice assistants are gradually acquiring the ability to engage in open-ended daily conversations with humans. However, current spoken dialogue systems often overlook multi-modal information in audio beyond text, such as speech rate, volume, emphasis, and background sounds. Relying solely on Automatic Speech Recognition (ASR) can lead to the loss of valuable auditory cues, thereby weakening the system’s ability to generate contextually appropriate responses. To address this limitation, we propose \textbf{VoxDialogue}, a comprehensive benchmark for evaluating the ability of spoken dialogue systems to understand multi-modal information beyond text. Specifically, we have identified 12 attributes highly correlated with acoustic information beyond words and have meticulously designed corresponding spoken dialogue test sets for each attribute, encompassing a total of 4.5K multi-turn spoken dialogue samples. Finally, we evaluated several existing spoken dialogue models, analyzing their performance on the 12 attribute subsets of VoxDialogue. Experiments have shown that in spoken dialogue scenarios, many acoustic cues cannot be conveyed through textual information and must be directly interpreted from the audio input. In contrast, while direct spoken dialogue systems excel at processing acoustic signals, they still face limitations in handling complex dialogue tasks due to their restricted context understanding capabilities. All data and code will be open source at \url{https://voxdialogue.github.io/}.

Published as a conference paper at ICLR 2025
VOXDIALOGUE: CAN SPOKEN DIALOGUE SYSTEMS
UNDERSTAND INFORMATION BEYOND WORDS?
Xize Cheng' Ruofan Hu Xiaoda Yang'
Jingyu Lu'
Dongjie Fu' Boyang Zhang'
Zehan Wang' Shengpeng Ji' Rongjie Huang'
Tao Jin'
Zhou Zhao
Zhejiang University' chengxize@zju.edu...
Code & Data: http:::/voxdillooggegggttttuut
ABSTRACT
With the rapid advancement of large models, voice assistants are gradually acquir- ing the ability engage in open-ended daily conversations with humans. How- ever, current spoken dialogue systems often overlook multi-modal information in audio beyond text, such as speech rate, volume, emphasis, and background sounds. Relying solely automatic speech recognition (ASR) can lead to the loss of valuable auditory cues, thereby weakening the system's 's ability to gen- propose erate contextually appropriate responses. To address this limitation, VoxDialogue, a comprehensive benchmark for evaluating the ability of spoken di- alogue systems to understand multi-moda. information beyond text. Specifically, we have identified 12 attributes highly correlated with acoustic information be- yond words and have meticulously designed corresponding spoken dialogue test sets for each attribute, encompassing a total of 4.5K multi-turn spoken dialogue samples. Finally, evaluated several existing spoken dialogue models, analyzing their performance on the 12 attribute subsets of VoxDialogue. Experiments have shown that in spoken dialogue scenarios, many acoustic cues cannot be conveyed through textual information and must be directly interpreted from the audio input. In contrast, while direct spoken dialogue systems excel at processing acoustic sig- nals, they still face limitations in handling complex dialogue tasks due to their restricted context understanding capabilities.
INTRODUCTION
Voice assistants (Ji et al. 2024a have rapidly evolved into a focal point of both academic research
and industry innovation, aiming facilitate daily conversations (Li et al. [017] Lee et al., 2023)
and task-oriented dialogues (Budzianowski et al. 2018 Si et al. 2024) with humans. Early itera- tions relied heavily automatic speech recognition (ASR) (Cheng et al. 2023b Fu et al. 2024
Lei et al. 2024 Huang et al.) 2023). combined with dialogue understanding and state management, to support basic, predefined tasks. However, these systems (Hoyl 2018) were constrained by their limited scope and inability to handle open-ended interactions. The advent of large language models (LLMs) (Touvron et al. 2023) with enhanced understanding and reasoning capabilities has revo- lutionized voice assistants, enabling them to engage in more dynamic and unrestricted dialogues with users AT 2024b). This marks a significant departure from their earlier, more constrained functionalities, opening up new possibilities for human-computer interaction.
Yet, despite these advancements, current spoken dialogue systems (Zhang et al., 2023 Xie & Wu 2024 Fang al. 20244 Cheng et al., 2025) often overlook the rich multimodal information em- bedded in audio beyond mere spoken words-such such as intonation, volume, rhythm, and background sounds. Relying solely ASR leads to the omission of valuable auditory cues, diminishing the system's 's ability to generate contextually appropriate responses. For example, a system might fail to adjust its language to match user's emotional state regional accent, such as responding with "Yes, madam" to a female voice or adopting british colloquialisms when detecting a British accent.
"Equal Contribution.
1 "oorreponniigg author.

---

Published as a conference paper at ICLR 2025
Table I: Comparison of spoken language and audio comprehension benchmarks in terms of data types and evaluation dimensions. SL. refers to Spoken Language. while Dig. indicates whether the benchmark evaluates on dialogue tasks. Aud. represents audio comprehension, and Mus. refers to music comprehension. Speaker Info includes attributes such as age (Age), gen- der (Gen), accent (Acc), and language (Lan). Paralinguistic Info covers aspects like emotion (Emo), volume (Vol), speech rate (Spd), speech fidelity (Fid), stress (Str), and non-verbal expres- sions (NVE). Althhough LeBenchmark includes a small amount of conversational data (29 hours out of 2933 hours), it does not evaluate the dialogue tasks. 'Please note that although AirBench can assess spoken language comprehension, its evaluation of conversational ability (AirBench-Chat) is based on text-based interactions and does not address spoken dialogue capabilities.
Types
Evaluation Dimensions
Benchmarks
SL.
Dig.
Aud.
Mus.
Speaker Info
Paralinguistic Info
SUPERB Yang et al
2021
X
(Emo)
SLUE Shon 2022
X
X
X
X
LeBenchmark Evain et al 2021
xi
X
X
X
(Fmo)
AF-Dialogue (Kong at
2024
x
AirBench Yang 2024g)
x3
V
(Age.Gen)
(Emo)
SPO oi
x
X
SD-EVAL et 2024
x
(Age.GennAAc))
(Eimo)
VoxDialogue (ours)
(Age.Gen,Aco.Lam)
(Emo, Wolpo.Vol..po..aa..s..vW
To address these limitations, recent research has shifted towards developing multimodal audio- language models that enhance system comprehension of audio inputs. Emotion2Vec (Ma et al, 2023). trained on vast emotional speech data, stands as the first high-quality pre-trained model for emotion recognition. Qwen-Audio 1/2 (Chu et al. 2023 2024) have been trained exten- sive datasets encompassing over 30 audio-related tasks, enabling them to understand various au- dio types-including speech, audio events, and music. Pushing the envelope further, FunAudi- oLLM (SpeechTeam) 2024) offers full-scene recognition capabilities, detecting non-verbal sounds like laughter and breathing within speech.
As large-scale audio-language models continue to evolve rapidly, the scientific community has in- creasingly recognized the urgent need for comprehensive benchmark to effectively evaluate spoken dialogue systems. While some progress has been made, existing benchmarks often exhibit notable shortcomings. For instance, SUPERB (Yang et al. 202T) the first benchmark specifically designed for spoken language, but it primarily focuses on coarse-grained semantic understanding tasks, over- looking the importance of various acoustic features. Other benchmarks, such as AirBench (Yang) (Kong et al., 2024), delve deeply into audio understanding, but
et al. 2024a) and Audio-Flaming their dialogue content limited to the textual modality, making them unsuitable for evaluating spo- ken dialogue tasks. SpokenWOZ (Si et al.) 2024), though valuable for its real human- numan-computer interaction data, is restricted to taskedriven dialogues and lacks detailed fine- grained labels. To ad- dress more specific attributes of spoken dialogue, SD-EVAL (Ao et al., 2024) shifts the focus to characteristics like gender, age, accent, and emotion, yet its effectiveness is limited by the use of speech utterances that are not derived from dialogue scenarios.
To better benchmark spoken dialogue systems, we analyzed non-textual multimodal acoustic infor- mation that may affect dialogue responses, which can be categorized into three main types: speaker information (age, gender, accent, language), paralinguistic information (emotion, volume, speed, fidelity, stress, and various non-verbal expressions), and background sounds (audio and music). In real-world dialogue scenarios, it is crucial to capture not only the semantic content of the speech but also these acoustic cues to generate more appropriate responses. For example, determining the speaker's age from their vocal tone can help select suitable form of address. We designed tai- lored spoken dialogue synthesis pipeline for each attribute to ensure that the synthesized dialogue data aligns accurately with the corresponding altribute. Leveraging the strong inference capabilities of large language models (LLMs) and high fighliiiiitty text-to- text-to-speech (TTS) synthesis (Ji et al.) 2024bb
Du et al. 2024), constructed the VoxDialogue benchmark, comprising 12 dialogue scenarios specifically tailored to different acoustic attributes. As shown in Figure to the best of our knowl- edge, this is the most comprehensive work focusing acoustic information in spoken dialogue
2

---

Published as a conference paper at ICLR 2025
benchmarks. Based VoxDialogue, evaluated several existing spoken dialogue systems, com- paring the performance of ASR-based dialogue systems and direct dialogue systems across various related tasks. The results demonstrate that ASR-based methods are limited in their ability to understand the diverse acoustic attributes present in spoken dialogues, highlighting the impor- tance of developing large-scale audio-language models. At the same time, existing direct dialogue systems (such as
still exhibit limitations in long-context reasoning, indicating the need for further improvement in their contextual understanding capabilities. All our code and data will be open-sourced Our main contributions are:
We present the first benchmark for evaluating the ability of spoken dialogue systems to under- stand acoustic information beyond speech content, VoxDialogue, which integrates 12 acoustic dimensions, including speaker attributes (age. gender, accent, language), paralinguistic features (emotion, volume, speed, fidelity, stress, non-verbal expressions), and environmental information (audio, music).
We were the first to develop distinct spoken dialogue data synthesis methods tailored for different acoustic attributes. This approach enables large-scale synthesis of spoken dialogue data, support- ing extensive training for spoken dialogue models and endowing them with more comprehensive acoustic understanding capabilities.
We conducted a systematic evaluation of existing spoken dialogue systems, comparing their per- formance in terms of understanding acoustic information, supplemented by a qualitative analysis using a GPT-based metric. Specifically, inspired by the MOS (Mean Opinion Score) evaluation mechanism, we provided GPT with descriptive criteria corresponding to different scores, enabling the evaluation model to more accurately assess each response in terms of both acoustic attributes and content quality.
RELATED WORKS
2.1 SPOKEN DIALOG SYSTEM
With the development of large-scale language models, increasingly powerful spoken dialogue mod- els have emerged, utilizing extensive training corpora for single tasks with LLM-based instructions to achieve comprehensive audio understanding capabilities. SpeechGPT (Zhang et al. 2023) inte- grates discrete speech units into large language models (LLMs), making it speech-centric model. Qwen-Audio 1/2 (Chu et al.) 2023) (024) established the first large-scale, comprehensive audio model for over 30 audio-related tasks. Similarly, Salmonn (Tang et al. 2023) addresses task com- plexity in audio models by introducing more intricate story generation tasks. Additionally, some directly use dialogue databases for training. StyleTalk (Lin et al., 2024b) focused on emotional di- alogue tasks and introduced the first spoken dialogue model capable of generating responses with varying emotional tones. Recent studies (Cheng al., [023c bla Lei al. 2024 Fu et al., [024] Yang et al., 20244b in conversational Al even have begun examining how visual data integration can enhance the contextual awareness of spoken dialogue systems.
However, existing spoken dialogue models (Xie & Wu, 2024 Fang et al. 2024) primarily focus on understanding speech content and audio information, with only few works specifically addressing detailed acoustic attributes within the speech. This oversight results in the loss of crucial information in spoken dialogue, which, as our experiments show, can significantly undermine the quality and effectiveness of response generation in daily dialogue.
2.2 SPOKEN LANGUAGE BENCHMARK
With the rapid development of large-scale audio models (Chu et al., 2024 SpeechTTemm 2024), , the scientific community has increasingly recognized the need for a comprehensive benchmark to eval- uate spoken dialogue systems. While some progress has been made, many existing benchmarks still fall short. For instance, SUPERB (Yang et al. 2021) was the first benchmark specifically de- signed for spoken language, but it primarily focuses on coarse-grained understanding tasks. Audio- Flamingo (Kong et al. 2024) explores audio comprehension but is limited to text-based dialogue data, lacking a focus on evaluating spoken dialogue. AirBench (Yang et al., 2024a) was the first propose using GPT-4 (OpenAl 2024b) to measure spoken dialogue system performance, but its
3

---

Published as a conference paper at ICLR 2025
Table 2: Dialogue example of VoxDialog. The bold black text highlights the acoustic and back- ground information unique to spoken dialogue, while the blue text emphasizes responses that depend on these information.
Attributes Example
I. Speaker Information
A: Excuse me, I'm looking for reliable smartphone that suits my needs. (Man Voice)
Gender
B: Certainly, Sir! ... excellent performance and long-lasting battery, perfect for business tasks,
A: What dishes do you recommend? (Young Voice)
Age
B: Sure! might enjoy our specialty heef burger, it's fantastic with fresh juice.
A: Could you help with my schedule? (British Accent)
Accent
B: course, assist you in organizing your timetable.
A: AAT (Speaking in Chinese)
Language
B: 4イス B....
II. Paralinguistic Information
A: Can you recommend good movie? (Sad Tone)
Emotion
B: get it. Maybe Soul-it's emotional and might bring you some comfort.
A: Can you tell me how to solve this problem? (Loud Volume)
Volume
B: Could you speak more quietly? People around are studying.
A: I'd like get the burger, soda, and also the pasta, please. (Speaking Quickly)
Speed
B: Sorry, could you say that more slowly? didn't i catch everything you said.
A: Can you tell the time of the meeting? (Poor Audio Quality)
Fidelity
B: I'm having trouble hearing you. Could you improve the audio quality?
A: really don't like making sushi. (Emphasis "making sushi"))
Stress
B: Oh, see. If you don't 't like making sushi, what other types cooking do you enjoy?
Non-verbal A: Could you help me move this box? (With a sigh before speaking)
Expressions B: Are you feeling okay? It seems like you' 're really tired. I can take care of it for you.
III. Environmental Information
Audio
What was that sound just now? (Background sound: airplane engine sound, explosion sound)
Events
That was loud explosion. It sounded like the plane exploded. Hope one was hurt.
A: Hey, what instrument is this song played on? (Music: Piano Song, Sad Song)
Music
B: It should be the piano, it sounds so sad.
evaluation set remains constrained to QA interactions. SpokenWOZ (i et al. 2024) is a large-scale task-oriented dataset that offers real human interaction data, making it valuable for evaluating task- driven dialogue systems. SD-Eval (Ao et al.) (024). which emphasizes acoustic attributes such as gender, age, accent, and emotion, uses raw audio from confessional-styee corpora, making less suitable for conversational scenarios.
However, due to the challenges associated with collecting spoken dialogue data in specific scenarios, existing benchmarks are unable to effectively evaluate whether spoken dialogue systems can under- stand various information beyond words. To address this limitation, we developed VoxDialogue, a benchmark built using synthetic data that focuses 12 acoustic dimensions that significantly influence dialogue content. These dimensions include speaker information (age, gender, accent, lan- guage), paralinguistic information (emotion, volume, speed, fidelity, stress, non-verbal expressions), and environmental information (audio events, music). Ultimately, VoxDialogue enables a compre- hensive evaluation of the ability of current spoken dialogue systems to process and interpret such detailed acoustic information.
VOXDIALOGUE
3.1 OVERVIEW
Spoken dialogue systems typically used in daily dialogues (Lin et al. 2024a). As shown in Table 2. we evaluate the performance of spoken dialogue systems across these three categories daily dialogue scenarios. Beyond understanding the speech content, spoken dialogue systems must also
4

---

Published as a conference paper at ICLR 2025
generate the most appropriate responses by considering the speaker's emotions, gender, and other acoustic-related information. Therefore, unlike traditional text- based dialogue benchmarks (Li al. 2017), we systematically analyze the acoustic characteristics that may influence response content and have developed a tailored evaluation set specifically for spoken dialogue systems. The evalua- tion set for daily dialogue is divided into the following categories: I. Speaker Information. (1) Age: Responses should be tailored to the speaker's 's age, adjusting salutations (e.g. Mrs. or suggest- ing content appropriate for their age group. (2) Gender: Responses should be gender-specific. mod- ifying salutations (e.g., Mr./Mrs.) or offering preferences based on gender. (3) Accent: Responses should account for the speaker's accent, selecting vocabulary that aligns with their speech (e.g,, British people may be more accustomed to using 'timetable' instead of 'schedule").
(4) Language: Responses should be adapted to the speaker's language, choosing the most appropriate language for the response. II. Acoustic Information. (5) Emotion: Responses should detect the speaker's s emo- tional state and provide a suitable reply (c.g., suggesting comforting music when sensing distress). (6) Volume: Responses should consider the speaker's ss volume, asking them to lower or raise their voice (e.g., requesting quieter speech in quiet environments). (7) Speed: Responses should adjust to the speaker's speech rate, asking them to slow down clarify if speaking too quickly for com- prehension. (8) Fidelity: Responses should detect poor audio quality and ask the speaker to repeat or improve the clarity of their speech for better understanding. (9) Stress: Responses should recog- nize emphasis on specific words and tailor replies to focus on the stressed content. (10) Non-verbal Expressions: Responses should account for non-verbal cues such as sighs, detecting emotions like tiredness or frustration, and offering assistance accordingly. III. Background Sound. (11) Au- dio Event: Responses should recognize relevant audio events and adapt accordingly. (12) Music: Responses should adjust to the type and mood of the background music.
3.2 SPOKEN DIALOGUE GENERATION
Dialogue Script Synthesis. Building on the methodology of previous studies (Lin et al,, 2024a Cheng et al. 2025), we employed large language models with advanced reasoning capa- bilities to synthesize spoken conversation scripts tailored to diverse scenarios and acoustic condi- tions. Specifically, we utilized GPT-4o (OpenAT 2024a) to pre-generate several rounds of historical conversations, followed by the generation of contextually appropriate responses under various con- trolled acoustic conditions. This approach ensures that the synthesized dialogue scripts capture a wide range of acoustic features, thereby enhancing their robustness and diversity.
Stagel:
Stage2: Spoken Dialogue Generation. We carefully tailored the most appropriate speech syn- thesis method for each attribute during the generation process We designed tailored spoken dia- logue synthesis pipeline for each attribute to ensure that the synthesized dialogue data aligns accu- rately with the corresponding attribute: (1) Gender, Speed and Emotion. We use CosY VOICE- 300M-INSTRUCT]] to achieve condition speech generation based on gender and emotion by adjust- ing style instructions. (2) Stress, Language, and Non-verbal Expressions. We achieved control over these aspects by adjusting the text content in the 3000- INSTRUCT (Stress, Non- verbal Expressions) and VOICE- SFT (Language). adding < stress >< /stress >, [aughterr), or changing the language of the text. (3) Volume, Fidelity, Audio Events, and Music. We used COSYVOICE-300M--3
to generate the basic speech, then applied post- post-processing tech- niques to fine-tune these specific attributes. The details of post-processing are shown in Stage 4. (4) Age. We randomly selected 1,000 speaker samples of different ages from Hechmi et al. (202T) and Tawara et al. (2021) reference timbres and used VOICE-OOI0300M for zero-shot TTS synthe- sis. (5) Accent. We used the industrial- grade TTS tool (edge-TTrs T which offers over 318 timbre references spanning various regions, languages, and genders to achieve precise accent generation.
Stage3: Automatic Verification for Spoken Dialogue. To ensure the quality of the synthesized spoken dialogue data, we first employed pre-trained model to automatically filter out unqualified samples, removing those with generation errors and inconsistent timbre. Specifically, we used the Whisper model (Radford 2023) to filter out all sentences with a word error rate (WER) greater
http:///waguuutrc.ecc////dldddss Instruct
http:///haiiiniee....oo///ooo/pp 300X-5FFF
5

---

Published as a conference paper at ICLR 2025
belle
Leve
neneaa sure
aughter
Abeöl
knds
right
Yean
eaa
take
2ood
peed
Love
S
S
bit
sound
ereat On
ee
ti me
khow
something n Totally def feilt
grina
(a) Word Cloud of VoxDialogue.
(d) Distribution of Each Attribute.
400
20, 11,00%
8.6.00%
(b) Duration Distribution Turns.
6
10
(c) The Duration Distribution of Dialogues.
(e) Distribution multi-turn dialogue.
Figure I: Visualization of static analysis of VoxDialogue.
than 5%, and applied speaker-diarization-3.. (Plaquet & Bredin 2023 Bredin 2023) to eliminate samples with timbre inconsistencies in speeches of the same speaker throughout dialogue sequence.
Stage4: Post- Post-processing for Specific Acoustic Attributes. For attributes such volume, fidelity, audio events, and music, we performed post-processing to ensure that the audio aligns with the re- quired expectations. For fidelity, according the Kyquist-Shannon sampling theorem, the sampling rate must be at least twice the highest frequency of the signal to ensure lossless reconstruction. To capture frequencies up 4 kHz, the minimum sampling rate should be kHz. Therefore, we downsampled the speech to 4 kHz (to simulate the loss of speech signal and represent 'poor' audio quality, resulting in the loss of some speech information) and then resampled it back to 16 kHz to simulate poor audio fidelity. For volume, dialogue turns labeled as 'loud' were amplified to simulate by increasing the power 8-fold. For dialogue turns labeled as 'low', the audio power was reduced to 50% of its original level to simulate poor microphone reception. For audio events, a large language model is used to classify events as either temporary or continuous. Temporary audio events, such as a door slamming a phone ringing, are brief sounds that occur momentarily and are spliced before the first voice segment. In contrast, continuous audio events, like background chatter or street noise, prolonged and looped as background sound throughout the conversation. For music, randomly spliced before the first speech segment or set to play in a loop as background sound.
Stage5: Human Verification. While large language models (LLMs) are effective following instructions and generating coherent conversation samples, they are primarily trained on text data and lack exposure human spoken conversations. As result, the automatically generated data may exhibit unnatural characteristics. To ensure the naturalness and logical consistency of the spo- ken conversation sample pairs with the audio features, we employ human annotators for additional quality checks.
3.3 DATASET STATISTICS
Distribution of Attribute Categories. As shown in Figure (d), the distribution of attribute cat- egories in VoxDialogue is balanced, allowing for comprehensive evaluation of spoken dialogue systems' understanding and dialogue capabilities across various acoustic attributes. In Figure [[ (a), also present word cloud of VoxDialogue, where it is evident that the dataset primarily

---

Published as a conference paper at ICLR 2025
Table 3: Detailed statistics the corresponding subsets of each attribute in VoxDialogue. Gray fonts indicate that samples of this attribute are included in other subsets. IN (India), CA (Canada), ZA (South Africa), GB (United Kingdom), SG (Singapore), US (United States), and AU (Australia). Turns represents the total number of turns in each subset, Dialog. indicates the number of dialogues each subset, Avg denotes the average number turns per dialogue each subset, and Dur. refers to the total duration (in hours) of all dialogues in each subset.
Attributes
Categories
Turns Dialog, Avg Dur
I. Speaker Information
Gender
Male, Female
2040
340
6.0
3.17
Age
Youth (15-30), Middle-Aged (30-60), Elderly (60+)
3096
447
6.9
6.05
Accent
IN, CA, ZAA, GB, SG, US, AU
1440
240
6.0
2.20
Language
Chinese, English
2892
482
6.0
3.51
II. Paralinguistic Information
Emotion
Neutral, Happy, Sad, Angry, Surprised, Fearful, Diagusted
1980
330
6.0
2.41
Volume
Loud Volume, Low Volume, Normal Volume
1824
304
6.0
2.08
Speed
High Speed, Low Speed, Normal Speed
2184
364
6.0
2.93
Fidelity
Low Fidelity, Normal Fidelity Stress, No Stress
2196
366
6.0
3.36
Stress
2354
392
6.0
2.51
NVE
Laughter, No Laughter
2046
341
6.0
3.68
III. Environmental Information
The caption of different audio.
Audio
5000
500
10.0 5.25
(e.g.. The wind is blowing and rustling occurs.)
The aspect list of different music pieces.
Music
3734
420
8.9
5.42
(e.g., [steeldrum, higher register, amateur recording))
Overall
30.7K
4.5K
6.8
42.56
sists of daily dialogue, featuring large number of natural spoken words such as "yeah," which representative of daily spoken interactions. This makes it suitable for assessing the performance of spoken dialogue systems real-world dialogue scenarios. Additionally, the dataset contains nu- merous acoustically relevant keywords, such as "heard," "loud," and "sound," further supporting the evaluation of acoustic-related aspects of dialogue understanding.
Distribution of Dialogue Turns and Duration. All dialogues in our dataset multi-turn dia- logues. In Figure | (e), we show the distribution of dialogue turns, with the majority consisting of 6 turns and maximum of 10 turns. This allows for comprehensive evaluation of spoken di- alogue systems' ability to understand contexts of varying lengths. In addition, Figures (b) and [ (c) illustrate the distribution of each turn and the overall dialogue length, respectively, showing that most sentences are approximately seconds long. This implies that the system must understand the context and reason effectively before generating a response.
Statistics for Subset of Each Attribute. We present the detailed statistics of each attribute in VoxDialogue Table 3 covering 35 different categories across 12 attributes. The average number of turns per dialogue exceeds 6, with each attribute containing more than 300 dialogues, ensuring comprehensive reflection of dialogue capabilities.
4 BENCHMARK FOR SPOKEN DIALOGUE SYSTEM
4.1
TASK DEFINITION
The task of a spoken dialogue system to generate appropriate responses based on the contextual information from the sequence of human dialogue (c.g., the user's utterance sequence) and the pre- ceding assistant response sequence, where the total number of dialogue turns is denoted by t. The goal of the spoken dialogue system is generate the most suitable response based on the previous

---

Published as a conference paper at ICLR 2025
Age
Gender
Accent
Language
Emotion
Volume
Speed Altributes
Fidelity
Stress
N'VE
Audio
Muscc
(a) Comparison of BLEU Across Methods and Attributes.
25
20
10
Age
Language
Emation
Ridelity
Audio
Altributes
(h) Comparison of ROUGE-L Across Methods and Attributes.
35
Shar Recidd
20
10
Age
Gender
Accent
Language ge
Emotion
Volume
Speed
Fidelity
Stress
Audio
Music
(c) Comparison of METEOR Across Methods and Attributes.
90
88
84
80
78
76
74
Age
Gender
Accent
Language
Emation
Volume Attri buttes
Speed
Ridelity
Stréss
NVE
Audio
Music
(d) Comparison of BERTScore Across Methods and Attributes.
Figure 2: The comparison of spoken dialogue performance across 12 different attribute-specific test sets on the VoxDialogue dataset.
l utterances the / 1 historical replies. In our work, we evaluate the performance the spoken dialogue system by focusing solely on the final utterance of each dialogue.
4.2 EVALUATION METRICS
To assess the model's performance, we conducted separate tests on subset of Voxdialogue. Draw- ing on previous research (Ao et al. 2024), we utilized both quantitative and qualitative metrics for comprehensive evaluation. The quantitative evaluation focused on two key aspects: content and style. For content evaluation, we employed widely recognized text generation metrics, including vocabulary-level measures such BLEU (Papineni al. 2002), ROUGE-L (Lin 2004). and ME- TEOR (Banerjee Lavie 2005), alongside semantic-level metrics like BERTScore (Zhang et al.) 2019
For style evaluation, calculated the weighted F1 score of speech sentiment.
In addition to these quantitative assessments, we conducted a qualitative analysis using GPT-based metric (Yang al. 2024a). The meaning of each score is as follows: 1: Contextually relevant but lacks attribute information.
2: Partially relevant to the context but feels unnatural, with no attribute information. 3: Partially relevant the context, with mention of the attribute. 4: Contextually
8

---

Published as a conference paper at ICLR 2025
Table 4: GPT-based Metric Comparison of Different Spoken Dialogue Models on VoxDialogue.
Speaker Info
Paralinguistic Info
Env Info
Method
Age Gen Ace Lan Emo Vol Spd Fid Str NVE Aud Mus
ASR- BaR-Based Spoken Dialogue System
FunAudioLLM
SpecchTeam
2024
4.32 4.39 3.57 4.61
4.09
1.82 1.92 1.79 3.13
2.87
3.47
3.59
Direct Spoken Dialogue System
(Kong et al.
2024
1.00 1,00 1.04 1.72 1.00 1.20 1.14 1.26 1.34
3.06
1.37
1.11
SALMONN Tang et
2023
1.99 1.64 1.78 3.50 1.84 2.88 2.27 2.29 3.86
2.59
2.15
2.23
Qwen-Audio (Chu et al.
2023
1.36 1.04 1.28 1.04
1.06
1.48 1.08
1.32
2.49
2.65
1.42
1.18
Qwen2-Audio
2024
3.46 4.18 2.71 4.43 3.73 3.06 3.29 2.98 3.93
3.46
3.81
3.98
relevant and natural, mentioning the attribute, but could be improved. 5: Contextually relevant, smooth, natural, and accurately addresses the attribute. We have included all the evaluated prompt templates in supplementary materials. Please refer the supplementary materials for more details.
4.3 SPOKEN DIALOGUE SYSTEM
In order to build a comprehensive benchmark, we evaluated two main types of spoken dialogue sys- tem approaches: (1) ASR-based dialogue systems (e.g., L (Fang et al. 2024)) and
(2) direct spoken dialogue systems (c.g.. Audio-Flamingo (Kong et al. 2024), SALMONN TTag
al. 2023). Qwen-Audio Instruct (Chu al., 2023 and Qwen2-Audio Instruct (Chu al. 2024)). ).
Figure 2 presents a comparative analysis using four metrics across various attributes on the VoxDi- alogue dataset. Based on the experimental results, we gained the following key insights:
ASR-based systems excel in context-sensitive tasks. In attributes that can be inferred through context understanding, ASR-based systems (such as FunAudioLLMM) show significant advantages. ASR systems first transcribe speech into text and then process it, allowing them to more effectively capture and analyze the context of conversation. For example, attributes like Emotion and Speaker Information(Age, Gender, Accent, Language), FunAudioLLM consistently outperforms direct spoken dialogue systems. The results from BLEU, ROUGE-L, METEOR, and BERTScore metrics indicate that LunAudioLL.M achieves higher scores, such in emotion (3.22 BLEU, 14.93 ROUGE-L, 19.31 METEOR, 86.92 BERTScore). This proves that most current direct spoken dia- logue systems lack adequate context understanding capabilities and are far weaker than text-based large language models. Additionally, although ASR-based models may have limitations in under- standing acoustic information, comparing them provides a valuable performance reference, repre- senting the upper bound performance without the integration of acoustic information.
Advantages of direct spoken dialogue systems in acoustic attribute processing. Although ASR- based systems can leverage the strong context understanding capabilities of large language models, they struggle with attributes that heavily rely on sound understanding (such as volume, fidelity, speed, and other paralinguistic information). ASR-based methods face challenges when addressing dialogue tasks related to these attributes. In contrast, direct systems like Qwen2-Audio excel in tasks involving these acoustic properties. The results show that Qwen2-Audin outperforms other systems in these categories. For instance, Owen2-AAdio achieved the highest scores for volume (4.36 BLEU, 16. ROUGE-L, 22.82 METEOR. and 87. $7.99 BERTScore), demonstrating its ability to handle loud and soft speech variations more effectively. Similarly, fidelity is another strong point for direct dialogue systems. Qwen2-Audio's 's excellent performance in handling varying fidelity levels (3.38 BLEU, 14.36 ROUGE-L, 12.78 METEOR, 85.66 BERTScore) confirms that spoken dialogue tasks, which heavily rely on acoustic information beyond words.
4.4 QUALITATIVE COMPARISON
Inspired by Yang et al. (2024)), we also attempted to use GPT-4 (OpenAl 2024b) for evaluation, focusing on whether the responses exhibit the specific attribute characteristics and whether they
Alll models used in the evaluation are -chat version.

---

Published as a conference paper at ICLR 2025
provide reasonable replies the previous context. As shown in Table 4 present the qualitative testing results of different methods across 12 attributes. Specifically, a score of represents mention of attribute information, 4 represents reasonable and natural response.
We observed that the conclusions from the qualitative tests largely align with those from the quan- titative evaluations. For context-driven attributes (such speaker information and emotion), ASR- based dialogue models continue to demonstrate the best performance. However, for attributes that highly dependent on acoustic information (such speed, fidelity, audio, and music), direct spo- ken dialogue models like Qwen2-Audio significantly outperform FunAudioLLMM underscoring the importance of developing direct spoken dialogue models.
Additionally, we found that Qwen-Audio often responds with descriptive sentences related to the query, which severely affects performance. The SALMONN model frequently repeats parts the query, leading to higher quantitative scores in some attributes (e.g.. BLEUScore of 87.53 for Stress, 0.53 higher than Qwen2-Audio, but its qualitative performance inferior to Qwen2-Aus Owen2-Audio (with GPT-4-based metric score 0.97 lower). This indicates that most current large audio-languas models are focused on QA-style interactions, and are not yet well-suited for dialogue-style conversations.
ETHICAL DISCUSSION
Our dataset incorporates certain attributes that may introduce bias (e.g., gender) as dimensions to evaluate the model's ability to process diverse acoustic information. However, this introduces po- tential risks of unfairness, such as biased stereotypical responses. In Appendix C fairness challenges faced by spoken dialogue models.
we outline the
To promote the development of unbiased spoken dialogue systems, conducted manual filtering of all potentially sensitive data to ensure that the dataset complies with [oolins & Clément (2012) and excludes examples that could cause harm due attribute-related biases. Looking ahead, we committed to proactively identifying and addressing these challenges, contributing to the creation of fairer and more inclusive spoken dialogue systems. Furthermore, we pledge to continually update this work to advance the development of equitable conversational Al.
6 CONCLUSION
In this work, we introduced VoxDialogue, a comprehensive benchmark designed to evaluate spo- ken dialogue systems' ability to understand information beyond words. By identifying 12 critical attributes tied acoustic cues such as speech rate, volume, emphasis, and background sounds, we constructed challenging test set of 4.5K multi-turn dialogue samples. Our experiments demon- strated that while ASR-based systems excel at context understanding and textual interpretation, they fail to capture important acoustic signals that essential for contextually appropriate responses. In contrast, direct spoken dialogue systems outperform ASR-based models in processing acoustic properties, but their limited ability to understand complex dialogue contexts remains significant shortcoming. The findings highlight the importance of acoustic information in enhancing the perfor- mance of spoken dialogue systems and reveal the current limitations in both ASR-based and direct spoken dialogue models.
REPRODUCIBILITY STATEMENT
All of our data, code, and model weights will be open-sourced.
Section 3 provides detailed instructions on the construction VoxDialogue, including a comprehensive list of all relevant open-source resources.
Section 4.1 outlines the detailed task definitions.
Section
4.2 elaborates on the evaluation metries and specific details.
All of our prompt templates included in the Supplementary Material.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by National Natural Science Foundation of China under Grant No. 62222211 and No.624B2128.
REFERENCES
Junyi Ao, Yuancheng Wang, Xiaohai Tian, Dekun Chen, Jun Zhang, Lu Lu, Yuxuan Wang, Haizhou Li, and Zhizheng Wu. Sd-eval: A benchmark dataset for spoken dialogue understanding beyond words. arXiv preprint arXiv:2406. 13340, 2024.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Hervé Bredin. pyannote audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. In Proc. INTERSPEECH 2023, 2023.
Pawel Budzianowski, Tsung- Hsien Wen, Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5016-5026, 2018
Xize Cheng, Rongjie Huang, Linjun Li, Tao Jin, Zehan Wang, Aoxiong Yin, Minglei Li, Xinyu Duan, Zhou Zhao, et al. Transface: Unit-based audio-visual speech synthesizer for talking head translation. arXiv preprint arXiv:23 151977 2023a.
Xize Cheng, Tao Jin, Rongjie Huang, Linjun Li, Wang Lin, Zehan Wang. Ye Wang, Huadai Liu, Aoxiong Yin, and Zhou Zhao Mixspeech: Cross-modality self-learning with audio-visual stream mixup for visual speech translation and recognition. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 15735-15745, 2023b.
Xize Cheng, Tao Jin, Linjun Li, Wang Lin, Xinyu Duan, and Zhou Zhao. Opensr: Open- Open-modality speech recognition via maintaining multi-modalit alignment. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6592- 6607, 2023c.
Xize Cheng, Dongjie Fu, Xiaoda Yang, Minghui Fang, Ruofan Hu, Jingyu Lu, Bai Jionghao, Zehan Wang, Shengpeng Ji, Rongjie Huang, et al. Omnichat: Enhancing spoken dialogue systems with scalable synthetic data for diverse scenarios. arXiv preprint arXiv:2501.01384 2025.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07912 2023.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint axXiv:2407.10759 2024.
Katherine A Collins and Richard Clément. Language and prejudice: Direct and moderated effects. Journal of Language and Social Psychology, 31(4):376-33 396, 2012.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma. et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based supervised semantic tokens. arXiv preprint 2407::4477005000 2024.
Solène Evain, Ha Nguyen, Hang Le, Marcely Zanon Boito, Salima Mdhaffar, Sina Alisamir, Ziyi Tong. Natalia Tomashenko. Marco Dinarelli, Titouan Parcollet, et al. Lebenchmark: A repro- ducible framework for assessing self-supe representation learning from speech. In INTER- SPEECH 2021: Conference of the International Speech Communication Association, 2021.
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. arXiv preprint arXiv:2409.06mm 2024.
11

---

Published as a conference paper at ICLR 2025
Dongjie Fu, Xize Cheng, Xiaoda Yang, Wang Hanting, Zhou Zhao, and Tao Jin. Boosting speech recognition robustness to modality modality-distorion distortion with contrast- -auirrate- prompts. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 3838-3888300 3888, 2024.
Khaled Hechmi, Trung Ngo Trong, Ville Hautamäki, and Tomi Kinnunen. Voxceleb enrichment for age and gender recognition. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 687-693. IEEE, 2021.
Matthew B Hoy. Alexa, siri, cortana, and more: an introduction to voice assistants. Medical refer- ence services quarterly, 37(1):81-88, 2018.
Rongjie Huang. Huadai Liu, Xize Cheng, Yi Ren, Linjun Li, Zhenhui Ye, Jinzheng He, Lichao Zhang, Jinglin Liu, Xiang Yin, et al. Av-transpeech: Audio-visual robust speech-to- trans- lation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pp. 8590-8604, 2023.
Shengpeng Ji, Yifu Chen, Minghui Fang, Jialong Zuo, Jingyu Lu, Hanting Wang, Ziyue Jiang, Long Zhou, Shujie Liu, Xize Cheng. al. Wavchat: survey of spoken dialogue models. arXiv preprint arXiv:2411.13577 2024a.
Shengpeng Ji, Jialong Zuo, Wen Wang, Minghui Fang, Siqi Zheng, Qian Chen, Ziyue Jiang, Hai Huang, Zehan Wang, Xize Cheng, et al. Controlspeech: Towards simultaneous zero-shot speaker cloning and zero-shot language style control with decoupled codec. arXiv preprint 2:Xiv:2406.01205 2024b.
Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: novel audio language model with few-shot learning and dialogue abilities. arXiv preprint arXiv:2402.0/83/ 2024.
Kcon Lee, Kyumin Park, and Dacyoung Kim. Dailytalk: Spoken dialogue dataset for conversational text-to-speech. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Songju Lei, Xize Cheng, Mengjiao Lyu, Jianqiao Hu, Jintao Tan, Runlin Liu, Lingyu Xiong, Tao Jin, Xiandong Li, and Zhou Zhao. Uni-dubbing: Zero-shot speech synthesis from visual articulation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 10822-1099,, 2024.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 986-995, 2017.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Guan-Time Ting Lin, Cheng- Han Chiang, and Hung-yi Lee. Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. In Lun-Wer Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedingy of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6626 6642, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. URL httpss://aclaamhhoooyy org/2024. acl1- long. 358
Guan-Ting Lin, Cheng-Han Chiang, and Hung-yi Lee. Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. arXiv preprint arXiv:2402.12786, 2024b.
Haochen Liu, Jamell Dacon, Wengi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. Does gender matter? towards fairness dialogue systems. arXiv preprint arXiv: 1910. 10486, 2019.
Xubo Liu, Egor Lakomkin, Konstantinos Vougioukas, Pingchuan Ma, Honglie Chen, Ruiming Xie, Morrie Doulaty, Niko Moritz, Jachym Kolar, Stavros Petridis, et al. Synthvsr: Scaling up visual speech recognition with synthetic supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18806-18815, 2023.
12

---

Published as a conference paper at ICLR 2025
Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. emotion2vec: Self-supervised pre-training for speech emotion representation. arXiv preprint arXiv:2312.15185. 2023.
OpenAI. Gpt-4o system card.
2024a.
OpenAl. Chatgpt can now see, hear, and hear-and-speak/, 2024b.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th anmual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Alexis Plaquet and Hervé Bredin. Powerset multi-class cross entropy loss for neural speaker diariza- tion. In Proc. INTERSPEECH 2023, 2023.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning. pp. 28492-28518. PMLR, 2023.
Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu Han. Slue: New benchmark tasks for spoken language understanding evaluation natural speech. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7927-7931. IEEE, 2022.
Shuzheng Si. Wentao Ma, Haoyu Gao, Yuchuan Wu, Ting-En Lin, Yinpei Dai, Hangyu Li, Rui Yan, Fei Huang, and Yongbin Li. Spokenwoz: A large-scale speech-text benchmark for spoken task-oriented dialogue agents. Advances in Neural Information Processing Systems, 36, 2024.
Tongyi SpeechTeam. Funaudiollm: Voice understanding and generation foundation models for natural interaction between humans and llms. arXiv preprint arXiv:2407.04051/ 2024.
Hsuan Su, Rebecca Qian, Chinnadhurai Sankar, Shahin Shayandeh, Shang- Chen, Hung-yi Lee, and Daniel M Bikel. Step by step to fairess:: Attributing societal bias in task-oriented dialogue systems. arXiv preprint arXiv:23/1 1.0553,, 2023.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv 13289,
preprint arXiv:2310. 13289, 2023.
Naohiro Tawara, Atsunori Ogawa, Yuki Kitagishi, and Hosana Kamiyama. Age-vox-celeb: Multi- modal corpus for facial and speech estimation. In ICASSP 2021-2021 IEEE International Con- ference Acoustics, Speech and Signal Processing (ICASSP), pp. 6963-6967. IEEE, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408. 16725, 2024.
Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. AlR-bench: Benchmarking large audio-language models via generative comprehension. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 1979-1998, Bangkok, Thailand, August 2024a. Association for Com- putational Linguistics. URL
/ttpaestttttttttpte/saeeeaaa httttp:://aclaathlo..ooo wwwpettwwwwoooooooooooooooooooee 2024 acl long. 109
Shu Wen Yang, Po Han Chi, Yung Sung Chuang, Cheng Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan Ting Lin, et al. Superb: Speech processing univer- sal performance benchmark. In 22nd Annual Conference of the International Speech Communi- cation Association, INTERSPEECH 2021, pp. 3161-3165. International Speech Communication Association, 2021.
13

---

Published as a conference paper at ICLR 2025
Xiaoda Yang, Xize Cheng, Dongjie Fu, Minghui Fang, Jialung Zuo, Shengpeng Ji, Zhou Zhao, and Jin Tao. Synctalklip: Highly synchronized lip-readable speaker generation with multi-task learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 8149- 8158, 2024b.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abil- ities. arXiv preprint arXiv:2305. 1000, 2023.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger. and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint 1rXiv:1104..000 09675, 2019.
14

---

Published as a conference paper at ICLR 2025
Table 5: Detailed Comparison of Spoken Dialogue Systems across Various Metries
(a) BLEU Scores
Speaker Info
Paralinguistic Info
Background
Method
Age
Gen
Acc
Lan
Emo
Vol
Spd
Fid
Str
NVE
Aud
Mus
LMMAudiOLLM
2.53
2.66
3.34
2.72
3.22
4.20
2.77
2.65
3.58
2.37
3.34
3.24
Audio-Flaminge
2.08
2.40
2.83
0.01
2.74
3.95
2.70
2.50
2.58
1.41
3.38
2.81
Qwen-Audio
2.26
2.56
3.05
1.74
3.01
3.78
2.61
0.54
3.02
2.85
3.60
2.87
SALMONN
2.29
2.35
2.88
3.09
2.88
4.44
2.73
2.82
2.33
2.04
3.55
2.86
Qwen2-Audh
2.22
2.52
3.20
3.18
3.11
4.56
2.92
3.38
2.93
2.10
2.97
2.97
(b) ROUGE-L Scores
Speaker Info
Paralinguistic Info
Background
Method
Age
Gen
Acc
Lan
Emo
Vol
Spd
Fid
Str
NVE
Aud
Mus
LuMAudioLLM
12.15
12.95
15.07
15.88
14.93
8.28
7.97
4.47
13.49
10.67
12.01
11.97
Audio-Flamingo
6.12
6.15
6.62
0.03
5.78
5.48
7.67
7.57
5.12
7.41
5.91
7.88
Qwen-Audio
8.34
9.62
7.12
12.09
8.24
0.71
6.61
14.36
7.76
12.36
7.29
9.01
SALMONN
11.52
11.43
10.51
14.80
11.81
13.30
10.56
10.22
15.71
11.01
10.05
10.51
Qwen2-Audio
11.51
11.44
13.18
15.66
14.18
23.13
17.34
9.58
13.45
11.36
12.23
12.18
(c) METEOR Scores
Speaker Info
Paralinguistic Info
Background
Method
Age
Gen
Acc
Lan
Emo
Vol
Spd
Fid
Str
NVE
Aud
Mus
FunAudioLLM
16.89
20.12
21.03
15.21
19.31
10.19
9.83
8.16
16.95
10.31
12.91
12.42
Audio-Flamingo
8.23
7.79
10.03
0.25
9.17
8.31
8.69
11.04
8.12
7.88
9.93
11.01
Qwen-Audio
12.87
14.16
12.92
11.06
13.12
1.41
5.28
6.11
10,92
21.41
11.92
11.68
SALMONN
11.02
10.81
11.21
10.35
11.13
11.78
10.14
10.17
11.84
9.03
11.18
11.08
Qwen2-Audio
12.96
16.15
18.24
14.37
17.05
22.11
19.08
12.78
14.01
13.11
13.21
13.39
(d) BERTScore
Speaker Info
Paralinguistic Info
Background
Method
Age
Gen
Ace
Lan
Emo
Vol
Spd
Fid
Str
NVE
Aud
Mus
FunAudioLLM
86.14
86.65
87.24
86.97 86.90
84.87
85.03 84.36 87.51
83.79 86.02
86.19
Audio-Flamingo
83.
83.84
83.86 75.28 83.78 84.91
84.71 84.81 83.78
82.89 83.78
84.74
Qwen-Audio
83.40
84,46
83.84
85.79
84.34
85.53
85.34
87.12
79.55
83.85
83.95
84.14
SALMONN
84.60
86.75
86.65
86.44
86.05
87.27
86.06
86.74
87.53
84.92
85.63
86.06
Qwen2-Audio
85.59
85.80
86.70
86.50
86.65
88.00
87.30
85.66
87.08
84.51
87.22
87.19
A MORE EXPERIMENT RESULTS
A. THE DETAILED PERFORMANCE COMPARISON
For comparison, the detailed performance corresponding to Figure 2 presented in Table 5
B LIMITATION
Our work heavily relies on synthetic datasets. Although prior research (Liu et al. 2023) has shown that synthetic data be effectively used for training and evaluation, domain gap persists between
15

---

Published as a conference paper at ICLR 2025
synthetic and real-world data. This gap may affect the generalization of models trained synthetic data when applied to real-world dialogue scenarios.
However, since focus is understanding acoustic information, synthetic data proves particu- larly useful in simulating various acoustic cues found in real conversational settings. Additionally, the synthetic dataset offers more diverse and controllable dialogue content, making it sufficient for evaluating whether spoken dialogue systems can understand information beyond text.
To properly assess the performance of dialogue systems in real-world scenarios, it is crucial to datasets based on authentic conversational environments. We believe that constructing a separate real-world dialogue evaluation benchmark, independent of our work, would be more effective in evaluating spoken dialogue systems' performance in real scenarios than using a single dataset to assess both acoustic information comprehension and real-world dialogue capabilities.
C ETHICAL DISCUSSIONS ON SPOKEN DIALOGUE SYSTEMS
C.I FAIRNESS CHALLENGES SPOKEN CONVERSATION.
Ensuring fairness in spoken dialogue systems involves several challenges, particularly when ad- dressing attributes like gender:
I. Difficulty in Identifying Gender Bias:
Beyond explicit expressions (e.g.. "sir" or "madam"), im- plicit biases (e.g., career or family-related topies) (Liu et al.) 2019) are deeply embedded in existing large language models, making it difficult to guarantee unbiased responses.
II Fairness and Attribute Understanding: While understanding gender attributes can enhance personalization and conversational relevance, over-reliance may reinforce stereotypes. Conversely, completely eliminating gender considerations could limit the model's ability to provide contextually appropriate responses in scenarios where gender information is explicitly relevant. Therefore, an appropriate balance between fairness and attribute understanding should be achieved, ensuring that biases do not cause harm while fostering diversity in responses and improving attribute-specific relevance.
III. Difficulty in Evaluating Bias: Current faimess metrics (Su et al. 2023 often fail capture nuanced and context-dependent biases in spoken dialogue systems, especially in open-ended and multi-turn conversations.
C.2 MITIGATION STRATEGIES.
To address these challenges, commit to implementing the following measures:
We conducted manual filtering all potentially sensitive data to ensure that the dataset complies with Collins & Clément (2012) and excludes examples that could cause harm due to attribute-related biases.
Manual Filtering:
IL Bias Warnings: Clear disclaimers will be included in the documentation to highlight potential gender biases and encourage developers to consider faimesss during model development.
III. Continuous Dataset Updates: We will continuously update and refine the dataset to address fairness issues. Any subsets or evaluation components found to introduce risks of bias will be removed or adjusted as necessary.
16


Reviews:
[{"review_id": "wixZSGRhQ8", "reviewer": "Reviewer_mqdD", "summary": "This paper introduces a new benchmark called VoxDialogue for evaluating the audio comprehension capabilities of voice dialogue systems. The benchmark encompasses 12 sound-related attributes, including speaker attributes (age, gender, accent, language), paralinguistic features (emotion, volume, speed, fidelity, stress, non-verbal expressions), and background sounds (audio, music). The paper also conducted a systematic evaluation of existing spoken dialogue systems, comparing their performance in terms of understanding acoustic information. Besides, the paper proposed a comprehensive method for constructing spoken dialogue data tailored to different acoustic attributes, enabling large-scale data synthesis to support model training.", "questions": "What do AVG and Dur. refer to in Table 3?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses the current issue of speech dialogue systems ignoring audio information by proposing a new benchmark test, which is a meaningful research endeavor.\n\n2. The evaluation of ASR-Based/Direct Spoken Dialogue Systems reveals the limitations of current ASR-based and direct speech dialogue models.\n\n3. The construction of a challenging test set containing 4.5K multi-turn dialogue samples can provide assistance to the voice dialogue systems research community.", "weaknesses": "1. Recommend to add the specific values of text generation metrics in the appendix.\n\n2. Suggest to include statistics on the duration of the dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zvqi0SdrKW", "reviewer": "Reviewer_mXjg", "summary": "This paper provides a dataset involving a spoken dialogue corpus between two humans. It is designed to evaluate spoken dialogue systems. The data is selected so as to include 12 different characteristics where speech would help, such as emotion.", "questions": "- the dataset may also be expanded with a target goal as in MultiWoz / ATIS. For example there is a frustrated call center example but the content is not there, it is just emotional exchanges.\n- can you elaborate further and think of some ablations especially regarding the difference of FunAudioLLM from direct dialogue models? why exactly is this model performing better and in which dimensions?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Overall, this is a clean paper proposing a valuable dataset for spoken language researchers. It is creative to use GenAI to create spoken data for target dimensions related to Speech. I also opened the examples in the github repo and it is very possible that this dataset will be employed by many researchers in this field.", "weaknesses": "The dataset does not include any task oriented dialogue. Hence the evaluation is limited to BLEU or GPT ratings. In many real life scenarios, the spoken dialogue systems are aiming at either an agent like scenario, like Google Home/Alexa style personal assistants, or call center automations, or outbound calls. Their performance cannot be evaluated using BLEU only and the target goal completion is critical. Maybe in the next version of the dataset the authors may want to extend this dataset with such data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jrCho1kLU6", "reviewer": "Reviewer_KJyA", "summary": "Voice assistants are systems that interact with human beings, but current systems usually only focus on linguistic information from text, neglecting useful verbal cues. This work provides a benchmark to evaluate current multimodal systems. In addition, they also identified 12 features that highly correlated to acoustic information and evaluated other dialogue systems on these features.", "questions": "Please refer to the weaknesses listed.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Other than semantic information from text, the authors proposed three directions worth investigating in dialogues: speaker information, paralinguistic information, and background sounds, which are valuable and well-designed.", "weaknesses": "1. On page 2, around line #100, 'For each of these attributes, we designed the most appropriate spoken dialogue synthesis pipelines' This needs more clarification; what is 'most appropriate'? How is it defined?\n2. The second and third contributions listed at the end of the 'Introduction' section need a little more details, such as metrics for evaluating performance. From the description of the third contribution, it is unclear how and why the way to construct spoken dialogue data is unique/beneficial.\n3. For the first two paragraphs under section 2.1, what are the differences between 'audio-language models' and 'spoken dialogue models'? There are no clear differences between the listed works and why they were separately discussed. In other words, I cannot find reasons to use two separate paragraphs, especially since they are all under the 'Spoken Dialogue System' subsection.\n4. I think the third paragraph under section 2.1 is not appropriately placed. This subsection mainly discusses spoken dialogue systems, but not why the lack of a comprehensive benchmark for different evaluation tasks. It will be more appropriate to place it at end of section 2.2. Currently, the contents for the last paragraph from sections 2.1 and 2.2 are heavily overlapped.\n5. I am confused for Table 2. For example, based on the examples given, 'business tasks' is dependent on 'man voice', and 'free juice' or/and 'beef burger' is dependent on 'young voice'. From the manuscript, I did not see how this is established for the speaker's information.\n6. Many details are missing under section 3. Under section 3.2, authors state that they are referring to [Lin et al., 2024a] for their implementation of LLMs with advanced reasoning capabilities to synthesize spoken scripts. They did not specify the exact reason for this, especially what LLMs were used. It is not clear if GPT-4o is the only one that has been used or if it is one among a few. Also, under stage 2, 'We carefully tailored the most appropriate speech synthesis method for each attribute during the generation process,' what does 'most appropriate' even mean here? And how was it compared? For instance, if another work also considered paralinguistic information for their data synthesis process, why is your approach more 'appropriate' than theirs? Under stage 4, the authors state that 'For attributes such as volume, fidelity, audio events, and music, we performed post-processing to ensure that the audio aligns with the required expectations.' how do we interpret 'music' as an attribute here? In addition, 'For music, we randomly apply two different methods to integrate the music with the dialogues.' What does this even mean? How is it post-processed to align with the required expectations? How is music processed? What exactly is the expectation?\n7. At the end of section 4.1 for task definition, is there a particular reason to use only the last response from the entire dialogue history for evaluation of the spoken dialogue systems?\n8. It is not counter-intuitive that ASR-based systems perform poorly compared to multimodal systems because they only take text at input. The authors demonstrate (from the abstract section, conclusion section, and all the experiments) that ASR systems fail to capture important acoustic signals; it is never a fair comparison in the first place.\n\nMinor issue:\nThe appearance of punctuation in the subsection titles could be more consistent. Why do sections 2.1 and 2.2 have periods in the titles?\n\nOverall, many details and justifications are unclear and missing. The manuscript cannot be accepted in its current form.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lUGJuzWltV", "reviewer": "Reviewer_NcGN", "summary": "This paper contributes to the design and creation of a novel dataset for dialogue systems benchmarking. Its design and creation relies on synthetic data and its main attempt is to close the gap on paralinguistic information understanding which its more appropriate to infer directly from acoustics than from the recognized text.\n\nThe paper summaries the current gap in SOTA benchmarks and the limitations of the existing data sets and evaluation protocols. This contribution is oriented for enhancing dialogue understanding. The protocol and steps how Voicebox is created is detailed into steps and quantitative and qualitative assessed.", "questions": "a) How do we ensure the synthetic data is faithful respect to human-human interactions? the emotion modelling and its realization is not a 1-1 mapping and its suitability given the input emotion from the user of the system is not trivial. \n\nb) Speech and spoken language communication is a 1 to many problem. Authors seem to define 1 single realization of a dialogue as the \"ground-truth\". Although multiple performances of the same dialogue (from the \"speech generation\" perspective would be acceptable. How authors and VoxDialogue datasets enable to consider the stochastic nature of human communication?\n\nc) How VoxDialogue is scalable to mutliple languages and low resource languages?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper presents a solid contribution to standardize, democratize, and accelerate the evaluation of spoken dialogue systems. The summary of the state of the art and status quo is well described and the voice attributes and dialogue rubrics is well covered. Authors also commit to open source their protocol and database, which will help the community to iteratively improve it beyond the current contribution. The mental model proposed on synthetic data is a strong tool to accelerate the capabilities of dialogue understanding on GPT-based foundational model.", "weaknesses": "The paper and background is slightly bias oriented to understanding, although in order to move the SOTA in dialogue modelling in human computer interaction, the faithful generation of spoken response is also critical. The paper lists the attributes pursued in the synthetic generation, and they dedicated effort in choosing sufficient good tools to generate the response, but there is not formal assessment of how faithful and suitable those target realizations are realistic. Without a human preference assessment between actual dialogues and the synthetic generated ones, the data set and benchmark presented in this work can limit the ceiling truth of the models developed using it as a benchmark. Still the work is valuable and will contribute to accelerate the foundational properties a the pre-training stage of Foundation models that power spoken dialogue systems.\n\nThe authors based there decisions on the audio generation side based only superficial knowledge of speech. Even their statements about the speech bandwidth is inaccurate and not scientifically supported. Authors should expand, improved and detailed describe the process and decision making on the fidelity and other speech attributes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BBUyvwC6ri", "reviewer": "Reviewer_CNoA", "summary": "The authors present a dataset called VoxDialogue to benchmark the ability of spoken dialogue systems to leverage acoustic information to adapt their interaction. Using LLMs and TTS, they created 4.5K multi-turn spoken dialogue samples according to 12 factors (e.g. age, language, accent, volume, noise...). The authors then evaluated several existing spoken dialogue systems on this benchmark showing that such systems struggle in these situations given their low BLEU and ROUGE scores.", "questions": "The generated dialogues I have listened and seen do not seem to have a goal (not task oriented) hence it is difficult to understand how a system should answer for reached a target. \n\nThe other question is related to the choice of the human behind. Assessing a gender from voice can be a bad idea simply because the system could be wrong and also because the human behind might not agree being attributed a title related to a gender. \n\nSome of the attributes (age for instance) might be seen as too much intrusiveness. What is the stance of the authors about this?\n\nHow many languages have been considered in the datasets? \n\n\ndetails \n\nSUPERB is not the only benchmark in 2021, Lebenchmark (Evain et al. 2021), was also there\n\nthere are repetitions of sentences between the introduction and section 2", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Making available such datasets  with controlled conditions over 12 factors (Gender, Speed, Emotion, Stress, Language, Non-verbal Expressions, Volume, Fidelity, Audio Events Music, Age, and Accent) is a very interesting contribution to the field. The creation process has been performed with care using the latest industrial methods (GPT-4o and GTP4Microsoft Edge's online text-to-speech). \n\nThe evaluation of the five systems (Audio-Flamingo, Qwen-Audio, SALMONN, Qwen-Audio2 and FunAudioLLM) is also very interesting, and forms the basis of the comparative analysis.", "weaknesses": "The paper does not present technical novelty nor original metrics to evaluate dialogue systems. For instance, the evaluation has been performed using n-gram metrics which fails to handle variations in answers. BERT-Score is  similar in all conditions and error bar is not provided. This makes it difficult to assess the difference between models. The evaluation with GPT4 is correlated to other metrics but with more extreme differences. Human evaluation would definitely be an added value here. \n\nAs raised by the authors using TTS and LLM-generated content might be too far from realistic settings. The benchmark might thus be useful for developing systems rather evaluating them. It is true that TTS is useful for training models (Liu et al. 2024 was about lip movements generation) it can be harmful when it is the only data available (Desot et al. 2020, Corpus generation for voice command in smart home and the effect of speech synthesis on End-to-End SLU). But if that informs about the training, it does not support using it for evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)