VoxDialogue: Can Spoken Dialogue Systems Understand Information Beyond Words?

Authors: Xize Cheng, Ruofan Hu, Xiaoda Yang, Jingyu Lu, Dongjie Fu, Zehan Wang, Shengpeng Ji, Rongjie Huang, Boyang Zhang, Tao Jin, Zhou Zhao
Primary area: datasets and benchmarks
Keywords: spoken dialogue system, paralinguistic information, benchmark

Abstract:
With the rapid advancement of large models, voice assistants are gradually acquiring the ability to engage in open-ended daily conversations with humans. However, current spoken dialogue systems often overlook multi-modal information in audio beyond text, such as speech rate, volume, emphasis, and background sounds. Relying solely on Automatic Speech Recognition (ASR) can lead to the loss of valuable auditory cues, thereby weakening the system’s ability to generate contextually appropriate responses. To address this limitation, we propose \textbf{VoxDialogue}, a comprehensive benchmark for evaluating the ability of spoken dialogue systems to understand multi-modal information beyond text. Specifically, we have identified 12 attributes highly correlated with acoustic information beyond words and have meticulously designed corresponding spoken dialogue test sets for each attribute, encompassing a total of 4.5K multi-turn spoken dialogue samples. Finally, we evaluated several existing spoken dialogue models, analyzing their performance on the 12 attribute subsets of VoxDialogue. Experiments have shown that in spoken dialogue scenarios, many acoustic cues cannot be conveyed through textual information and must be directly interpreted from the audio input. In contrast, while direct spoken dialogue systems excel at processing acoustic signals, they still face limitations in handling complex dialogue tasks due to their restricted context understanding capabilities. All data and code will be open source at \url{https://voxdialogue.github.io/}.

Decision: accept (poster)
Reviewer scores: 8, 8, 3, 8, 6

Reviews and discussion:
<thread>
creation date:
1729788504687

thread_type:
official_review

<thread_object>
creator: reviewer_cnoa

<content>
summary:
The authors present a dataset called VoxDialogue to benchmark the ability of spoken dialogue systems to leverage acoustic information to adapt their interaction. Using LLMs and TTS, they created 4.5K multi-turn spoken dialogue samples according to 12 factors (e.g. age, language, accent, volume, noise...). The authors then evaluated several existing spoken dialogue systems on this benchmark showing that such systems struggle in these situations given their low BLEU and ROUGE scores.

strengths:
Making available such datasets  with controlled conditions over 12 factors (Gender, Speed, Emotion, Stress, Language, Non-verbal Expressions, Volume, Fidelity, Audio Events Music, Age, and Accent) is a very interesting contribution to the field. The creation process has been performed with care using the latest industrial methods (GPT-4o and GTP4Microsoft Edge's online text-to-speech). 

The evaluation of the five systems (Audio-Flamingo, Qwen-Audio, SALMONN, Qwen-Audio2 and FunAudioLLM) is also very interesting, and forms the basis of the comparative analysis.

weaknesses:
The paper does not present technical novelty nor original metrics to evaluate dialogue systems. For instance, the evaluation has been performed using n-gram metrics which fails to handle variations in answers. BERT-Score is  similar in all conditions and error bar is not provided. This makes it difficult to assess the difference between models. The evaluation with GPT4 is correlated to other metrics but with more extreme differences. Human evaluation would definitely be an added value here. 

As raised by the authors using TTS and LLM-generated content might be too far from realistic settings. The benchmark might thus be useful for developing systems rather evaluating them. It is true that TTS is useful for training models (Liu et al. 2024 was about lip movements generation) it can be harmful when it is the only data available (Desot et al. 2020, Corpus generation for voice command in smart home and the effect of speech synthesis on End-to-End SLU). But if that informs about the training, it does not support using it for evaluation.

questions:
The generated dialogues I have listened and seen do not seem to have a goal (not task oriented) hence it is difficult to understand how a system should answer for reached a target. 

The other question is related to the choice of the human behind. Assessing a gender from voice can be a bad idea simply because the system could be wrong and also because the human behind might not agree being attributed a title related to a gender. 

Some of the attributes (age for instance) might be seen as too much intrusiveness. What is the stance of the authors about this?

How many languages have been considered in the datasets? 


details 

SUPERB is not the only benchmark in 2021, Lebenchmark (Evain et al. 2021), was also there

there are repetitions of sentences between the introduction and section 2
</content>
</thread_object>

</thread>
<thread>
creation date:
1730558890423

thread_type:
official_review

<thread_object>
creator: reviewer_ncgn

<content>
summary:
This paper contributes to the design and creation of a novel dataset for dialogue systems benchmarking. Its design and creation relies on synthetic data and its main attempt is to close the gap on paralinguistic information understanding which its more appropriate to infer directly from acoustics than from the recognized text.

The paper summaries the current gap in SOTA benchmarks and the limitations of the existing data sets and evaluation protocols. This contribution is oriented for enhancing dialogue understanding. The protocol and steps how Voicebox is created is detailed into steps and quantitative and qualitative assessed.

strengths:
This paper presents a solid contribution to standardize, democratize, and accelerate the evaluation of spoken dialogue systems. The summary of the state of the art and status quo is well described and the voice attributes and dialogue rubrics is well covered. Authors also commit to open source their protocol and database, which will help the community to iteratively improve it beyond the current contribution. The mental model proposed on synthetic data is a strong tool to accelerate the capabilities of dialogue understanding on GPT-based foundational model.

weaknesses:
The paper and background is slightly bias oriented to understanding, although in order to move the SOTA in dialogue modelling in human computer interaction, the faithful generation of spoken response is also critical. The paper lists the attributes pursued in the synthetic generation, and they dedicated effort in choosing sufficient good tools to generate the response, but there is not formal assessment of how faithful and suitable those target realizations are realistic. Without a human preference assessment between actual dialogues and the synthetic generated ones, the data set and benchmark presented in this work can limit the ceiling truth of the models developed using it as a benchmark. Still the work is valuable and will contribute to accelerate the foundational properties a the pre-training stage of Foundation models that power spoken dialogue systems.

The authors based there decisions on the audio generation side based only superficial knowledge of speech. Even their statements about the speech bandwidth is inaccurate and not scientifically supported. Authors should expand, improved and detailed describe the process and decision making on the fidelity and other speech attributes.

questions:
a) How do we ensure the synthetic data is faithful respect to human-human interactions? the emotion modelling and its realization is not a 1-1 mapping and its suitability given the input emotion from the user of the system is not trivial. 

b) Speech and spoken language communication is a 1 to many problem. Authors seem to define 1 single realization of a dialogue as the "ground-truth". Although multiple performances of the same dialogue (from the "speech generation" perspective would be acceptable. How authors and VoxDialogue datasets enable to consider the stochastic nature of human communication?

c) How VoxDialogue is scalable to mutliple languages and low resource languages?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730602984577

thread_type:
official_review

<thread_object>
creator: reviewer_kjya

<content>
summary:
Voice assistants are systems that interact with human beings, but current systems usually only focus on linguistic information from text, neglecting useful verbal cues. This work provides a benchmark to evaluate current multimodal systems. In addition, they also identified 12 features that highly correlated to acoustic information and evaluated other dialogue systems on these features.

strengths:
Other than semantic information from text, the authors proposed three directions worth investigating in dialogues: speaker information, paralinguistic information, and background sounds, which are valuable and well-designed.

weaknesses:
1. On page 2, around line #100, 'For each of these attributes, we designed the most appropriate spoken dialogue synthesis pipelines' This needs more clarification; what is 'most appropriate'? How is it defined?
2. The second and third contributions listed at the end of the 'Introduction' section need a little more details, such as metrics for evaluating performance. From the description of the third contribution, it is unclear how and why the way to construct spoken dialogue data is unique/beneficial.
3. For the first two paragraphs under section 2.1, what are the differences between 'audio-language models' and 'spoken dialogue models'? There are no clear differences between the listed works and why they were separately discussed. In other words, I cannot find reasons to use two separate paragraphs, especially since they are all under the 'Spoken Dialogue System' subsection.
4. I think the third paragraph under section 2.1 is not appropriately placed. This subsection mainly discusses spoken dialogue systems, but not why the lack of a comprehensive benchmark for different evaluation tasks. It will be more appropriate to place it at end of section 2.2. Currently, the contents for the last paragraph from sections 2.1 and 2.2 are heavily overlapped.
5. I am confused for Table 2. For example, based on the examples given, 'business tasks' is dependent on 'man voice', and 'free juice' or/and 'beef burger' is dependent on 'young voice'. From the manuscript, I did not see how this is established for the speaker's information.
6. Many details are missing under section 3. Under section 3.2, authors state that they are referring to [Lin et al., 2024a] for their implementation of LLMs with advanced reasoning capabilities to synthesize spoken scripts. They did not specify the exact reason for this, especially what LLMs were used. It is not clear if GPT-4o is the only one that has been used or if it is one among a few. Also, under stage 2, 'We carefully tailored the most appropriate speech synthesis method for each attribute during the generation process,' what does 'most appropriate' even mean here? And how was it compared? For instance, if another work also considered paralinguistic information for their data synthesis process, why is your approach more 'appropriate' than theirs? Under stage 4, the authors state that 'For attributes such as volume, fidelity, audio events, and music, we performed post-processing to ensure that the audio aligns with the required expectations.' how do we interpret 'music' as an attribute here? In addition, 'For music, we randomly apply two different methods to integrate the music with the dialogues.' What does this even mean? How is it post-processed to align with the required expectations? How is music processed? What exactly is the expectation?
7. At the end of section 4.1 for task definition, is there a particular reason to use only the last response from the entire dialogue history for evaluation of the spoken dialogue systems?
8. It is not counter-intuitive that ASR-based systems perform poorly compared to multimodal systems because they only take text at input. The authors demonstrate (from the abstract section, conclusion section, and all the experiments) that ASR systems fail to capture important acoustic signals; it is never a fair comparison in the first place.

Minor issue:
The appearance of punctuation in the subsection titles could be more consistent. Why do sections 2.1 and 2.2 have periods in the titles?

Overall, many details and justifications are unclear and missing. The manuscript cannot be accepted in its current form.

questions:
Please refer to the weaknesses listed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681053185

thread_type:
official_review

<thread_object>
creator: reviewer_mxjg

<content>
summary:
This paper provides a dataset involving a spoken dialogue corpus between two humans. It is designed to evaluate spoken dialogue systems. The data is selected so as to include 12 different characteristics where speech would help, such as emotion.

strengths:
Overall, this is a clean paper proposing a valuable dataset for spoken language researchers. It is creative to use GenAI to create spoken data for target dimensions related to Speech. I also opened the examples in the github repo and it is very possible that this dataset will be employed by many researchers in this field.

weaknesses:
The dataset does not include any task oriented dialogue. Hence the evaluation is limited to BLEU or GPT ratings. In many real life scenarios, the spoken dialogue systems are aiming at either an agent like scenario, like Google Home/Alexa style personal assistants, or call center automations, or outbound calls. Their performance cannot be evaluated using BLEU only and the target goal completion is critical. Maybe in the next version of the dataset the authors may want to extend this dataset with such data.

questions:
- the dataset may also be expanded with a target goal as in MultiWoz / ATIS. For example there is a frustrated call center example but the content is not there, it is just emotional exchanges.
- can you elaborate further and think of some ablations especially regarding the difference of FunAudioLLM from direct dialogue models? why exactly is this model performing better and in which dimensions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709994081

thread_type:
official_review

<thread_object>
creator: reviewer_mqdd

<content>
summary:
This paper introduces a new benchmark called VoxDialogue for evaluating the audio comprehension capabilities of voice dialogue systems. The benchmark encompasses 12 sound-related attributes, including speaker attributes (age, gender, accent, language), paralinguistic features (emotion, volume, speed, fidelity, stress, non-verbal expressions), and background sounds (audio, music). The paper also conducted a systematic evaluation of existing spoken dialogue systems, comparing their performance in terms of understanding acoustic information. Besides, the paper proposed a comprehensive method for constructing spoken dialogue data tailored to different acoustic attributes, enabling large-scale data synthesis to support model training.

strengths:
1. This paper addresses the current issue of speech dialogue systems ignoring audio information by proposing a new benchmark test, which is a meaningful research endeavor.

2. The evaluation of ASR-Based/Direct Spoken Dialogue Systems reveals the limitations of current ASR-based and direct speech dialogue models.

3. The construction of a challenging test set containing 4.5K multi-turn dialogue samples can provide assistance to the voice dialogue systems research community.

weaknesses:
1. Recommend to add the specific values of text generation metrics in the appendix.

2. Suggest to include statistics on the duration of the dataset.

questions:
What do AVG and Dur. refer to in Table 3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733302553815

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Thank you for your invaluable efforts and constructive feedback on our work. As the rebuttal period comes to a close, we would like to take this opportunity to summarize the discussions and outline the improvements we have made based on your suggestions.

## **Major Contributions of Our Work**

We are grateful for the reviewers’ recognition of our work. Below, we summarize the key contributions highlighted:

1. **Motivation:**
    - This work introduces a novel spoken dialogue system benchmark, VoxDialogue, to explore whether current systems can comprehend various acoustic properties. To the best of our knowledge, we are the first to provide such a comprehensive benchmark, making it a valuable resource to advance research in spoken dialogue systems. **[mqdD, mXjg, KJyA, NcGN, CNoA]**
    - We systematically analyzed 12 acoustic attributes that may influence spoken dialogue responses. For each of the 12 acoustic attributes, we designed a dedicated data synthesis pipeline with careful verification to ensure controllable manipulation of these attributes. **[mqdD, mXjg, NcGN, CNoA]**
2. **Experiments:**
    - All reviewers agreed that our benchmark represents a fair and well-designed experiment, providing an objective evaluation framework for spoken dialogue systems. **[mqdD, mXjg, KJyA, NcGN, CNoA]**
    - The experiments conducted on five distinct spoken dialogue systems are highly insightful, highlighting the limitations of both ASR-based systems and direct spoken dialogue systems. **[mqdD, CNoA]**

3.	**Presentation and Impact:**
    - All reviewers commended the clarity and quality of our demonstration. **[mqdD, mXjg, KJyA, NcGN, CNoA]**
    - This work lays a solid foundation for the standardization, democratization, and acceleration of spoken dialogue system evaluations. It provides a thorough review of the current state-of-the-art while comprehensively addressing speech attributes and dialogue rules. **[mqdD,mXjg,NcGN,CNoA]**

## **Revisions and Improvements Based on Reviewer Suggestions**

In response to the reviewers’ insightful comments, we have revised and enhanced the manuscript with the following updates:

1. **Additional Explanations**
    - We provided more detailed explanations for potentially confusing attributes in the benchmark, emphasizing that music is an environmental information beyond the language itself, and refining the construction details for fidelity. **[NcGN,KJyA]**
    - We elaborated on the design rationale behind the synthetic scripts and GPT evaluation metrics. Additionally, we included all relevant prompts in the appendix to clarify the details of our LLM-based methodology. **[CNoA,KJyA]**
2. **Improved Figures and Tables**
    - We added Figure 1(c) to illustrate the distribution of dialogue duration, further clarifying the details of our benchmark. In addition, we provided Table 5 in the appendix to supplement Figure 2 with detailed evaluation metrics, enhancing the credibility of our experimental results. **[mqdD]**
    - We added a comparison with LeBenchmark in Table 1. **[CNoA]**

3. **Writing Refinements**

    - We clarified that the baseline experiments used the Chat version and further emphasized the importance of acoustic information in spoken dialogue models. We also unified the punctuation throughout the titles of each subsection. **[KJyA]**
    - We defined and explained abbreviations such as AVG and Dur., improving the readability of the paper. **[mqdD]**

4. **Fairness and Ethical Discussions**

    - We reviewed the literature on “dialogue bias” and revised potentially offensive expressions in the VoxDialogue dataset. **[CNoA]**
    - Additionally, we added an Ethical Discussion section in Appendix C to further address fairness-related concerns, enhancing the usability and societal impact of our work. **[CNoA]**

## **Conclusion**

Once again, we deeply appreciate the reviewers’ dedication and valuable feedback, which have greatly contributed to refining this work. We are honored by the opportunity to present our research and look forward to any further suggestions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a benchmark, VoxDialogue, for evaluating the ability of spoken dialogue systems to understand acoustic information beyond lexical content. VoxDialogue integrates12 acoustic dimensions, including speaker attributes (age, gender, accent, language), paralinguistic features (emotion, volume, speed, fidelity, stress, non-verbal expressions), and environmental information (audio, music). The paper includes comparison of VoxDialogue to several other benchmarks with respect to many evaluation dimensions. The work also proposes spoken dialogue data synthesis methods for various acoustic attributes and compares the performance of existing spoken dialogue systems in terms of understanding acoustic information using a GPT-based metric. Reviewers acknowledged that these contributions of the paper constitute its strengths. They also asked several clarification questions and requested explanations which the authors responded to during the rebuttal period, and integrated them into the paper already.