Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics

Authors: Siddhant Arora, Zhiyun Lu, Chung-Cheng Chiu, Ruoming Pang, Shinji Watanabe
Primary area: datasets and benchmarks
Keywords: Turn-taking, Conversation AI, Audio Foundation Models, Evaluation Metric, Evaluation Benchmark

Abstract:
The recent wave of audio foundation models (FMs) could provide new capabilities for conversational modeling. However, there have been limited efforts to evaluate these audio FMs comprehensively on their ability to have natural and interactive conversations. To engage in meaningful conversation with the end user, we would want the FMs to additionally perform a fluent succession of turns without too much overlapping speech or long stretches of silence. Inspired by this, we ask whether the recently proposed audio FMs can understand, predict, and perform turn-taking events? To answer this, we propose a novel evaluation protocol that can assess spoken dialog system's turn-taking capabilities using a supervised model as a judge that has been trained to predict turn-taking events in human-human conversations. Using this protocol, we present the first comprehensive user study that evaluates existing spoken dialogue systems on their ability to perform turn-taking events and reveal many interesting insights, such as they sometimes do not understand when to speak up, can interrupt too aggressively and rarely backchannel. We further evaluate multiple open-source and proprietary audio FMs accessible through APIs on carefully curated test benchmarks from Switchboard to measure their ability to understand and predict turn-taking events and identify significant room for improvement. We will open source our evaluation platform to promote the development of advanced conversational AI systems.

Decision: accept (poster)
Reviewer scores: 8, 6, 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1730624096388

thread_type:
official_review

<thread_object>
creator: reviewer_mo78

<content>
summary:
This paper introduces a novel evaluation framework for assessing turn-taking capabilities in audio foundation models (FMs). The authors first propose metrics for five core conversational abilities: determining when to speak up, backchannel, interrupt, convey turn-taking cues, and handle interruptions. They develop a supervised model trained on human-human conversations to serve as a judge for evaluating these turn-taking events. Using this framework, they conducted a user study with different spoken dialogue systems (full-duplex E2E spoken dialogue system Moshi and VAD-based cascade dialogue system) and evaluated them. They evaluate several open-source and proprietary audio FMs on their ability to understand and predict turn-taking events.

strengths:
1. The evaluation protocol is novel and well-motivated.
2. The experimental analysis provides valuable insights into turn-taking capabilities of audio foundation models (FMs).
3. The user study reveals noteworthy observations about current spoken dialogue systems.

weaknesses:
1. Turn-taking prediction models used in evaluation protocol require training, which limits scalability and applicability. 
2. The paper does not thoroughly address how its proposed evaluation protocol compares with previous turn-taking approaches, such as Ekstedt and Skantze (2022).

Reference
* Ekstedt, Erik, and Gabriel Skantze. Voice activity projection: Self-supervised learning of turn-taking events. Interspeech 2022

questions:
1. What is the main difference between the proposed evaluation protocol and the previous approach by Ekstedt and Skantze (2022)? Is it impractical to apply the metrics from prior turn-taking evaluation methods to audio FMs?
2. While the turn-taking prediction model has been evaluated on an out-of-domain task-oriented spoken dialogue corpus, could you evaluate it on additional non-task-oriented spoken dialogue datasets to assess the generalizability of the model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730653223784

thread_type:
official_review

<thread_object>
creator: reviewer_bktk

<content>
summary:
The paper presents an evaluation protocol designed to assess the turn-taking capabilities of spoken dialogue systems. It evaluates the exact timing of these events using a supervised model trained to predict them. The experimental results reveal interesting insights about existing spoken dialogue systems and offer valuable suggestions for their future development.

strengths:
1. This paper proposes a comprehensive evaluation protocol and well-designed metrics to assess the turn-taking capabilities of spoken dialogue systems. The evaluation framework and metrics are thoughtfully developed and provide valuable insights.
2. The paper extends the evaluation of turn-taking capabilities of spoken dialogue systems from corpus-level statistics to a more granular assessment of the timing of turn-taking events. This fine-grained approach enables a more accurate reflection of a spoken dialogue system’s turn-taking capabilities.
3. The proposed evaluation metrics provide insights into the limitations of current systems in achieving interactive and natural conversations, highlighting areas for potential improvement.

weaknesses:
1. In Metric (E), the judge labels show low consistency with human relevance judgments, indicating that this metric may have limited reliability in assessing the model's ability to handle user interruptions effectively.
2. My primary concern is the relatively low agreement between the majority of judge labels and human judgments, with most falling below 80%. This raises questions about the strength of the claim that the proposed metrics maintain high consistency with human decisions.
3. GPT-4o was not evaluated.

If my above concerns are resolved, I would consider increasing my rating.

questions:
My questions are listed above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692233304

thread_type:
official_review

<thread_object>
creator: reviewer_cjr2

<content>
summary:
This paper proposes an evaluation protocol to measure the turn-taking capabilities of audio foundation models.

strengths:
The strengths of this paper are as follows:
 1. This paper provides an automated turn-taking protocol for audio foundation models
 2. The evaluation platform will be open-sourced.

weaknesses:
The weaknesses of this paper are as follows:
 1. The study aims to measure precise turn-taking, but the thresholds are set to arbitrary values.
 2. The participants introduced in Sec 3 seem biased, consisting of the authors and related individuals.
 3. The confidence in some evaluations (Fig. 3(b), (e)) appears high, but no explanation is provided.

questions:
Here are questions for the authors:
 - The thresholds in Sec. 4.4-4.8 seem arbitrary. Is there a specific reason for choosing these values? All units appear to represent likelihoods, yet they range from negative ($threshold_3$ = -0.45) to positive values ($threshold_2$ = 0.1).
 - There are concerns about the reliability of the judge model. Since all results are based on comparisons with this model, is there concrete evidence supporting its credibility? Specifically, the conclusion that Moshi[1] is "too aggressive" lacks persuasiveness if it relies solely on comparisons with the judge model.

[1] Defossez et al. Moshi: a speech-text foundation model for real-time dialogue
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713380415

thread_type:
official_review

<thread_object>
creator: reviewer_ddi6

<content>
summary:
This paper addresses the challenges of evaluating the turn-taking capabilities of audio foundation models (FMs) in conversational settings.
It defines 6 types of Turn-Taking Events and evaluates the performance of end-to-end speech dialogue models as well as cascaded systems.
Through the results obtained from this study, the authors discovered numerous issues with existing AI dialogue systems in handling turn-taking, such as sometimes failing to intervene in conversations at appropriate times or excessively interrupting others. Furthermore, the authors conducted tests on multiple open-source and closed-source audio foundation models, revealing their limitations in understanding and predicting turn-taking events, and highlighting areas for improvement.

strengths:
1. The definition of turn-taking is detailed and clear.

2. The evaluation protocol proposed in this paper contributes to better assessing the performance of audio foundation models in dialogues, providing strong support for the development of voice dialogue systems.

3. This paper reveals many issues existing in current AI dialogue systems when handling turn-taking, offering valuable references for future research.

weaknesses:
1. This study only tested a few open-source and closed-source audio FM models.

2. There is a lack of comprehensive performance evaluation and summary.

questions:
NA.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714246413

thread_type:
official_review

<thread_object>
creator: reviewer_wgpu

<content>
summary:
The paper proposes a new evaluation protocol to assess the spoken dialog system's turn-taking capabilities, i.e., the Moshi and Cascaded model. They use a supervised model as a judge which is trained to predict turn-tasking events in human-human conversation (i.e., Switchboard). The paper presents a comprehensive user study that evaluates the Moshi and Casaded model on their ability to perform turn-taking events, and it finds that they sometimes do not understand when to speak up, can interrupt too aggressively, and rarely backchannel. The main contributions are:
1. A new evaluation protocol to assess the spoken dialog system's turn-taking capabilities.
2. Some insight about existing spoken dialogue systems through user study.
3. Additionally create a test benchmark using Switchboard dataset to evaluate SALMONN, Qwen2-audio-instruct, Qwen-audiochat, Whisper+GPT-4o on their ability to understand and predict turn-taking events.

strengths:
1. The originality of the work is commendable. The authors propose a novel evaluation protocol to assess the turn-taking capabilities of spoken dialog systems.
2. The paper is well-written and provides sufficient experimental details in the Appendix.
3. The authors plan to open source the evaluation platform.

weaknesses:
1. The evaluation protocol is highly expensive, as it requires a supervised dataset to train the judge model. This approach is not feasible if we lack a supervised dataset in other languages, such as Chinese.
2. The filler word set for backchannel detection is heuristic and may miss some backchannel cases that are not included in the filler word set.

questions:
1. The Fisher dataset is a common dataset comparable to Switchboard. What is the performance of the supervised turn-taking prediction model on this dataset?
2. How can your evaluation protocol be adapted or applied in scenarios where supervised datasets are not available for other languages, such as Chinese?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732208460024

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all the reviewers for their thoughtful feedback. We are pleased that our approach is recognized as novel (@R WGPU, @R mo78), thoughtfully designed (@R BKtK), and clearly presented (@R WGPU, @R Ddi6). We appreciate the recognition of its value in assessing conversational capabilities of audio FMs (@R Ddi6, @R BKtK), highlighting AI dialogue system limitations, and suggesting future research directions (@R Ddi6, @R BKtK, @R mo78). We will incorporate the reviewers’ constructive suggestions into the paper. 

---

# 1. Results on Fisher

Reviewers (@R WGPU, @R mo78) suggested evaluating our judge turn-taking model on an out-of-domain, non-task-oriented spoken dialogue dataset, such as Fisher. To address this, we created a random test set comprising 23 hours of audio and 138 conversations, similar in size to Switchboard test set, and we will make this test split publicly available. The Fisher dataset's transcriptions were created using the Quick Transcription specification, which introduced inaccuracies and left significant portions untranscribed as also noted in [1]. We developed heuristics to identify such audio segments with large untranscribed content and made sure to exclude these audio segments from the test set. Further manual analysis revealed errors in the ground truth timestamps and we corrected these timestamps using speaker diarization outputs from Pyannote.

Our results show that our model demonstrates strong generalization on this corpus, achieving a **ROC-AUC of 91.0** comparable to the performance on the in-domain test set, i.e., Switchboard (overall ROC-AUC score of 92.0). These results and a detailed discussion have been included in the updated draft (Tab. 1, Sec. 4.1, Sec A.4), and we will make all our data preparation code and model publicly available. Thank you for the suggestion, as it helped us further validate the robustness of our approach. 

---

# 2. Evaluating more audio FMs
We acknowledge the reviewer's (@R Ddi6, @R BKtK) concern about benchmarking only a limited set of audio FMs for evaluating conversational capabilities. At the time of conducting this study, there were indeed very few audio FMs capable of performing turn-taking events, limiting our selection. We emphasize that this is a developing field, and we plan to expand our benchmarking as more audio FMs with turn-taking capabilities emerge. We explicitly mention this in the updated paper (Limitations in Sec. 6).

@R BKtK mentioned that we should have evaluated GPT-4o. At the time of submission, GPT-4o’s Advanced Voice Mode was not publicly available, preventing its inclusion in our evaluations. Based on the feedback, we have started collecting human-GPT-4o conversation data, accumulating 55 minutes and 38 seconds of audio across six speakers. This preliminary test trial has revealed some interesting insights:

- Turn-Taking Latency: GPT-4o exhibits a moderate gap between speaker turns (16.1% of cumulative duration), smaller than the Cascaded system (32.5% in Fig. 2b) but larger than Moshi (11.8% in Fig. 2b), indicating intermediate latency.
- Overlap: Similar to a cascaded system, GPT-4o has minimal overlapping speech (0.5% of cumulative duration in Fig. 2b), resulting in less interactive conversations.
- Turn-Yielding Behavior: GPT-4o has a high number of pause events (18.0 per minute) and fewer gap events (3.2 per minute) compared to the other dialogue systems (Fig. 2a), indicating that it sometimes speaks for very long without yielding it’s turn, which makes the conversation bland and less engaging for end users.
- Metric C (Sec. 4.6): GPT-4o rarely interrupts users (0.1%, lower than the other systems in Table 2), but when it does, its agreement with the judge label (75.0%) is significantly higher than Moshi (35.7% in Fig. 3c) and the Cascaded system (24.2% in Fig. 3c).
- Metric D (Sec. 4.7): GPT-4o is significantly better at conveying users when it wants to keep the conversation floor, having 68.9% agreement with judge label which is much higher than Moshi (32.7% agreement in Fig. 3d) and Cascaded system (40.8% agreement in Fig. 3d).

We are currently expanding this effort to collect 4 hours of GPT-4o conversation data, comparable to the results reported for other dialogue systems in the draft. This analysis with GPT-4o will be included in the final paper.

We agree with the reviewers (@R Ddi6, @R BKtK, @R mo78) that, while our analysis is limited to a few audio FMs, it effectively highlights issues in current AI dialogue systems, offering valuable insights for future research and underscoring the utility of our protocol.

---

References:

[1] Generative spoken dialogue language modeling (https://doi.org/10.1162/tacl_a_00545 )
</content>
</thread_object>

</thread>
<thread>
creation date:
1733178746748

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response regarding the agreement between judge labels and human decisions

comment:
We sincerely thank the reviewers for their thoughtful feedback. Some reviewers (@R cJr2, @R BKtK, @R mo78). raised concerns about the relatively low agreement (<80%) between judge labels and human judgments. We would like to clarify that, with the exception of Metric E, the judge labels demonstrate a consistently high level of agreement (>70%, as shown in Figure 3) with human decisions on the in-domain test set. Additionally, prior works have reported **similar levels of agreement when using LLMs as judges, with around 70% [1] and 66% [2] agreements**. Turn-taking, by nature, is a subjective task influenced by individual user behavior, making complete alignment neither expected nor feasible. Moreover, our judge labels also show moderate agreement with human decisions even on an out-of-domain test set.
 
To further demonstrate the robustness of our trained turn-taking model, we evaluated it objectively on its ability to predict upcoming turn-taking events in human-human conversations. This evaluation was conducted on both an in-domain test set (Switchboard) and two out-of-domain datasets (Columbia Games and Fisher). Our results indicate that the model not only performs on par with prior works but also generalizes strongly to OOD spoken dialogue corpora in zero shot manner, achieving comparable performance across datasets.
 
These findings suggest that the model can reliably evaluate the precise timing of turn-taking decisions made by AI dialogue systems. Its decent consistency with human decisions indicates that it serves as an effective proxy for human judgment, addressing concerns regarding its reliability and utility. We will incorporate these discussions into the paper.
 
References

[1] Yiang et al. AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension (https://arxiv.org/pdf/2402.07729 )

[2] Kolchinski et al. Approximating Human Judgment of Generated Image Quality. (https://arxiv.org/pdf/1912.12121 )
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel evaluation protocol for assessing turn-taking capabilities in audio-based spoken dialogue systems and audio foundation models. By defining metrics to capture when to speak, backchannel, interrupt, and handle interruptions, the authors introduce a supervised judge model trained on human-human conversations. Reviewers appreciated the motivation and the effort to formalize a challenging, subjective aspect of conversational interaction. They also welcomed the clarification that while perfect human-model alignment is unrealistic, the judge model achieves moderate to high agreement with human judgments across datasets. Although some thresholds and methods could be justified, and certain baselines were missing, additional evaluations indicate that the approach is sound and generalizes to out-of-domain scenarios.
Given these points and the constructive improvements offered by the authors, I recommend acceptance.