Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics
Authors: Siddhant Arora
Zhiyun Lu
Chung-Cheng Chiu
Ruoming Pang
Shinji Watanabe
Venue: ICLR 2025 Poster (2025)
Keywords: Turn-taking
Conversation AI
Audio Foundation Models
Evaluation Metric
Evaluation Benchmark
Primary area: datasets and benchmarks
TL;DR: We propose a novel evaluation protocol using a turn-taking judge model to automatically assess spoken dialog systems, providing valuable insights into their turn-taking capabilities.

Abstract: The recent wave of audio foundation models (FMs) could provide new capabilities for conversational modeling. However, there have been limited efforts to evaluate these audio FMs comprehensively on their ability to have natural and interactive conversations. To engage in meaningful conversation with the end user, we would want the FMs to additionally perform a fluent succession of turns without too much overlapping speech or long stretches of silence. Inspired by this, we ask whether the recently proposed audio FMs can understand, predict, and perform turn-taking events? To answer this, we propose a novel evaluation protocol that can assess spoken dialog system's turn-taking capabilities using a supervised model as a judge that has been trained to predict turn-taking events in human-human conversations. Using this protocol, we present the first comprehensive user study that evaluates existing spoken dialogue systems on their ability to perform turn-taking events and reveal many interesting insights, such as they sometimes do not understand when to speak up, can interrupt too aggressively and rarely backchannel. We further evaluate multiple open-source and proprietary audio FMs accessible through APIs on carefully curated test benchmarks from Switchboard to measure their ability to understand and predict turn-taking events and identify significant room for improvement. We will open source our evaluation platform to promote the development of advanced conversational AI systems.

Published as a conference paper at ICLR 2025
TALKING TURNS: BENCHMARKING AUDIO FOUNDATION MODELS ON TURN-TAKING DYNAMICS
Siddhant Arora', Zhiyun Lu², Chung- Chenn Chiu', Ruoming Pang", Shinji Watanabe'
I Carnegie Mellon University, USA,² Apple si didhhnaaaaddrew. cmu. edu
ABSTRACT
The recent wave of audio foundation models (FMs) could provide new capabil- ities for conversational modeling. However, there have been limited efforts to evaluate these audio FMs comprehensively on their ability to have natural and in- teractive conversations. To engage in meaningful conversation with the end user, we would want the FMs to additionally perform a fluent succession of turns with- out too much overlapping speech or long stretches of silence. Inspired by this, we ask whether the recently proposed audio FMs can understand, predict, and perform turn-taking events? To answer this, we propose novel evaluation pro- tocol that can assess spoken dialog system's turn-taking capabilities using a su- pervised model judge that has been trained to predict turn-taking events in human-human conversations. Using this protocol, we present the first compre- hensive user study that evaluates existing spoken dialogue systems their ability to perform turn-taking events and reveal many interesting insights, such as they sometimes do not understand when to speak up, can interrupt too aggressively and rarely backchannel. We further evaluate multiple open-source and proprietary audio FMs accessible through APIs on carefully curated test benchmarks from Switchboard to measure their ability to understand and predict turn-taking events and identify significant room for improvement. We will open source our evalua- tion platform to promote the development of advanced conversational AI systems. n
INTRODUCTION
When humans interact with an audio foundation model (FM), it is a two-way communication. Sim- ilar to human-human human conversations, the model listens and speaks, and more importantly, does both at the same time. Turn management is at the core of real-world conversations, This means that when the user speaks, the system should know when to listen and when to speak (Gravano & Hirschberg 2011) The system should additionally provide subtle cues. referred to backchannels (Fujie al. 2005 to indicate that it is "listening" and make relevant follow-up up interruptions (Lee & Narayanan 2010 ensuring that the conversation feels interactive. Furthermore, when the system speaks, should formulate its output in a way that conveys to the user whether it wants to keep the conversa- tion floor (Ekstedt et al. 2023) and also appropriately address user interruptions (Ma al. 2024).
Commercial voice assistants (i et al. 2017) use traditional silence duration-based end-of-turn pre- diction models for turn management. However, this is insufficient for having natural conversations because silence is often not the main cue for humans to switch turns. In fact, research shows that silence within the same speaker's turns (pauses) tends be longer than silence between different speaker's 's turns (gaps) (Ten Bosch et al. 2005). Hence, this continues to remain a challenging task for muman-machine interaction. As a result, there has been lot of interest in automatically pre- dicting turn-taking events. Prior research works (Hara et al. 200 8 Fujie et al.) 2005) have explored automatically predicting turn change and backchanneling from text, audio, and multimodal input. More recently, audio FMs (Xie & Wu 2024; Défossez et al. 2024) have been proposed that can automatically take turns while interacting with the end user.
Work done during internship at Apple.

---

Published as a conference paper at ICLR 2025
However there have been limited efforts to evaluate these FMs on their conversational capabilities. Furthermore, this requires more than simply testing on standard speech recognition (ASR), text-to- speech (TTS), and text-based dialogue benchmarks. To engage in meaningful conversation with the end user, we would want the FMs to additionally perform a fluent succession of turns without too much overlapping speech or long stretches of silence. Finally, the model should be able to engage in dynamic and fluid conversations where it should use cues to confirm listening and encourage the speaker, and it should listen for cues continue or to be interrupted while speaking. To the best of our knowledge, there has not been any prior effort to empirically evaluate the turn-taking capabilities of audio FMs. As a result, the community still lacks a fine tine-granedd understanding of the relative merits and limitations of different audio FMs on their conversational capabilities. Motivated by this, our work presents the first empirical investigation to evaluate audio FMs on their ability to understand, predict, and perform tur-takiin events. Our key contributions are:
An evaluation protocol to assess audio FM's capability to perform turn-taking events: Prior work (Nguyen et al., 2023) uses corpus level statistics for automatic evaluation. While these metrics capture how well the distribution of the turn-taking events globally in the generated dialogue matches with ground truth, they fail to evaluate the exact timing when turn-taking event happens within the local context. We propose evaluating the timing of turn-taking events by training model on human-human conversations to effectively predict the turn change, backchannel, and interruption. We use this predictor model as the judge since getting human relevance judgments for each turn-taking event expensive and timecconsuming Using the judge model, we propose automated metrics corresponding to each of the core conversational abilities, thus building a com- prehensive suite of diverse metrics that can empower systemic understanding of any audio FM with conversation capability on its ability to manage turns. Finally, we plan to publicly release our evaluation platform so that researchers can easily test their own pre-trained audio FMs.
Contribution 1:
Interesting insights about existing spoken dialogue systems: We run a user study with different spoken dialogue systems, namely full-duplex E2E spoken dialogue system Moshi and VAD>based cascaded dialogue system and evaluate these systems using both corpus level statistics and our proposed metrics, unveiling many interesting observations: (a) Both dialogue systems sometimes do not speak up even when user wants to yield its turn. (b) Moshi interrupts too aggressively and its interruptions occur at unlikely instances. (c) Both dialogue systems rarely backchannel. (d) Both dialogue systems do not give users enough cues about when they want keep the conversation floor. (e) Moshi mostly continues speaking even after the user interrupts, whereas a VAD-based dialogue system is more likely to yield its turn when interrupted.
Contribution 2:
Contribution 3: Evaluation of audio FMS on understanding and predicting turn-taking events: We additionally curate test benchmark using human- conversation datasets to evaluate audio FMs on their ability to understand and predict turn-taking events. This benchmark facilitates an understanding of the relative merits of different FMs, how this trend varies across different turn- taking events, and quantify the room for improvement.
2 RELATED STUDY
Turn-Taking Prediction:
There have been many prior works on predicting turn change (Gravano
& Hirschberg 2012, Hara al. 2018 Li et al. 2022), backchannel (Fujie et al., 2005) and inter- ruptions (Lee & Narayanan] 2010 Lee et al. 2008) from audio, text and other multimodal infor- mation (Morency et al. 2010 Scherer et al. 2012) (detailed related work discussion in Appendix A.I. The Voice Activity Projection ( (VAP) models (Ekstedt & Skantze 2022a Inoue et al., 2024) are trained an unsupervised manner on spoken dialogue data to predict future "speech" activity.
Other recent works (Wang et al., 2024) use pseudo labels to predict turn-taking and backchanneling locations in spoken dialogue by integrating a neural acoustic model with a large language model (LLM). Recently, audio FMs like Moshi (Défossez et al. 2024) and GPT-4o "have been proposed that can perform turn-taking to achieve simultaneous real-time 2-channel conversation. However, these FMs have still not been quantitatively evaluated on their ability to perform turn-taking events.
Benchmarks for Audio FM: Benchmarks like Dynamic SUPERB (Huang et al. 2023) and AIR- Bench Yang et al. 2024 have been valuable for evaluating audio FMS on standard speech process-
2

---

Published as a conference paper at ICLR 2025
ing tasks, but they do not specifically assess conversational capabillitis. SD-Eval (Ao et al. 2024) claims to be an evaluation benchmark for spoken dialogue understanding and generation, evaluat- ing the quality of response generation using automated metrics like BLEU, GPT-4o, or subjective evaluation. However, in this work, we specifically focus on the turn-taking capabilities of audio FMs, particularly their ability understand, predict, and perform turnttaking events. In addition to previously proposed corpus-level statistics (Nguyen et al. [002),, we introduce our own metrics that evaluate the timing of turn-taking events. These metries help us gain insights into the limitations of existing systems in engaging in interactive and natural conversations.
3 TURN-TAKING ANALYSES BASED ON CONVENTIONAL CORPUS-LEVEL STATISTICS
In the given task setting, we evaluate the recently proposed dialogue systems with turn-taking capa- bilities using corpus-level statisties (Nguyen et al., (023). The authors organize user study and compute these statistics from the collected human- Al conversation data.
Dataset and Audio FM details: Participants were hired to engage in conversations with the dia- logue systems to mirror real-world usage of these systems. As a case study, we evaluate the follow- ing spoken dialogue systems: (a) Mosh which claims be fully E2E duplex spoken dialogue system; (b) Cascaded system of open-source models, including Silvero VAD (Team 2024). Whis- per tiny (Radford et al. 2022), SmolL.M-135MIInmne
(Allal et al. 2024), and Melo TTS (Zhao eet al. 2023c). Our selection of these two systems is driven by the goal of understanding the relative strengths and weaknesses of fully duplex E2E system compared
traditional cascaded pipeline. We collect roughly 4 hours of human-Al conversation data for each dialogue system, where each session lasts nearly minutes across 11 different participants. Our analysis confirms that the col- lected data is sufficient to derive statistically significant insights for all our evaluations (including Sec. D.. The participants primarily included the authors and their research colleagues. To minimize potential variations in results arising from different conversation topics and to facilitate smoother interactions, we provided participants with predefined list of topics to choose from. More details about our user study in Sec. A.2
3.1
EVENTS
Human-human conversations contain a spontaneous succession of turns where overlap and silences occur naturally. Let us assume a two speaker conversation. For simplicity, we divide entire con- versation audio into a sequence of N non-overlapping chunks the voice-activity of the first speaker can be represented by y' by = I,...) where y if speaker 1 is speaking in chunk i. Similarly, the voice-activity of speaker can also be represented as Y2. We show the turn-taking events that occur in spontaneous spoken conversations in Fig. (1) Inter-Pausal Unit (IPU) is a continuous stretch of speech in one speaker's channel, delimited by silence of more than 200ms from both sides, such that IPU (i [a, blly; 1), where and b denote the start and end chunk indexes of IPU* respectively and k can be or 2.
{U; | i I......! .). Then
Turn
IPU
Flking
(SIL)
Internuption
+1
b Dollo
(2) Silence (S IL) part of the conversation where there is no audio from both sides i.e. SII. = (i [asil, b... Jby = and y = 0), where اله and bail de- note the start and end chunk indexes.
(3) Pause is the silence that occurs between the successive IPUs of the same speaker. Succes- sive IPUs by the same speaker separated by a pause are grouped (U,1P0))(b)-1.4 ayl c SII.,, where b-- is the end chunk index of IPU*_, and ay is the start chunk index of IPU (4) Gap is the silence that occurs between IPUs of different speakers, i.e., preceding turn change.
Gap SSL
BC Butting-in
Figure 1: Overview of turn- furn-taking events in form a turn i.e. TURN human-human conversation
https://us moshi. moshi.chat/
https://github hww.hh/ww....ugugggggng///geeeee speech
3

---

Published as a conference paper at ICLR 2025
Nimber of ocoumencee min
Human Pet (witchooodd Modhi Cassanded
B7.3
Human Rel (Bwiichbbard) Moshi
46.00 of Cumsiited duration
$1.0
Cascaded
15.7
11.3
7.0
6.7
6.6
3.8
4,8
1.0
5.77
10.3
6.3
11.8
0.4
Gap
Overiap
Gap
(a) Number of turn-taking events per minute
(b) % of Cumulated durations of tum-takinn
Figure 2: Results of audio foundation models on engaging in conversation with humans based on corpus-level statistics proposed in prior works (Nguyen et al. 2023).
(5) Overlap occurs when both speakers talk simultaneously, i.e. y! and y² =1
(6) A backchannel is a short utterance such as "um" and "right", which the listener utters without taking the speaker's turn to acknowledge the current speaker. The backchannels typically occur dur- ing the speaker's turn. In this work, we denote the backchannel label sequence for the listener k as BC = (bc]li I...N where be 1 listener & backchannels in chunk
(7) An interruption is defined as overlapping speech between two speakers where both speak- ers are trying to take the turn. We can also define in terms of both speaker's 's IPU such that IPU* = fi E [at, D']I and IPU = li [a _b1], I]. where at < a" and a < b such that K' is the interrupting speaker. An interruption can further be classified into (i) Floor-taking / Successful interruption: Where the speaker who starts speaking within the IPU of the first speaker takes over the conversation i.c. 1² b². (ii) Butting-in Unsuccessful Interruption: Where the speaker who starts speaking within the IPU of the first speaker will be forced to wait for natural break or pause before speaking, and the first speaker will continue i.e. b" b².
3.2 CORPUS-LEVEL STATISTICS
The conversation is passed through a VAD model using pyannotate (Bredin et al. 2020) library. We use the VAD output to get voice activity vectors yAo and Hummn and analyze the statistics of IPU, Pause, Gap and Overlap events, focusing on the number of events per minute and their cumulated duration percentage of the total duration of the conversation. The results are shown in Fig. 2a and Fig. respecttiv For reference, we also include statistics derived from human-human con- versations the switchboard dataset (Godfrey et al. 1992). which serve as ground truth. We observe that L Moshi has a small gap (1 (11.8% of the cumulative duration in Fig. 2b) between speaker turns and some overlapping speech; however, the overlap rate is much lower than that seen in natural human dialogues. 2 In contrast, the cascaded system shows higher latency, resulting in a larger gap (32.4% of the cumulative duration) and minimal overlap, making the conversation feel less natural.
While these metrics capture how well the global turn- turn-taking events distribution in the generated dialogue matches with ground truth, it cannot evaluate the exact timing when turn-taking event happens. For instance, Fig. sssows that Moshi can generate overlapping speech; however, it remains unclear whether this overlap is appropriate and supportive of the user's statements unexpected and disruptive to the natural flow of the conversation.
4 PROPOSED TURN- TAKING ANALYSES WITH CEINGICENTRIC METRICS
Inspired by the limitations of existing corpus-level statistics, we propose to evaluate the timing of the turn-taking event by training model that predicts after every 40 ms chunk and is causal, i.e., trained to predict what turn-taking event will happen in the next 40 ms. We then use this predictor model proxy of human relevance judgments.
4.1 JUDGE TURN- TAKING MODEL
We take inspiration from prior work (Wang al. 2024 that passes single-channel input speech through the SSL speech foundation model and predicts the listener's 's behavior as turn-taking deci- sions that an ideal system should make. However, this approach cannot handle overlapping speech. Hence, we extend their labeling annotation scheme to model both the listener and speaker's 's behavior and instead use single-channel mixed speech input. We define the corresponding label sequence
4

---

Published as a conference paper at ICLR 2025
for turn turn-taking events as L = 14,1 I... where the label for each chunk can be one of the fol- lowing: (1) NA: If no one is talking i.e. i e SIL. (2) BC ("Backchannel"): If listener backchannels during the speaker's turn i.c. be, = and i E TURN where k can be either 1 or 2, (3) I ("Interrup- tion"): If both speakers are talking to take the turn and none of the speakers are hackchanneling i.e. y 1 and y/ and bc! 0 and be? 0 (4) T ("Turn Change"): If there is turn change. Special case for interruption: if it follows a floor-taking interruption (see Sec. 3.1. J. it is also considered a turn change. (5) C ("Continuation"), Otherwise.
Our model is causal such that it is trained to make future predictions, i.e., what turn-taking event I, will happen in the next chunk by conditioning on only the first i chunks i.e., U1--1 We further simplify by conditioning only on a prior context window O; of size W (See Eq. / in Appendix for more details). The audio in the context window is passed through encoders of the pre-trained speech foundation model, namely Whisper (Radford et al. (022), to generate acoustic representation h,- The acoustic representations are then passed through linear layer (Out(.) followed by softmax:
f(O;, ()) = Softmax(Out(h:))
(1)
where f(O;, is the likelihood that the turn-taking event I; will happen at chunk
i as predicted by our supervised turn-taking model. More details about the prob-
lem formulation and architecture of our model have been provided in Sec. A.3
Dataset details: We train our turn-taking pre- diction model on Switchboard dataset. Simi- lar to prior work (Wang et al.) 2024), we split dataset by conversations into 2000:300:138 for train, validation, and test respectively. We evaluated our model on the in-domain switch- board test set and additionally on 2 out-of- domain (OOD) datasets: the Columbia Games Corpus Gravano & Hirschberg, 2011) and the Fisher Corpus (Cieni et al. 2004). We report the model's performance using the ROC-AUC score (Area Under the Receiver Operating Characteristic Curve) similar to Wang et al. The context win- dow of the supervised model W 30 seconds. More details about the experiment setup and super- vised model hyperparameters can be found in Sec. A.4
Tum-Taking Taking Label
Switchboard (T)
Columbia Games (T) Fisher
(Continaation)
95.2
BC (Backchamee))
89.4
94,0
83.3
TTTum change)
90.8
$1.6
91.6
(liicruption)
91.3
92.6
91.8
(Silence)
Overall
92.0
91.5
91.0
Table I: Performance (ROC-AUC values) of su-
pervised turn-taking prediction model.
Table D shows that our supervised turn-taking prediction model can effec- tively handle interruptions and performs similar to prior approaches for other turn-taking events, such as those in Wang et al., which reports ROC-AUC scores of 90.29, 81.84, and 91.97 on Switch- board for C (continuation), BC (backchannel). and T (turn change), respectively Further, the model shows strong OOD generalization, achieving similar performance on the OOD spoken dia- logue corpora. This suggests that it can be reliably used to evaluate the precise timing of turn-taking decisions made by Al dialogue systems and serve as a proxy for human relevance judgments.
Results and Analysis:
4.2 CORE CONVERSATION CAPARILITIES
To develop metrics that comprehensively evaluate audio FMs' conversational capabilities, we review prior literature (Gravano & Hirschberg 201T Skantze 2021 Raux et al. (2066) and identify the key turn-taking abilities required for a conversational agent to interactively engage with an end user. We design metrics (Sec. 4.8) corresponding to each capability to ensure complete coverage of the skills required for natural human-A Al dialogues. These capabilities are categorized depending on whether they are useful for user input recognition system output generation, as shown below:
When user speaks: (a) When system should speak up? This means that when the user speaks, the system should know when to listen, when to speak, and when to pause, ensuring that the conversation feels natural and not rushed, If the FM speaks too early without much pause, it makes FM sound too eager. On the other hand, if the FM leaves too big of gap, makes the conversation unnatural. (b) When system should backchannel? The AI system should also provide subtle cues to the user that indicate the assistant is "listening" processing the user's input. (c) When system should interrupt?
The AI system could additionally make relevant follow-up interruptions that encourage further dialogue, making the conversation feel more interactive. However, if it interrupts too aggressively,
"However, results are not directly comparable since Wang et al. predicts at word boundaries, whereas we predict 40ms chunks.
5

---

Published as a conference paper at ICLR 2025
it can come across rude.
When system speaks: (d) Convey user when it can speak up? While generating speech output, should ensure that it conveys the end user if it wants to yield its turn or keep the conversation floor. (e) Handle user's 's interruptions: When a user interrupts the AI, it should address the interruption or gently steer the conversation back track.
It is important to note that these conversation capabilities are sensitive to the timing, and cannot be evaluated using the corpus-level statistics in Section 3.2 The following section shows that we can create the corresponding metrics by using our judge turn-taking model (Section
4.3 EVALUATION PROTOCOL
In this work, we design automated metrics for each conversation capability using our judge pre- dictor model (Section 4.1). Consistent with prior works on evaluating turn-taking classification models (Ekstedt & Skantze, 2022a), we initially propose metries that perform a pairwise compar- ison between surn-taking decisions, for instance, when the user pauses, can the audio FM infer when the user wants to hold the conversation floor when it is ok for assistant to speak up. It is important to note that while prior evaluation metrics (Sec. (Sec.A..1.I) focus on assessing turn-taking model's ability to predict upcoming turn-taking events, evaluation protocol presents the first ef- fort to assess audio FMs' ability to perform turn-taking events in human-Al conversations. We also compute the mean and standard deviation of each metric for each conversation session and show 95% confidence intervals assuming normal distribution.
The sequence of these conversational exchanges is denoted as Idialogue. YAi and ylluman can be computed similar to Sec. 3.2 The input speech for each utterance passed through an ASR model (Radford al. 2022) get the corresponding ASR transcript. We define filler word set as the most frequent isolated one and two-word phrases such as "hmm"", "oh", "okay", see", etc. We follow Wang et al.) and label filler words uttered by the listener in the other speaker's turn backchannel. Using this labeling scheme, we get backchannel sequence BCAI and BCHuman Using YAi, yHumam BCA BCHuman, we follow our turn-taking label annotation scheme (Sec. 4.1 to compute L dialogue as the turn-taking decisions that were actually made by the human and AI during the conversation. We can similarly define the turns of the Al dialogue system and the human user as TURNAL and TURN respectively. Furthermore, recognize turn-taking events at chunk as decisions made by the AI system if i-1 "TURyHuumal and as decisions made by Human if i - 1 TURNAL This distinction is motivated by prior work (Wang it al. 2024) where the listening system's is behavior is considered as turn-taking decisions.
Turn-taking decisions in collected user data:
Judging consistency of metries: To quantify the feasibility of using our trained turn-taking model (Section 4.1 as a judge, we take inspiration from prior works (Yang et al., 2024; Zheng et al., 2024] that have experimented with using an LLM as judge. These studies justify this approach by showing high consistency between LLM predictions and human relevance judgments. Similarly, we evaluate the consistency of our judge model with human judgments by examining instances in a human-human conversation dataset that correspond to each of the proposed metrics. We consider the decisions made by humans during the conversation as proxy for human judgments. We tune the thresholds defined below for all proposed metrics on an in-domain validation set to maximize the agreement between the judge labels and human judgments (Details in Sec. A.4). Finally, we report the agreement of judge label with human judgments on the in-domain and OOD test sets in Fig. and show that our proposed metrics mostly have high consistency (>60%) with human decisions.
4.4 METRIC (A) WHEN USER SPEAKS: WHEN SYSTEM SHOULD SPEAK UP?
To compute this metric, we look at those cases when the user is speaking and then pauses, i.e., i-l E Huma and 1:-1
dialogue = NA. Using f( dialogge (f) (Eq. 1b computed by the judge model, we
hypothesize that i) the turn change likelihood f( (()) (idilloogu [T] should be high for an ideal AI system when it decides to speak up, i.e., L,
dialogue T and ii) the continuation likelihood f( ((O,dinoogne [O[C] should be high when the system lets user continue i.e. 4. dialogue
C. The judge Label J! is:
T. if [O]T] 1(O,iialoge OO[C] > threshold,
5 = C.
(2)
otherwise

---

Published as a conference paper at ICLR 2025
Human Ref (In-domain)
Human Ref (OOD)
Moshi
Cascaded
80.2 81.8
92.8
47.6
74.9
71.7
74.9
82.1
91.7
78,
72.6
78.2
85:0
Alignmentt Judge
61.9
B6:2
65.5
52.8
30.4
24.2
9.1
When speaker continue
does not interrupt
decides to speak
backchannels (b) When to backchannel?
not backchennel
interrupts
(a) When to speak up?
(c) When to interrupt?
Alignment with Judge Label
80,2 84.4
74:9
70.8
61.9
63.1
52.6
55:9554:5
56.7
10.5
When listener decides to speak up
When speaker continue
When interrupting speaker takes the floor
When interrupting speaker does not take the floor
(d) Convey user to speak up?
(e) Handle user interruptions?
Figure 3: The results show the consistency of the AI dialogue system's 's turn-taking decisions with judge labels across our proposed metrics. The first graphs correspond to when Al system is listener and the remaining 2 graphs correspond to when Al system is speaker. Additionally, 95% confidence intervals are provided for AI system with all metrics (also in Appendix Tab. 77.. For each graph, the first two bars represent the consistency of our computed judge labels with human relevance judgments obtained from both an in-domain and out-of-domain spoken dialogue corpus.
where threshold, is hyperparameter. To compute human judgment, we look at instances where the speaker pauses i.e., 4-1 =NA and the listener decides whether to speak up i.e., (T C).
Results and Analysis: Fig. 33 show that I Moshi has significantly lower agreement with judge labels compared to humans when it allows users to continue speaking. This finding contrasts with Fig. which indicates that Moshi maintains gap similar to humans when it initiates speaking, thereby not highlighting any differences in turn-change capabilities. Upon manual inspection, we observed that Moshi occasionally fails speak up when the user is ready to yield its turn, consistent with our proposed metric. According to Moshi's 's architecture (Défossez et al.j 2024), there is explicit boundary to indicate changes, i.e., it listening and generating audio tokens at all time, simply initiates speaking when predicts the special token EPAD and it stays silent when it predicts the special token PAD. Increasing the EPAD logit bias when the user pauses- thus encouraging the selection of the EPAD token-coul improve Moshi's turn- surn-taking capabilities. ② The cascaded dialogue system performs even worse than Moshi in deciding when to speak up and when to allow the user to continue. It speaks up 37.1% of the time when the user pauses (Table much higher than that seen in human-human conversations, aligning with prior research that silence alone is often not reliable cue for turn-switching.
4.5 METRIC (B) WHEN USER SPEAKS: WHEN SYSTEM SHOULD BACKCHIANNEL?
Similarly, we look at when it is the user's turn, i.e., E TURNHummn We hypothesize that i) backchannel likelihood dialogee ()[BC] should be high when the system decides to backchannel, dialogue # BC. Judge Label J? is:
i.e., 4,
dialogue BC and ii) low when the system does not i.e. L,
BC.
if .0[BC] threshold,
(3)
# BC,
otherwise
where threshold, a hyperparameter. Human judgments are obtained from instances where either the listener begins backchanneling. i.e., 4-1 # BC and BC or it does not backchannel.

---

Published as a conference paper at ICLR 2025
Dialogue System
is Listener (b) % Backehannel
Al Speaker
(a) % Turn Change
(c) & Internuttion
(d) % Turn Change
% Interrustion ((e) & Floer (loor-taking)
Human Ref (in domain)
15.9
0.30
0.4
15.9
0.4 (63.6)
(OOD)
32.9
0.09
0.4
32.9
0.4 (60.8)
Moshi
32.7
0.5
Cuscaded
37.1
0.01
0.2
24.2
0.1 (59.4)
Table 2: Percentage of instances for each proposed metric where the Al makes a specific turn-taking decision. Brackets in the last column show the percentage of interruptions that result in floor-taking.
Results and Analysis: Fig. 3p show that both the dialogue systems do ! not backchannel at ap- propriate points, as indicated by low agreement with judge labels. The confidence intervals are large. which potentially results from small sample size @ both systems rarely backchannel (Tab. 2. Moshi's backchanneling capability probably results from its fine-tuning on speech conver- sations Défossez et al., 2024) and recommend increasing the size of this fine-tuning data.
4.6 METRIC (c) WHEN USER SPEAKS: WHEN SYSTEM SHOULD INTERRUPT?
Similarly, we hypothesize that i) the interruption likelihood ((O,ialogue .T[T]] should be high when the system decides to interrupt, i.e., L, dialogue and ii) the continuation likelihood I(Opdinlogne [O[C] should be high when the system does not i.e. L dialogue
C. Judge Label J
II. if E(C)dialouue ,0][ 1(O,dialogue .O[C] > threshold,
(4)
otherwise
where thresholds is a hyperparameter. To compute human judgments, we examine instances where only one speaker is active (1/1 =C) and then observe if both the interrupting and active speakers are talking simultaneously (4, =I) or if the same speaker continues speaking alone (4; =C).
Results and Analysis: Fig. 3 shows that i the agreement with judge labels is much lower when interruptions are made by Moshi. While corpus-level statistics (Fig show that both humans and Moshi exhibit overlapping speech, our proposed metric differentiates between them by revealing that interruptions made by humans are expected and collaborative, whereas interruptions made by Moshi are surprising and can be rude. 2 The cascaded system rarely interrupts (Table 2) and when it does, they are not well-timed (Fig. 3c), as they typically result from errors in the VAD output.
4.7 METRIC (D) WHEN SYSTEM SPEAKS: CAN CONVEY WHEN USER CAN SPEAK UP?
To compute this metric, we look at when the system is speaking and then pauses, i.e., i-1 TURNAL and L dialogue
NA. We hypothesize that the i) turn change likelihood f( I(O Oilialogue ()[T) should be high when the user believes that an ideal system can speak up, i.e., L, dialogue T and ii) the continuation
likelihood i (())llalogue [ should be high when the system continues i.e. L, diulogue C. Judge label (J. in Eq. 2 and human relevance judgments calculated similar to Metric (1) (Sec. [44).
Results and Analysis: Fig. 3 demonstrates that ① Moshi does not give users enough cues when it
wants to keep the conversation floor, leading to the lower agreement with judge labels and the user speaks up more frequently (46.5% in Tab. ( when Moshi pauses. Similar to Sec. 4.5 fine-tuning
on additional speech conversations could improve its turn willingness capability. 2 The cascaded
dialogue system was slightly better conveying turn willingness than Moshi.
4.8 METRIC (E) WHEN SYSTEM SPEAKS: HANDLE INTERRUPTIONS?
Ma et al.) has proposed Interactive capability metric to evaluate audio FM's capability to handle user's interruptions, which looks whether the system yields its turn when the user interrupts. Mo- tivated by this, we look at instances when the user has already made an interruption, i.e., 1
dialogue I. We hypothesize that i) the turn change likelihood I( E(O:liilogue, .0)TT] should be high when ideal AI system lets the user take the conversation floor (Sec. 3.1 i.e., L, dialogue and ii) the continuation likelihood f( r(),
dialoguc C. Judge Label J dialogue ()[ should be high when it keeps the floor i.e, L.
T. f( (ijalooo .O)TT - F(O) dialogue [O[C] > threshold,
(5)
C, otherwise
8

---

Published as a conference paper at ICLR 2025
C58.22 0.17 0.12 2.09 3.58
CB6.60
0.06
0.09
0.47
3.16
Judge Label
T 0.35 0.10 0.01
0.21 2.21
Judge Label
0.10 0.01
0.00
0.02
1.62
BC 0.17 0.01 0.00 0.04 0.10
BC
0.04 0.00
0.00
0.00
0.04
1 0.21 0.03 0.01 0.73 0.05
0.00
0.00 0.00 0.03 0.00
NA 3.74 0.34 0.01 0.08 27.44
NA 2.71 0.27 0.01 0.02 54.70
T BC I NA
C
BC NA
Decisions made by Moshi
Decisions made by Cascaded (b) Cascaded dialogue system
(a) Moshi
Figure 4: Confusion Matrix showing the performance of the turn-taking decisions dillogue made by the Al systems using the supervised turn-taking model as judge (i.e. L as the ground truth). The numbers the confusion matrix represent percentages.
where threshold, is a hyperparameter. To compute human relevance judgments, we identify in- stances where both the interrupting and active speakers are simultaneously speaking (4-1 =I) and then evaluate the consistency of the judge's label with speakers decision to allow a turn change. Results and Analysis: Compared to other metries, judge labels have lower consistency with human relevance judgments for this particular metric as shown in Fig. 3e. suggesting that insights derived using this metric may be less reliable. I Tab. 2 shows that only 17.4% of user interruptions led to turn change, with Moshi continuing to speak most of the time. Increasing the PAD (See Sec. 4.4) logit bias when the user interrupts could encourage Moshi to occasionally become silent, which could help it to avoid ignoring user interruptions entirely. ? Moshi's decisions do not differ sig- nificantly in agreement with judge labels compared to humans. ④ The cascaded dialogue system yielded its turn more than 50% of the time (Tab. 2 when the user interrupted, even when the user was not trying to take over the conversation but merely supporting the Al's thoughts. However, the number of user interruptions too few (0.1% shown Tab. 2 to draw strong conclusions,
resulting in large confidence intervals in Fig. 3
4.9 SINGLE-LABEL EVALUATION
our judge model in single answer grading setup (Zheng al. 2024) where we directly assign a turn-taking event label to each chunk, thus generating the pseudo ground truth labels _ by tuning operating points (thresholds, see Sec.
Additionally,
A.4] for the predicted likelihood on the
validation set 5) Using the generated ground truth labels gea
we compute a confusion matrix of
I. diallguue for decisions made by Al system. Fig. 44 and 4b show the confusion matrices for the decisions made by Moshi and the cascaded dialogue system, respectively. ! Moshi has high rate of false positives for interruptions (2.09% of total decisions). indicating that it interrupts the user too aggressively when it should have allowed the user to continue speaking (i.e., C). Decreasing the EPAD logit bias (Défossez et al. 2024) when the user actively speaking could reduce the frequency of these interruptions, making Moshi's behavior less rude. 2 Moshi is also not effective at backchanneling, with our judge model predicting few opportunities for human-Al interactions.
3 The accuracy of turn changes is low, mainly due to false negatives (2.21% % of total decisions), meaning Moshi often fails to speak up when it should, (4) For the cascaded system, there are too few possibilities of either interruption or backchannel showing that the conversation is not interactive.
5 ADDITIONAL EVALUATION ON UNDERSTANDING AND PREDICTING TURN-TAKING EVENTS
While most existing open source and proprietary audio FMs cannot perform turn-taking events, many audio FMs (Chu et al. 2024 2023) claim multi-turn dialogue capabilities where end-of-turn is manually specified by the user. Hence, we additionally investigate whether training on multi-turn audio dialogues can enable these FMs that cannot perform tunnttaking events to least understand the meaning of turn-taking events and recognize the acoustic and semantic cues that precede their occurrence. Sec. A.5 in the Appendix contains details about experiment setup and audio FMs.
'Due the significant class imbalance among turn-taking events, using a simple argmax is suitable.

---

Published as a conference paper at ICLR 2025
How Well Audio Foundation Models Understand Turn- Taking?: In this task setting, the audio FM is provided with the first chunks of audio and must predict the turn-taking event that occurred within the given audio. We focus on three specific turn-taking events: turn change T, backchannel BC, and interruption I. We present the performance of audio FMs in Table Among the open-source audio FMs, Qwen-Audio-Chal achieves the best performance. A cascade of Whisper and GPT-4o generally outperforms the open-source models. However, all models perform close to a random-guess baseline when it comes to understanding backchannels.
Model
Turn Change (7)
Backchannel (1)
Interruption (1)
Random Baseline
50.0
50.0
50.0
SALMONN
41.4
50.1
51.3
Owen2-Audio-Instuue
48.8
48.5
50.3
Qwen- AadiA-Adio-iii
56.7
52.7
69.5
Whisper GPT-4o
66.3
49.1
52.9
Table 3: Accuracy of audio foundation models on test bench- marks evaluating their ability to understand turn-taking events
How Well Audio Founda- tion Models Predict Turn- Taking?: In the given task setting, the audio FM is provided access the first chunks audio, and has to predict the turn- taking event that will hap- pen next i.e. 4, We focus on predicting four specific turn-taking events: turn change, backchannel, interruption, and whether the interruption is floor- taking. To identify areas for improvement, we compare the audio FM's performance against our supervised turn-taking model, using J! (Eq J² (Eq Jdd (Eq 4h and J (Eq(5) to predict turn change, backchannel, interruption, and floor-taking interruption, respectively. We present find- ings in Table 4 The cascade of Whisper and GPT4-o API is very good at predicting turn change, which is not very surprising since prior works (Ekstedt & Skantze 2020) have shown that text-based LM can effectively predict turn shifts. However, it cannot predict backchannel, interruptions, and whether the interruption will lead to turn change. All the open-source audio FMs perform close to a random-guess baseline at all tasks. Our findings indicate that there is substantial room for improvement the audio FM's ability understand and predict tur-ttakin events.
Model
Tum Change
BackChannel
Interruption
Floor- TakinTaking Interruption (f)
(1)
(T)
(1)
Random Baseline
50.0
Supervised Topline (Sec. (Sec...3
4.3
78.6
75.1
74.9
65.6
SALMONN
49.3
50.4
Qwen2-Andio Audio-
46.5
49.3
$1.5
$4.4
Qwen Audn--AnioCChat
49.9
Whisper+GPT-to
62.2
48.6
49,3
50.0
Table 4: Accuracy of audio foundation models test benchmarks evaluating their ability to predict future turn-taking events.
DISCUSSIONS AND CONCLUSIONS
Through an extensive survey of prior work, this work identifies the core turn management capabil- ities required to build an interactive voice assistant. To quantify the performance of existing audio FMs these conversational aspects, we propose automated metrics corresponding to each capabil- ity, representing the first effort to assess the quality of turn-taking decisions made by audio FMs. Our findings offer valuable insights into the strengths and limitations of existing spoken dialogue systems. We plan to make our evaluation platform public, hoping that these proposed metrics will be adopted by future research to develop audio FMs that can engage in more natural and interactive conversations with users.
I Scalability and Applicability; Our evaluation protocol needs a supervised dataset to train the judge model, limiting its scalability and applicability. This model can however be trained on any spoken conversation dataset containing speaker turns, transcripts, and timestamps, which are often available even for non-English languages as discussed in Sec. A.7 ② Modelling BackChan- nels: Our current approach for identifying backchannels relies on heuristics (S.4. and may miss certain backchannels which is limitation of our model. ② Performance of Turn Taking Model: The agreement between the judge labels and human judgments is below 80% for many metrics (Fig. (3). The primary contribution this work lies our novel and adaptable evaluation protocol, capable of integrating any turn-taking prediction model. Future efforts to improve model accuracy would further improve the protocol's 's reliability. ④ Limitations of User Study: This study currently tests only few audio FMs (Sec. A.9). and we plan to expand our benchmarking more audio FMs with turn-tat taking capabilities emerge. There is the potential for bias in our user study, as participants were primarily the authors and their research colleagues (see Sec. A.2.
Limitations:
10

---

Published as a conference paper at ICLR 2025
7 REPRODUCIBILITY STATEMENT
To ensure reproducibility of our results and provide researchers with the ability to evaluate their own pre-trained audio FMs using our evaluation platform, we will publicly release full codebase as part of the ESPnet (Watanabe et al., 2018) toolkit. The release can be followed here: https: //github. coom/espne//sppeet pu 1/5948 Specifically, we provide: ! The code to train judge tur-takkin model (Sec. 4.1) 2 The inference code for our judge model to generate turn- taking pseudo-labels for each chunk i. 3 The evaluation platform to compute our proposed metrics (Sec. (.3) using the pseudo-labels from our judge model. Researchers can then use our evaluation platform to benchmark their own audio FMs by collecting human-Al conversation data and comparing performance against other models using our evaluation metrics.
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. Smollm blazingly fast and remarkably powerful, 2024.
Junyi Ao, Yuancheng Wang, Xiaohai Tian, Dekun Chen, Jun Zhang, Lu Lu, Yuxuan Wang. Haizhou Li, and Zhizheng Wu. Sd-eval: A benchmark dataset for spoken dialogue understanding beyond words. CoRR, abs/2406. 13340, 2024. doi: 10.48550/ARXIV.2406.6 133400 URL https://doi. org/10. 485.0.4850//00 arxiv 2406. 13346
Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie- Phillipe Gill. Pyannote. audio: neural building blocks for speaker diarization. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 7124-7128. IEEE, 2020.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919 2023.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen2-audio technical report. arXiv preprint arXiv:2407. 10759, 2024.
Christopher Cieri, David Miller, and Kevin Walker. The fisher corpus: A resource for the next generations of speech-to-text. In LREC, volume pp. 69-71, 2004.
Samuele Corelll, Jee-weon Jung, Shinji Watanabe, and Stefano Squartini. One model to rule them all towards end-to-end joint speaker diarization and speech recognition. CoRR, abs/2310.01 688, 2023. doi: 10.48550/ARXIV. 11..550/A 10.01 URL https://doi.....o org/10..0011... /tt.///../5000..000 48550/8555//00 arviv. 2310. 01688
Edmilson da Silva Morais, Matheus Damasceno, Hagai Aronowitz, Aharon Satt, and Ron Hoory. Modeling turn-taking in human-to-human spoken dialogue datasets using self-supervised fea- tures. In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023, pp. 1-5. IEEE, 2023. doi: 10. 109/CSSSS4499 2023.10096777 URL http::://ooi wrw. 10. http://11111111111111 1119//CASSP99355/ 2023. 10096775
Alexandre Défossez, Laurent Mazaré, Manu Orsini. Amélic Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dia- logue. Technical report, Kyutai, September 2024. URL httphttttt///tta.orrg//// seishh.pdf.. pdf
Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen. Sophie Rosset, Eneko Agirre, and Mark Cieliebak. Survey on evaluation methods for dialogue systems. Artificial Intelligence Review, 54:755-810, 2021.
Erik Ekstedt and Gabriel Skantze. Turngpt: a transformer- language model for predict- ing turn-taking in spoken dialog. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 Novem- ber 2020, volume EMNLP 2020 of Findings of ACL, pp. 2981-2990. Association for Com- putational Linguistics. 2020. doi: 10. (0.18653/11/2200FFNDD SENDDING-EEMNLP.2... URL https: //doi . org 10. 18653/11122200 www.eiiinndnnnniii 2m8.

---

Published as a conference paper at ICLR 2025
Erik Ekstedt and Gabriel Skantze. Voice activity projection: Self-supervised learning of turntakking events. In Hanscok Ko and John H. L. Hansen (eds.), 23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022, pp. op-5519-51949 ISCA, 2022a. doi: 10.21 (022144///TRRSPPHHHH.O 10955. URL https://do/
10955
Erik Ekstedt and Gabriel Skantze. How much does prosody help turn-akking? investigations using voice activity projection models. In Oliver Lemon, Dilek Hakkani-Tür, Junyi Jessy Li, Arash Ashrafzadeh, Daniel Hernández García, Malihe Alikhani, David Vandyke, and Ondrej Dusek (eds.), Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL 2022, Edinburgh, UK, 07-09 September 2022, pp. 541-551. Association for Computational Linguistics, 2022b. doi: 10. //doi. //o0..org/10.11655/122222 http:///22222..iia111111111111 sigdial-1....
1.51. URL https:
Erik Ekstedt, Siyang Wang, Êva Székely, Joakim Gustafson, and Gabriel Skantze. Automatic eval- uation of turn-taking cues conversational speech synthesis. In Naomi Harte, Julie Carson- Berndsen, and Gareth Jones (eds.), 24th Annual Conference of the International Speech Com- munication Association, Interspeech 2023, Dublin, Ireland, August 20-24, 2023, pp. ISCA, 2023. doi: 1022437/NTTERSPEEHH.O0 2064- URL https::///ot/ org/ 10. 21437/ Interspeech.2021
Shinya Fujie, Kenta Fukushima, and Tetsunori Kobayashi. Back-channel feedback generation using linguistic and nonlinguistic information and its application spoken dialogue system. In 9th European Conference on Speech Communication and Technology. INTERSPEECH Eurospeeeh 2005, Lisbon, Portugal, September 4-8, 2005, pp. 889 892. ISCA, 2005. doi: 10.21437/ INTERSPPEECH.205-4000 htts::/tp:/// org/10....10.. 10. 21437 /
Inteerppeeeeehh.tee 21t.s//tttttteeeeeehc 2005-400)
Yusuke Fujita, Naoyuki Kanda, Shota Horiguchi, Yawen Xue, Kenji Nagamatsu, and Shinji Watan- abe. End-to-end neural speaker diarization with self-attention. In IEEE Automatic Speech Recognition and Understanding Workshop. ASRU 2019, Singapore, December 14-18, 2019, pp. 296-303, IEEE, 2019. doi: 10. 1109/ASRU460 201999003999
URL https:/:: org/10.
J.J. Godfrey, E.C. Holliman, and McDaniel. Switchboard: telephone speech corpus for re- search and development. In [Proceedings] ICASSP-92. 1992 IEEE International Confer- ence on Acoustics, Speech, and Signal Processing, volume 1, pp. 517-520 vol.1, 1992. doi: 10. N0.1109/CCASSPP1192.. 225858.
Agustín Gravano and Julia Hirschberg. Turn-taking cues in task-orie dialogue. Comput. Speech Lang.. 25(3):6011 634, 2011. doi: 10.10166.CSL.2210010000 URL hhttps://doi.org// http:///oo..rr//....
1016/j.cs1.200 100003]
Agustín Gravano and Julia Hirschberg. A corpus-based study of interruptions in spoken di- alogue. In /3th Annual Conference of the International Speech Communication Associa- tion, INTERSPEECH 2012, Portland, Oregon, USA, September 9-13, 2012, pp. 855-858. ISCA, 2012. doi: 1021147//NTERSPPEECH.... Interspeech.2011 193
URL https://doi. http::///di.or/11..114444/
Kohei Hara, Koji Inoue, Katsuya Takanashi, and Tatsuya Kawahara. Prediction of turn turn-taking using multitask learning with prediction of backchannels and fillers. In B. Yegnanarayana (ed.). 19th An- nual Conference of the International Speech Communication Association, Interspeech 2018, Hy- derabad, India, September 2-6, 2018, pp. 1442. URL httph:tpp://// //dt:::://do.... orgii.rr//// 21437 211377 2018
ISCA, 2018. doi: 10.2145/INTERSPEHCC
Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang. Chi- Yuan Hsiao, Chun-Yi Kuan, Haibin Wu, Sid- dhant Arora, Kai-Wei Chang. Jiatong Shi, Yifan Peng, Roshan S. Sharma, Shinji Watanabe. Bhik- sha Ramakrishnan, Shady Shehata, and Hung-yi Lee. Dynamic-superb: Towards A dynamic, col- laborative, and comprehensive instrctiion-tuing benchmark for speech. CoRR. abs/ abs/2309,0 2023. doi: 10.44550/ARXII.2300.0 190000 URL battps://don org/10 10. 48550/ w...v.2399. 10510.
12

---

Published as a conference paper at ICLR 2025
Michimasa Inaba, Yuya Chiba, Ryuichiro Higashinaka, Kazunori Komatani, Yusuke Miyao, and Takayuki Nagai. Collection and analysis travel agency task dialogues with age-diverse speak- In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 5759- 5767, 2022.
Koji Inoue, Bing'e er Jiang, Erik Ekstedt, Tatsuya Kawahara, and Gabriel Skantze. Multilingual turn- taking prediction using voice activity projection. In Nicoletta Calzolari, Min- Yen Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pp. 11873-11883. ELRA and ICCL,, 2024. URL
2024. wtw.:r/02..eeemamaiii.i.. -ree-maiinn 1036
Ryo Ishii, Xutong Ren, Michal Muszynski, and Louis-Philippe Morency. Can prediction of turn- management willingness improve turn-changing modeling? In Stacy Marsella, Rachael Jack, Hannes Högni Vilhjålmsson, Pedro Sequeira, and Emily S. Cross (eds.), /VA '20: ACM Inter- national Conference on Intelligent Virtual Agents, Virtual Event, Scotland, UK. October 20-22, 2020, pp. 28: 28:1-22::8 ACM, 2020. doi: 10. 10.1145/3883662.3422990 URL https:///oi..om 10. 0145/3383652 3423907
Ryo Ishii, Xutong Ren, Michal Muszynski, and Louis-Philippe Morency. Multimodal and mul- titask approach to listener's backchannel prediction: Can prediction of turn-changing and turn- management willingness improve backchannel modeling? In IVA '21: ACM International Con- ference Intelligent Viriual Agents, Virtual Event, Jupun, September 14-17, 2021, pp. 131-138. URL https: org 10. 1145/3472366
ACM, 2021. doi: 10.1145/34723066347833 3478360
Chi-Chun Lee and Shrikanth S. Narayanan. Predicting interruptions in dyadic spoken interactions. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Process- ing. ICASSP 2010, 14-19 March 2010, Sheraton Dallas Hotel, Dallas, Texas, USA, pp. 5250- 5253. IEEE, 2010. doi: 10.100//CASS7220005449999 ICASSP 2010. 5494991
URL htttp::///oi.orrtteere 100 1109/
Chi-Chun Lee, Sungbok Lee, and Shrikanth S. Narayanan. An analysis of multimodal cues of interruption in dyadic spoken interactions. In 9th Annual Conference of the International Speech Communication Association, INTERSPEECH 2008, Brisbane, Australia, September 22-26, 2008, pp. 1678-167666 1681. ISCA, 2008. doi: 1022147/INTERSSPEECHH.OO 2006-336.. URL htthh::////d: org/10. 211/3/0..113//// (tt.44/////ntepeeehh...a 2008 366
Bo Li, Tara N. Sainath, Arun Narayaman, Joe Caroselli. Michiel Bacchiani, Ananya Misra, Izhak Shafran, Haşim Sak, Golan Pundak, Kean Chin, Khe Chai Sim, Ron J. Weiss, Kevin W. Wilson, Ehsan Variani, Chanwoo Kim, Olivier Siohan, Mitchel Weintraub, Erik McDermott, Richard Rose, and Matt Shannon. Acoustic modeling for google home. In INTERSPEECH, pp. 399-403, 2017.
Siyan Li, Ashwin Paranjape, and Christopher D. Manning. When can I speak? predicting initiation points for spoken dialogue agents. In Oliver Lemon, Dilek Hakkani-Türr, Junyi Jessy Li, Arash Ashrafzadeh, Daniel Hernández García, Malihe Alikhani, David Vandyke, and Ondrej Dusek (eds.), Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL 2022, Edinburgh, UK, 07-09 September 2022, pp. 217-224. Association for Computational Linguistics, 2022. doi: 10. (0.18853/V11/0222SSGDDD //doi.org/10.
1.22. URL https:
Yi Liu, Pascale Fung, Yongsheng Yang, Christopher Cieri, Shudong Huang, and David Graff. Hkust/mts: A very large scale mandarin telephone speech corpus. In Chinese Spoken Language Processing: 5th International Symposium, ISCSLP 2006, Singapore, December 13-16, 2006. Pro-
ceedings. pp. 735-735. Springer, 2006.
Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, and Xie Chen. Language model can listen while speaking, 2024. URL hwtp::ttt/:// org org/abs// 2408. 02622
13

---

Published as a conference paper at ICLR 2025
Ryo Masumura, Tomohiro Tanaka, Atsushi Ando, Ryo Ishii, Ryuichiro Higashinaka, and Yushi Aono. Neural dialogue context online end-of-turn detection. In Kazunori Komatani, Diane J. Litman, Kai Yu, Lawrence Cavedon, Mikio Nakano, and Alex Papangelis (eds.), Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, Melbourne, Australia, July 12- 14, 2018, pp. 224-228 Association for Computational Linguistics, 2018. doi: 10.18653/V1/ W18-5024 URL httptttts///ioorrrg///
18653/v1/w18- 5024
Louis-Philippe Morency. Modeling human communication dynamics [social sciences]. IEEE Signal Process. Mag., 27(5):112-116, 2010. doi: 10.1109/MSP22010995330.. URL. https://doi.
org/10. 1109/MSP. 937500
Louis-Philiipe Morency, Iwan de Kok, and Jonathan Gratch. A probabilistic multimodal approach for predicting listener backchannels. Auton. Agents Multi Agent Syst. 20(1):70 84. 2010. doi: 10. 00.1007/111458-00900
Y. URL http:://////ooooo htg////0oooor//.... 1007/s10458 009 9092-y
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, AbdeIrahman Mohamed, and Emmanuel Dupoux. Generative spoken dialogue language modeling. Trans. Assoc Comput. Linguistics, 1 11:50-2666, 266, wet.o.1122tcccccc 00545
2023. doi: 10.1162/TACL) A) 00545.. URL https:
Paul Novello, Gaël Poëtte, David Lugato, and Pietro Marco Congedo. Goal-oriented sensitivity analysis of hyperparameters in deep learning, 2022. URL httpss//arriiv.orrgbb//22222 06216
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McL.eave, and llya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URL httlhhttt//aiiiii org/ abs/2212. 04356
Antoine Raux, Dan Bohus, Brian Langner, Alan W Black, and Maxine Eskenazi. Doing research on deployed spoken dialogue system: one year of let's go! experience. In INTERSPEECH, 2006.
Saman Razavi, Anthony Jakeman, Andrea Saltelli, Clémentine Prieur, Bertrand Iooss, Emanuele Borgonovo, Elmar Plischke, Samuele Lo Piano, Takuya Iwanaga, William Becker, Stefano Taran- tola, Joseph H.A. Guillaume, John Jakeman, Hoshin Gupta, Nicola Melillo, Giovanni Rabitti, Vincent Chabridon, Qingyun Duan, Xifu Sun, Stefán Smith, Razi Sheikholeslami, Nasim Hos- seini, Masoud Asadzadeh, Arnald Puy, Sergei Kucherenko, and Holger R. Maier. The future of sensitivity analysis: An essential discipline for systems modeling and policy support. Environ- mental Modelling and Software, 137:104954, 2021. ISSN 1364-8152. doi: https://oii.or//I1 10. URL
1016/j.envsoft.220 2020. 104954. pi1/$1364415222310112
Zahra Sadeghi and Stan Matwin. A review of global sensitivity analysis methods and a comparative 16975
case study digit classification, 2024.
Stefan Scherer, Derya Ozkan, and Louis-Philippe Morency. Investigating the influence of pause fillers for automatic backchannel prediction. In Feedback Behaviors in Dialog, 2012.
Gabriel Skantze. Turn-taking conversational systems and human-robot interaction: review. 2021.
Computer Speech & Language,
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview hett/forummidd14mm/hhp.
Judith M. Tanur. Margin of Error, pp. 765-765. Springer Berlin Heidelberg, Berlin, Heidelberg. 2011. ISBN 978-3-642-04899-2. doi: 10.1007/993.336422-448898
URL https://doi.
org/10 10. 01117/9988---64---488-----
Silero Team. Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier. hittp:://github. httpaa/ooooononnanns//sseeeereee
2024.
Louis Ten Bosch, Nelleke Oostdijk, and Lou Boves. On temporal aspects of turn taking in conver- sational dialogues. Speech Communication, 47(1-2):80-88) 86, 2005.
14

---

Published as a conference paper at ICLR 2025
Jan N. Rijn and Frank Hutter. Hyperparameter importance across datasets. In Proceedings of the 24th ACM SICKDD International Conference on Knowledge Discovery and Data Mining, KDD 18. pp. 2367-2376 ACM, July 2018. doi: 10.1145//2219119:220053 URLhttp:///o
org/10 10.
3220058
Jinhan Wang, Long Chen, Aparna Khare, Anirudh Raju, Pranav Dheram, Di He, Minhua Wu, An- dreas Stolcke, and Venkatesh Ravichandran. furn-taking and backchannel prediction with acous- tic and large language model fusion. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12121-12125. IEFE, 2024.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. Espnet: End-to-end speech processing toolkit. In B. Yegnanarayana (ed.). In- terspeech 2018, 19th Annual Conference the International Speech Communication Association, Hyderabad, India, 2-6 September 2018, pp. 2207-22 ISCA, 2018. doi: 10.21147//Inerspeeeee 2018- 1456. URL https: peattp::/doi..o http://ooo.oog/////// 21437 http///nnettreehehheccee 2018-11111 1456
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming, 2024. URI. wtt.htttt///ai.v.ooot org/ http://.rrrgab/////00// 16725
Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. Air-bench: Benchmarking large audio-language models via generative comprehension. CoRR, abs/2402.07729, 2024. doi: 10.48550/ARXIV. 2402.07729. URL https://:
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan Guan-Ting Lin, al. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2105.01051, 2021.
Guanlong Zhao, Quan Wang, Han Lu, Yiling Huang, and Ignacio López-Moreno. Augmenting waanformer-transdiie based speaker change detection with token-level training loss. In IEEE In- lernational Conference on Acoustics, Speech and Signal Prucessing /CASSP 2023, Rhodes Island, Greece, June 4- 2023, pp. 1-5. IEEE, 2023a. doi: 10. (00100//CASSS9495520000.0000 URL httts:://pooo.... org/ 10. 1.0.11000 1109 1C09///CSSS49357 2023. 10094955
Guanlong Zhao, Yongqiang Wang, Jason Pelecanos, Yu Zhang, Hank Liao, Yiling Huang, Han Lu, and Quan Wang. USM-SCD: multilingual speaker change detection based on large pretrained foundation models. CoRR, abs/2309.08022 090022,, 2023b. doi: 10.4850/ARXIV.2300000000 URL
https://doi. 0rg/10.c 48550 arXiv 2309. 08023
Wenliang Zhao, Xumin Yu, and Zengyi Qin. Melott: High-quality multi-lingual multi-accent text- to-speech, 2023c. URL https:/::: //gi thub cww/myshell wwwpssheellaaaaMeoeeaaaaa ai/MeloTTS
Liammin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PRIOR WORKS ON PREDICTING TURN-TAKING EVENTS
There have been a lot of prior efforts in predicting turn change, backchannel, and interruptions from spoken conversations. Prior works (Gravano & Hirschberg 2012 2011) have shown that spoken conversations consist of Transition- suggests that the listener is likely to speak up particular places in conversation, i.e. during or after the speech, with certain acoustic and prosodic properties. Gravanno & Hirschberg further, introduces detailed annotation scheme label various turn-taking events naturally occurring in spoken conversation and releases a Columbia Games corpus, which contains approximately 10 hours of annotated speech using this scheme.
place (TRP), where a turn-shift usually occurs. This
Predicting turn change: Prior works (Masumura et al., 2018 Ekstedt & Skantze 2020
da Silva Morais et al. 2023) have focused on predicting end-of-turn (EOT) events help voice assistants determine when to speak. These studies aim to identify, based on the conversation history, whether the current speaker's 's utterance has concluded. Consequently, their primary objective is to distinguish between gaps and pauses, and most of these models are not designed to handle overlap- ping speech. Another recent work (Li et al.] 2022) focuses on predicting the appropriate timing for when the assistant should speak up using acoustic and prosodic features.
Another related line of research involves detecting speaker changes in spoken conversations (Zhao et al., 2023b a). typically assuming minimal or no overlaps. In this approach, special speaker change token is inserted between different speakers' transcripts to generate training targets. Recent works (Fujita et al. 2019; Coreel et al. 2023) have extended this formulation by predicting the exact timestamps when a given speaker is active, a task referred to as speaker diarization. Some studies (Cornell et al. 2023) have also attempted to model speech recognition and speaker diariza- tion jointly. However, unlike these approaches, a turn-taking prediction model must operate causally i.c. predicting whether the speaker should speak up now based solely on the conversation history.
Most early works (Fujic et al., 2005) aim to predict backchannel only from acoustic information. They show that acoustic features and prosody are more beneficial for knowing when to generate backchannel than the actual content of speech. Building on these findings, there has similarly been interest in predicting backchannel from multimodal in- formation [Morency et al. 2010; Morency 2010 Scherer et al. (012).. Similarly, efforts have been made to predict interruptions and their types (competitive/cooperatiee by manually annotating the IEMOCAP dataset (Lee & Narayanan 2010 Lee et al. 2008).
Predicting backchannels and interruptions:
Multi-task prediction of Turn-Taking Events: Recent works often adopt a multi task learning approach to jointly predict various turn-taking events. For example, one study (Hara et al., 2018) uses multitask learning to predict turn changes along with backchannels and fillers. Other stud- ies (Ishi et al.) 2021 2020) explore joint modeling of turn changes, turn management willingness, and backchanneling Turn management willingness is related to the mental states of the speaker and listener in spontaneous conversations and can be categorized as follows: (1) turn-holding speaker's willingness to continue speaking, (2) turn-yielding - the speaker's willingness to listen, (3) turn-grabbing the listener's willingness to start speaking, and (4) listening the listener's willingness to continue listening.
the
While these works required supervised labeled data, recent works like the Voice Activity Projection (VAP) models (Ekstedt & Skantze 2022a Inoue et al. 2024) are trained in an unsupervised manner on spoken dialogue data to predict upcoming "speech" activity for each speaker in a future time window using pre-train speech representations and voice activity detection (VAD) features. The latest version of VAP (Inoue et af.) (024)) no longer requires VAD features and multilingual. The outputs from the two channels are fed into a cross-attention Transformer, similar to the Dialog GSLM model (Nguyen et al. 2023), and the system performs multitask learning on both VAP and VAD. They then successfully trained turn-taking models on Chinese (Mandarin) and Japanese using publicly available datasets (Liu et al. 2006; Inaba et al., (022). Interestingly, multilingual models trained on English, Chinese, and Japanese perform comparably to monolingual models despite the diverse turn- tan-itaing behaviors of these languages. EEstedt & Skantze investigate the role of prosody in turn-taking using the VAP model
16

---

Published as a conference paper at ICLR 2025
Multimodal FM: Recently multimodal FMs (Dfoossez al. 2024; Xie & Wu 2024) have been proposed that claim to be able to engage in real-time conversation with a user. Moshi (Défossez et al. 2024) is a full duplex E2E spoken dialogue framework based on a text language model backbone and models separately its own speech and that of the user into parallel streams. Further, Moshi does not explicitly model speaker turns, i.e., it is listening and generating audio tokens all time, it simply initiates speaking when it predicts the special token EPAD and it stays silent when it predicts the special token PAD.
A.1.1 PRIOR TURN TAKING EVALUATION METRICS
Most prior works (Mauumura et al. 2018: Wang et al. 2024) evaluate turn-taking models based on their ability to predict whether a turn-takin event, such as turn change or backchannel, will occur in the near future. This involves assessing how well turn-taking model forecasts when specific turn- taking events will happen in human-human conversations based on contextual cues. VAP (Ekstedt) & Skantze, (022a) similarly evaluates turn-taking models using four key metries: (a) SHIFT vs. HOLD (S/H), assessing how well a model predicts whether the current speaker will hold the turn or whether the turn will shift to the other speaker during mutual silence; (b) SHIFT prediction, evaluating the ability to predict an upcoming speaker change, while a speaker is still active; (c) Backehannel prediction, measuring how well upcoming backchannels are predicted; and (d) SHORT vs. LONG (S/I.), evaluating whether speaker change part of backchannel or a proper turn change. Additionally, [kstedt et al.] propose automatic evaluation method based on VAP to measure turn management behaviors (e.g., hold/yield) for conversational speech synthesis.
Our work distinguishes itself from these prior evaluation metrics as the first effort to evaluate Al dialogue systems' ability to perform tum-taakin events in spontaneous, interactive conversations with human users. This involves assessing how well the Al system actively decides to take the con- versation floor, yield it's turn, backchannel or interrupt the user during live human-AI interactions, reflecting its capability to engage in natural dialogue with the end user. Due to this fundamental difference in application focus, several challenges arise when attempting to apply prior turn-taking evaluation metrics (Ekstedt & Skantze 2022a) our setting:
I A Absence of Ground Truth:
In prior turn-taking evaluation methods, human-human turn-taking decisions serve as ground truth to assess how accurately model predicts upcoming turn-taking events. However, in Au conversations, there is no inherent ground truth to evaluate the quality of Al's turn-tuakiin tunnta decisions. Ground truth must be generated through human relevance judgments, requiring annotators to listen to entire human-Al conversations and determine whether the Al's turn-taking decisions were appropriate. For example, Fkstedt Skantze 's SHIFT vs. HOLD (S/H) metric evaluates whether turn-taking model can predict whether the current speaker will hold or yield its turn. We adapted this to assess whether an audio FM can correctly decide when to speak or when to allow the user to continue when the user pauses in interactive human-AI AI conversation. Since no predefined ground truth exists to judge these decisions, we introduced a protocol to generate pseudo ground- truth labels using a judge turn-taking model trained suman-huma conversations. The agreement between the Al's decisions and these pseudo labels serves as a measure of the quality of the Al system's 's turn-taking behavior.
② Dual Metrics for Human and AI Turns: Unlike prior works, our evaluation protocol requires separate modeling of turn-taking decisions for when the AI is the listener (human's turn) versus when the Al is the speaker (Al's turn). For example, in addition to evaluating the Al's behavior as a listener, we also assess it based on the turn taking events made by the user when it pauses during its turn. Specifically, we examine whether the Al provides clear cues to the end user to convey its willingness to yield the floor or retain it. This distinction crucial for understanding the Al's ability to appropriately formulate it's output to manage turn-taking effectively.
Distinet Metrics for Human and AI Turns: It is important to recognize that the same set of metrics cannot be applied uniformly to both the Al and human turns. For instance, during the human's turn, we evaluate the Al's decision to ceterupp-dterrmnnnng whether its interruptions are relevant and timely or overly aggressive. Conversely, during the Al's turn, it is not meaningful to assess the appropriateness of user interruptions. Instead, we focus on how the AI responds to user interruptions. Specifically, we evaluate whether the AI ignores interruptions entirely or, conversely, always becomes silent, even when the user is merely providing supportive feedback rather than attempting to take the conversation. To address these nuances of what metrics are appropriate,
17

---

Published as a conference paper at ICLR 2025
Roursom
them, what's gnlng net gaed.
Figure 5: Screenshot of Moshi demo as shown to the participants during user study
conducted thorough survey of prior work (see Sec. 4.2) to identify the key turn-taking abilities required for a conversational agent to engage effectively with end users. Based on these findings, we designed appropriate metrics for each ability, adapting existing metries when feasible and creating new ones where necessary.
④ Introducing New Metrics: Due to the difference in application focus, existing metries could not be fully adapted to capture the complete range of conversational capabilities. For example, prior metrics did not address the handling of interruptions. In this work, designed a labeling sequence for our judge model to explicitly differentiate between takr-taking (successful interruption) and butting-in (unsuccessful interruption) interruptions. This enhancement ensures that our evaluation protocol comprehensively assesses all key aspects of human-Al AI turn- turn-taking behavior.
In summary, our protocol builds on prior work by identifying core turn-taking abilities for human- Al interaction, adapting existing metrics, and introducing new ones for comprehensive evaluation across all turn-take turn-taking abilities.
A.2
USER STUDY
As discussed in Sec. 3 we user study where we hire participants to have conversation with spoken dialogue systems. For Moshi, we use their publicly available demo webpage https: momoshi.chate for participants to converse with the dialogue systems and collect their conver- sation recordings. The screenshot of Moshi's demo is shown in Fig. 5 For the cascaded dia- logue system, since the publicly available implementation https: thub. cww com./hugiingface.... speech-to-spedn
could only be run on the command line terminal, we built a gradio demo to facilitate more user-friendly interface for participants. A screenshot of the demo has been shown in Fig. [ and the demo will be made publicly available upon acceptance. We collect exactly 4 hours minutes and 3 hours 35 minutes of spoken conversation data across 11 different speakers with Moshi and cascaded dialogue system, respectively. The participants primarily included the au- thors and their research colleagues. Similar lab experiments are common in prior work (Deriu et al.) 2021}, enabling focused testing of specific functionalities (eg. prompting participants to interrupt the AI system) and facilitating deeper insights through discussions about their subjective experiences. However, acknowledge that this controlled setup may introduce bias. We hope our evaluation protocol inspires future studies to conduct larger-scale evaluations with more diverse participants. The instructions provided the participants are shown below:
18

---

Published as a conference paper at ICLR 2025
confitrt consider which danstc aandwicche
||
Delect- Mactbook
Clear
Figure 6: Screenshot of Cascaded demo shown to the participants during study
Turn Taking Event Label for next chunk (i.e. U3)
Linear Layer
Encoder Layer
Whisper Encoder Layers (x Ne)
Encoder Layer
Downsampled audios
U1
U2
Previous Acoustic Context
Figure 7: Architecture of our causal turn-taking model predicting at every 40ms chunk
Instruction for User Study: We are interested in studying and evaluating real-time dia- logue interactions during human-AI (audio FMs) conversations. To this end, we recruit par- ticipants to converse with the dialogue system and will analyze your conversation record- ings gain insights the dialogue interactions. If you are interested, please follow these guidelines to have a short conversation session minutes) on one of these topics:
(1) Feedback your pronunciation.
(2) Good places to eat Manhattan.
(3) Sightseeing in New York.
(4) Ask Moshi for help in learning python.
(5) Ask Moshi for help in preparing go for hike.
(6) Any topic of your choice.
Please label the audio with the topic you choose. Please do discuss anything you are not comfortable to share.
19

---

Published as a conference paper at ICLR 2025
Switchboard ROC-AUC
Turn- Mun-Takinn Label
F1
C (Continuation)
93.1
84.1
BC (Backchannel)
89.1
68.6
T (Turn change)
90.3
54.8
I (Interruption)
91.0
70.2
NA
95.2
86.6
Overall
91.7
72.9
Table 5: Validation performance of supervised turn-taking prediction model (Sec. 4.1
A.3 ARCHITECTRRE JUDGE TURN-TAKING MODEL
The input to the model is single-channel mixed speech X which can be represented as T T-length sequence of features = |x,]r I..... T). Through the maximum a posteriori (MAP) decision theory, tur-takking model estimates the label sequence L by maximizing posterior probability P(4X). By product rule, we have P(L[X) = IT; P(11/15-1, For simplicity, we divide the entire conversation audio into sequence of N non-overlapping chunks U = (U, U = XB; rl:s where B; = i Nylock and Nytock is the size of the chunk. Our model is causal such that is trained make future predictions. i.c., what turn-tale oun-ttaking event will happen in the next chunk? Hence, we condition on only the first chunks i.c., U1--1
......N) such that
P(LX) [ PiMl-ai. ULa-1).
(6)
From Sec. 3.1
we have U1s-1 = X1:8.1- By C.I. assumption
44-1 I U15-1, we get P(4X)
I P(I))..... We can further simplify by conditioning only on a prior context window O, of size W:
0, = X(B; -W)R
Using Ú, and CI assumption, we simplify Eq. 6 to:
P(4X)) P(JÚO.
1=1
Fig. shows the architecture of our causal turn-taking model. The input speech is first passed through encoders (Encoder(-)) of the pre-trained speech foundation model, namely Whisper RRad- ford et al. 2022). We perform weighted sum of the encoder's hidden states (Yang et al. 2021) and then use the encoder output of the last audio frame acoustic representation h, Encoder(U,). The acoustic representations are then passed through linear layer (Out(.) followed by softmax:
f(O,, 0) Softmax(Out(h)).
where f(O,, (4] is the likelihood that the turn-taking event will happen chunk as predicted by our supervised turn-taking model. The entire model is then trained using cross entropy loss. It important to note that while some prior approaches (Ekstedt & Skantze (2022a) can model interruptions, they require separate channels for each speaker, which might not always be available. Our approach, however, can effectively model both speakers' turn-taking behavior using single- channel mixed speech, thereby overcoming this limitation.
A.4 EXPERIMENT SETUP or SUPERVISED TURN-TAKIN MODEL.
The size of the chunk, i.c., Notock is 40msec. We use the Whisper medium encoder to generate acoustic representations. The context window of the supervised tum-taking model W is 30 seconds. To get backchannel annotations, we follow Wang et al.) and use the most common isolated one and two-word phrases backchannels Following (Wang al. 2024), downsample chunks with
backchannels...
20

---

Published as a conference paper at ICLR 2025
Switchboard Margin of Error
Turn-Taking Metric
Agreement
When to speak up?
When listener decides speak up
81.2
+6.43
When listener lets speaker continue
75.5
+7.46
When to backchannel?
When listener backchannels
71.6
+5.75
When listener does not backchannels
74.8
+3.49
When to interrupt?
When listener interrupt
76.6
+5.13
When listener does not interrupt
72.8
+1.80
Handle user interruptions?
When interrupting speaker takes the floor
61.3
+5.90
When interrupting speaker does not take the floor
57.0
+5.30
Table 6: Results presenting alignment of judge label with human decisions in in-domain validation set. We also show the Margin of Error (ME) (Tanur) (011) for 95% confidence intervals to quantify uncertainty in agreement between judge labels and human relevance judgments due to sensitivity to threshold values as discussed in Sec, A.8
Turn- Turn-Taking Metric
Moshi
Cascaded
(a) When to speak up?
When listener decides to speak up
[88.7, 94.2] [48.9, 56.5]
[84.2, 88.9]
When listener lets speaker continue
[38.4, 48.3]
(b) When to backehannel?
When listener backchannels
[22.9, 63.4] [85.7, 87.8]
0.0, 59.5]
When listener does not backchannels
[90.7, 92.5]
(c) When interrupt?
When listener interrupt
[31.1, 42.0]
[11.8, 32.6]
When listener does not interrupt
[77.3, 80.6]
[83.9, 85.5]
(d) Convey user to speak up?
When listener decides to speak up
[79.0, 84.9] [24.9, 36.2]
[77.3, 86.2]
When listener lets speaker continue Handle user interruptions?
[36.9, 44.1]
(e)
When interrupting speaker takes the floor
[26.2, 84.0]
0.0, 35.3]
When interrupting speaker does not take the floor
[37.2, 65.2]
[89.2, 100.0]
Table 7: Confidence intervals for the consistency of the AI dialogue system's 's turn-taking decisions with judge labels across our proposed metrics.
21

---

Published as a conference paper at ICLR 2025
Dialogue System
Speaking Rate
Backchannel rate
Switchboard
204.5
4.00
Moshi
223.7
0.25
Cascaded
206.5
0.13
Table 8: Performance of audio foundation models on engaging in conversation with humans based on Natural Dialogue Event Statistics i.e. Speaking rate and Backchannel rate.
label class C, NA, BC and from training set such that there are roughly similar numbers of samples for each label class. The validation performance shown Table [5)
We evaluated our model's OOD generalization on two out-of-domain (OOD) datasets: (1) the
Columbia Games Corpus (Gravano Hirschberg. 2011). 10-hour task-oriented spoken dialog corpus, and (2) the Fisher Corpus (Cieri et al. 2004), non-task-oriented spoken dialog corpus. The Fisher dataset's 's transcriptions were created using the Quick Transcription specification, which introduced inaccuracies and left significant portions untranscribed. To address this, developed heuristics to identify and exclude audio segments with large untranscribed content from the test set. We created random test split of 138 conversations, comprising 23 hours of audio, which will make publicly available. Further manual analysis revealed errors in the ground truth timestamps. We corrected these timestamps using speaker diarization outputs from Pyannote. All data preparation code and our turn-taking model will also be made publicly available.
For our proposed metrics (Sec. 4.4 we get the following values for threshold i.e. threshold, O, threshold2 = 0.1, threshold; = -0.4, threshold, = -0.1. The agreement between the judge labels and human judgment on the in-domain validation set shown in Table We discuss the impact of threshold tuning on our judge model's agreement with human judgment in Sec. A.8
For single-label evaluations (Sec. 4.9 operating points or thresholds for the predicted likelihood of label C = 0.2, NA = 0.45, = 0.4, BC = 0.4," T = 0.4. The validation F1 using these thresholds is shown in Tab. 5
A.5 DATASET AND AUDIO FM DETAILS FOR ADDITIONAL EVALUATION
Understand Turn- Tarn-Taking Events: To evaluate this, we create test bed where positive samples are those where 3i I; =l and negative samples are those where Vi 4, #I where I is either T, BC or I. The audio FM is then prompted answer simple Yes/No question. We create a test benchmark from 1500 samples from Switchboard's test set, such that there are 750 positive and 750 negative samples. Each audio sample has maximum length of 10 seconds. We evaluate both open-source Audio FMs, such as Qwen-Audio-Char (Chu et al. (202)) (7B). Siven-AAudo-Instruue 2024) (7B), and SALMONN (Tang et al.J 2024) (I3B), as well as cascade system combining Whisper and GPT-40 using the OpenAl API. We manually construct five prompts to query these FMs and present the results using the most effective prompt. The best prompt for each audio FM detailed in Tab. [2]
(Chu et al.)
Predict Turn- Turn-Taking Events: The samples in this evaluation benchmark for predicting turn change, backchannel, interruption, and floor-taking interruption are obtained from instances used to compute human relevance judgments for Metries (1), (2), (3), and (5), respectively. Similar to Setting 1, the audio FM is again prompted to answer simple Yes/No question. Similar to before, each evaluation benchmark consist of 1500 sample with equal positive and negative instances. Each audio sample has maximum length of W 30 seconds, matching the context window of the supervised topline model. We evaluate the same audio FMs as before. We again manually construct five prompts to query these FMs and the best prompt for each audio FM is detailed in Tab. 13 and
[4
A.6 NATURAL DIALOGUE EVENT STATISTICS
The input speech for each utterance is passed through an ASR model (Radford et al. 2022) to get the corresponding ASR transcript. We use the generated transcript to get the speaking rate (i.e.
Except benchmark of floor-taking interruptions, which only has 250 samples since interruptions are rare.
22

---

Published as a conference paper at ICLR 2025
number of words/minute). We compute backchannel sequence BC and HC (See Sec. 4.3
and report the backchannel word rate (i.e. number of backchannel words/minute) Tab. shows that ! both dialogue systems speak faster than the average human, with Moshi, in particular, speaking at a significantly higher rate, and 2 both dialogue systems do not produce a similar number of backchannels as humans.
A.7 SCALABILITY APPLICABILITY NON ENGLISH LOW RESEOURCE LANGUAGES
As noted in Appendix A.I, prior work (Inoue et al. 2024) successfully trained turn-taking models on Chinese (Mandarin) and Japanese using publicly available datasets. Interestingly, multilingual mod- els trained on English, Chinese, and Japanese perform comparably to monolingual models despite the diverse turn taking behaviors of these languages.
To address scenarios without supervised datasets, we propose low-cost solution: collecting a small spoken dataset for the target language and generating annotations through human efforts or using tools like PyAnnote (speaker diarization) and Whisper (ASR). We can then train multilingual turn- taking models that leverage high-resource language data to improve performance on low-rews low-resource languages. This approach is still far more cost-effective than collecting human relevance judgments for every turn-taking event.
A.8
IMPACT OF TUNING THRESHOLDS FOR JUDGE LABELS
The number of turn taking events are extremely unbalanced with continuation (C) and silence (NA) making up the label set for more than 95% of instances. As result, it is common practice in prior literature (Ekstedt& & Skantze 2022a) tune the threshold for each label on the validation set. We further surveyed prior literature to better quantify the impact of threshold tuning on our judge model's s performance and identified two main approaches:
! Sensitivity Analysis (Novello et al., 2022, Razavi et al. 2021 Sadeghi & Matwin, 2024): Prior works have experimented with varying key hyperparameters systematically over a range and ana- lyzed validation performance as a function of hyperparameters. Inspired by this, we vary thresholds from -0.5 to 0.5 for metrics where judge labels are computed using the difference of likelihoods (all metrics (Sec. (.8) except metric (B) (see Eq.3) 33) and from 0 to for others (i.e., for metric B in Sec. 4.5). The thresholds are incremented in steps of 0.01. We then calculated the Margin of Er- (ME) (Tanur 2011) for 95% Confidence Intervals to quantify uncertainty in agreement between judge labels and human relevance judgments due to sensitivity to threshold values. Margin of error (ME) is calculated as:
ME
(10)
Where (z = 1.96): Z-score for a 95% confidence level, (or): Standard deviation and (n): Sample size (i.e. Size of threshold range over which performance is computed =100 for our experiments) The observed margin of errors, along with the agreement on the in-domain validation set, are shown in Tab. (6 Generally, a margin of error of below 5% is considered excellent for high-accuracy needs, and less than 10% is acceptable for most studies. Our analysis shows that the agreement with human judgements does not undergo huge fluctuations with changes in threshold (i.e., margin of error is always less than 10%), and hence, our approach does not require extensive hyperparameter tuning.
the judge label with human decisions
② Validation on Multiple Datasets (yan Rijn & Hutter 2018): Prior works have argued the reli- ability of their model by showing consistent performance across datasets, demonstrating that the model generalizes well without dataset-specifee tuning. In Figure 3, our judge labels achieve good agreement with human judgment even on the OOD spoken dialog dataset, i.e., Columbia Games Corpus, without threshold being specifically tuned for this dataset. This result shows that our model achieves consistent performance without dataset-specific tuning.
A.9 RESULTS WITH GPT-4o
When GPT-4o's Advanced Voice Mode became publicly available, we initiated data collection of human-GPT conversations, accumulating 2 hours and 36 minutes of audio across six speakers. Our experiments have revealed many interesting insights.
23

---

Published as a conference paper at ICLR 2025
Turn taking event
Number of events/minute
% Cumulated Duration
IPU
13.9
71.9
Pause
11.4
13.2
Gap
2.3
15.2
Overlap
0.3
0.2
Table 9: Results of GPT-4o on engaging in conversation with humans based on corpus-level statistics
proposed in prior works (Nguyen al. 2023).
Al is Listener
Al Speaker
(a) % Turn Change
(b) Backehannel
(c) Interrutiion
(d) % Turn Change
% Interruption ((e) % Floor-taking)
37.2
0.003
0.06
13.6
0.04 (67.3)
Table 10: Percentage of instances for each proposed metric where GPT-4o makes specific turn- taking decision. Brackets in the last column show the percentage of interruptions that result in floor-taking.
Corpus-level Statistics: The corpus level statistics for human GPT-4o conversation are shown in Tab. ① Turn- Tun--akinn Latency: GPT-4o exhibits moderate gap between speaker turns (13.2% of cumulative duration in Tab. 9). smaller than the Cascaded system (32.5% in Fig. 2b) but larger
than Moshi (11.8% Fig. 2bp. indicating intermediate latency. 2 Overlap: Similar to a cascaded system (Fig. 2 GPT-4o has minimal overlapping speech (0.2% of cumulative duration in Tab. 1). resulting in less interactive conversations. 3 Turn- Muin-iiediig Behavior: GPT-4o has high number of pause events (1 4.4 per minute) and fewer gap events (2.3 per minute) compared the other dialogue systems (Fig. 2ah. indicating that it sometimes speaks for very long without yielding it's turn, which makes the conversation bland and less engaging for end users.
Our Proposed Metrics: ! Tab. 10 show that instances of GPT-4o backchanneling and GPT-4o and user interrupting each other too few. This leads to large confidence intervals for Metries B, C and E, making it challenging to derive conclusive insights. 2 Metric A: GPT-4o demonstrates strong performance in determining when to allow the user to continue speaking, achieving 71.6% agreement with human judge labels (Tab. I.. This is significantly higher than Moshi (52.8%) and the Cascaded system (44.7%) (Fig. Dd. 3 Metric GPTT-4o is similarly significantly better at conveying users when it wants to keep the conversation floor, having 70.5% agreement with judge labels (Tab. TT] which is much higher than Moshi (32.7% agreement in Fig 3) and Cascaded system (40.8% agreement in Fig 3d).
While our analysis limited few audio FMs, this a developing field of research, and we plan to expand our benchmarking more audio FMs with turn-taking capabilities emerge.
24

---

Published as a conference paper at ICLR 2025
Tukn-naking Metric
% Consistency with Judge
Confidence Interval
When speak up?
When listener decides speak up
84.8 71.6
70.8. 89.0] 62.9, 81.4]
When listener lets speaker continue
(b) When to backchannel?
When listener backehannels
0.0
[ 0.0, 0.0] 16,44, 90.7]
When listener does not backchannels
89.3
When interrupt?
When listener interrupt
59.3
[-12.7, 58.3]
When listener does not interrupt
88.2
85.9, 90.6]
(d) Convey user to speak up?
When listener decides speak up
86.9
80.6, 91.4]
When listener lets speaker continue
70.5
60.8, 75.2]
(c) Handle user interruptions?
When interrupting speaker takes the floor
51.4
26.2, 83.8] 14.1, 93.5]
When interrupting speaker does not take the floor
70.6
Table 11: Consistency of the GPT-4o's tum-taking decisions with judge labels across proposed metrics along with 95% confidence intervals
25

---

Published as a conference paper at ICLR 2025
Turn-taking event
Model
Prompts
Example Output
SALMONN
No. the given audio
does not have turn change.
Turn Change
Qwen2-
Does the given audio has turn change anywhere? Briefly explain your answer. You must reply with (a) Yes or No.
No, there is turn change in the speech as the pitch and tone remain consis- tent throughout the recording.
Audio-
Instruct
Qwen-Audio- Chat
Yes. there is turn change in the given audio. The male speaker starts speak- ing and then the fe- male speaker starts speaking.
Whisper+GPT-
Does the given audio has turn change anywhere? Briefly explain your answer. You must reply with (a) Yes (b) No.
(b) No
4o
SALMONN
Backchannels are brief words or sounds (like 'um," 'right,' or 'okay") that a listener makes to show they're re paying attention. Do the 2 speakers backchannel anywhere in the provided audio? Briefly explain your answer. You must reply with (a) Yes or (b) No.
No, the speakers do not backchannel in the provided audio. No, there are backchannels this audio.
Backchannel
Qwen2-
Audio-
Instruct
Qwen-Audic
Yes, the two speakers backehannel briefly
Chat
in the provided au- dio.
Whisper+GPT-
Do the 2 speakers backchan- nel anywhere in the pro- vided audio? Briefly explain your answer. You must reply with (a) Yes (b) No.
(b) No
4o
SALMONN
Do the 2 speakers interrupt each other anywhere in the provided audio? Briefly explain your answer. You must reply with (a) Yes or (b) No.
No, the 2 speakers do interrupt each other in the provided audio.
Interruption
Qwen2- Audio-
No, the speakers do not interrupt each other. They speak at different times and their overlapping minimal.
Instruct
Qwen-Audio-
Yes, the two speakers interrupt each other.
Chat
Whisper+GPT
the 2 speakers interrupt each other anywhere in the provided audio? Briefly ex- plain your answer. You must reply with (a) Yes or (b) No.
(b) No
4o
Table 12: Prompts for audio foundation models test benchmarks evaluating their ability to derstand turn-taking events. All open source FMs have same best prompt.
26

---

Published as a conference paper at ICLR 2025
Tum-takkin
Model
Prompts
Example Output
SALMONN
You are provided the audio of a 2 speaker conversation. The current speaker takes pause the end of current audio. Given the provided audio, do you think that after the pause, the current speaker will (A) continue talking or (B) the socond speaker will speak up. Output only A B, nothing elsc. Among
(B) Thee second speaker will speak up The current speakeer will continue talking,
Qwen2-
Tur Change
Audio-
Instruct
Qwen-Audio-
Chat
answer is
Whisper+GPT-
You are provided the audio of a 2 speaker conver- sation. The current speaker takes pause at the end of current andio. Given the provided audio, da you think that after the pause, the current speaker will (a) continue talking or (b) the sccond speaker will speak up. Briefly explain your answer. You must reply with one of 2 options.
(b) the second speaker will speak up. The current speaker has asked direct which typi- cally prompts response from the second speaker.
4a
SALMONN
You are given the audio of two speaker conversation. Based on provided audio, do you predict that cither speaker will produce a backechnnel g., short ac acknowleet ledgeeent response) immediutely after the end of the andio! Please explain your reasoning brielly. You must answer (a) Yes (b)
No, I do not predict that either speaker will pro- duce a backchannel immediately after end the audio. Yes
Backehannel
Qwen2-
Andio-
Instruct
Qwen-Audio-
Based on the provided andio, likely that ther speaker will produce a backchunnel response immesiately after the end cause the conversation between two people who discussing topic and exchanging ideas, and it is common for people to respand or acknow each other after pouse when the conversation is wrapping up.
Chat
audio. This be-
Whisper+GPT-
You are provided the audio of 2 speaker conver- sation. Given the provided andio, do you think any of the 2 speaker will backchammel exactly uf- ter the end of the andio. Answer no. Briefly explain your answer.
(b) no. Explanation: The provided text does not indicate any clear cues signals would prumpi an immediate backchumnel response from the other speaker. The speaker expressing per- sonal thoughts and opinions, and there is no ev- ident pause question that would naturally lead to backchannel respouse right at the end of the audio.
yes
SALMONN
During spontaneous haman conversation, interupptions occur frequently and often correspund to breaks between conversation partacrs. You are analyzing brief audio clip from two-person conversation. The clip ends ahruptly while the current speaker still the middle of sentence. Based ONLY on the provided audio. is it more likely that: (a) The other speaker will interrupt BEFORE the current speaker linishes their sentence?? The other speaker will let the current speaker to continue speaker? Justify your choice, considering the factors that might lead to an
(a) other speaker will interrppt BEPORE the current speaker finishes their sentence.
Interruption
QwenZ- Audio-
information flow
is not possible deternine with certainty whether the other speaker will interrupt before the current speaker finishes their sentence based solely on the provided aadio.
Instruct
Owen-Audio-
(a) The other speaker will interrupt BEFORE the current speaker finishes their sentence. The con- versation ends abruptly. and the other speaker is inter- rupted. Additionally. the other speaker mentions "that wouldn 't be fair" and "what they would have to do," indicating that they may be about to make argument or disagreement, which could lead an interrpptinn.
Chat
still speaking. suggesting that they
interruption encourage someone to wait their turn.
Whisper+GPPT
During spontancous buman conversation, inter- ruptions occar frequently and often correspond to breaks in the information flow between conversa- tion partners. Interruptions can happen duc variety of reasons like urgency, disagreement, etc. Interruptions be broadly classified into two categories: competitive and cooperative interrup- tions. Occurrences of competitive interrupiions are usually dismupting to flow conversation between speakers while cooperative interruptions are more supportive the flow: You mnalying brief audio clip from a two-person conversation. The clip ends abruptly while current speuker is still the middle of a sentence. is unclcur from the provided audio whether the speaker about to pause, is striggling find right word, or will continue speuking uninternupted. Based ONLY on the provided audio, is it more likely The other speaker will interrupt BEFORE the cur- speaker finishes their sentence? The other speaker will let the current speaker to continue speaker? Justify your choice, considering the fac- that might lead to an intermpption or encourage someone to wait their turn.
Based the provided audio, it is more likely that (b) other speaker will let the current speakker continue speaking. Justification: 1. +eFlow of Conversation The current speaker is narrating a story with clear sequence of events. The nar- rative
40
to reuching conclusion, indi- cated by the speaker's 's expression of salisfaction (1 just had the biggest grin tener's Engagerment ; other speaker has ready shown engagement and support by interject- ing with I'll bet you did, which terruption. This indicates that the listener is fol- lowing the to finish their thought. 3. **Lack of Urgency or Disagreement **; There are signs of urgency disugreemmnt the provided audio. The tone of the conversation is casual and friendly, which reduces likelihood of competitive interrup- tion. 4, *Naarrative Complction The current speaker in the middle of sentence that seems to he wrapping up the story. Interrupting at this point would disrupt the natural conclusion of the narrative. Given these factors, able to expect that the other speaker will wait for the current speaker to finish their sontence before responding.
faco). 2. **Lis-
cooperative in-
and is likely allow the speaker
(a)
more reason-
Table 13: Prompts for audio foundation models on test benchmarks evaluating their ability to predict turn-taking events. All open source FMs have same best prompt.
27

---

Published as a conference paper at ICLR 2025
Turn laking event
Model
Prompts
Example Output
Successful
SALMONN
You are examining a short midio clip from a
(B) The second speaker's overlap is a mistimed attempt start their turn, and the first speaker will continue, forcing the second speaker wait for a natural break or pause before speaking
conversation between people. The audio ends second speaker begins talk, overlapping with the first speaker. It's escerpt the
saddenly RIGHT
Qwen2-
impussible to tell from this first speaker was about to pause, was having trouble fioding a word,
Audio-
Instruct
would have continued speaking. Based ONLY Y on the provided audio, is more likely that: (A) The second speaker's 's overlap is successful interruption, and they will take over the conversation? (B) The second speaker's es overlap their turn, and the first speaker will continue. forcing the second speaker to for natural break or pause before speaking? Output only A nothing else. Among A or B. unswer is
Interruption
Qwen-Andio-
Based solely on the provided audio, it is more likely that the second speaker's overlap cessful interruption. and they will take over the conversation. This bocause the audio eads ahrupily as the second speaker begins speak. sugeesting that the first speaker was interrupted. Additionally, the second speaker's confident and assertive, which is characteristic of who trying take over conversation. On the other hand, the first speaker's '''s respouse is short and does not indieate that they are about pause or continue speaking.
Chat
mistimed altempt
Whisper+GGPT
During spontaneous human conversation, inter. ruplions occur frequently and often currespond lo breaks the information flow hetween conver- sation pariners. Interruptions
Rased on the provided audio excerpt, it more likely that: The second speaker's overlup is a mistimed attempt to start their turn, and the first speaker will continue, forcing the second speaker to wait for a natural break or pause before speak- ing. Justification: I. * *oonte and Flow The first speaker is in the middle of detailed expla- nation about weather pattems and their impact on snowfall. They have not yet reached clear con- clusion or natural pause in their narrative. The sentence But have playedsnggests that the first speaker is about to continue with a related point ancedote, 2. "<Timing of Overlap* to; The over- lap occurs just as the speaker is transitioning to new but related point (But I have played). speaker not finished and is likely to continue speaking. The sccond speaker's attempt interject ment is mistimed because it interrupis the flow of the first speaker's 's ongaing thought.
4a
huppen due la variety of reasons like urgency, disagreement, etc. Interruptions can be breadly classified into two catepories: competitive and cooperative terruttions Occurrences of competitive interrup- tions usually distupting to the flow of con- versation between speakets while cooperative in- terrpttonns more supportive
the flow. You are examining a short audio clip from a conver- sation between two people. The audio ends sud- denly RIGHT the second speaker begins talk, overlapping with the first speaker. It's impossible from brief excerpt if the first speaker was about to pause, was having trouble finding word, would have continued speaking. Based ONLY on the provided audio, is it more likely that: (a) The second speaker's overlup is suc- cessful interpptiion,
timing suggests that
**Con- versational Dynamics " spontanenus conver- sution, speakers often signal the end of their turn clear pause or a concluding statement. The first speaker has not provided such signal, mak- ing
they take over conversation? (b) The second speuker's 's overlap is mistimed attempt start their turn, and first speaker will continue, forcing the second speaker wait for natural break pause hefore speak- ing? Justify your choice, considering the fac- tors that might lead to successtul interruption or cacourage somcone to wait their turn. but fo- cus on how the SPECIFIC TIMING of this over- lap makes one outcome more likely the other.
likely that they will cantinue speak- ing despite the second speaker's 's allempt to inter- ject. Given these factors, the specific timing the overlap makes more likely that the first speaker will continue, and the second speaker will need to more appropriste moment take turn.
Table 14: Prompts for audio foundation models on test benchmarks evaluating their ability to predict turn-taking events. All open source FMs have same best prompt.
28


Reviews:
[{"review_id": "KX8UUn8uwL", "reviewer": "Reviewer_WGPU", "summary": "The paper proposes a new evaluation protocol to assess the spoken dialog system's turn-taking capabilities, i.e., the Moshi and Cascaded model. They use a supervised model as a judge which is trained to predict turn-tasking events in human-human conversation (i.e., Switchboard). The paper presents a comprehensive user study that evaluates the Moshi and Casaded model on their ability to perform turn-taking events, and it finds that they sometimes do not understand when to speak up, can interrupt too aggressively, and rarely backchannel. The main contributions are:\n1. A new evaluation protocol to assess the spoken dialog system's turn-taking capabilities.\n2. Some insight about existing spoken dialogue systems through user study.\n3. Additionally create a test benchmark using Switchboard dataset to evaluate SALMONN, Qwen2-audio-instruct, Qwen-audiochat, Whisper+GPT-4o on their ability to understand and predict turn-taking events.", "questions": "1. The Fisher dataset is a common dataset comparable to Switchboard. What is the performance of the supervised turn-taking prediction model on this dataset?\n2. How can your evaluation protocol be adapted or applied in scenarios where supervised datasets are not available for other languages, such as Chinese?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The originality of the work is commendable. The authors propose a novel evaluation protocol to assess the turn-taking capabilities of spoken dialog systems.\n2. The paper is well-written and provides sufficient experimental details in the Appendix.\n3. The authors plan to open source the evaluation platform.", "weaknesses": "1. The evaluation protocol is highly expensive, as it requires a supervised dataset to train the judge model. This approach is not feasible if we lack a supervised dataset in other languages, such as Chinese.\n2. The filler word set for backchannel detection is heuristic and may miss some backchannel cases that are not included in the filler word set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FY0OtTM7Xi", "reviewer": "Reviewer_Ddi6", "summary": "This paper addresses the challenges of evaluating the turn-taking capabilities of audio foundation models (FMs) in conversational settings.\nIt defines 6 types of Turn-Taking Events and evaluates the performance of end-to-end speech dialogue models as well as cascaded systems.\nThrough the results obtained from this study, the authors discovered numerous issues with existing AI dialogue systems in handling turn-taking, such as sometimes failing to intervene in conversations at appropriate times or excessively interrupting others. Furthermore, the authors conducted tests on multiple open-source and closed-source audio foundation models, revealing their limitations in understanding and predicting turn-taking events, and highlighting areas for improvement.", "questions": "NA.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The definition of turn-taking is detailed and clear.\n\n2. The evaluation protocol proposed in this paper contributes to better assessing the performance of audio foundation models in dialogues, providing strong support for the development of voice dialogue systems.\n\n3. This paper reveals many issues existing in current AI dialogue systems when handling turn-taking, offering valuable references for future research.", "weaknesses": "1. This study only tested a few open-source and closed-source audio FM models.\n\n2. There is a lack of comprehensive performance evaluation and summary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NNDKxnkP88", "reviewer": "Reviewer_cJr2", "summary": "This paper proposes an evaluation protocol to measure the turn-taking capabilities of audio foundation models.", "questions": "Here are questions for the authors:\n - The thresholds in Sec. 4.4-4.8 seem arbitrary. Is there a specific reason for choosing these values? All units appear to represent likelihoods, yet they range from negative ($threshold_3$ = -0.45) to positive values ($threshold_2$ = 0.1).\n - There are concerns about the reliability of the judge model. Since all results are based on comparisons with this model, is there concrete evidence supporting its credibility? Specifically, the conclusion that Moshi[1] is \"too aggressive\" lacks persuasiveness if it relies solely on comparisons with the judge model.\n\n[1] Defossez et al. Moshi: a speech-text foundation model for real-time dialogue", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strengths of this paper are as follows:\n 1. This paper provides an automated turn-taking protocol for audio foundation models\n 2. The evaluation platform will be open-sourced.", "weaknesses": "The weaknesses of this paper are as follows:\n 1. The study aims to measure precise turn-taking, but the thresholds are set to arbitrary values.\n 2. The participants introduced in Sec 3 seem biased, consisting of the authors and related individuals.\n 3. The confidence in some evaluations (Fig. 3(b), (e)) appears high, but no explanation is provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OkKQQtTSx3", "reviewer": "Reviewer_BKtK", "summary": "The paper presents an evaluation protocol designed to assess the turn-taking capabilities of spoken dialogue systems. It evaluates the exact timing of these events using a supervised model trained to predict them. The experimental results reveal interesting insights about existing spoken dialogue systems and offer valuable suggestions for their future development.", "questions": "My questions are listed above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "1. This paper proposes a comprehensive evaluation protocol and well-designed metrics to assess the turn-taking capabilities of spoken dialogue systems. The evaluation framework and metrics are thoughtfully developed and provide valuable insights.\n2. The paper extends the evaluation of turn-taking capabilities of spoken dialogue systems from corpus-level statistics to a more granular assessment of the timing of turn-taking events. This fine-grained approach enables a more accurate reflection of a spoken dialogue system’s turn-taking capabilities.\n3. The proposed evaluation metrics provide insights into the limitations of current systems in achieving interactive and natural conversations, highlighting areas for potential improvement.", "weaknesses": "1. In Metric (E), the judge labels show low consistency with human relevance judgments, indicating that this metric may have limited reliability in assessing the model's ability to handle user interruptions effectively.\n2. My primary concern is the relatively low agreement between the majority of judge labels and human judgments, with most falling below 80%. This raises questions about the strength of the claim that the proposed metrics maintain high consistency with human decisions.\n3. GPT-4o was not evaluated.\n\nIf my above concerns are resolved, I would consider increasing my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wIMAkFntRF", "reviewer": "Reviewer_mo78", "summary": "This paper introduces a novel evaluation framework for assessing turn-taking capabilities in audio foundation models (FMs). The authors first propose metrics for five core conversational abilities: determining when to speak up, backchannel, interrupt, convey turn-taking cues, and handle interruptions. They develop a supervised model trained on human-human conversations to serve as a judge for evaluating these turn-taking events. Using this framework, they conducted a user study with different spoken dialogue systems (full-duplex E2E spoken dialogue system Moshi and VAD-based cascade dialogue system) and evaluated them. They evaluate several open-source and proprietary audio FMs on their ability to understand and predict turn-taking events.", "questions": "1. What is the main difference between the proposed evaluation protocol and the previous approach by Ekstedt and Skantze (2022)? Is it impractical to apply the metrics from prior turn-taking evaluation methods to audio FMs?\n2. While the turn-taking prediction model has been evaluated on an out-of-domain task-oriented spoken dialogue corpus, could you evaluate it on additional non-task-oriented spoken dialogue datasets to assess the generalizability of the model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The evaluation protocol is novel and well-motivated.\n2. The experimental analysis provides valuable insights into turn-taking capabilities of audio foundation models (FMs).\n3. The user study reveals noteworthy observations about current spoken dialogue systems.", "weaknesses": "1. Turn-taking prediction models used in evaluation protocol require training, which limits scalability and applicability. \n2. The paper does not thoroughly address how its proposed evaluation protocol compares with previous turn-taking approaches, such as Ekstedt and Skantze (2022).\n\nReference\n* Ekstedt, Erik, and Gabriel Skantze. Voice activity projection: Self-supervised learning of turn-taking events. Interspeech 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)