Reinforcement Learning Enhanced Full-Duplex Spoken Dialogue Language Models for Conversational Interactions
Authors: Chen Chen
Ke Hu
Chao-Han Huck Yang
Ankita Pasad
Edresson Casanova
Weiqing Wang
Szu-Wei Fu
Jason Li
Zhehuai Chen
Jagadeesh Balam
Boris Ginsburg
Venue: COLM 2025 (2025)
Keywords: Full-Duplex model
Spoken Dialogue Models
Speech-to-Speech model
TL;DR: Use Reinforcement Learning to optimize the Spoken Dialogue Models

Abstract: Mainstream spoken dialogue language models (SDLMs) primarily handle turn-based interactions by alternating between processing user speech and generating responses. Recently emerging full-duplex SDLMs have showcased more natural and engaging conversational performance by simultaneously listening and speaking. However, the complex dynamics of human conversation introduce unique challenges to full-duplex SDLMs: Beyond generating reasonable responses, these models must exhibit diverse and prompt conversational behaviors in real-time interactions with the user. In this work, we present an efficient full-duplex SDLM optimized by Online Reinforcement with Interactive Speech Evaluation (ORISE). In ORISE, we design a customized reward function derived from automated annotations of online generated speech to guide the model toward well-formed and speech-text aligned responses. Experimental results show that ORISE effectively improves robustness and accuracy in handling conversational dynamics, including turn-taking, user barge-in, and backchanneling. Furthermore, ORISE enables the model to adapt to unseen noise conditions without relying on any labeled data, demonstrating the generalization of ORISE in real-world scenarios.

Published as a conference paper at COLM 2025
Reinforcement Learning Enhanced Full-Duplex Spoken Dialogue Language Models for Conversational Interactions
Chen Chen
Kevin Hu
Chao-Han Huck Yang
Ankita Pasad Jason Li
Edresson Casanova
Weiqing Wang Jagadeesh Balam
Szu-Wei Fu Boris Ginsburg
Zhehuai Chen
NVIDIA
Abstract
Mainstream spoken dialogue language models (SDLMs) primarily han- dle turn-based interactions by alternating between processing user speech and generating responses. Recently emerging full-duplex SDLMs have showcased more natural and engaging conversational performance by si- multaneously listening and speaking. However, the complex dynamics of human conversation introduce unique challenges to full-duplex SDLMs: Beyond generating reasonable responses, these models must exhibit di- verse and prompt conversational behaviors in real-time interactions with the user. In this work, we present an efficient full-duplex SDLM optimized by Online Reinforcement with Interactive Speech Evaluation (ORISE). In ORISE, we design a customized reward function derived from automated annotations of online generated speech guide the model toward well- formed and speech-text aligned responses. Experimental results show that ORISE effectively improves robustness and accuracy in handling conversa- tional dynamics, including turn-taking, user barge-in, and backchanneling. Furthermore, ORISE enables the model to adapt to unseen noise conditions without relying on any labeled data, demonstrating the generalization of ORISE in real-world scenarios.
1 Introduction
Spoken dialogue is the most intuitive form of human-computer interaction, enabling users to communicate effortlessly and naturally with Al agents (Rebman Jr et al., 2003). Con- spoken dialogue language models (SDLMs) has emerged as
sequently, research important topic within conversational AI (Freed, 2021), exemplified by popular voice assis- tants such as Siri and Alexa, which facilitate various aspects of daily life.
As shown in Figure (a), most existing SDLMs rely on turn-based interactions and process speech inputs one segment at time to generate speech textual responses (Kulkarni et al., 2019). However, these half-duplex systems, which must alternate strictly between "listening" and "speaking" modes, struggle to handle complex conversational dynamics commonly observed in human dialogue, such as user barge-in or backchanneling (Lin et al., 2025a). Therefore, developing full-duplex SDLMs has emerged as a more attractive goal (Wang et al., 2024b) to simultaneously process streaming user inputs and generate agent voices, shown in Figure (b). This simultaneous modeling of input and output channels facilitates more responsive and flexible form of interaction (Défossez et al., 2024).
Beyond comprehending speech content, full-duplex SDLMS are expected learn timing information in spoken conversations to provide prompt responses. As illustrated in Figure (b), we highlight three key conversational behaviors: (1) User barge-in: agent should gracefully allow users to interrupt its ongoing speech and switch seamlessly to listening mode; (2) Turn-taking: the agent should determine appropriate moments to begin speaking once the user finishes the utterance; and (3) Backchanneling: the agent should remain unaffected by users' short affirmative response (e.g., "yeah", and "Mm-hmm", etc.) or third-party voice. Additionally, considering the variability of conversational environments,

---

Published as a conference paper at COLM 2025
Agent turn-1
Agent stream
(a) Half-duplex SDLMs
(b) Full-duplex SDLMs
3
User tum-1
User tum-2
User stream
Figure 1: (a) Half-duples SDLMs with turn-based interaction. (b) Full-duplex SDLMs can listen and speak simultaneously, which allows the system to handle conversttion dynamics, including ! User barge-in, ② turn-taking, and ( user backchanneling.
the overall system needs to demonstrate robustness against background noise. Collectively, these factors significantly increase the complexity of conversation modeling and pose challenges for full-duplex SDLMs.
To address these challenges, this work aims to develop a robust SDLM to effectively handle real-time conversational dynamics. We first present an efficient and direct speech-to-speech modeling approach for adapting a text-based LLM into a full-duplex SDLM. Building on this model, we focus on applying reinforcement learning (RL) as post-training optimization to refine the agent's s conversational behaviors. This motivation stems from the long-standing utilization of RL to mitigate the mismatch between training and evaluation in sequence generation problems (Bahdanau et al., 2016; Rennie et al., 2017; Prabhavalkar et al., 2018), a.k.a., exposure bias introduced by the Teacher-forcing algorithm (He et al., 2019b).
We propose ORISE, which adopts an online sampling and evaluation strategy that leverages voice activity detection (VAD) model (Team, 2021) and automatically assigns multiple rewards for the predicted speech. Through policy gradient optimization, the likelihood of sequences exhibiting desirable conversational behaviors is promoted, allowing the model to pursue higher reward expectation. Experimental evidence demonstrates that ORISE effectively improves the robustness in handling conversational behaviors, including turn- taking, user barge-in, and backchanneling. Furthermore, as our rule-based reward function does not requires labeled examples, we apply ORISE unseen background noise and improve the accuracy of barge-in and backchanneling in a fully unsupervised adaptation manner. In summary, our contribution is as follows:
We propose ORISE, the first RL-based optimization method specifically designed for full-duplex SDLMs. ORISE leverages automated annotations to guide the model's conversational behaviors using self-generated samples, effectively mitigating the exposure bias commonly observed in sengsee generation under SFT.
Enhanced by ORISE, our SDLM is more robust and efficient, achieving higher accuracy and lower latency in handling conversational dynamics such barge-in and backchanneling. Our SDLM delivers high-quality responses with only 1/6 the parameter size of Moshi (Défossez et al., 2024).
ORISE is also empirically shown to mitigate the model performance degradation in unseen background noises without relying on any supervised labels, demonstrating its practical value for deploying conversational Al in real-world scenarios.
2 Related Work
2.1 Full-Duplex Spoken Dialogue Language Models
To address the limitation of half-duplex models, recent works such GPT-4o voice mode (Hurst et al., 2024) have been proposed to build full-duplex models capable of simultaneous speaking and listening capacities (Lin et al., 2025b). According to modeling
2

---

Published as a conference paper at COLM 2025
approaches, existing full-duplex models can be categorized into two types: cascaded models and end-to-end models.
Cascaded approaches break down the process into distinct stages-find converting speech to text and then handling the linguistic content. For example, Wang et al. (2024a) propose using control, speak, and listen tokens a large language model with additional perception and motor modules, although this approach still depends on separate speech-to- text-to-speech components. Similarly, MiniCPM-Duplex (Zhang et al., 2024c) and MiniCPM- Duo (Xu et al., 2024) incorporate time-sliced text tokens via time-synchronous methods, while systems like VITA (Fu et al., 2024) and Freeze-Omni (Wang et al., 2024c) accept raw speech inputs to speed up inference, though they still require text to complete the loop.
and
In contrast, end-to-end two-channel models aim to jointly process both user and agent channels without intermediate conversion. For instance, dGSLM (Nguyen et al., 2023) employs a Siamese network with cross-attention to handle two-channel dialogue, and Moshi (Défossez et al., 2024) utilizes parallel streams to separately process speaker and listener speech. SyncLLM (Veluri et al., 2024) introduces a time-synchronous mechanism for autoregressive decoding across channels, while OmniFlatten (Zhang et al., 2024b) adopts progressive post-training to jointly process flattened sequences of speech and text tokens. Other works, including SALMONN- (Yu et al., 2024), MinMo (Chen et al., 2025), Parrot (Wang et al., 2025), RTTL-DG (Mai & et al., 2024b) explore techniques such augmenting state tokens, next-token-pair prediction, and dialogue management to refine two-channel processing.
2025), and neural FSM in (Wang
2.2 Reinforcement Learning in Language Generation
Before the advent of LLMs, optimization methods based on reinforcement learning had already been widely applied to language generation tasks (He et al., 2019a;b). For exam- ple, in the speech domain for ASR tasks (Prabhavalkar et al., 2018), image domain for image captioning (Rennie et al., 2017), and NLP for machine translation (Bahdanau et al., 2016), these methods aim to optimized SFT models based on the traditional REINFORCE algorithm (Williams, 1992) improved variants (Zhang et al., 2021).
In the LLM era, RL-based optimization methods are increasingly explored in the form of RLHF technique (Ouyang al., 2022). Given that LLMs already exhibit strong generation capabilities, RLHF primarily serves as a post-processing technique to align model outputs with human preferences in various scenarios (Dai et al., 2023; Dong et al., 2024). Early RLHF algorithms rely on an independent reward model to score generated samples (Zheng et al., 2023), whereas later approaches adopt implicit reward modeling, directly learning human preferences from mhosen/rejecter pairs (Rafailov et al., 2024; Meng et al., 2024).
3 Speech to speech (S2S) Full-Duplex Modeling
An overview of our proposed full-duplex SDLM is illustrated in Figure 3. Given a (possibly noisy) user stream X, our duplex $25 model designed to concurrently process the incoming signal while streaming predictions for both text yl and speech Y2 outputs. This simultaneous "listening" and "speaking" autoregressive generation process is formalized as follows:
I
P& x2 x)
= I (v., 1=1
y Xst),
(1)
where Xet represents the user stream up to time step 1, and yt and yf denote the text and speech outputs at time step f, respectively. We hereby highlight following key factors of modeling a text-only LLM to a full-duplex SDLM:
Speech Tokenization. We employ the neural audio codec to represent the speech predicted by SDLMs, i.e., the agent stream can be represented as an acoustic matrix with length T and K channels of codebooks. Unlike commonly used RVQ in TTS technique (Wang et al., 2023), we adopt NanoCodec (Anonymous, 2025) with Finite Scalar Quantization (Casanova et al., 2025) to ensure independence among multiple codebooks. This independence allows
3

---

Published as a conference paper at COLM 2025
all K codebooks to be predicted in parallel at each timestep, thereby enabling fully parallel modeling with low latency.
Alignment between generated text and speech. Due to different modalities, y2 is typically longer than YI, making synchronous prediction unfeasible. To align Y and Y2, we introduce padding mechanism for the text channel coordinate a bi-modal response. As illustrated in the Figure 3, after the text output is predicted, the model continues to generate a special (pad) token until the predicted audio codecs reach the end of sequence. This approach eliminates the need for word-level alignment like Moshi (Défossez et al., 2024), and even if the prediction process of Y2 is interrupted by the user, the generated Y can serve as context for subsequent dialogue.
User stream perception. We employ a pretrained streaming speech encoder to extract continuous embeddings X' RTxd from the user channel in real time. The frame rate of the encoder is set to match the frame rate of the audio codec model. Consequently, for each time step I, a projection layer maps X to match the embedding dimension of LLM, d2 after which (v/-1.y?-11 X)) are fused the embedding level as the input to the LLM. Notably, compared to discretizing user speech using an audio codec (Défossez al., 2024), the streaming speech encoder can be trained end-to-end, capturing subtle nuances in the user stream and more flexibly adapting to diverse acoustic environments.
LLM backbone. To adapt any text-based LLM to receive and predict audio tokens, we first expand its original word embedding matrix W R0 with K layers of codebooks, resulting in a new word embedding matrix W t Kixd,, After obtaining the hidden representation ht from the final transformer block, the original LLM's linear layer is reused to map ht the logits of vocabulary size U1, while parallel linear layers independently map h; to the logits with size for each audio codec codebook. In practice, we observe that the text channel typically learns knowledge first and predominantly influences conversational behavior. Therefore, two techniques are implemented to enhance model performance. First, the text channel is assigned a higher weight when computing the cross-entropy loss. Second, one-token delay (i.e., 80ms) is introduced between the text channel and the audio channel. Our SDLM is trained with cross-entropy loss as follows:
T
LCF = - Σ [ logp (yl h:) + A2 logp (y? 1 ht)]
(2)
where ê and A2 are the hyper-parameters to balance the weights between speech and text channels.
S2S Training Data. Existing publicly available conversational datasets, such as Fisher (Cieri et al., 2004), capture human dialogue behaviors but are insufficient for training helpful agent to effectively respond to a wide range of inquiries. To address this gap, we construct dialogue-style data featuring two speech streams: one for the user and one for the agent. When the user begins speaking, the agent is expected to predict silence; once the agent takes over the turn, appropriate silence is inserted into the user stream to allow the agent to respond. Further details are provided in the Appendix A.1.
Online Reinforce with Interactive Speech Evaluation (ORISE)
Given an SFT model, reinforcement learning aims to improve its generation quality by maximizing customized reward R. In the mainstream RLHF, R is typically determined by an independent reward model trained with human feedback, or directly learns human preference from annotated chosen/ booee//rjjcca rrjectedd pairs. To circumvent the high cost associated with human annotations, this paper focuses on automated metrics R to measure the conver- sational performance of generated samples, thus eliminating the need for any ground truth involvement throughout the optimization process. In general, the maximizing objective can be written as:
E ExED (
R(X x3) - (DoL(TO((Y)) (2) [)ll ((2,)22(1))
(3)
4

---

Published as a conference paper at COLM 2025
Table 1: Ideal agent behaviors in respond to different user speech intents, where 0 and 1 in transition denote "silence" and "speaking" modes of agent, respectively.
Index
User intent
Transition
Agent Behavior Description
Reward
C211
normal query
0 0-1 1
take the turn within delay Af1 stop speaking within delay At2
C2
barge-in
1
i
C23
backchanneling
1
continue speaking until t' + At3
1
where reference model Fe with KI. divergence penalty is introduced to prevent the model My from making radical update, and ß is balancing weight. To annotate the quality of generation, we define three automatic reward criteria for evaluating generated samples:
C₁ Turns Consistency. In multi-turn dialogues, the number of turns predicted by the agent should be equal to the number of user turns, excluding those backchan- nel responses: R1 = Number of turns X) Number of turns (Y2)l, where the Number of turns (Y²) is examined by a VAD model Silero (2024).
Cz: Conversational behavior. Given segment of user speech X' with ending timestamp of f', we define speaking state of agent as 1 and silence state as 0, and the desirable behaviors of agent are illustrated in Table 1. Due the awareness of I' and transition of user speech, the generated agent speech y2 can be detected by VAD model to determine whether the desirable action has been executed, without any reliance on ground-truth. Since C2-1 is easy to learn from SFT, we define the R2 = [[222] + [C223), where the indicator function I returns 1 if the condition inside is satisfied for the whole conversation, and 0 otherwise.
C3: Alignment between y and Y2, Since the agent generates text and speech independently, their content should remain consistent. To evaluate this, we employ an ASR model (Galvez et al., 2024) to transcribe the generated speech into text and then compute the word error rate (WER) between the predicted transcription and the generated text. Note that R3 is applied only to those samples that were not interrupted, as when a user barges in, the text channel, y' may have more content than the speech channel, y2, In such cases, a high WER does not necessarily indicate a mis-alignment between the two predicted modalities.
To avoid introducing excessive hyperparameters, we directly sum the R1 ~ R3 to measure the quality of Y2, Additionally, preference-basee optimization can also be conducted in terms of positive positive/negative nngaatiee samples selected from these criteria. We consider preference optimization methods as baseline and introduce more details in the Appendix A.2.
For optimization, we adapt typical REINFORCE algorithm (Williams, 1992) with online sampling strategy. Specifically, based on user speech X, we perform online decoding to sample N pairs of (Y1, y2) using ne. Then the expectation in Eq. (3) is approximated using an empirical average over these samples, and the optimization objective is written as:
1 N C= -= Σ [M1P)(T)) X) + (2 N ==1
x)) An BBDL || True)
(4)
where A₁ and A are keep consistent with SFT process in Eq. (2). Po ( X) and Po(Y, X) A are normalized by the sequence length A is the advantage function (Schulman al., 2017), and KL-item is estimated using the approximator introduced by (Schulman, 2020):
A, = R(x, mean R((X 82)
(5)
std 82))
R R1 R₂ + R3,
(6)
T
The (vl,yy
V2r)) - -log To (vl.y? x, x.
X, K
nht (yl. X, x,y 22r)
IDKt. [Ttp || [
Σ
-
(7)
no (vl, x
1=1
5

---

Published as a conference paper at COLM 2025
In Eq. (4), Y obtain higher rewards than other samples, its likelihood generated by πa would be increase due to the positive value of An, and vice versa. Additionally, although the reward is computed based on y2, the behavior of Y remains largely consistent with it, as text channel is predicted one token ahead of the speech channels. The pseudo code of ORISE is summarized as Algorithms 1.
Algorithm 1 ORISE Algorithm
Require: SFT model πe, VAD model Φ, Unlabeled dataset D = (Xi, (Xi+
Ensure: Optimized Model To
1: Duplicate Thg to get reference model Te
2: for each training step do
3: Sample user input X
4: Obtain N pairs of (Y',, Y2) online samples with 7t
5: Calculate reward R for Y? using 4, f', and states of X
6: Compute loss C in Eq. (4)
7: Update parameters: 0
Ve
8: end for
9: return Optimized model parameters
Experimental Setup
5.1 Dataset
Training Data. To generalize spoken QA capability,
multi-speaker TTS model (Hus- sain et al., 2025) to synthesize context, question, and answer from text-based datasets such as Alpaca (Taori et al., 2023) and MS-MÁRCO (Bajaj al., 2018). To prevent the model from overfitting to synthetic speech, we follow the approach proposed (Noroozi et al., 2024) to construct additional QA dataset using the Mistral LLM, where the user speech is mixture of real and TTS-generated audio, referred to as the ASR-QA set. Furthermore, we construct two multi-turn dialogue datasets, Topic and UltraChat, to enhance the agent's capabilities to leverage context information. In UltraChat, the original text data (Ding et al., 2023) is reformulated into dialogue format, while Topic simulates user-agent conversa- tions based on specified topic. For RL, only the UltraChat user speech is employed for optimization. Detailed data statistics shown in Appendix A.1 Table 6.
Background Noise. We utilize WHAM (Wichern et al., 2019) to create noisy dataset, which are dynamically mixed into the user channel during training in an online manner. The signal-to-noise ratio (SNR) is randomly sampled from range of 10dB to 30dB. For unseen noise setting, we randomly sample some noise from MUSAN (Snyder et al., 2015).
User Backchannel. To simulate user backchanneling, 20 affirmative short utterance that commonly occurs in dialogue backchanneling. We then sample 200 speakers from the LibriTTS dataset and randomly paired them with the generated phrases. Using multi-speaker TTS model (Du et al., 2024), we have synthesized total of 4,000 distinct backchannel audio samples. Next, these backchannel samples are randomly selected during training and inserted into silent segments of the user channel that exceed 4 seconds. The insertion point set to seconds after the agent begins speaking to simulate real backchannel. During inference, since the agent's response time is uncertain, we ensure that the user backchannel is inserted before the agent completes its response.
first use GPT-4o to generate
5.2 Training Details and Baselines
Our model is implemented in PyTorch using the NeMo Toolkit (Kuchaiev et al., 2019) and trained on 32 A100 (80G) GPUs, with each GPU handling batch duration of 1000 seconds. The speech encoder is derived from a 100M-parameter streaming pretrained model with an 80ms right context (NVIDIA, 2023), while the LLM is initialized from the 1.1B TinyLlama

---

Published as a conference paper at COLM 2025
Turn-Taking Latency (s)
Barge-in Latency (s)
Backchannel Acc (%))
Model
E2E
MOS
Acc (%)
Freeze-Omni
1.17*
79.5*
1.20*
4.3
dGSLM
0.57
85
0.86
2.2
Moshi
n.a.
55.1
0.81
3,9
ORISE
0.43
96.8
0.61
95.7
4.2
Table 2: Conversational Behaviors of different models under various conditions. Turn-taking and Backchannel are evaluated on UltraChat, and Barge-in is evaluated on Impatient. The number with denotes it is reported in (Lin et al., 2025a), and "_" indicates that the model does not support the corresponding functionality.
(Zhang et al., 2024a). For tokenization, employ 32k SentencePPiece model for text and customized 0.6 kbps NanoCodec (Anonymous, 2025) for speech. The speech representation utilizes four codebook layers, each containing vocabulary of 4,037 entries. The training loss is balanced across modalities, with weights set to A₁ = 3 for text and A2 = for speech. Optimization is performed using FusedAdam with an inverse square root annealing learning rate schedule, beginning with peak learning rate of 3e-4 for SFT and le-5 for RL. To maintain stability, gradient clipping is enforced threshold of 1.0. The balance the KL-item is set as 0.2.
We adopt two preference optimization baselines DPO (Rafailov al., 2024) and IPO (Azar et al., 2024), as well as a baseline that does not rely on pairwise preferences, KTO (Ethayarajh et al., 2024). Since these methods are originally designed for LLMs, several necessary adaptations are made to apply them to SDLMs. Further implementation details are provided in the Appendix A.2.
5.3 Evaluation Dataset and Metric
Response Quality. We evaluate our model in two scenarios: 1) multi-turn conversations: UltraChat, Roleplay, and Topic, and 2) spoken QA reasoning: ASR-QA and Alpuca. All these test sets are unseen during training. To measure the response quality, we utilize GPT score" ranging from to 10 based the hypotheses and references of all the agent turns. The agent speech response transcribed by ASR model2 as GPT input. We also report the UTMOS automatically evaluate by pre-trained UTMOS model (Saeki et al., 2022). As S2S model, we do not report the quality of the predicted text, while generally better than the predicted speech content in terms of quality and coherence.
User Barge-in. Since the response time of agent is unpredictable, the natural occurrence probability of user barge-in is relatively low. To evaluate the barge-in performance, we create a test set based on ASR-QA QA, called impatient, where the agent is given barely an average of 2 seconds to respond. Then the user barge-in occurs in more than 95% cases, showcasing a obvious mismatch with training set. The At is set as 1.5 second, meaning that if the agent does not stop speaking within 1.5 seconds, it considered barge-in failure. We report both accuracy and average latency as metric to measure the barge-in performance.
User Backchanneling. Different with training, insert random backchannel utterance in every response of agent turn. UltraChat is selected as test set due to its long agent response, and if agent stops speaking within 1.5 second, it is considered backchannel failure. We report the average accuracy calculated across the all backchannels.
Turn-taking. As the basic behavior for the model, we evaluate it by measuring the latency between the end of the user's 's speech and the start of the agent's response. Notably, this we introduce 0.64-second delay in the training label make the conversation more natural.
hnttss///openaa.com//iii winde////iie/////eee www.aiiadvannnnnooooooeeiiiiiiie
Phtpss//huugiigffac..oo /vviaaa tat ctc-1100m
7

---

Published as a conference paper at COLM 2025
Multiturn Conversation Roleplay
Spoken QA
ID
Models
LLM Backbone
UltraChat
Topic
ASR-QA
Alpaca
Moshi
Helium-7B
3.4
1.7
2.8
1.9
1.7
SFT Model
Qwen2-1.5B
3.7
3.7
6.1
6.3
3.3
SFT Model
3.5
4.6
6.1
7.8
2.9
TinyL.aMMA-12BB
4
ORISE
4.0
4.1
5.8
7.7
2.9
GT+LLM
TinyLLaMA-1.2B
6.4
4.9
5.5
5.8
5.0
Table 3: GPT-score of multi-turn conversation and spoken QA. GT+LLM denotes an oracle cascaded system which feeds every ground-trule user turn to the LLM.
SFT Augmented by
RL Optimization
Metric
Test set
0% %
20%
80%
DPO
IPO
KTO
ORISE
BLEU Barge-in-Acc
10.5
10.0
10.1
6.2
8.3
5.5
9.8
UltraChat
83.0
74.1
70.7
66.9
70.9
75.5
83.7
Backchannel-Acc
85.8
86.4
88.0
92.7
96.0
95.7
Barge-in-Acc
Impatient
94.5
94.1
96.5
94.7
94.7
95.2
96.8
Table 4: Comparison of conversational behavior improvements using RL, where "Aug- mented by 80%" is the starting point of RL optimization.
6 Result and Analysis
6.1
Main Results
We first report the conversational behavior in Table 2, where Moshi (Défossez et al., 2024), Freeze-Omni (Wang al., 2024c), and dGSLM (Nguyen et al., 2023) are employed as baselines. We observe that: (1) For turn-taking, end-to-end models exhibit lower latency compared with cascaded model. The "n.a." entries for Moshi are due to its frequent behav- ior of responding before the user finishes speaking. In contrast, our model introduces 0.64-second delay during training, while during inference, the latency dynamically varies between 0.4 and 0.9 seconds across different datasets. (2) For barge-in, our model demon- strates strong robustness compared with other baselines. Under both clean and noisy conditions, it significantly outperforms the baselines in terms of both accuracy and latency. (3) For backchanneling, our model achieves high accuracy, indicating its effectiveness in distinguishing normal user query and affirmative response.
In Table 3, we report our model's reasoning capabilities by comparing it against E2E baseline Moshi and an oracle cascaded system, where ground-truth transcriptions of user turns are directly provided to a language model (denoted as GT+LLM in Table 3). The dGSLM is not included in the comparison it cannot function as agent capable of responding to user requests. By comparing System ID-2 and System ID-3, we demonstrate that our S2S modeling can efficiently adapt different text-based LLM into full-duplex SDLM. Furthermore, the comparison between System ID-3 and System ID-4 shows that ORISE effectively enhances dialog capabilities without compromising the model's reasoning performance. We attach listening examples are in anonymous link3.
6.2 Effect of ORISE
We then evaluate the efficacy of RL-based optimization and report the accuracy in Table 4. We use the SFT model augmented by certain percentage of synthesized user backchannel speech inserted into the training data. When it is 0%, any user voice is considered as barge-in for agent. Building upon the model trained with 80% backchannel augmentation,
8

---

Published as a conference paper at COLM 2025
we apply various post-training optimization methods, including DPO, IPO, KTO, and our proposed ORISE. Additionally, since the reward function focuses solely on conversational behaviors, report the BLEU score on the text channel to assess the model's ability to generate high-quality response content and to monitor whether the quality of the generated responses degrades due to the conservation-speciff optimization.
From Table 4, we observe that post-training methods consistently improve backchannel accuracy, backchannel events occur frequently and the model can simply continue speak- ing to receive high reward. The R1 and R₂ (backchanneling) is visualized in Figure 2. However, for barge-in, different methods exhibit distinct behaviors across the UlfraChat and Impatient evaluation settings. Despite the limited response window in Impatient, the agent demonstrates strong robustness and is often successfully interrupted across differ- ent models. In contrast, in the multi-turn conversations of UlfraChat, the success rate of barge-ins coops-particulartie as the proportion of backchannel examples increases in the training data. Compared to preference-based optimization methods, approach offers two key advantages. First, it achieves balanced improvements in both backchannel and barge-in handling. Second, it demonstrates higher BLEU score, indicating that the quality of generated responses remains stable.
1.0
Accuracy
Noise
Model
TT
B1
BC
Avg. 32 samples
0.8
SFT Model
79.1
62.0
94.2
W.
+ORISE
84.1
72.6
96.7
R, (Dc)
0.4
SFT Model
75.4
76.3
94.3
M.
+ORISE
77.2
84.0
92.5
0.0
110
Table 5: Accuracy of conversational be- haviors on unseen noises. "TT", "BI", and "BC" denote turn-taking, barge-in, and backchanneling, "W." and "M." indi- cate noise from Wham and MUSAN.
400
800
Training Iterations
Figure Visualization of R1 (turns consis- tency) and R₂ (backchanneling) during training.
6.3 Adaptation on Unseen Noises
In this section, we first analyze the impact of unseen noise on SDLM behaviors, and then examine the efficacy of ORISE to alleviate it. As shown in Table 5, "SFT Model" is trained with online augmented noisy segments from Wham training set, and we evaluate the model on Wham evaluation set and MUSAN noises. From the results, we observe that when noise is introduced during training, backchannel performance improves significantly 886.4%%94+%% comparing with result in Table 4, as the model becomes desensitized to third- party voices. However, the performance of both turn-taking and barge-in degrades notably under the same conditions. ORISE effectively mitigates these challenges without relying on labeled data, thereby demonstrating its capacity to preserve balanced conversational behaviors under domain shift.
7 Conclusion
In this paper, we introduce robust full-duplex SDLM enhanced by ORISE, a reinforcement learning method to improve model's conversational behaviors real-time interaction. The proposed ORISE leverages automated annotations of online generated speech to guide the model toward well-formed and speech-text aligned responses. Through extensive experiments, our model demonstrates strong performance in handling user barge-in, turn- taking, and backchanneling, while remaining resilient to unseen background noise without the reliance of labeled data. These results highlight that our approach enables SDLMs to handle diverse interactive spoken dialogue dynamics within a unified system, thereby serving as universal optimization framework for next-generation conversational AL.

---

Published as a conference paper at COLM 2025
Ethics Statement
Our research utilizes publicly available and synthetic data, ensuring that no private or sensitive information is employed. We have carefully considered potential biases inherent in conversational models and have designed our approach to minimize harmful behaviors. The proposed system intended for academic exploration and serves as prototype for further research; its future deployment in real-world applications should be accompanied by comprehensive ethical review and oversight. We also encourage future studies to further investigate the societal and ethical implications of next-generation conversational AI systems.
References
Anonymous, Nanocodec: Towards high-quality ultra fast speech Ilm inference. Preprint (Under Review), 2025.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Slatistics, pp. 4447-4455. PMLR, 2024.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. ar arxiv preprint ar wxxi:1607.000888 2016.
P Bajaj, D Campos, N Craswell, L Deng, Gao, X Liu, R Majumder, A McNamara, B Mitra, Nguyen, al. A human generated machine reading comprehension dataset. arXiv preprint ar Xii::1611.99266 2018.
Edresson Casanova, Ryan Langman, Paarth Neekhara, Shehzeen Hussain, Jason Li, Sub- hankar Ghosh, Ante Jukić, and Sang-Gil Lee. Low frame-rate speech codec: codec designed for fast high-quality speech Ilm training and inference. In ICASSP 2025-2025 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, et al. Minmo: A multimodal large language model for seamless voice interaction. arXio preprint ar Xiv:2501.06282. 2025.
Christopher Cieri, David Graff, Owen Kimball, Dave Miller, and Kevin Walker. Fisher english training speech part 1 transcripts. Philadelphia: Linguistic Data Consortium, 2004.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXio:2310.12773, 2023.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. Technical report, Kyutai, September 2024. URL http: org/ Mdf.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. ar Xiv preprint ar Xiv:2305 142333, 2023.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. ar Xiv preprint ar Xiv::245.007863 2024.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to- speech synthesizer based on supervised semantic tokens. Xiv preprint arXio:2407.0544 2024.
10

---

Published as a conference paper at COLM 2025
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
Andrew Freed. Conversational ai. Simon and Schuster, 2021.
Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, et al. Vita: Towards open-source interactive omni multimodal Ilm. arviv preprint (rXit:2408005211 2024.
Daniel Galvez, Vladimir Bataev, Hainan Xu, and Tim Kaldewey. Speed light exact greedy decoding for rnn-t speech recognition models on gpu. arXiv preprint arXio:2406.03791, 2024.
Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurell Aurelio Ranzato. Revisiting self-training for neural sequence generation. arv preprint arXiv:1909.13788 2019a.
Tianxing He, Jingzhao Zhang, Zhiming Zhou, and James Glass. Quantifying exposure bias for neural language generation. 2019b.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arv preprint arXi:24100221276 2024.
Shehzeen Hussain, Paarth Neekhara, Xuesong Yang, Edresson Casanova, Subhankar Ghosh, Mikyas T Desta, Roy Fejgin, Rafael Valle, and Jason Li. Koel-tts: Enhancing 11m based speech generation with preference alignment and classifier free guidance. arXiv preprint drXiv:2502.05236, 2025.
Oleksii Kuchaiev, Jason Li, Huyen Nguyen, et al. Nemo: toolkit for building applications using neural modules. Xrv preprint ar XrXiv:1909.09577, 2019. 2019.
Pradnya Kulkarni, Ameya Mahabaleshwarkar, Mrunalini Kulkarni, Nachiket Sirsikar, and Kunal Gadgil. Conversational ai: An overview of methodologies, applications & fu- ture scope. In 2019 5th International conference on computing, communication, control and automation (ICCUBEA), pp- 1-7. IEEE, 2019.
Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchipalli, Alexander H Liu, and Hung-yi Lee. Full-duplex-bench A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities. arXiv preprint arXit:2503.04721 2025a.
Yu-Xiang Lin, Chih-Kai Yang, Wei-Chih Chen, Chen-An Li, Chien-yu Huang, Xuanjun Chen, and Hung-yi yi Lee. A preliminary exploration with gpt-4o voice mode. arXiv preprint ar arXiv:2502.09940 009900000099,,,,0 2025b.
Long Mai and Julie ar (rX:::2501.0433 2025.
Real-time textless dialogue generation. arv preprint
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Adoances in Neural Information Processing Systems, 37:124198-122235, 2024.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. Genera- tive spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11:250-26,, 2023.
Vahid Noroozi, Zhehuai Chen, Somshubra Majumdar, Steve Huang, Jagadeesh Balam, and Boris Ginsburg. Instruction data generation and unsupervised adaptation for speech language models. ar Xrx preprint ar arXiv:2406.12980 12946,, 2024.
NVIDIA. STT En FastConformer Hybrid Transducer-CTC Large Streaming 80ms, 2023. URL https: http://caallaaggggeeeeeoa ngc. nwidia. com/orgs/nvidia httpiiiiiiaaaaa///iiiaaaaaaaaaee motess/std en fastconformer hybrid large streaming 80ms. Version 1.20.0, Released June 22, 2023.
11

---

Published as a conference paper at COLM 2025
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong C Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-277444 2022.
Rohit Prabhavalkar, Tara N Sainath, Yonghui Wu, Patrick Nguyen, Zhifeng Chen, Chung- Cheng Chiu, and Anjuli Kannan. Minimum word error rate training for attention-based sequence-to-squeene models. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), PP. 4839 4843. IEEE, 2018.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and
Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Carl M Rebman Jr, Milam W Aiken, and Casey Cegielski. human-computer interface. Information & Management, 40(6):509-519 2003.
Speech recognition in the
Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7008-7024, 2017.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. ar Xiv preprint Xiv:2204.02152. 2022.
John Schulman. Approximating kl divergence, 2020. URL http://joschu.nnet/blg// kl-approx. html.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint ar arXie:1707.06347, 2017.
Silero. Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier. https: // //gi thub. wwwccoom//acecoeeececc wtwm///aaerr///sleroooeeee 2024.
David Snyder, Guoguo Chen, and Daniel Povey. Musan: music, speech, and noise corpus. ar Xiv preprint ar axXto:1510.08444, 2015.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-folloving llama model, 2023.
Silero Team.
Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier. Retrieved March, 31:2023, 2021.
Bandhav Veluri, Benjamin Peloquin, Bokai Yu, Hongyu Gong, and Shyamnath Gollakota.
Beyond turn-based interfaces: Synchronous Ilms full-duplex dialogue agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 21390-214D0 2024.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. Xiv preprint Xiv:2301.02111, 2023.
Peng Wang, Songshuo Lu, Yaohua Tang, Sijie Yan, Wei Xia, and Yuanjun Xiong, A full-duplex speech dialogue scheme based on large language model. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https://openreviee .nt/ Forum?id==YaWKY6miii
Peng Wang, Songshuo Lu, Yaohua Tang, Sijie Yan, Wei Xia, and Yuanjun Xiong. A full-duplex speech dialogue scheme based on large language models. Xrviv preprint #rXio22405.19887 2024b.
12

---

Published as a conference paper at COLM 2025
Qichao Wang, Ziqiao Meng, Wenqian Cui, Yifei Zhang, Pengcheng Wu, Bingzhe Wu, Zibin Zheng, Irwin King, Liang Chen, and Peilin Zhao. Parrot: Seamless spoken dialogue interaction with double-channel large language models, 2025. URL https::/opennreview. net/fforum2id777EDG666mm
Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: A smart and low latency speech-to-speets dialogue model with frozen llm. arx preprint (rx:::2411.0077 2024c.
Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. Whaml: Extending speech separa- tion to noisy environments. Xiv preprint (Xiv:1907.01116, 2019.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Machine learning, 8:229-256, 1992.
Wang Xu, Shuo Wang, Weilin Zhao, Xu Han, Yukun Yan, Yudi Zhang, Zhe Tao, Zhiyuan Liu, and Wanxiang Che. Enabling real-time conversations with minimal training costs. ar Xiv preprint ar xrXit:2409.11727, 2024.
Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, and Chao Zhang. Salmonn-omni: codec-free Ilm for full-duplex speech understanding and generation. arXiv preprint arXio:2411.18138, 2024.
Junzi Zhang, Jongho Kim, Brendan O'Donoghue, and Stephen Boyd. Sample efficient reinforcement learning with reinforce. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 10887-10895, 2021.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024a.
Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, Chaohong Tan, Zhihao Du, al. Omniflatten: An end-to-end gpt model for seamless voice conversation. arXiv preprint ar xrXio:2400.17999 2024b.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Xu Han, Zihang Xu, Yuanwei Xu, Weilin Zhao, Maosong Sun, and Zhiyuan Liu. Beyond the turn-based game: Enabling real- time conversations with duplex models. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11543-11557, Miami, Florida, USA, November 2024c. Association for Computational Linguistics. doi: 11186633v112224.em// main.644.....4 URL https: //aclanthology..oo org/ 2024. emnlp-main. 644/.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf in large language models part i: Ppo. ar Xiv preprint ar Xiv:2307 0.4944444444 2023.
13

---

Published as a conference paper at COLM 2025
(3 Vi
W vi
Audio-Text Alignment
Text
Channel
Ta
<pad>
kslkklli
Text-based LLMs
1-token delay
sall
You'
right.
<sip
Streaming Speech Encoder
Aduio Channels
Fails
Yoa
You'te
richt
<5l
Kallz
<all
User stream
Causal Mask
Yoa
Xoutze
richt
<sib
Figure 3: Full-duplex Model structure (left) and data alignment between audio and text channels (right). We visualize turn agent response to illustrate the predicted channels of LLM. "T1 T3" are the text tokens and the blue chunks are composed by audio tokens.
A Appendix
A.1 Details of Training Data
Figure illustrates the format of our training data. Since the agent cannot anticipate the user's behavior in advance, we introduce offset of ==0.64 second into the training labels for both turn-taking and barge-in. This offset enables the agent to detect the appropriate timing from the user's 's speech input and exhibit the correct conversational behaviors. Each token in the figure corresponds to 80 ms in real-world time.
Statistics of our training data are presented in Table 6. For MS MARCO and Alpaca, adopt their predefined test splits. For other synthetic datasets, we reserve an unseen shard (approximately 5 hours) as the test set.
Table Synthetic training data with multi-turn and barge-in.
Task
Dataset
#Hours
Speech
Multi-turn
Barge-in
ASR-QA
20k
Mix
Augment
X
Spoken QA
MS MARCO
0.2k
TTS
Augment
X
Alpaca
0.2k
TTS
Augment
X
Internal SFT
3k
TTS
Real
V
Conversation
UltraChat
3k
TTS
Augment
Topic
0.3k
TTS
Augment
A.2 Preference-ppiinization Baseline
To enable preference optimization, we utilize the implicit reward modeling (Rafailov et al., 2024) though the probaabiiyy between policy model 7ig and reference model πf
Mtg (Y + A2 log
TTa (Y
Y2 A₁ log
(8)
Jree (Y
Aeee (Y2
where A and A2 keep consistent with SFT to balance the weight of channels. Given user input we utilize online sampling to obtain batch of text-speech pairs {(Y). Y? 1 Using C and C2, we filter positive and negative samples from B:
(YY) Y?) E B (G (G ()), B = [Y)] x?) e B| -C (
(9)
(10)
where A is the logical "and" and - is logical "not". For DPO baseline (Rafailov et al., 2024), reduce the for negative examples in its loss function, observed that in our system,
14

---

Published as a conference paper at COLM 2025
'Can you tell me a joke about...' !
'Wai!! can you explain more ...'
"Great!
User stream (input)
course! Here is joke about...
<sil>
Agent stream (to be predicted)
<sll> <sit>
Figure 4: Multi-turn training data formulation, where A is set as 0.64 second for agent to take the turn and handle user bargi-in. The backchannel utterance (i.e., "great") is synthesized and insert into the training data points with 4 silence.
positive and negative samples frequently yield identical predictions (i.e., the pad token). Maintaining the same & for both positive and negative samples introduces high instability during training.
A.3 Limitations and Future works
One limitation of this work is that our speech-to-spene modeling framework requires full fine-tuning of the LLM, which restricts our current exploration to relatively small models around 1.5B parameters. In future work, we plan to adopt larger LLMs as backbones to investigate whether they lead to substantial improvements in reasoning capabilities. An- other important research direction is how to effectively preserve the real-world knowledge encoded in pretrained LLMs during SDLM modeling. As observed in challenging datasets such UltraChat, there remains a performance gap between our system and the GT+LLM baseline as shown in Table 3. This highlights a promising avenue for future work aimed at better integrating knowledge retention with interactive spoken dialogue generation.
Furthermore, we find that evaluating dialogue data is inherently complex and challenging. Beyond the content generated by the agent, wide range of paralinguistic cues-such as emotion, speaker identity, and interaction style-must also be considered for a intelligent AI system. These factors play critical role in reinforcement learning, as they directly impact reward design and model behavior. In future work, we plan to develop a more comprehensive evaluation framework to support continued progress in this field.
15


Reviews:
[{"review_id": "uijn5EDqRX", "reviewer": "Reviewer_PVVb", "summary": "This paper presents a novel method for better training full-duplex spoken dialogue language models (SDLMs) with automatically generated supervision. Specifically, the model consumes input audio, previous text outputs, and audio outputs (as tokens), and produces the next text tokens and audio output (as tokens). To better handle complex scenarios where the user interrupts the AI assistant (“barge-in”), utter affirmative sounds (”backchanneling”), this paper proposes a reward modeling approach that leverages pretrained ASR and VAD models to compute the rewards of the model outputs. Then, they utilize reinforcement learning to maximize the reward. Empirical results show that their method outperforms existing full-duplex SDLMs in terms of latency and prediction accuracy for barge-in and backchanneling situations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper address an important research question and the empirical results are great.\n- The experiments clearly validates the effectiveness of the proposed model.\n- The paper is generally easy to follow and the motivation is clear.", "reasons_to_reject": "- There are three reward functions, but there is no ablation studies to verify the effectiveness of each of them. Thus, it is hard to know the effectiveness/necessity of each of the rewards.\n- While I understand that the text outputs are not the focus of this paper, I think providing the evaluation of the output text quality is very important for understanding the implications of this paper. I suggest, at the very least, to include such results in the Appendix.\n- It seems that the model requires an ASR model to produce the text labels for the audio, which is a kind of supervision. Does this align well with the statement in the Introduction that ORISE do not rely on any supervised labels?\n- To better understand the results, I suggest including the performance of non-full-duplex SDLMs.", "questions_to_authors": "- Line 105: “optimized” → “optimize”.\n- Table 4: The BLEU score seems very low, and applying RL further reduces it. Is this normal?\n- It seems that the reward modeling in ORISE does not take into account the audio-specific features, such as emotions. Is this true?\n- Are there other kinds of backchanneling other than affirmative short utterances?\n- In Algorithm 1, I think the ASR model required for computing $R_3$ should be included.\n- Line 298: System ID-2, 3, and 4 have not been introduced, and I have a hard time understanding what they refer to.\n- Figure 2: I think it’s better to report $R_1$ instead of $-R_1$.\n- Table 5: I suggest including the score for the model without unseen noise for reference.", "extra_scores": {}}, {"review_id": "X7w493HTul", "reviewer": "Reviewer_W4Zs", "summary": "The paper applies the reinforce algorithm to a duplex (speech in, speech out) conversational dialogue system. The authors show that after defining a set of rewards (which do not require human labels but are based on table stakes properties of the conversation which can be measured automatically with existing systems like VAD) and a means of combining these, that the resulting system is better at turn taking, handling barge ins and backchannels.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is clearly written and shows improved results on barge ins on 2 public datasets. Results are mixed on a QA task, with the proposed method improving upon the SFT starting point in some cases and regressing on others. \nI don't know the current literature in this space very well, but I believe there is enough in this paper that it will be of interest to the multimodal LLM community.", "reasons_to_reject": "No significant reasons. This is the application of existing techniques to a (to my knowledge) new area. \nThe results as noted are mixed. There does appear to be a trade off across the turnTaking, bargeIn and backChannel properties that are aimed to be optimised here, but overall the proposed rewards and optimisation of them via reinforce does lead to benefits in most cases, particularly with noise introduced into the speech (table 5).", "questions_to_authors": "- Regarding C-3 and the alignment between text and speech: if perfect alignment (as in words are the same) is desired, is a speech system needed? Can an independent TTS system not be used in such a case?\n- in equation 2: why the need for 2 hyperparameters? Isn't one sufficient? \n- S2S section (starting line 159): why is the conversational data insufficient? Not pushing back on the claim, but asking for a brief explanation/reason why it's insufficient. \n- Further in the same paragraph, around line 162: why does silence need to be inserted? \n- nit: there's a typo \"an\" -> \"a\" on line 160 too.", "extra_scores": {}}, {"review_id": "qpHB4v8apB", "reviewer": "Reviewer_nQWA", "summary": "This paper presents ORISE (Online Reinforcement with Interactive Speech Evaluation), a reinforcement learning framework for enhancing full-duplex spoken dialogue language models (SDLMs). The authors focus on optimizing conversational behaviors such as turn-taking, user barge-in, and backchanneling through a customized reward function derived from automated annotations. The paper makes contributions in adapting text-based LLMs into full-duplex SDLMs and demonstrating how RL can improve conversational dynamics without relying on labeled data, even in unseen noise conditions.\n\nThe work represents a meaningful step toward more natural and responsive conversational AI systems that can better handle the complex dynamics of human conversation. The empirical results show improvements in both accuracy and latency for handling important conversational behaviors compared to baseline models.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- **Technical Innovation**: The paper introduces a novel approach combining RL with automated speech evaluation to enhance conversational behaviors, which is valuable for advancing SDLMs.\n\n- **Forward-looking Vision**: Full-duplex SDLMs represent an important frontier in conversational AI, and this work makes progress toward more natural human-computer interaction.\n\n- **Efficient Architecture**: The authors demonstrate strong performance with a relatively lightweight model (1/6 the parameter size of comparable models), which is important for practical applications.\n\n- **Adaptability**: The framework shows promising results in handling unseen noise conditions without needing labeled data, demonstrating practical utility for real-world deployment.", "reasons_to_reject": "- **Data Limitations**: The training methodology relies heavily on synthetic data with artificially constructed turn-taking behaviors. The paper acknowledges the use of TTS models to synthesize conversations from text datasets and introduces artificial timing delays (such as Δ=0.64 second which seems arbitrary). This synthetic approach may not capture the natural variability and nuance of real human conversational dynamics.\n\n- **Evaluation Methodology Concerns**: The reward mechanisms are primarily based on automated metrics from VAD models without sufficient human evaluation of conversational naturalness. This creates a risk of reward hacking where the model optimizes for the metrics rather than for genuinely improved conversation quality.\n\n- **Reward Design Limitations**: The paper combines multiple rewards (R1, R2, R3) through simple summation without exploring how different weightings might affect model behavior. There's no ablation study or analysis showing how these weights interact or whether the chosen combination is optimal.\n\n- **Limited Real-world Validation**: While the paper demonstrates improvements on automated metrics, there's insufficient evidence that these improvements translate to better subjective experiences in real human-computer interactions.", "questions_to_authors": "Some minor typos:\n\npage 3, line 105, aim to optimized -> aim to optimize\n\npage 5, line 206: are keep consistent -> are kept consistent\n\nAlso, I suggest you move Figure 3 and 4 to the main text to help the readers have a better view of the paper.", "extra_scores": {}}, {"review_id": "1LxVwn3OVc", "reviewer": "Reviewer_mDwK", "summary": "This paper addresses the challenge of enhancing full-duplex spoken dialogue language models by introducing an online reinforcement learning optimization method (ORISE) for more robust and natural conversational interactions. The main contributions include the development of ORISE, a reinforcement learning-based framework that utilizes automated annotations to refine conversational behaviors such as turn-taking, user barge-in, and backchanneling, alongside demonstrating robustness in noisy and unseen conditions. The paper compares this approach against relevant full-duplex baselines (e.g., Moshi, Freeze-Omni), effectively considering LLM baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Introduces a novel reinforcement learning-based optimization framework specifically tailored for improving conversational dynamics in full-duplex SDLMs.\n2. Provides comprehensive experimental evaluation showing significant improvements in critical interaction metrics (e.g., user barge-in accuracy, turn-taking latency, backchannel handling).\n3. Demonstrates the generalization capability of the proposed approach to unseen background noise conditions without reliance on supervised labels.", "reasons_to_reject": "1. Limited clarity on how exactly automated annotations from the speech evaluations are guaranteed to consistently reflect desirable conversational qualities, potentially affecting reproducibility.\n2. While noise robustness is demonstrated, the analysis on how varying noise conditions specifically impact conversational performance metrics remains somewhat superficial.\n3. Lack of human evaluation or user studies to validate that improvements in metrics truly reflect a better user experience, potentially limiting real-world impact.", "questions_to_authors": "1. Could you clarify in more detail how the automated annotation method reliably differentiates between subtle conversational behaviors like backchanneling and user barge-in, especially in noisy environments?\n2. Have you considered or experimented with human-in-the-loop evaluations or human preference modeling to further validate your automated annotation-based reward scheme?\n3. How sensitive is ORISE to hyperparameter selection, particularly regarding reward weights and KL divergence terms, and how was stability during training ensured?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a new RL method to enhance full-duplex spoken dialogue language models.

All reviewers appreciate the contributions of this paper including important task, experimental results, and clear writing with marginal acceptance scores.

AC also agrees with the reviewers' consensus, so recommends accepting this paper.

AC asks the authors reflect the reviewers' comments in the final version.
Decision: Accept