SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation
Authors: Wenyi Yu
Siyin Wang
Xiaoyu Yang
Xianzhao Chen
Xiaohai Tian
Jun Zhang
Guangzhi Sun
Lu Lu
Yuxuan Wang
Chao Zhang
Venue: NeurIPS 2025 poster ()
Keywords: speech large language model
spoken dialogue model
full duplex model
Primary area: applications

Abstract: In order to enable fluid and natural human-machine speech interaction, existing full-duplex conversational systems often adopt modular architectures with auxiliary components such as voice activity detectors, interrupters, conversation state predictors, or multiple LLMs. These systems, however, suffer from error accumulation across modules and struggle with key challenges such as context-dependent barge-in and echo cancellation. Recent approaches, most notably Moshi, simplify the pipeline by injecting audio codecs into the token space of a single LLM. However, such methods still incur significant performance degradation when operating on the speech rather than text modality. In this paper, we introduce SALMONN-omni, the first single, standalone full-duplex speech LLM that operates without audio codecs in its token space. It features a novel dynamic thinking mechanism within the LLM backbone, enabling the model to learn when to transition between speaking and listening states. Experiments on widely used benchmarks for spoken question answering and open-domain dialogue show that SALMONN-omni achieves at least 30\% relative performance improvement over existing open-source full-duplex models and performs highly competitively to half-duplex and turn-based systems, despite using substantially less training data. Moreover, SALMONN-omni demonstrates strong performance in complex conversational scenarios, including turn-taking, backchanneling, echo cancellation and context-dependent barge-in, with further improvements achieved through reinforcement learning. Some demo conversations between user and SALMONN-omni are provided in the following repository https://github.com/bytedance/SALMONN.

SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation
Wenyi Yu!.2+
Siyin Wang 1.2- Xiaoyu Yang²
Xianzhao Chen²
Xiaohai Tian
Jun Zhang²
Guangzhi Sun
Lu Lu
Yuxuan Wang²
Chao Zhang'
'Tsnnguua University "ByteDance
University of Cambridge
(ywy22, wwngsiyyn3) (atessiyyyymmmmaallleaaa @mails. tsinghua. edu.
cz277@tsinghua edu. cn
Abstract
In order to enable fluid and natural numan-machinn speech interaction, existing full-duplex conversational systems often adopt modular architectures with auxiliary components such as voice activity detectors, interrupters, conversation state predic- tors, or multiple LLMs. These systems, however, suffer from error accumulation across modules and struggle with key challenges such as context-dependent barge- in and echo cancellation. Recent approaches, most notably Moshi, simplify the pipeline by injecting audio codecs into the token space of single LLM. However, such methods still incur significant performance degradation when operating on the speech rather than text modality. In this paper, we introduce SALMONN- the first single, standalone full-duplex speech LLM that operates without audio codecs in its token space. It features novel dynamic thinking mechanism within the LLM backbone, enabling the model to learn when to transition between speaking and listening states. Experiments on widely used benchmarks for spoken ques- tion answering and open-domain dialogue show that SALMONN-omin achieves least 30% relative performance improvement
existing open-source full- duplex models and performs highly competitively to half-duplex and turn-based systems, despite using substantially less training data. Moreover, SALMONN-ome demonstrates strong performance in complex conversational scenarios, including turn-taking, backchanneling, echo cancellation and context-dependent barge-in, with further improvements achieved through reinforcement learning. Some demo conversations between user and SALMONN-omii are provided in the following repository https http:/bb..com/yyyyeearee////////
Introduction
Large language models (LLMs) T 3 have revolutionized problem-solvine and task execution through text-based interaction. However, speech, being more natural and intuitive modality, offers smoother and more human-centric interface for human-machine communication. As a result, there increasing interest in extending text-based LLMs with speech- processing capabilities. These efforts span range of tasks, including advanced speech and audio understanding 5 text-driven speech synthesis 7 8 and interactive spoken dialogue systems [TO]
1I
12
While based speech LLMs [TO] 12 13
15] support conversational artificial intelligence (AI) in half-duplex manner, human conversation is inherently full-duplex, requiring the ability to listen, think, and speak simultancously. This dynamic is evident in natural conversational behaviours such as frequent turn-taking, backchanneling, overlapping speech and barge-ins, all of which contribute to more fluid and engaging interactions. These rich, interactive communication patterns have spurred
"Thee authors contributed equally to this work. Conresoonding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

growing interest in full-duplex speech LLMs, which seek to improve the naturalness, responsiveness and overall quality of human-Al conversations.
The recent release of GPT-4o (GPT-4omni) has demonstrated low-latency, emotionally expressive speech capabilities, further increasing global interest in full-duplex speech technologies. Among open research efforts, models such as Moshi [T]] SyncLLM [16) and OmniFlatten (17) inject discrete audio codee tokens into pre-trained LLM backbones and fine-tune them to support flexible speech input and output. However, these approaches typically require large-scale training data to bridge the modality gap between text and speech and to mitigate catastrophic forgetting of original LLM textual knowledge. As result, the speech synthesizer must infer whether to produce speech or silence based solely on the codec stream. While these models avoid the need for annotations of conversational dynamics, they frequently struggle with accurate timing of dialogue transitions (18). Other approaches [9 20 21 22 explicitly predict transitions between speaking and non-speaking states and integrate streaming speech encoders and synthesizers to enable real-time interaction. However, these systems typically rely on complex mechanisms rather than a standalone L.LM to handle full-duplex functionality, resulting in added system complexity and suboptimal context integration. Some methods, such as [2| 22], propose cascaded systems with external modules like voice activity detection (VAD) and dialogue controllers to manage conversational flow, which suffers from error propagation. Others, including VITA
[9 and Freeze-Oma (20). follow a "model-as-a server" paradigm, running two interdependent LLM processes, with one dedicated to listening and the other to speaking, thus introducing additional computational and memory overhead.
This paper introduces SALMONN-omni, standalone speech interaction model based on novel codec-free, full-duplex spoken dialogue framework. By seamlessly integrating streaming speech encoder, a single LLM backbone without audio codec injection, and a streaming speech synthesizer, SALMONN-omni supports simultaneous speech input and output within unified end-to-end archi- tecture. To autonomously manage dialogue state transitions in full-duplex interactions, we propose periodic synchronisation mechanism coupled with a novel "thinking" strategy, endowing the model with the temporal awareness that enables coherent and responsive real-time speech communication.
Our key contributions are summarized as follows:
the first standalone speech LLM that enables full-duplex Auman-AI conversations without injecting audio codecs into the LLM backbone. Unlike previous models, GALLONN-omm directly integrates a streaming speech encoder, an LLM, and speech synthesizer, requiring careful synchronisation to process input and output speech simultancously. As standalone model, the model must also decide when to start and stop speaking on its own. To achieve this, we train the LLM generate special tokens that control dialogue timing, treating these state transitions just like generating normal text tokens, based on the idea that "your LLM is secretly a full-duplex predictor."
We present SALMONN-omai
Experiments widely used spoken question answering (QA) and conversation benchmarks demonstrate the strong performance of MALMONN-omii In full-duplex mode, SALMONN- omni sets new sfate-oftthe-e art (SOTA), achieving least 30% relative improvement over previous best results. In half-duplex evaluation, it delivers highly competitive performance compared recent turn-based models, including those trained on substantially larger datasets - some with up to 13 million hours of audio (1T).
-ALMONN-omni shows strong performance in predicting the timing of turn-taking, backchanneling, and context-dependent barge-ins in free-form conversations. Additionally, to the best of our knowledge, we are the first to incorporate reinforcement learning (RL) to further improve the modeling of dialogue dynamics.
2 Related Work
2.1 End-to-end speech (interaction) LLMs
Compared with traditional modular conversational AI systems [23] 24 25
end-to-end speech interac-
tion LLMs have attracted great attention for their ability to support fluent, expressive, and emotionally rich spoken interactions. Depending on whether the model can listen and speak simultaneously, core characteristic of human communication, recent end-to-end conversational AI systems can be broadly categorized into two types: half-duplex (turn-based) and full-duplex speech LLMs.
2

---

Currently, most speech LLMs operate in a half-duplex manner [3 14 19 26 27 including models such as GLM-4- Voice [[5]. Qwen2.5-Omni [22]. and Kimi-Audio 21]. While these models can engage in turn-based speech conversations, they lack an internal duplex strategy for modeling dialogue dynamics like turn-taking. Instead, they rely on external VAD modules to alternate between listening and speaking states. As result, they struggle with key aspects of natural conversation (e.g., barge-ins and backchanneling) which require the ability to listen and speak simultaneously.
can process streaming speech input and output simultaneously, while also determining when to speak and when to stop. One straightforward approach to building such models involves injecting audio codecs into the LLM vocabulary, adopted by Moshi (IT) syncLLM [16] and OmniFlatten (17). Despite its conceptual simplicity, this approach demands large-scale speech-text paired data to prevent catastrophic forgetting. Even with extensive training, they typically lag behind similarly sized text-only L.MS in knowledge and reasoning and suffer from the modality gap. An alternative strategy used by VITA (19), Freeze-Omni (20) and MinMo (28) is to connect the LLM backbone with speech encoder and synthesizer through embeddings, without significantly hurting the LLMs. However, they are not standalone full-duplex since one LLM instance can only listen or speak, and they require two separate LLM processes to manage simultaneous listening and speaking. In contrast, SALMONN-ommi introduces novel duplex strategy that enables a single LLM to perform standalone full-duplex speech interaction.
Full-duplex speech LLMs [T]
17 20 28
Table 1: Categorization of speech interaction L.LMs by full-duplex support. Full-duplex
Half-duplex (turn-based)
standalone
non-standalone
BLMM--Voicee 15 Owen2.5-Omml 22
dGSLM (30), Moshi 11
Freeze-Omni 20
miniCPM-o 26 Baichuan-Audio 27
SALMONN-omm
VITA 19
MinMo 28
2.2 Reinforcement learning (RL) for speech LLMs
RL has long been applied to advance speech tasks, for example, to minimize word errors for automatic
speech recognition (ASR) [3T 32 33 34), to optimize task-oriented spoken dialogue procedure (35)
36). and to enhance emotional expressiveness in speech synthesis [37 38 39]. As for speech LLMss
recent RL methods such as proximal policy optimisation (40) and direct preference optimisation (DPO) [41] have been adopted by speech understanding models like Qwen2-Audio (6] and Step-
Audio [42] to align speech LLM behaviour with human preference. However, RL remains unexplored in the context of full-duplex speech LLMs. To the best of our knowledge, SALMONN-ome is the
first work to apply RL to improve the modeling of dialogue dynamics for full-duplex speech LLMs.
3 Methodology
Four challenges must be solved when implementing a full-duplex speech LLM. To implement SALMONN- as a standalone full-duplex model without codec injection, novel codec-free (no audio codecs in LLM vocabulary) full-duplex spoken dialogue framework is proposed.
The model must support streaming speech input and output. Without relying on codecs, SALMONN-ommt achieves real-time speech interaction by integrating an LLM backbone with a streaming speech encoder and a streaming speech synthesizer through hidden embeddings.
The model must process both environmental sounds and the assistant's speech simultancously. In full-duplex conversations, the speech LLM must generate speech responses as the assistant stream while concurrently processing all incoming sounds (background sounds, user speech, assistant echo and etc.) as the environment stream. SALMONN- response tokens, environment stream embeddings, and assistant stream embeddings into a single sequence, enabling the LLM backbone to model them jointly in an autoregressive manner. Note that Moshi [I] also jointly models these two streams, but with different purposes and design choices.
achieves this by interleaving LLM text
The model must incorporate sense of "time" to align audio and text modalities. SALMONN- omni employs a periodic synchronisation mechanism, processing a fixed duration of input speech while generating matching duration of speech responses in each time block for smooth interactions.
3

---

The model must handle natural conversation dynamics such as turn-taking and barge-ins. This requires the model decide when start and stop speaking based on contextual understanding. SALMONN-ommi learns these state transitions using novel "thinking" strategy. Instead of adding separate full-duplex predictor, we propose that your L.LM secretly full-duplex predictor and train it to generate state transition tokens as part of its normal output.
Notably, the last three features are key distinctions between SALMONN-ommn and models such as VITA, Freeze-Omni, and MinMo, whose LLM decoders cannot listen and speak simultancously. This limitation also prevents these models from achieving full-duplex dialogue in a standalone manner. The structure of SALMONN-On is illustrated in Fig. [ with detailed explanation of its key components provided in the following subsections.
-0
Environment stream hidden
Assistant stream hidden
Text hidden
Streaming Speech Synthesizer
Large Language Model
<bos>
<think>
<shift>
It's raining umbrello.
<shift>
<think>
Mamba Streaming Speech Encoder
- [
|I Mom
(What's weather todoy?)
(It's raining You'd better bring on umbrella.) (Thanks for your aduice!)
The architecture of SALMONN-omni. Two input streams, the environment stream and the assistant stream, are processed by streaming speech encoder. Speech embeddings from both streams, along with textual embeddings, are fed into the LLM backbone in interleaved manner. When in the speaking state, the streaming speech synthesizer takes the textual embeddings derived from the LLM backbone input to produce speech responses.
Figure 1:
3.1 Mamba streaming speech encoder
It is demonstrated that streaming general-purper encoder can be effectively trained using knowl- edge distillation from multiple pre-trained non-streaming encoders (43). Following this approach, SALMONN-omni employs streaming speech encoder designed to enhance generalisation in long conversational contexts, with Whisper-large-v. (44) serving as the teacher model. Specifically, log-Mel filter bank features are extracted at 100Hz frame rate, then downsampled to 50Hz using two convolutional layers. Next, two adjacent embeddings are concatenated into a single embedding. which is fed into number of Mamba language model blocks [45). An LI loss function is applied to match the output features of the streaming encoder to those of the teacher model.
3.2 Streaming speech synthesizer
Our streaming speech synthesizer builds upon CosyVoicet Voice2 [46). adopting its fixed-length interleaved speech generation strategy. In this approach, input text and output speech tokens are interleaved so that every N text tokens, the model generates end-to-end integration with the LLM, we replace the original text input of the synthesizer with the L.M backbone's 's output embeddings. Several linear transformations is then applied to align these LLM embeddings with the input space expected by the streaming speech synthesizer.
speech tokens in sequence. To support

---

3.3 Full-duplex strategy
Interleaving multiple streams into single stream. LLMs are traditionally trained to model single sequence in an autoregressive manner. To enable pre-trained LLM backbones to effectively handle multiple concurrent streams in natural conversations, we propose interleaving these streams into a unified soft token sequence. Specifically, we divide conversation into two primary streams: the assistant stream, representing the model's generated responses, and the environment stream, which includes all incoming audio, such user speech, background noise, and echoes from the assistant's own speech. full-duplex model must autoregressively generate the assistant stream while continuously attending to the environment stream. To facilitate this, segment the conversation into fixed-duation time blocks. Within each block, SALMONN-Ome from the environment stream, then generates the corresponding assistant response. To further stabilise training and enhance performance, a dual-chame channel input mechanism is employed: in each time block, the model also receives speech embeddings from its prior assistant outputs (including echo information) before generating the next textual response.
first processes speech embeddings
test hidden
Eavirooment stream haddes
Alstes
disleo
eapeskos
<speaks
<speolic
sistaas
Lange Language Model
Large Language Model
calillo
<ttiib
*
Livioning
Speaking
Listening
Linwning
Speoking
Livnoing
(a) Implicit thinking
(b) Explicit thinking
Figure 2:
Illustration of "Implicit" and "Explicit" thinking strategies. The tokens on the top of the LLM are predicted by speech embeddings, while the bottom ones are predicted by textual embeddings and fed back to the input sequences to the LLM.
"Thinking" strategy for handling dynamics in natural conversations. The difference between turn-based and full-duplex spoken dialogue models is the latter's 's ability to dynamically determine when to start and stop speaking. To enable this, propose novel "thinking" strategy that allows SALMONN- oomni to learn the transitions between listening and speaking states. Within each time block, the model must decide whether state transition should occur. We explored two variants of this strategy, which differ in whether the predicted state token from each time block autoregressively fed input the next time block. An illustration of the two strategies is shown in Figure 2
Implicit "thinking". Under this strategy, the state determined by the model in each time block not carried over in the LLM's i input sequence. Whenever receiving audio segment, the model determines the state as "listen" or "speak". "listen", it continues to ingest the next audio time block. If "speakk, it generates a specified number of textual tokens the response before consuming the next audio segment. Notably, when a state transition from "listen" to "speak" occurs, a special token <shift> is sent to the input manually to initiate the text generation process.
Explicit "thinking". This strategy does not distinguish between the prediction of state-related special tokens and the generation of the textual response. Consequently, special token predicted by the model is autoregressively used as input for the subsequent time block. Two special tokens, <think> and <shift>, scenarios. First, when the model is the listening state, it should generate <think> each time block, which mimics the human thought process of deciding when to speak in conversation. Second, due to the large difference in frame rates between text and speech modalities, the model generates <think> during the time blocks after completing the text response while waiting for state transition from speaking to listening. The special token <shift> marks the state transitions of both "listening + speaking" and "speaking - listening".
introduced in this strategy. The special token <think> is utilized in two
The comparison of these two strategies is performed in Section 5.1 and the better performance achieved through explicit "thinking" demonstrates that training the LLM to generate whole sequence
5

---

consisting of both normal responses and teransiiioo-relateeee tokens naturally aligns with its intrinsic mechanism. other words. your LLM is secretly a full-duplex predictor.
3.4 Training scheme
We propose a three-stage training procedure for SALMONN-omni, as illustrated in Fig
Connecting streaming encoder. In stage 1, focus enabling the model with streaming speech understanding ability by connecting the Mamba streaming encoder with the LLM. The connector between the streaming encoder and the LLM is MLP. Only the connector and LoRA on LLM are trained on ASR and QA tasks, with the encoder and LLM backbone frozen.
Stage 1:
Stage 2: Connecting streaming synthesizer. In stage 2, SALMONN-omni is equipped with the speech generation capability and the whole model is trained in an end-to-end manner. In this stage, the streaming encoder and LLM backbone remain frozen. The connectors for the encoder and synthesizer, LoRA on LLM and the streaming synthesizer are trained on ASR, QA and multi-turn dialogue, including conversation dynamics such barge-ins and backchanneling.
Stage 3:
RL for full-duplex modeling. After the first two-stage training, SALMONN-omni demon- strates the capability to manage complex dynamics in free-form conversations, particularly in handling context-aware barge-ins and backchanneling. However, after the above supervised fine-tuning (SFT), the model exhibits clear bias towards being interrupted and shows insufficient understanding of the dialogue semantics. Thus, we incorporate reinforcement learning to further enhance the full-duplex modeling ability. Concretely, DPO (11 is applied to barge-in and backchanneling tasks, while subset of the SFT data from other tasks is retained to preserve overall performance.
Lapeess (CE)
Lapeuct
Speech Synthesizer
Speech Synthesizer
Luoe
Luoe/CE) (CE)
Synthesizer Adapter
Liod (PPO)
Syuthesizer Adapter
Large Language Model
LoRA
Large Language Model
LoRA
Large Language Model
LoRA
System
Encoder Adapter
System
Encoder Adapter
System Prompt
Encoder Adapter
Prompt
Speech Encoder
Prompt
Speech Encoder
Speech Encoder
Environment streom
Environment stream assistont streom
Enviroement stream assistant stream
(a) Stagel
(b) Stage2
(c) Stage3
Connecting Encoder
Connecting Synthesizer
RL. for full-duplex modeling
Figure 3: Three-stage training strategy for SALMONN-omni.
4 Experimental Setups
4.1 Model specifications
The Mamba streaming encoder in SALMONN-ommi consists of 32 Mamba 45 LM blocks with a lis--dimensional hidden state. The encoder generates embeddings at a frame rate of 25Hz, requiring downsampling of Whisper-large-v3 embeddings from 50Hz to 25Hz. This is achieved by concatenating every two adjacent embeddings into a single embedding. To ensure dimensional compatibility before applying LI loss, introduce linear layer to align the embedding dimensions 3-B-n---tttttt
of the two models. We use Alama-3-883 Instruct as the LLM backbone and use LoRA [47] with a rank of 32 and scaling factor of 1.0 when finetuning the LLM backbone. Our streaming speech synthesizer is finetuned based on CosyVoice2-00 Voice2-0.5B [46).
We set 80 milliseconds (ms) the time block size and the model generates one textual token after listening to ms of audio. Once the L.LM backbone generates tokens, 12 speech tokens will be generated by the streaming speech synthesizer, namely 480 ms speech. Thus, delay of 320 ms

---

oceurs between the moment the model initiates speech and the onset of audio output. The current design strikes an effective balance between response latency and speech generation quality.
4.2 Training tasks
SALMONN-omin is trained on ASR, spoken QA and multi-round conversation tasks. For ASR task, LibriSpeech-90 (48] corpus and GigaSpeech-M (4) with ~480k training samples are utilized. For spoken QA task, we gather questions from a variety of sources, including Alpaca-52k (50), Web Questions [5] TriviaQA (52). SQuAD (53), Natural Questions [54]. YoiceAssistant-4000
from Mini-Omni [14] and UltraChat from SLAM-Omni [55]. Overall, there are ~730k QA samples in our training datasets. For multi-round conversation, the conversation is generated by L-ama-3-8B-Insiie based on a theme selected from TriviaQA and Natural Questions. Then the conversation is synthesized by Cosy Voice2-0.5B. There are ~80k multi-round conversation samples in our training datasets.
Moreover, we consider scenarios involving barge-ins and backchanneling in multi-turn conversations. The dataset includes two types of harge-ins. For context-independent barge-ins, we prompt GPT-44 to generate 10 direct interrupting sentences (e.g. Please stop talking right now.). For context-dependent barge-ins, design task evaluate the model's ability to determine whether to stop speaking based on contextual cues. Specifically, we prompt Llama-3-8B-Instie are either closely related to the ongoing dialogue (as meaningful interruptions) or entirely unrelated (as distractors). For backchanneling, we include 7 commonly used words or phrases (e.g. Uh-huh.). Notably, our experiments also take into account multi-speaker barge-in scenarios. More details about our training dataset can be found in Appendix A
to generate questions that
4.3 Evaluation
Evaluation datasets and metrics. We evaluate the speech interaction ability on four commonly used datasets, I.lama Questions (56). Web Questions (5T), TriviaQA (52) and AlpacaEval from Voicebench [57]. For Web Questions, the questions
converted into speech using commercial TTS model from Volcano Engine. For triviaQA, we use the subset with 1000 samples from Ope- nAudioBench (27). For knowledge QA datasets, Llama Questions, Web Questions and TriviaQA, accuracy are calculated as evaluation metric. For oral conversation dataset AlpacaEval, GPTScore by GPT-411-0025004-44 is reported score the appropriateness and reasonableness of the answer, rang- ing from 1 to 5 with indicating the worst and 5 the best. Both answer text and speech are evaluated, noted as S2T and S2S. For S2S, answer speech is transcribed into text using Whisper-large-- v3 first. The quality of the response speech is evaluated using the widely used UTMOS [58].
For evaluating full-duplex modeling capahilities, first, we report the success rate of turn-taking timing prediction, which reflects the model's ability to determine the appropriate moment to begin speaking. Second, assess the model's ability to decide whether it should be interrupted, measured by the F1 score. For the context-indepeendent setting, we collect 100 direct interrupting sentences - 50 spoken by the user and 50 by a third speaker - as positive examples, and 100 user-spoken backchanneling phrases as distractors. In the context-dependent setting, we collect 100 contextually relevant questions intended interrupt the model (again, 50 from the user and 50 from a third speaker), while using 50 unrelated questions spoken by a third speaker and 50 silent samples as negative examples. Noted that the sentences used for evaluation are different from those during training.
To assess turn-taking statistics and real-world latency, we conduct live human-machine conversa- tions across three scenarios: multi-turn dialogue, cerreettal-indppedddde dependent barge-in. A total of 60 dialogue sessions are collected (20 for each scenario). On average, conversations in multi-turn scenario consist of turns with mean duration of 75 seconds, whereas the dialogues in barge-in settings are shorter, averaging 3 turns. Statistics of turn-taking events including Inter-Pausal Unit (IPU), Pause and Gap are only evaluated in the multi-turn scenario.
barge-in and contextual-
Baselines. We select a large scale of performant open- source half-duplex or full-duplex speech LLMs as baselines, including Moshi [TT], Prrezze-mmit (20). GLM [15]. Owen2.5-0mmi 22), VITA 1.5 19), miniCPM-o 266. Kimi-Audio 21 and Baichuan-Audio [27]. All models are
tested in an "oracle turn-taking" setting. in which models are forced to start speaking at the end of
the question, except Moshi since it's difficult to force Moshi speak. Moreover, we also experiment
with more practical "predicted turntakking setting, in which the model needs to determine when to
7

---

start speaking by itself, which only full-duplex models can achieve. Moshi and Freeze- Omiz serve as baselines in this setting.
Experimental Results
5.1 Comparison between implicit "thinking" and explicit "thinking"
We first compare the models trained with implicit and explicit "thinking" strategies in stage As shown in Table 2 explicit "thinking" consistently outperforms implicit "thinking". We attribute this to training the L.M to output sequences containing both normal responses and state-transition-relae tokens better aligns with its autoregressive nature. In contrast, the implicit "thinking" lacks complete feedback from output back to the input of the L.M.. An interesting observation is that after Stage models achieve only around 70% turn-taking success rates on AlpacaEval. However, with the introduction the assistant stream in Stage 2, success rates increase to approximately 90%, which highlights the importance of incorporating the assistant stream in full-duplex modeling for more robust turn-taking prediction. According to above results, we adopt explicit "thinking" as full-duplex strategy throughout this work. Actually, some variants can be derived from these two basic strategies, which are further analyzed in the Appendix C
Table 2: Comparison of implicit and explicit thinking strategy.
Full-duplex strategy
Llama Q.
Web Q.
TriviaQA
AlpacaEval succ. %
succ. %
S2T
succ. %
S2T
succ. %
S2T
S2T
Implicit Explicit
97.3 100.0
76.3
99.9
50.8
95.8
62.0
67.3
3.73
82.0
99.9
52.4
92.3
67.5
68.3
4.48
5.2 Comparison of the speech interaction capabilities between different speech LLMs
The main results of speech interaction abilities evaluation are shown in Table Under the predicted turn-taking setting, SALMONN -ommn achieves SOTA performance across all datasets, delivering an average 35.9% relative improvement over previous open-source full-duplex speech interaction LLMs. This substantial gain highlights AALMONN-omni) strong capabilities in handling wide range of conversational scenarios from general knowledge queries to everyday casual dialogue. In the oracle turn-taking setting, we further compare SALMONN-mme with more recent turn- turn-based speech interaction LLMs. As illustrated in Table 3 SALMONN- performance across all models, with best speech-to-text performances across 4 datasets and best speech-to-speech performances on Web Questions and TriviaQA. The average UTMOS across all datasets indicates that the speech generated by SALMONN-omin is of high quality. These results demonstrate that SALMONN-omm not only excels in speech understanding but also maintains highly stable and coherent speech generation.
achieves the best overall
5.3
Evaluation of full-duplex modeling capabilities
5.3.1 Turn-taking prediction
The turn-taking performances of different full-duplex speech interaction LLMSs are presented in Table The results show that SALMONN-Ome achieves the highest success rate in accurately predicting turn-taking moments, averaged across multiple datasets. Notably, SALMONN- maintains strong performance even when handling short and fast-paced speech inputs, such as those in TriviaQA and AlpacaEval, scenario in which Moshi and Freeze-Omni suffer a significant performance decline.
4
5.3.2 Distinguish true barge-ins from false barge-ins and backchanneling
Comparison between different models on contextuualinddeeender setting, As shown in Table 5, SALLONN-onni demonstrates more robust performance on handling nontextualinndeeedder barge-ins and backchanneling. Moreover, both SALMONN-omni and Moshi can perform well when listening to their own echoes simultaneously, while Freeze-Omn exhibits significant performance drop. It highlights the drawback of not implementing full full-duplex models in a standalone manner.
8

---

Table 3: The performance of different speech interaction LLMs. For Llama Questions, Web Questions
and Tr,viaQA, the evaluation metric is Acc.% For AlpacaEval, the evaluation metric is GPTScore. S2T is the speech-to-fex performance and S2S is the speech-to-speech performance. The average UTMOS across all datasets is also reported with a more detailed version in Table 13
Llama Q.
Web Q.
TriviaQA
AlpacaEval S2T
Avg. UTMOS
Model
S2T
S2S
S2T
$2$
S2T
S2S
S2S
Predicted turn-taking
Moshi
60.8
54.5
23.4
22.1
25.6
16.7
1.84
1.76
3.178
Freeze- Omni 20
74.2
56.2
40.8
27.9
45.1
28.5
3.90
2.46
4.277
SALMONN-omni
79.3
73.6
49.7
43.7
63.6
56.0
4.01
3.22
4.255
Oracle turn-taking
GLM-4-Voice Voice
15
75.0
65.7
38.5
37.0
50.8
47.5
3.82
3.58
4.081
Qwen2.5-0mmi
22
78.7
75.7
41.9
39.6
55.5
52.2
3.04
2.61
4.085
VITA-1.5 [9
77.3
56.0
43.9
31.0
54.8
41.3
3.92
1.94
4.022
miniCPM-o 26
76.3
70.0
47.1
42.7
65.4
56.0
3.99
3.44
4.147
Kimi-Audio
21
79.7
68.3
44.0
37.3
63.6
51.2
3.92
2.95
2.679
Baichuan-Auu Aucio
27
76.0
74.0
43.1
40.7
56.9
53.0
3.76
3.32
4.182
Freeze-Omni (20)
77.0
61.0
42.3
29.1
53.5
36.5
3.71
2.42
4.276
SALMONN-omit
80.0
75.0
50.5
45.4
66.0
58.8
4.05
3.33
4.261
Table 4: Turn-taking success rate (%) of full-duplex speech interaction LLMs.
Model
Llama Q.
Web Q.
TriviaQA
AlpacaEval
Moshi
85.0
76.0
37.1
83.4
Freeze-Omni
99.7
99.8
72.0
87.9
SALMONN-omni
99.7
99.6
92.8
92.0
Since Freeze-Ome Omni relies on an additional VAD module, its performance is severely affected when the VAD module fails provide accurate speech segmentation.
Table 5: Comparison between different models on contextual-inndeeender barge-ins and backchan- neling. Echo factor means the model can hear its own echo.
Model
Echo factor
Precision
Recall
F1 score
Moshi
x 11.
0.84
0.77
0.80
Freeze-Omni
x0,0
0.64
0.72
0.68
Freeze-Omni
x0.1
0.19
0.86
0.31
Freeze-Omni
x
0.10
0.69
0.17
SALMONN-omni
x1.0
0.84
0.92
0.88
DPO further enhances model's full-duplex modeling capabilities. We train SALMONN-omni on both context-independent and -epeedttee settings, and turns that the model can handle both scenarios effectively, which demonstrates our framework can handle complex context-aware dynamics in free-form conversations. However, after the SFT stages, the low precision shows that SALMONN-ome -mmi equips a tendency to be interrupted, which reflects the model's 's insufficient understanding of context. Thus, incorporate DPO for post-training the model, and the overall performance improvement confirms the effectiveness of this approach. As shown in Fig. 4 and Table 6, an interesting phenomenon observed during DPO training that model initially becomes extremely conservative, rarely getting interrupted at all. However, as training progresses, the model's capabilities gradually recover and eventually surpass those of the SFT-trained models. More details can be found in Appendix D

---

Table The performance SALMONN ommi on nuntextual-dependeee and contextual-independeee settings through the DPO training stage with batch size set to 256.
Contextual-Independeen
Contextual-Dependdnt
Overall
Stage
Precision
Recall
FI score
Precision
Recall
Fl score
Fl score
SFT (Stage 2)
0.68
0.93
0.79
0.88
0.99
0.93
0.86
DPO 10 steps
0.88
0.07
0.13
0.95
0.21
0.34
0.24
DPO 40 steps
0.84
0.92
0.88
0.89
0.98
0.93
0.90
0.8
Overall F1 scoeee
0.6
Table 7: Statistics of turn-takings events (IPU, Pause
and Gap) in real-world human- human-machine machine dialogues.
0.4
DPQ [bs2-128)
IPU
Pause
Gap
0.2
DPO (bs2=256)
(bsz=512)
Ground Truth
61.89s
5,49s
6,50s
0.0
SALMONN-omni
63.30s
4.68s
5.01s
40
number of steps
The overall Fl score of SALMONN- when trained with dif- ferent batch sizes during the DPO stage.
Figure 4:
5.4 Turn-taking statisties and latency evaluation in real-world deployment
The turnttaking statistics in the multi-turn dialogue scenario (normalized to 5-turn dialogues) shown in Table 7 which is largely consistent with the statistical properties the training data. As for latency, SALMONN-omni exhibits average Time-to-First-Audin (TTFA) of 405ms. This latency is primarily consequence of its design, which mandates waiting for four time blocks to accumulate sufficient context for the speech synthesizer. We posit that this latency is not fundamental limitation and could be substantially reduced through synthesizer optimization. In contrast, the model demonstrates its capacity for rapid interaction in the barge-in scenarios, achieving response latencies of approximately 128 ms and 144 ms for contextual mentextttlaidddeeeeeee and contextual- dependent internuptions, respectively. These results confirm the model's l's ability react swiftly to user interjections.
Conclusion
In this work, we present SALMONN-ommi, the first standalone full-duplex speech LLM that operates without injecting audio codec tokens into its vocabulary. By seamlessly integrating streaming speech encoder, a unified LLM backbone, and streaming speech synthesiser, and introducing a novel explicit "thinking" strategy for predicting dialogue state transitions, SALMONN-omni autonomously learns when to listen and when to speak. Extensive experiments on standard benchmarks for spoken QA and open-domain dialogue show that SALMONN-omni achieves average 35.9% relative accuracy improvement over existing open-source full-duplex models predicted turn-taking scenarios, while remaining highly competitive with half-duplex models trained significantly larger datasets. Furthermore, SALMONN-omia demonstrates robust modeling of full-duplex conversational dynamics, including turn-taking, backchanneling, echo cancellation, and context-dependent barge-ins. Additional RL further enhances the model's 's ability to handle real-time dialogue behaviours with improved fluency and responsiveness.
10

---

References
[1] OpenAL GPT-4 technical report. 2024. URL: httph:t/atapiiiiia org/abs/ /2003/233...m 08774
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, al. "I.LaM:: Open and efficient foundation language
models". In: arXiv preprint arXiv:2.02. /.97/ (2023).
[3] Daya Guo, Dejian Yang, Haowei Zhang, al. "DeepScek-Ree R1: Incentivizing reasoning capability in LLMs via reinforcement learning". : In: arXiv preprint arXiv:2501. /2948 (2025).
[4] Changli Tang. Wenyi Yu, Guangzhi Sun, al. "SALMONN: Towards generic hearing abilities for large language models". In: Proc. ICLR, Vienna, 2024.
[5] Yuan Gong, Hongyin Luo, Alexander H. Liu, al. "Listen, think, and understand". In: Proc. ICLR. Vienna, 2024.
[6] Yunfei Chu, Jin Xu, Qian Yang, al. "Qwen2-Audio technical report". In: arXiv preprint arXiv:2407. 10759 (2024).
[7] Hongkun Hao, Long Zhou, Shujie Liu, et al. "Boosting large language model for speech synthesis: An empirical study". In: arXiv preprint arXiv:2401.002 (2023).
[8] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. "The Llama herd models". In; arXiv preprint arXiv:2407.2 200.2177777 (2024).
[9] Zhihao Du, Yuxuan Wang, Qian Chen, et al. VCosyVoice 2: Scalable streaming speech synthesis with large language models". In: arXiv preprint arXiv:2412. 10/ (2024).
[10] Dong Zhang. Shimin Li, Xin Zhang, et al. "SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities". In: Proc. EMNLP Findings. Singapore, 2023.
[11] Alexandre Défossez, Laurent Mazaré, Manu Orsini, et al. "Moshi: A speech-text foundation model for real-time dialogue". In: arXiv preprint arXiv:2410.0000
(2024).
[12] Qingkai Fang. Shoutao Guo, Yan Zhou, et al. I.LaMA-Ommi Scamless speech interaction with large language models". In: Proc. ICLR. Singapore, 2025.
[13] Tu Anh Nguyen, Benjamin Muller, Bokai Yu, et al. "Spirit-LM: Interleaved spoken and written language
model". In: TACL 13 (2025), pp. 30-52.
[14] Zhifei Xie and Changqiao Wu. "Mini-Omni: Language models can hear, talk while thinking in streaming". In: arXiv preprint arXiv:2408. 16725 (2024).
[15] Aohan Zeng. Zhengxiao Du, Mingdao Liu, et al. "GLM-4- Towards intelligent and human-like end-to-end spoken chathot". In: arXiv preprint arXiv:2412.0266 (2024).
[16] Bandhav Veluri, Benjamin Peloquin, Bokai Yu, al. "Beyond turnbbased interfaces: Synchronous LLMs
as full-duplex dialogue agents". In: Proc. EMNLP. Miami, 2024.
[17] Qinglin Zhang, Luyao Cheng, Chong Deng, et al. "OmniFlatten: An end-to-end GPT model for seamless voice conversation". In: arXiv preprint arXiv:2440. (2024).
[18] Siddhant Arora, Zhiyun Lu, Chung-Cheng Chiu, al. "Talking turns: Benchmarking audio foundation models on turn-taking dynamics". In: Proc. ICLR. Singapore, 2025.
[19] Chaoyou Fu, Haojia Lin, Xiong Wang. et "VITA-1.5: Towards GPT-4o level real-time vision and speech interaction".. In: arXiv preprint arXiv:2501.0195 (2025).
[20] Xiong Wang, Yangze Li, Chaoyou Fu, al. "Freeze-Omni: A smart and low latency speech-to-specch dialogue model with frozen LLM In: arXiv preprint arXiv:2411.00774 (2024).
[21] Ding Ding, Zeqian Ju, Yichong Leng, et al. "Kimi-Audio technical report". In: arXiv preprint arXiv:2504. 18425 (2025).
[22] Jin Xu, Zhifang Guo, Jinzheng He, et al. "Qwen2.5 technical report". In: arXiv preprint arXiv:2533.200 (2025).
[23] Steve Young. Milica Gašić, Blaise Thomson, et al. WOMMP-based statistical spoken dialog systems: A review". In: Proceedings of the IEEE 101.5 (2013), pp. 1160-1179.
[24] Oriol Vinyals and Quoc V. Le. "A neural conversational model". In: Proc. ICML Deep Learning Workshop. Lille, 2015.
[25] lulian V. Serban, Alessandro Sordoni, Yoshua Bengio, et al. "Building end-to-end dialogue systems using generative hierarchical neural network models". In: Proc. AAAI. Phoenix, 2016.
[26]
OpenBMB. MiniCPM-O 0 2.6: A GPT-4u level MLLM for vision, speech, and multimodal live streaming your phone. htttps:///ppenaeee notion..onoooooooooooooooooo...t
2025. [27] Tianpeng Li, Jun I. Liu, Tao Zhang, et al. "Baichuan-Audio: A unified framework for end-to-end speech interaction". In: arXiv preprint arXiv:2502. /7239 (2025).
[28] Qian Chen, Yafeng Chen, Yanni Chen, et al. "MinMo: A multimodal large language model for seamless voice interaction".. In: arXiv preprint arXiv:2501.06288 (2025).
[29] Ziyang Ma, Yakun Song, Chenpeng Du, al. "Language model can listen while speaking". In: Proc. AAAL Philadelphia, 2025.
11

---

[30] Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, et al. "Generative spoken dialogue language modeling". In: TACL 11 (2023), pp. 250-266.
[31] Matt Shannon. "Optimizing expected word error rate via sampling for speech recognition". In: Proc. Interspeech. Stockholm, 2017.
[32] Dongcheng Jiang, Chao Zhang, and Philip C. Woodland. "Variable Frame Rate Acoustic Models Using
Minimum Error Reinforcement Learning"' " In: Interspeech. 2021, pp. 2601-2603
[33] Ye Bai, Jingping Chen, Jitong Chen, et al. "Seed-ASR: Understanding diverse speech and contexts with LLM-based speech recognition". In: arXiv preprint arXiv:2407. 04675 (2024).
[34] Chirag Nagpal, Subhashini Venugopalan, Jimmy Tobin, et al. "Speech recognition with LLMs adapted to disordered speech using reinforcement learning". In: Proc. ICASSP. Hyderabad, 2025.
[35] Satinder Singh, Michael Kearns. Diane Litman, et al. "Reinforvement learning for spoken dialogue systems". In: Proc. NIPS. Denver, 1999.
[36] Konrad Scheffler and Steve Young. "Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning". In: Proceedings of HLT. Vol. 2. 2002. p. O.
[37] Dong Zhang. Zhaowei Li, Shimin Li, et al. "SpeechAlign: Aligning speech generation to human prefer-
ences". In: Proc. NeurlPS. Vancouver, 2024.
[38] Chen Chen, Yuchen Hu, Wen Wu, et al. "Enhancing zero-shot text-to-speech synthesis with human feedback". In: arXiv preprint arXiv:2406.0066
(2024).
[39] Philip Anastassiou, Jiawei Chen, Jitong Chen, et al. "Seed-TTS: A family of high-quality versatile speech generation models". In: arXiv preprini urXiv:2406. 02430 (2024).
[40] John Schulman, Filip Wolski, Prafulla Dhariwal, et al. "Proximal policy optimization algorithms". In: arXiv preprint arXiv: 7707.06347 (2017).
[41] Rafael Rafailov, Archit Sharma, Eric Mitchell, et al. "Direct preference optimization: Your language model is secretly a reward model". In: NIPS. New Orleans, 2023.
[42] Ailin Huang, Boyong Wu, Bruce Wang, et al. "Step-Audio: Unified understanding and generation in intelligent speech interaction". In: arXiv preprint arXiv:2502. 1946 (2025).
[43] Xisoyu Yang, Qiujia Li, Chao Zhang, et al. "MT2KD: Towards general-purpose purppse encoder for speech, speaker, and audio events". In: arXiv preprint arXiv:2409. 17010 (2024).
[44] Alec Radford, Jong Wook Kim, Tao Xu, al. "Robust speech recognition via large-scale weak supervi- sion". In: Proc. ICMI. Honolulu, 2023.
[45] Albert Gu and Tri Dao. "Mamba: Linear-time sequence modeling with selective state spaces". In: Proc. COLM. Montreal, 2024.
[46] Zhihao Du. Yuxuan Wang, Qian Chen, al. "CosyVoice 2: Scalable streaming speech synthesis with large language models". In: arXiv preprint arXiv:2412.1011 (2024).
[47] Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, "LoRA: Low-rank adaptation of large language models". In: Proc. ICLR, 2022.
[48] Vassil Panayotov. Guoguo Chen, Daniel Povey, et al. "Librispeech: An ASR corpus based on public domain audio books". In: Proc. ICASSP. South Brisbane, 2015.
[49] Guoguo Chen, Shuzhou Chai, Guanbo Wang, et al. "GigaSpeech: An evolving, multi-domain ASR corpus with 10,000 hours of transcrihed audio". In: Proc. Interspeech. Brno, 2021.
[50] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, et al. Stanford Alpaca: An Instruction-folloiinn LLaMA model. / /tt.bub.tttu..... wtw/tatsu- lab/ htt/:sstnodddaaappa
2023.
[51] Jonathan Berant, Andrew Chou, Roy Frostig, et al. "Semantic parsing on freebase from question-answer
pairs". In: Proc. EMNLP. Seattle, 2013.
[52] Mandar Joshi, Eunsol Choi, Daniel Weld, et al. "TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension". In: Proc. ACL. Vancouver, 2017.
[53] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, et al. "SQuAD: 100,000+ Questions for Machine Comprehension of Text". In: Proc. EMNLP. Austin, 2016.
[54]
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, et al. "Natural questions: benchmark for question answering research". In: Transactions of the Association for Computational Linguistics 7 (2019).
pp. 453 466.
[S5]
Wenxi Chen, Ziyang Ma. Ruigi Yan, al. "SLAM-Omni: Controllo Voice Interaction System with Single-Stage Training". In: arXiv preprint arXiv:2412. 1564999 (2024).
[56] Eliya Nachmani, Alon Levkovitch, Roy Hirsch, et al. "Spoken Question Answering and Speech Continu- ation Using Spectrogram- Poweerdd LLM". In: Proc: ICLR. Vienna, 2024.
[57] Yiming Chen, Xianghu Yue, Chen Zhang, et al. "Voicebench: Benchmarking llm-based voice assistants".
In: arXiv preprint arXiv:2410.17/1 (2024).
[58] Takaaki Sacki, Detai Xin, Wataru Nakata, et al. "UTMOS: UTokyo-Sarulaab System for VoiceMOS Challenge 2022". In: Proc. InterSpeech. Incheon, 2022.
12

---

[59] Wei Kang, Xiaoyu Yang, Zengwei Yao. et al. "Libriheavy: A 50,000 hours ASR corpus with punctuation casing and context". In: Prec. ICASSP. Seoul, 2024. [60] Ilya Loshchilov and Frank Hutter. "Decoupled Weight Decay Regularization". In: Proc. ICLR. New Orleans, 2019.
13

---

A Further experimental details
A.1 Details for training data
For VoiccAssistant-400K and UltraChat, we only select the first-round QA and filter repeated questions. We utilize the question speech in the dataset and re-synthesize answer speech based on the text answer by CosyVoice22 Voice2-0.5B (46) for training. For other dataset, text responses are generated by Llama-3 --s-trrrutt and both question and speech audio are synthesized by Cosy CosyVoice2-0.55
Table 8: Training dataset details.
Task
Dataset
#Samples
LibriSpeech GigaSpeech
28 lk
ASR
200k
Alpaca-52k
39k
Web Questions
4k
TriviaQA
58k
QA
SQuAD
127k
Natural Questions
301k
Voice/ssistantt4400
79k
UltraChat
120k
multi-round
TriviaQA
25k
conversation
Natural Questions
56k
A.2 Training specifications
The Mamba streaming encoder is pre-trained on LibriHeavy 59 and GigaSpeech [4] for 300k steps with batch size of 512. The AdamW (60) optimizer with 31 0.9 and B3 = 0.95 were used. In the first two stages, the batch size is set to 128 and the learning rates are 4 x 10 and x 10 respectively. In the third stage, batch sizes include 128, 256 and 512 are compared with learning rate set
10 All training processes are performed on 32 A100 GPUs with 50k and 30k steps for stage and 2. Model checkpoints with the best validation accuracy are used for evaluation.
B Prompts
Prompts for Llama-3-8B-Inmn Instruct to generated spoken QA response
Please answer the following question (} in conversational style. Keep your response concise, to the point, and within 300 tokens.
Prompts for
to generated multi-turn dialogues
Based on the topic: (). Write the transcript of conversation between two people A and B. Use short turns. The conversation should be spoken form (especially the numbers or dates should be in spoken form) and doesn't 't include *xxx* [xxx] or (xxx).
14

---

You don't need Prompts for Llama-3-8B-Ingn Insttucct to generated context-dependent barge-in data
The following is conversation between A and B.
Please ask B question based the conversation. This question needs to be strongly related to what B said. The question should concise, short and in spoken form. Please only output your response.
Prompts for
to evaluate oral conversation results on AlpacaEval
I need your help to evaluate the performance of several models in the speech interaction scenario. The models will receive a speech input from the user, which they need understand and respond to with a speech output.
Your task is rate the model's responses solely based their accuracy and reasonableness, using the provided user input transcription [Instruction] and the model's output transcription [Response]. Please evaluate the response on scale of I to 5:
point: The response is clearly incorrect, illogical, or entirely fails to address the user's request.
points: The response contains major inaccuracies or flaws reasoning, even it appears somewhat related to the question.
points: The response mostly reasonable and factually correct but may include minor
mistakes or unsupported assumptions.
points: The response is accurate and logically sound, with only negligible or borderline issues. The response is fully accurate and logically consistent. It reflects correct and understanding of the user's input.
5 points:
well-reasoned
Below are the transcription of user's instruction and models' response:
### [Instruction]
()
### [Response]
After evaluating, please output the score only without anything else. to provide any explanations.
Prompts for wwwp:uudddd--.eeie-22e0--------0
evaluate emotion intensity
Given speech audio clip, evaluate whether the speaker expresses any emotion, and if so, how strongly and clearly that emotion is conveyed. Focus on #vocal features* such as pitch variation, intonation, rhythm, volume, and prosodic dynamics. Do *not= infer from semantic content only judge the presence and intensity of emotion in how the #voice sounds*, Output a single integer score from
based on the following scale: ### Scoring Rubric:
#*1 No Emotion#:: Completely flat and monotone. Voice sounds like neutral, mechanical
reading or dry recitation.
**2 Barely Emotionals:: Emotion may feel suppressed or accidental. Difficult to confidently
say any emotion present.
**3 Mild Emotion**: Emotion is noticeably present but limited in strength. Vocal delivery
includes some clear variation in tone or rhythm suggesting emotional engagement.
**4 - -Clear Emotion=*: Emotion is clearly conveyed through dynamic changes in pitch, volume, and timing. Voice reflects emotional involvement (e.g., excitement, frustration, warmth). #*5 Strong and Expressive Emotion*:; Emotion is vivid, intense, and unmistakable. Rich, nuanced control over prosody (e.g., expressive pitch arcs, pauses, stress) highlights emotional intensity.
After evaluating, please output the score only without anything else. You don't 't need to provide any explanations.
15

---

C Other Variants of Thinking Strategy
We also explore other thinking strategies with different thinking contents. We experiment with small dataset of only LibriSpeech and Alpaca-52k. The ASR results are reported in Table
Implicit-ASR. Implicit-ASR is variant of the implicit "thinking" strategy. Here only one special token <shift>. At those steps to keep silent, the model trained to perform streaming ASR. Moreover, within those time blocks that the model has finished the text response and is waiting for speech synthesizing, we utilize an LLM to generate some inner thoughts related to the current conversation as the label. At those points for state transition, it should output <shift> marking the state transition.
Explicit-ASR. Explicit-ASR is similar to Implicit-ASR, apart from that the thinking contents are sent back the imput autoregressively.
Explicit-NS has two special tokens <think> and <shift>. For the time blocks without generating text response, <think> is used the input and <shift> is used the label. However, the weight of the loss negative except for those time blocks in which the model needs to perform state transition.
Explicit-NS.
Table 9: The performance of different "thinking" strategies on ASR task. succ. is the success rate of tum-taking prediction.
test-clean % BWWR (4)
test-other
"Thinking" Strategy
succ.
succ.
%WER (L)
Implicit
93.3
2.67
94.6
6.63
Implicit-ASR
94.7
3.48
95.7
8.26
Explicit
95.6
2.40
95.6
6.09
Explicit-ASR
91.8
5.09
94.2
9.85
Explicit-NS
91.8
7.65
92.9
10.5
As shown in increasing the diversity of labels during the thinking process (listening state) hurts performance. We assume that the performance differences different thinking labels may be partially attributed to the full-duplex finetuning. In LLM, the model doesn't 't need generate explicit responses upon receiving user input. However, for full-duplex speech interaction LLMs, the model must consistently generate responses to manage turn-taking in natural dialogues, alternating between speaking and listening states, During full-duplex fine-tuning fine-tuning, the model must learn two critical skills: first, the ability to model state transitions: second, the ability distinguish between the operational modes of the speaking and listening states. When using the simple Explicit thinking strategy. the listening state (where the model generates only a single special token) easily distinguishable. This allows the model to focus on learning state transitions and optimizing the speaking state's operation mode, leading to high turn-taking precision and improved content quality. However, as the diversity of thinking labels increases, the operation modes of the two states become less distinct, which sometimes results undesirable messy generation during the speaking state. Meanwhile, while oracle "thinking" ASR content can enhance speech generation, incorrect "thinking" content during inference may introduce biases and hurt the quality of generated speech due to the error propagation in response generation.
D More Results on DPO Post-Training for Full-duplex Modeling
As shown in Table 10 II 12 the intriguing observation, consistent across different training batch sizes, is that during DPO training, the model initially becomes overly cautious- raanslrrrrryyyaa getting interrupted. However, as training progresses, its performance steadily improves and eventually surpasses that of the SFT-trained model.
16

---

Table 10: The performance SALMONN omni on peneextual-eeendeen and inneextualinndeeeddent settings through the DPO training stage with batch size set to 128. (detailed version)
Contextual-Indeeendee
Contextual-Dependeen
Overall Fl score
Stage
Precision
Recall
Fl score
Precision
Recall
Fl score
SFT (Stage 2)
0.68
0.93
0.79
0.88
0.99
0.93
0.86
DPO 10 steps
0.88
0.14
0.24
0.97
0.38
0.55
0,41
DPO 20 steps
0.78
0.75
0.77
0.91
0.96
0.94
0.85
DPO 30 steps
0.83
0.88
0.85
0.90
0.98
0,93
0.89
DPO 40 steps
0.81
0.84
0.82
0.90
0.95
0.93
0.88
Table 11: The performance of SALMONN- on contextual-dependene and contextual-independeer settings through the DPO training stage with batch size set to 256. (detailed version)
Contextual-Indeeendeat
Contextual-Dependee
Overall F1 score
Stage
Precision
Recall
F1 score
Precision
Recall
F1 score
SFT (Stage 2)
0.68
0.93
0.79
0.88
0.99
0.93
0.86
DPO 10 steps
0.88
0.07
0.13
0.95
0.21
0.34
0.24
DPO 20 steps
0.92
0.45
0,60
0.98
0.78
0.87
0.75
DPO 30 steps
0.84
0.88
0.86
0.92
0.97
0.95
0.90
DPO 40 steps
0.84
0.92
0.88
0.89
0.98
0.93
0.90
E Speech Quality Assessment
Detailed UTMOS on all the datasets of each model is reported in Table 13
F
Emotional Speech Generation
We also explore equipping SALMONN- with emotional speech generation support more natural and human-like speech interactions. To achieve this, the LLM backbone is prompted to produce an emotion indicator prior to generating its verbal response, and the speech synthesizer operates in instruction TTS style. Emotional speech samples are synthesized using commercial TTS model provided by Volcano Engine. The emotion intensity of the emotion enhanced SALMONN- omni, along with baseline models, is evaluated 2024-12-17, with results summarized Table [4] Findings show that SALMONN -omi achieves the highest emotion intensity among all evaluated systems. However, its emotional expression remains inconsistent, occasionally producing responses with inappropriate or mismatched emotions.
in-house dataset using gpt-4o-audio- -at---aad------iive---
G Limitation
While SALMONN-omni is high-performance full-duplex speech interaction L.M,, it still faces several limitations. First, although it effectively predicts turn-taking and can distinguish between user barge-ins and backchanneling. it rarely generates backchanneling responses during user speech, primarily due to limitations in the training data. Enhancing this capability is essential for achieving more natural, human-like conversational behavior. Second, SALMONN-omni's training sequence is limited to three minutes due to memory constraints. As result, its performance noticeably degrades when interaction exceeds this duration. Future work involves enabling long-term memory for full- dullex speech interaction LLM.
H Impact Statement
This paper aims explore robust full-duplex speech interaction framework, advancing the frontier of real-time, fully interactive conversational AI. We believe that, alongside enhancing the performance of conversational AI, essential prioritize Al safety. This includes ensuring that all generated content is free from harm, discrimination, and bias, as well developing detection models to identify
17

---

Table 12:
The performance SALMONN omni on contextual -eepettdelddeeeeenttee and contextual-indeenddeee settings through the DPO training stage with batch size set to 512. (detailed version)
Kontextual-Independee
Contextual-Dependent
Overall Fl score
Stage
Precision
Recall
FI score
Precision
Recall
Fl score
SFT (Stage 2)
0.68
0.93
0.79
0.88
0.99
0.93
0.86
DPO 10 steps
1.0
0.01
0.02
1.0
0.01
0,02
0,02
DPO 20 steps
0.88
0.29
0.44
0.98
0.96
0.55
0.58
DPO 30 steps
0.82
0.86
0.84
0.93
0.97
0.95
0.89
DPO 40 steps
0.82
0.88
0.85
0.91
0.95
0.93
0.89
Table 13: UTMOS of SALMONN-omii and other speech LLMs on spoken QA task.
Model
Llama Q.
Web Q.
TriviaQA
AlpacaEval
Predicted turn-taking
Moshi
3.245
3.257
3.017
3.193
Freeze-Omni 20
4.288
4.273
4.293
4.255
SALMONN-omni
4.256
4.250
4.272
4.243
Oracle turn-taking
GLM-4-
15
4.180
4.107
4.145
3.890
Qwen2.5- Omni 22
4.140
4.124
4.127
3.950
VITA-1.5
[19
4.042
4.091
4.104
3.852
MiniCPM-o
26
4.171
4.152
4.155
4.110
Kimi-Audio
21
2.882
2.713
2.793
2.327
Baichuan-Audio
27
4.261
4.216
4,292
3.959
Freeze-Omni 20
4.300
4.281
4,278
4.245
SALMONN-OR -omii
4.262
4.263
4.279
4.240
Al-generated content. It is also crucial that all users made aware when they interacting with conversational Al chatbot.
I Case Studies
USER
(Which Olympic sport also known "table tennis"?)
(You're thinking of table tennis, ch? Well, the Olympic sport you're referring to is actually called table tennis, but it's also commonly known ping-pong! It's super popular sport, the Olympics have been featuring it since 1988. The name "ping-pong" actually comes from the sound the ball makes when it bounces off the table. Pretty cool, right? So, there you have it table tennis, ping-pong, is Olympic sport you were thinking of!)
SALMONN-omai
Figure Spoken QA: -ALMONN-omní can handle turn-taking in spoken question answering scenarios with "thinking" mechanism.
18

---

Table 14: Emotion intensity of speech interaction LLMs.
Model
Emotion intensity
GLM-4-Voice
3.30
Qwen2.5-Ong
3,27
VITA-1.5
3.03
miniCPM-o
3.16
Kimi-Audio
3.39
Baichuan-Audio
3.14
Freeze-Ommt
3.00
SALMONN-omni
3.49
USER
want to get better networking at work.)
(That's great goal! Start by being approachable and friendly. Show
genuine interest others ask open-ended questions to spark conversations. Attend company events and participate in team-butt mun-building activities build connections. Be proactive in offering help support to your colleagues. Remember, networking is about building relationships, so focus on being a supportive and helpful person. Lastly, follow up with contacts keep the connection alive.)
SALMONN-omni
Figure 6: Open-domain dialogue: SALMONN-omni can handle turn-taking in spoken question answering scenarios with "thinking" mechanism.
USER
(What's the the sea's ccosystem like?)
(It's in pretty bad shape. The sea's 's biodiversity is declining, and there concerns about the long-term sustainability fishery. There are efforts to ....
SALMONN-omni
USER
(What's causing the decline the sca's biodiversity?)
(Stop)
-ALMONN-ommi
Figure Context-dependert barge-in: When the user quick to buzz in with the response, SALMONN- ommi can stop generating speech.
19

---

USER
(Could you tell me something about the Tikal City?)
(Sure! Tikal was major city in the Maya region during the Classic period.)
SALMONN-omni
USER
(I see.)
(Continue)
-aju Iou |
SALMONN-
(And is abandoned now.)
Figure 8: Distinguishing backchanneling: SALMONN ommi can continue speaking.
When the user interjects with backchannel cue,
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In both the abstract and the introduction section, have summarized the scope and the claims of this paper accurately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: A limitation section is provided in Appendix G which describes the limitation of this work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [NA]
Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Detailed experimental setups provided in Section
San gand Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We'll open source the code once accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipssccc/pill wttpcccpppppppppeeeeeeeeeeeeeeee httpa///guddeee/eeeeabiiissssss /Cc////oddiissssseeeeeee ssiiiiiiccyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Section provides general descriptions of the experimental setup, further details on training data is provided in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
This item not necessary in our experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information about the computational resources used in the experi- ment in Section 4 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our research adheres to the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential impact of our work in Appendix H
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We believe our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have used assets in accordance with the license and terms.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new assets introduced in the paper have well-documented records.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work involves crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work involves crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
26

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use LLM to generate training data oral conversation results for AlpacaEval (GPT-4.1-2025-04-- which is stated clearly in Section 4 and Appendix A
and evaluate the
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hattap///aauuppscccc///oferrrrrp for what should or should not be described.
27


Reviews:
[{"review_id": "e8AeymoJAT", "reviewer": "Reviewer_sYnD", "summary": "This paper introduces SALMONN-omni, a system designed to enable full-duplex spoken dialogues without relying on neural codecs, unlike several prior approaches. The method introduces a “thinking” mechanism to alternate between speaking and listening modes within a single sequence; however, the explanation of this mechanism is vague, and the core idea remains difficult to grasp. Additionally, it leverages DPO to improve naturalness in barge-ins and backchanneling, resulting in a three-stage training pipeline. While the model demonstrates state-of-the-art results on several public benchmarks compared to existing open-weight systems, the comparisons are not entirely straightforward. The training data, architecture, and training methodology differ significantly from other models, making it unclear which factors are responsible for the performance gains.", "questions": "- Unclear justification for avoiding neural codecs: The paper does not clearly explain the limitations of neural codecs or provide experimental validation for avoiding them. Are there any ablation studies or comparisons demonstrating the effectiveness of the proposed method without a neural codec?\n- Sound quality evaluation is lacking: One of the main advantages of neural codecs is their ability to generate high-quality audio. Therefore, the proposed method without neural codes should include sufficient discussion or analysis of the sound quality produced by the proposed method. Are there any objective or subjective evaluations to support its adequacy? A comparison would strengthen this point.\n- Ambiguity in the \"thinking\" mechanism: The core concept of the proposed “thinking” strategy is difficult to understand. Notations such as <bos> and <and> are introduced without adequate explanation. More detailed clarification, especially in relation to Figure 2, is necessary to make this part accessible to readers.\n- Reproducibility concerns: Although the paper appears to rely mostly on public datasets, it also uses TTS-generated spoken text. If the TTS resources or generated data are not publicly available, full reproduction of the experiments may be challenging. Please clarify the availability of these resources.\n- Lack of structured dataset summary: Section 4.3 could be improved by presenting the dataset information in a structured format, such as a table. This would help readers understand the experimental setting more easily.\n- Insufficient baseline details: The paper lacks important details regarding the setup of each baseline method. Without this, it is difficult to assess the fairness and significance of the comparisons. Please provide more comprehensive descriptions to support the experimental findings.", "limitations": "While the paper touches on certain technical limitations, it lacks discussion of the broader societal impacts. For instance, techniques of this nature often raise concerns about hallucinations, yet this issue is not addressed. Additionally, potential biases—such as skewed conversational topics or limited speaker demographic coverage—should be considered and discussed more explicitly.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Proposes a full-duplex dialogue system that avoids the use of neural codecs, enabling a simpler and unified LLM-based modeling approach.\n- Demonstrates state-of-the-art performance on several benchmarks.\n\nWeaknesses\n- The explanation of the core “thinking” strategy is vague and difficult to follow, making the overall method hard to understand.\n- The training pipeline is complex, involving a three-stage process including DPO which may hinder reproducibility and practical adoption.", "was_revised": true, "final_justification": "I still have concerns regarding the clarity of the “thinking” strategy, which I believe requires major revision, and I cannot fully assess it without carefully reviewing the revised version. However, regarding my other concerns about the complex training strategies, the authors’ point that similar complexity exists in other approaches has changed my perspective. Therefore, I have adjusted my recommendation from 3: Borderline Reject to 4: Borderline Accept.", "rebuttal": "We sincerely thank the reviewer for the effort. We hope the following responses will help address your concerns.\n- **Q1: The explanation of the core “thinking” strategy is vague ...**\n\nThank you for pointing out these issues, and we will improve the presentation in the final version. Here, we would like to clarify the motivation behind our \"thinking\" strategy, which is designed to handle the dynamics of natural conversations in full-duplex speech dialogue systems.\n\nFull-duplex models alternate between two states: listening, where the model remains silent but attentive, and speaking. In the speaking state, the LLM generates speech while continuing to listen, which is required to manage natural conversation behaviours such as barge-ins. In the listening state, the model still needs to perform internal computation to monitor ambient sounds and detect potential incoming speech, even though it does not produce spoken output. We refer to this process as thinking, drawing an analogy to human behaviour.\n\nIn SALMONN-omni, we explored different thinking strategies, primarily distinguishing them based on whether the model feeds the thinking content back into the input, and present the results in Table 2. Implicit thinking strategy requires the model to make decisions between the \\<listen\\> and \\<speak\\> state tokens in each time block and these tokens are not fed back into the model’s input. However, the model trained with explicit thinking strategy feeds all the generated tokens (both \\<think\\> and its response tokens) into its input and uses \\<shift\\> to mark the state switching between listening and speaking. Experiments show that explicit thinking consistently outperforms implicit thinking, likely due to better alignment with the autoregressive nature of LLMs.\nWe also experimented with different forms of thinking content, as detailed in Appendix C. Among them, the single \\<think\\> token outperformed alternatives such as ASR transcriptions and random text, suggesting it provides the most effective signal for this internal state.\n\nLastly, we were unsure what the notation \"[object Object]\" in your comment was intended to reference. We would greatly appreciate it if the reviewer could clarify this point.\n\n- **Q2: The training pipeline is complex, ...**\n\nWe will open-source our training data and source code to facilitate reproducibility. Our three-step training procedure is straightforward and closely follows the standard paradigm of (multimodal) LLM training. Each stage is essential for achieving high-performance speech dialogue interaction: 1) the first stage focuses on speech understanding; 2) the second on speech generation; 3) the final DPO stage enhances full-duplex capabilities to enable more natural conversational experiences.\n\n- **Q3: Unclear justification for avoiding neural codecs ...**\n\nInjecting codecs into the LLM is certainly an alternative approach. We are not drawing any definitive conclusions about which paradigm future duplex models must adopt and merely exploring a framework that has not been investigated in prior work. However, major challenges in integrating autoregressive LLMs with neural codecs include the following:\n\n1) Mismatch in unit durations: Injecting speech codecs often degrades the language processing performance of LLMs [1].  This is believed to be largely attributed to the mismatch between the variable-length word pieces used by LLMs and the short, fixed-duration units typically produced by speech codecs.\n\n2) Content inconsistency: In neural codec-based approaches [2], speech tokens are often generated sequentially after text tokens. As a result, the content of the speech response can noticeably differ from its corresponding text response. This issue (commonly referred to as the \"modality gap\" problem), leads users to perceive that the spoken output lacks the intelligence or coherence of the text response.\n\nFor our approach, which does not require injecting the codec into the LLM, the above two issues are significantly less problematic. As shown in the following table, we compared the performance of our model using both text and audio inputs with the original performance of LLaMA-3. It turns out that the performance degradation of the LLM is not as significant as Moshi and our model exhibits comparable performance when using text and audio inputs.\n\n||Llama Q.|Web Q.|Trivia Q.|AlpacaEval|\n|---|---|---|---|---|\n|Llama-3|0.85|0.56|0.73|3.83|\n|Ours-text|0.84|0.52|0.68|3.82|\n|Ours-wav|0.80|0.51|0.66|4.05|\n\nMoreover, we found that the \"modality gap\" problem becomes severer for codec-based models in multi-turn dialogue scenarios. We followed the multi-turn dialogue setup used in SLAM-Omni to evaluate the multi-turn conversational capabilities of different models. The results (numbers in parentheses indicate S2S results) show that while GLM-4-Voice demonstrates stable performance in S2T, its S2S performance is significantly inferior. We observed that the generated audio responses often begin with phrases like \"You want a friend? You got one…\", which are not present in the corresponding textual responses. \n\n|N Turn|2|3|4|5|\n|---|---|---|---|---|\n|GLM-4-Voice|4.02 (3.52)|4.66 (3.98)|4.52 (3.84)|4.36 (3.62)|\n|Ours|4.24 (4.04)|4.48 (4.18)|4.44 (4.16)|4.34 (4.02)|\n\n[1] Jiatong Shi, Chunlei Zhang, Jinchuan Tian, et al. “Balancing Speech Understanding and Generation Using Continual Pre-training for Codec-based Speech LLM”. In: arXiv preprint arXiv:2502.16897 (2025).\n\n[2] Aohan Zeng, Zhengxiao Du, Mingdao Liu, et al. “GLM-4-Voice: Towards intelligent and human-like end-to-end spoken chatbot”. In: arXiv preprint arXiv:2412.02612 (2024).\n\n- **Q4: Sound quality evaluation is lacking ...**\n\nWe compared the UTMOS scores of different models across various datasets, and the results indicate that the audio generated by our model is of high quality.\n\n|                  | Llama Q.| Web Q. | Trivia Q. | AlpacaEval |\n|------------------|-------|-------|--------|--------|\n| **Predicted turn-taking** |       |       |        |        |\n| moshi            | 3.245 | 3.257 | 3.017  | 3.193  |\n| freeze-omni      | 4.288 | 4.273 | 4.293  | 4.255  |\n| SALMONN-omni     | 4.256 | 4.250 | 4.272  | 4.243  |\n| **Oracle turn-taking**    |       |       |        |        |\n| GLM-4-Voice      | 4.180 | 4.107 | 4.145  | 3.890  |\n| Qwen2.5-Omni     | 4.140 | 4.124 | 4.127  | 3.950  |\n| VITA-1.5         | 4.042 | 4.091 | 4.104  | 3.852  |\n| miniCPM-o        | 4.171 | 4.152 | 4.155  | 4.110  |\n| Kimi-Audio       | 2.882 | 2.713 | 2.793  | 2.327  |\n| Baichuan-Audio   | 4.261 | 4.216 | 4.292  | 3.959  |\n| Freeze-Omni      | 4.300 | 4.281 | 4.278  | 4.245  |\n| SALMONN-omni     | 4.262 | 4.263 | 4.279  | 4.240  |\n\n- **Q5: Reproducibility concerns ...**\n\nWe promise to open-source all training data and source code used in our paper to facilitate reproducibility. For our self-generated dialogue data, all models involved in the pipeline are publicly available. The questions and topics are selected from existing text datasets such as WebQuestions and Natural Questions. Text responses or dialogues are generated using Llama-3-8B-Instruct, while both the questions and corresponding speech audio are synthesized using CosyVoice2-0.5B. Both models are open-source.\n\n- **Q6: Lack of structured dataset summary ...**\n\nThanks for your advice. Due to the page limit, we put the dataset details in Appendix A, shown in Table 7. We will consider rearranging our presentation to enable better understanding.\n\n- **Q7: Insufficient baseline details ...**\n\nWe will add more details about baselines in the Appendix in the final version. In our paper, all baseline models are opensourced models and we have correctly referenced all baseline papers. For all baseline models, we test them following the official recipes with preferenced prompts and default decoding options to ensure all models demonstrate the best performance of their own. And we believe it can support a relatively fair comparison.\n\n- **Q8: Limitations**\n\nWe thank the reviewer for highlighting these important limitations related to societal impact, and we will incorporate them into the limitations section of the paper. These are fundamental non-technical challenges shared by all conversational AI systems and will require sustained, long-term efforts from both the research community and society at large to address."}, {"review_id": "CmhP65Mc27", "reviewer": "Reviewer_ExHa", "summary": "This paper introduces SALMONN-omni, a standalone, full-duplex speech Large Language Model that innovatively operates without injecting audio codecs into its vocabulary. It achieves the ability to listen and speak simultaneously by seamlessly integrating a streaming speech encoder, a unified LLM backbone, and a streaming speech synthesizer. The model adopts a novel \"explicit thinking\" strategy, training the LLM to generate special control tokens to autonomously decide when to start speaking and when to remain listening, thereby managing conversational state transitions. Experimental results show that SALMONN-omni excels at handling complex conversational dynamics such as turn-taking, backchanneling, and context-dependent barge-ins, with its performance significantly surpassing existing open-source full-duplex models.", "questions": "- The model generates a large number of `<think>` tokens due to the mismatch in sequence length between text and speech, particularly when it has finished generating a text response but is still in the \"speaking\" state. A key question is whether this process of generating numerous non-substantive tokens could affect the model's core text-based abilities, especially in the multi-turn conversation scenarios.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Pros\n\n- The proposed model handles both speech understanding and generation outside of the LLM, which largely preserves the LLM's inherent capabilities. Results on several spoken QA benchmarks show that it outperforms many existing models, including strong models like Kimi-Audio and Qwen2.5-Omni.\n  \n- The paper is well-written and easy to understand.\n  \n\n## Cons\n\n- The model needs to synthesize its output tokens into speech and then re-encode that speech through its own encoder as an \"assistant stream\" input. Compared to models like Moshi, which only need to perform auto-regression on codec tokens , this synthesis and re-encoding loop adds inference overhead and impacts latency.\n  \n- For the full-duplex evaluation, the authors focused on metrics like success rate and F1 scores but did not report measurements of actual inference latency. A more thorough evaluation should assess the actual time of the gap or overlap in different full-duplex scenarios (turn-taking, interruption, backchanneling).\n  \n- Certain design choices, such as the use of a Mamba-based speech encoder, are not accompanied by sufficient experimental analysis to justify their necessity—for example, by detailing its impact on response quality and inference speed.\n  \n- The current evaluation is limited to single-turn interactions. I believe that an evaluation in multi-turn scenarios is also necessary for a more comprehensive assessment.", "was_revised": true, "final_justification": "The authors report multi-turn and barge-in evaluations in the rebuttal, which addresses the original review’s core concerns.", "rebuttal": "Thank you for your efforts—your suggestions are extremely valuable for improving our paper. We hope that the additional experiments we have provided will help address your concerns.\n\n- **Q1: The model needs to synthesize its output tokens into speech and then re-encode that speech through its own encoder as an \"assistant stream\" input. Compared to models like Moshi, which only need to perform auto-regression on codec tokens , this synthesis and re-encoding loop adds inference overhead and impacts latency.**\n\nThank you for the insightful comment. We would like to clarify that the Environment Stream and Assistant Stream are processed in parallel which does not necessarily increase the latency. Furthermore, during real-world deployment, our approach does not require the Environment Stream and Assistant Stream to be fully aligned, and thus does not incur additional latency. While a potential concern is the overhead introduced by interleaving all embeddings into a single sequence, we note that this design choice allows the LLM backbone to model the flexible interaction between the two streams. In contrast, Moshi adopts a RQ-Transformer component, which compresses information into a more compact representation. Our interleaving approach trades off some efficiency for improved alignment and interaction modeling, which we found beneficial in our full-duplex dialogue setting. In real-world deployment, the latency introduced by this design is also acceptable. Processing one time block (80 ms) of input with the encoder takes approximately 55 ms, while the LLM requires around 70 ms to decode and generate the response for one time block.\n\n- **Q2: For the full-duplex evaluation, the authors focused on metrics like success rate and F1 scores but did not report measurements of actual inference latency. A more thorough evaluation should assess the actual time of the gap or overlap in different full-duplex scenarios (turn-taking, interruption, backchanneling).**\n\nThank you for your suggestion. Here, we conducted a detailed latency evaluation and analyzed the turn-taking statistics of the generated dialogues. We deployed our model in a real-world setting and conducted human-model interactions through live conversations to evaluate its performance across three main scenarios: multi-turn dialogue, contextual-independent barge-in and contextual-dependent barge-in. We collected 20 dialogue sessions for each scenario. In the multi-turn dialogue setting, conversations averaged 5 turns with a mean duration of 75 seconds. In the barge-in scenarios, the dialogues averaged 3 turns.\n\n|               | IPU     | Pause   | Gap     |\n|---------------|---------|---------|---------|\n| Ours          | 61.89s  | 5.49s   | 6.50s   |\n| Ground Truth  | 63.3s   | 4.68s   | 5.01s   |\n\nThe turn-taking statistics in a multi-turn dialogue scenario are shown in the table (normalized to 5-turn dialogues), and our model exhibits an average response latency of **405 ms**, primarily due to the current design, which requires waiting for four time blocks to provide sufficient context for the speech synthesizer. We believe this latency can be further reduced by optimizing the synthesizer. In contextual-independent and contextual-dependent barge-in scenarios, the response latency when interrupted is approximately **128 ms** and **144 ms**, respectively, indicating that our model can quickly react to user interruptions.\n\n- **Q3: Certain design choices, such as the use of a Mamba-based speech encoder, are not accompanied by sufficient experimental analysis to justify their necessity—for example, by detailing its impact on response quality and inference speed.**\n\nThis represents an alternative design choice rather than the central focus of our work. Any architecture capable of handling streaming speech processing—such as CNNs or causal transformers—can certainly be employed. We opt for Mamba due to its ability to perform streaming sequence modeling with comparatively lower computational overhead than causal transformers, while also demonstrating satisfactory performance in our preliminary evaluations.\n\n- **Q4: The current evaluation is limited to single-turn interactions. I believe that an evaluation in multi-turn scenarios is also necessary for a more comprehensive assessment.**\n\nWe evaluated the model’s multi-turn dialogue capabilities in both simulated and real-world scenarios. In simulated settings, we follow the evaluation protocol of SLAM-omni, assessing the model’s final-turn output using the ChatGPT Score  [1]. The evaluation covered dialogues ranging from 2 to 5 turns. Test data were collected using the same method employed for constructing the multi-turn dialogue training data, with 50 samples gathered for each dialogue length, resulting in a total of 200 test samples.\n\n|   N Turn     |  2        | 3        | 4        | 5        |\n|----------------|--------------|--------------|--------------|--------------|\n| GLM-4-Voice     | 4.02 (3.52)  | 4.66 (3.98)  | 4.52 (3.84)  | 4.36 (3.62)  |\n| MiniCPM-o       | 4.38 (4.20)   | 4.40 (4.16)   | 4.20 (3.96)   | 4.44 (4.16)  |\n| Qwen2.5-omni    | 4.28 (4.02)  | 4.44 (4.20)   | 4.30 (4.08)   | 4.10 (3.92)   |\n| Freeze-Omni     | 3.58 (2.94)        | 4.00 (3.12)         | 3.44 (2.58)        | 3.34 (2.46)        |\n| ours            | 4.24 (4.04)  | 4.48 (4.18)  | 4.44 (4.16)  | 4.34 (4.02)  |\n\nWe compared our model with several full-duplex and half-duplex models. The results (numbers in parentheses are S2S performance) show that our model maintains stable performance across dialogues of varying lengths. Moreover, it significantly outperforms the open-source full-duplex model Freeze-Omni and achieves performance comparable to other half-duplex models, consistent with the findings from other evaluations presented in our paper.\n\nAs for real-world settings, we transcribed the dialogues into text and evaluated aspects such as naturalness and fluency using the ChatGPT Score. **Naturalness** evaluates how human-like, smooth, and contextually appropriate the responses are; while **Fluency** evaluates the grammatical correctness, flow, and ease of understanding of the language used. All aspects were rated on a scale from 1 to 5, with higher scores indicating better performance. Comparisons with the ground truth further demonstrate the multi-turn dialogue capability of our model in real-world conversational settings. A slight discrepancy in fluency compared to the ground truth can be attributed to unavoidable errors introduced by the ASR system, as well as the tendency of our model’s responses to be more conversational in nature and less constrained by formal grammatical conventions.\n\n|               | naturalness | fluency | overall |\n|---------------|-------------|---------|---------|\n| Ours          | 3.80        | 4.00     | 3.65    |\n| Ground Truth  | 4.14        | 4.62    | 4.18    |\n\n[1] Wenxi Chen, Ziyang Ma, Ruiqi Yan, et al. “SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training”. In Proc. ACL, Vienna, 2025.\n\n- **Q5: The model generates a large number of \\<think\\> tokens due to the mismatch in sequence length between text and speech, particularly when it has finished generating a text response but is still in the \"speaking\" state. A key question is whether this process of generating numerous non-substantive tokens could affect the model's core text-based abilities, especially in the multi-turn conversation scenarios.**\n\nThank you for your question. All speech dialogue models with streaming generation capabilities must address the challenge of sequence length mismatch. A common solution is to pad the text sequence to match the length of the speech sequence; accordingly, models such as Moshi generate padding tokens in the text stream output. We further explored various options for this \"thinking content\" (the padding portion of the text sequence), including simple pad tokens, question ASR transcriptions, and random text, as detailed in Appendix C. Our results show that generating a dedicated \\<think\\> token as padding yields the best performance. We hypothesize that using a special token (e.g., \\<think\\>) when the model is not producing a meaningful answer makes state transitions more distinct and easier for the model to learn. We also compared our model’s performance, using both text (\"Ours-text\") and audio inputs (\"Ours-wav\"), to the original LLaMA-3. The performance degradation for the LLM was minimal. Furthermore, multi-turn dialogue evaluations confirm that this design does not suffer from noticeable performance loss as the number of dialogue turns increases.\n\n||Llama Q.|Web Q.|Trivia Q.|AlpacaEval|\n|---|---|---|---|---|\n|Llama-3|0.85|0.56|0.73|3.83|\n|Ours-text|0.84|0.52|0.68|3.82|\n|Ours-wav|0.80|0.51|0.66|4.05|"}, {"review_id": "d5AAbnN3lM", "reviewer": "Reviewer_2yL6", "summary": "This paper introduces SALMONN-omni, a novel full-duplex speech-language model designed for real-time, naturalistic spoken dialogue. Unlike prior systems that rely on codec token injection (e.g., Moshi) or modular dual-LLM architectures (e.g., Freeze-Omni), SALMONN-omni is a standalone codec-free solution that integrates a streaming speech encoder, a single LLM, and a speech synthesizer in an end-to-end trainable framework. The model handles simultaneous speech input/output and autonomously manages dialogue transitions through a new “thinking” strategy involving explicit state-transition tokens (e.g., <think>, <shift>).\n\nThe authors implement a three-stage training process: (1) encoder-LLM alignment, (2) speech synthesis integration, and (3) reinforcement learning (via DPO) for improved dialogue dynamics. Experimental results on speech QA, dialogue, and turn-taking benchmarks show a significant performance gain; up to 35.9% relative improvement over previous open-source full-duplex models. SALMONN-omni also achieves competitive results against strong half-duplex models trained on much larger corpora. The model further demonstrates robustness in backchannel detection, echo cancellation, and barge-in handling.", "questions": "1. Have you evaluated model behavior with spontaneous, noisy user inputs beyond TTS-generated samples?\n\n2. Can you provide latency measurements under real-time deployment (e.g., wall-clock delay per turn)?\n\n3. How would the model scale to multilingual or code-switched dialogue?\n\n4. Could you share insights on where the model fails, especially in ambiguous barge-in situations?", "limitations": "- Evaluation uses synthetic speech inputs, which may not capture conversational noise, hesitation, or accent variation.\n\n- While turn-taking is well-evaluated, the paper lacks a detailed ablation or error analysis that would provide insight into specific failure cases.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n- **Novel architecture:** SALMONN-omni is the first open-source standalone full-duplex speech LLM without audio codec injection.\n\n- **Well-motivated design:** The “thinking” strategy and stream interleaving offer a clean integration of speech and dialogue dynamics.\n\n- **Competitive with large-scale models:** Despite being trained on far less data than models like Kimi-Audio, SALMONN-omni delivers strong results across QA and dialogue tasks.\n\n- **RL-based modeling of conversation flow:** Application of DPO to barge-in and turn-taking tasks is novel and well-justified.\n\n- **Comprehensive evaluation:** Model is tested across oracle and predicted turn-taking, S2T and S2S, and real-time conversational phenomena.\n\n**Weaknesses:**\n\n**Lack of human evaluation:** The paper reports GPTScore and ASR-based metrics but no human judgments for naturalness, fluency, or appropriateness of the dialogue.\n\n**Synthetic evaluation conditions:** Many input utterances are generated via TTS and text prompts, which might not reflect real-world speech variability.\n\n**No analysis of failure cases:** For example, in barge-in detection or speech overlap, it’s unclear where the model fails and how errors propagate.", "was_revised": true, "final_justification": "The work is technically solid and all my concerns have been appropriately addressed.", "rebuttal": "We sincerely thank the reviewer for the recognition of our work. To ensure clarity, we have summarized the reviewer’s comments into the following key points. We hope this presentation addresses your concerns. If you feel that any aspect has not been adequately captured, we would greatly welcome further discussion.\n\n- **Q1: Lack of human evaluation: The paper reports GPTScore and ASR-based metrics but no human judgments for naturalness, fluency, or appropriateness of the dialogue.**\n\nThank you for your invaluable suggestions. We agree that incorporating human evaluation is highly important. However, due to time constraints, it is challenging to organize a rigorous human assessment. As a result, we have temporarily relied on the use of ChatGPT Score to evaluate the model’s conversational capabilities in real-world scenarios. In the following real-world scenario evaluations, we provide a detailed explanation of our evaluation methodology.\n\n- **Q2: Synthetic evaluation conditions: Many input utterances are generated via TTS and text prompts, which might not reflect real-world speech variability.**\n\nWe sincerely appreciate the reviewer’s suggestion. In response, we deployed our model and evaluated its performance in real-world human-machine interaction scenarios. The experimental results are presented below.\n\n- **Q3: Latency measurements under real-time deployment**\n\nIn the description of the real-time deployment experiments that follow, we evaluate the model’s turn-taking statistics, response latency to users, and latency in reacting to user interruptions.\n\n- **Q4: Insights on failure cases**\n\nHere we would like to share some humble insights on some failure situations of turn-taking and barge-ins. \n\n1) In turn-taking scenarios, when the speech question sample is very quick and short, the model tends to fail to determine when to start speaking. As shown in Table 4, the AlpacaEval turn-taking success rate indicates that this phenomenon can be observed across many models. We suppose that short questions require the model to switch between listening state and speaking state quite fast, which is hard to learn.\n\n2) In real-world testing, it is best to maintain a quiet environment, as our model was not trained with any data augmentation techniques involving additive noise. When the environment is noisy, the model’s responses are often unreasonable. In future work, we plan to further explore strategies to enhance the model’s robustness in noisy conditions.\n\n3) Regarding the sentences used for contextual-independent barge-in, for the sake of implementation simplicity, we limited the training set to only 10 interruption phrases. While the model demonstrated a certain degree of generalization during testing, some sentences still failed to successfully interrupt the model—particularly when the interruption was very short or spoken at a low volume. In such cases, the model may confuse the input with backchanneling.\n\n- **Q5: How would the model scale to multilingual or code-switched dialogue?**\n\nThis is indeed an interesting and worthwhile direction to explore. However, we acknowledge that at the current stage, SALMONN-Omni focuses solely on English speech input and output due to the data constraint. We believe it's valuable to extend SALMONN-omni to handle multilingual situations by adding multilingual or code-switched data. However, it will need careful exploration of how to balance multiple languages.\n\n- **Real-time deployment user-machine interaction experiments**\n\nWe deployed our model in a real-world environment and assessed its performance through live human-model interactions across three primary scenarios: multi-turn dialogue, context-independent barge-in, and context-dependent barge-in. For each scenario, we collected 20 dialogue sessions due to time constrain. In the multi-turn dialogue scenario, conversations consisted of an average of 5 turns, with a mean duration of 75 seconds. In both barge-in scenarios, the dialogues had an average length of 3 turns.\n\n1) *Turn-taking statistics and latency analysis*\n\n|               | IPU     | Pause   | Gap     |\n|---------------|---------|---------|---------|\n| Ours          | 61.89s  | 5.49s   | 6.50s   |\n| Ground Truth  | 63.3s   | 4.68s   | 5.01s   |\n\nThe results here (normalized to 5-turn dialogues) illustrate the turn-taking statistics of SALMONN-omni compared with our multi-turn training data (noted as Ground Truth). Furthermore, our model demonstrates an average response latency of **405 ms**, largely attributed to its current architecture, which necessitates a wait for four time blocks to accumulate adequate context for the speech synthesizer. We anticipate that this latency can be further minimized through targeted optimization of the synthesizer. In the context-independent and context-dependent barge-in scenarios, the response latency upon interruption is approximately **128 ms** and **144 ms**, respectively, suggesting that the model is capable of responding promptly to user interruptions.\n\n2) *Quality assessment of the generated content*\n\nAlthough time constraints prevented us from conducting a rigorous human evaluation, we adopted the reviewer’s suggestion and assessed the quality of the model’s responses in real-world conversations using ChatGPT Score from three perspectives: naturalness, fluency, and overall quality. \n\n**Naturalness** evaluates how human-like, smooth, and contextually appropriate the responses are; while **Fluency** evaluates the grammatical correctness, flow, and ease of understanding of the language used. All aspects were rated on a scale from 1 to 5, with higher scores indicating better performance.\n\n|               | naturalness | fluency | overall |\n|---------------|-------------|---------|---------|\n| Ours          | 3.80        | 4.00     | 3.65    |\n| Ground Truth  | 4.14        | 4.62    | 4.18    |\n\nThe results indicate that our model is still able to produce reasonable responses during real-world testing. However, a slight gap in fluency compared to the ground truth is observed, which can be attributed to inevitable errors introduced during ASR, as well as the responses generated by our model tend to be more conversational and may not strictly adhere to formal grammatical rules."}, {"review_id": "hFkSD2VXFU", "reviewer": "Reviewer_tsTQ", "summary": "This paper proposes SALMONN-omni, a full-duplex speech LLM that achieves speech-text modality fusion and natural conversational interaction without compromising LM capabilities. The framework integrates a streaming speech encoder and synthesizer into the LLM backbone, enabling simultaneous processing of environmental audio and assistant speech. An explicit \"thinking\" strategy and subsequent DPO optimization enhance duplex interaction abilities. Experiments on spoken QA and open-domain dialogue validate the model’s duplex capabilities.", "questions": "- Q1 Compared to the backbone LLM, does the final model exhibit degradation in reasoning/understanding due to added modules (LoRA, encoder adapter, etc.)\n- Q2 For the streaming speech encoder: Why choose Mamba blocks? \n- Q3 Lines 205–206 (Stage 3): Could you clarify the conclusion where the model \"exhibits a clear bias toward being interrupted and shows insufficient understanding of dialogue semantics\"?\n- Q4 For DPO, it's better to detail the data scale, construction process, and reward design criteria for better understanding.\n- Q5 Line 232–233: For synthesized conversation data, how were speaker attributes (timbre/accent/emotion) selected in the generation process?\n- Q6 This question does not affect scoring. Existing works in the field of speech generation, especially in speech dialogue, often employ different pre-training models and dataset with different designed complexity. How to compare more fairly in the research, and how to better identify the effectiveness of the proposed method.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- S1: A well-executed study: Efficiently enables end-to-end speech via encoder/synthesizer integration while freezing the LLM backbone. The explicit state strategy reduces duplex system complexity, and DPO further boosts performance.\n- S2: Comprehensive experiments covering duplex dynamics, QA, daily dialogue, and ablation studies on state transition strategies.\n  \nWeaknesses\n- W1: Some methodological details and experimental setups could be clearer.\n- W2: Lacks evaluation of generation quality (e.g., subjective/objective metrics for speech quality).", "was_revised": true, "final_justification": "The author addressed my concerns one by one in the discussion, and 5 (Accept) can be given after the author's stated revisions to the paper.", "rebuttal": "We sincerely appreciate the high score you have given to our work. We hope the following content will further address and alleviate any remaining concerns you may have.\n\n- **Q1: Some methodological details and experimental setups could be clearer.**\n\nThank you for your suggestions. We will improve the presentation of the paper accordingly. Several experimental setups are detailed in the Appendix: Training data and specifications are in Appendix A; Prompts used for generating the training data and ChatGPT Score evaluation are in Appendix B. If the paper is accepted, we also plan to release our source code, training data, and pipeline to further clarify the implementation details. Please let us know if any additional aspects of our setup require clarification.\n\n- **Q2: Lacks evaluation of generation quality (e.g., subjective/objective metrics for speech quality).**\n\nThanks for your valuable advice. Speech quality is also an important evaluation aspect for speech generation. Here we compute the UTMOS score (a widely used automatic MOS prediction model, also used by GLM-4-Voice, the score is between 1 and 5, the higher the better) for answer samples.\n\n|                  | Llama | Web   | Trivia | Alpaca |\n|------------------|-------|-------|--------|--------|\n| **Predicted turn-taking** |       |       |        |        |\n| moshi            | 3.245 | 3.257 | 3.017  | 3.193  |\n| freeze-omni      | 4.288 | 4.273 | 4.293  | 4.255  |\n| SALMONN-omni     | 4.256 | 4.250 | 4.272  | 4.243  |\n| **Oracle turn-taking**    |       |       |        |        |\n| GLM-4-Voice      | 4.180 | 4.107 | 4.145  | 3.890  |\n| Qwen2.5-Omni     | 4.140 | 4.124 | 4.127  | 3.950  |\n| VITA-1.5         | 4.042 | 4.091 | 4.104  | 3.852  |\n| miniCPM-o        | 4.171 | 4.152 | 4.155  | 4.110  |\n| Kimi-Audio       | 2.882 | 2.713 | 2.793  | 2.327  |\n| Baichuan-Audio   | 4.261 | 4.216 | 4.292  | 3.959  |\n| Freeze-Omni      | 4.300 | 4.281 | 4.278  | 4.245  |\n| SALMONN-omni     | 4.262 | 4.263 | 4.279  | 4.240  |\n\nThe results show that SALMONN-omni can generate high-quality speech, with average UTMOS quality around 4.25 (For MOS score, 4 means good, 5 means excellent). It indicates SALMONN-omni can not only generate speech faithfully (as shown in high S2S performance), but also generate high-quality speech.\n\n- **Q3: Compared to the backbone LLM, does the final model exhibit degradation in reasoning/understanding due to added modules (LoRA, encoder adapter, etc.)**\n\nWe compared the performance of our model using both text and audio inputs with the original performance of LLaMA-3. It turns out that the performance degradation of the LLM is not significant, and our model exhibits comparable performance when using text (\"Ours-text\") and audio inputs (\"Ours-wav\"), which suggests that avoiding codec injection into the LLM indeed results in only minimal degradation to the model’s performance. Notably, we did not incorporate any pure text data during training. Such an approach is beneficial for preserving LLM performance in works like Qwen2.5-Omni. [1]\n\n||Llama Q.|Web Q.|Trivia Q.|AlpacaEval|\n|---|---|---|---|---|\n|Llama-3|0.85|0.56|0.73|3.83|\n|Ours-text|0.84|0.52|0.68|3.82|\n|Ours-wav|0.80|0.51|0.66|4.05|\n\n[1] Jin Xu, Zhifang Guo, Jinzheng He, et al. “Qwen2.5-Omni technical report”. In: arXiv preprint arXiv:2503.20215 (2025).\n\n- **Q4: For the streaming speech encoder: Why choose Mamba blocks?**\n\nIt is an alternative design choice and  not the primary focus of this work. One can definitely select any architecture that can support streaming speech processing, such as CNN and casual transformer. We choose Mamba since Mamba can support streaming sequence modeling with a relatively low computational cost compared to the casual transformer and it shows reasonable performance in our preliminary experiments.\n\n- **Q5: Lines 205–206 (Stage 3): Could you clarify the conclusion where the model \"exhibits a clear bias toward being interrupted and shows insufficient understanding of dialogue semantics\"?**\n\nThis is an empirical observation: we found that after Stage 2, the model exhibited high recall but relatively low accuracy on conversational dynamics evaluations (where we consider being interruptible as a positive case). We believe this is primarily because the model determines whether it is being interrupted, not based on semantic understanding, but rather on the presence of another speaker’s voice. The model tends to prefer being interrupted because the resulting output sequence is shorter, which may lead to a higher overall probability for the complete sentence.\n\n- **Q6: For DPO, it's better to detail the data scale, construction process, and reward design criteria for better understanding.**\n\nThank you for your suggestion. We clarify the composition of the DPO data here and will include these details in the next version. First, we obtained the Stage 2 model on the dialogue training data from Stage 2, collecting 11,000 samples involving errors in handling dynamics such as barge-in and backchanneling. These 11k samples evenly cover five categories of errors: incorrect handling of contextual-dependent barge-in, user-triggered contextual-independent barge-in, third-party-triggered contextual-independent barge-in, backchanneling, and early stop in the absence of any interruptions. For positive samples (with reward 1), we constructed them by directly modifying the position of the shift token in the negative samples (with reward 0), ensuring the accuracy of the reward signal. Moreover, we randomly sampled another 11k examples from other tasks within the Stage 2 dataset to prevent the model from forgetting knowledge related to those tasks. Using these 22k samples, we applied DPO loss to tasks related to barge-in and backchanneling, while continuing to use SFT for the remaining tasks.\n\n- **Q7: Line 232–233: For synthesized conversation data, how were speaker attributes (timbre/accent/emotion) selected in the generation process?**\n\nOur speaker prompts are selected from the LibriHeavy dataset. Although this dataset does not provide explicit speaker attribute annotations, we believe it offers a diverse range of speaker characteristics, as LibriHeavy was specifically designed to support the development of robust ASR models.\n\n- **Q8: This question does not affect scoring. Existing works in the field of speech generation, especially in speech dialogue, often employ different pre-training models and dataset with different designed complexity. How to compare more fairly in the research, and how to better identify the effectiveness of the proposed method.**\n\nIt's a good question. And it's the fact that almost all comparisons between different speech dialogue models are not strictly fair, since different models utilize different datasets and training specifications (such as batch size, training steps and so on). Under different setups, it's hard to say where the preference gain comes from, maybe the training dataset, maybe the architecture design, maybe the training strategies. It's hard or even impossible to disentangle the contribution of each part. In one paper, we can do ablation studies to search for a better design. But we can not do ablation studies between papers. Open-sourcing training data and model weights is certainly a critical prerequisite for fair comparisons. While many company-developed models currently rely on proprietary data, we also advocate for researchers in the academic community to make their training data publicly available whenever possible."}]

Author rebuttal:

Meta-review: This paper proposes SALMONN-omni, a standalone full-duplex speech LLM that eliminates audio codecs in token space (as was used by Moshi and other earlier systems) and incorporates a dynamic thinking mechanism for seamless turn-taking within the LLM backbone. It outperforms existing open-source full-duplex models by at least 30%, performs competitively to half-duplex and turn-based systems despite using less training data, and shows strong performance in complex conversational scenarios like barge-in, echo cancellation, and backchanneling.

Summary of Strengths:
- The work introduces a novel, clean and unified architecture, with a well-motivated design including a novel “thinking” mechanism and RL-based training (DPO) for modeling turn-taking and barge-in dynamics.
- The experiments demonstrates that the approach is also effective and outperforms strong baselines, despite using less data.
- The paper is mostly clearly written and easy to follow, though there are some parts that are less clear (please see the weaknesses).

Summary of Weaknesses:
- The core “thinking” strategy can be better explained. There were useful discussions with the reviewers during the rebuttals, which I believe will help with clarifying these parts of the paper.
-  The three-stage training pipeline (including DPO) adds complexity. During the rebuttals authors suggested that they will release their code, which will help with reproducability.
- Reviewers highlighted several issues with the evaluation, such as lack of evaluation on multi-turn datasets. Authors included additional results to address these concerns during the rebuttals.

The discussions during the rebuttals resulted in three reviewers accepting the author responses and increases in their scores. Once these  clarifications and additional results are integrated into the paper, it will be an interesting contribution.
Decision: Accept (poster)