DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representations
Authors: Chao-Hong Tan
Qian Chen
Wen Wang
Chong Deng
Qinglin Zhang
Luyao Cheng
Hai Yu
Xin Zhang
Xiang Lyu
Tianyu Zhao
Chong Zhang
Yukun Ma
Yafeng Chen
Hui Wang
Jiaqing Liu
Xiangang Li
Jieping Ye
Venue: ICLR 2026 Poster (2026)
Keywords: voice conversation model
parallel speech-text
end-to-end
dual-resolution
Primary area: foundation or frontier models, including LLMs

Abstract: Recent studies on end-to-end (E2E) speech generation with large language models (LLMs) have attracted significant community attention, with multiple works extending text-based LLMs to generate discrete speech tokens. Existing E2E approaches primarily fall into two categories: (1) Methods that generate discrete speech tokens independently without incorporating them into the LLM’s autoregressive process, resulting in text generation being unaware of concurrent speech synthesis. (2) Models that generate interleaved or parallel speech-text tokens through joint autoregressive modeling, enabling mutual modality awareness during generation. This paper presents DrVoice, a parallel speech-text voice conversation model based on joint autoregressive modeling, featuring dual-resolution speech representations. Notably, while current methods utilize mainly 12.5Hz input audio representation, our proposed dual-resolution mechanism reduces the input frequency for the LLM to 5Hz, significantly reducing computational cost and alleviating the frequency discrepancy between speech and text tokens and in turn better exploiting LLMs’ capabilities. Experimental results demonstrate that DrVoice-7B establishes new state-of-the-art (SOTA) on prominent speech benchmarks including OpenAudioBench, VoiceBench, UltraEval-Audio and Big Bench Audio, making it a leading open-source speech foundation model in ∼7B models.

Published as a conference paper at ICLR 2026
DRVOICE: PARALLEL SPEECH-TEXT VOICE CON- VERSATION MODEL VIA
SPEECH
REPRESENTATIONS
Chao-Hong Tan, Qian Chen, Wen Wang, Chong Deng, Qinglin Zhang, Luyao Cheng, Hai Yu, Xin Zhang, Xiang Lv, Tianyu Zhao, Chong Zhang, Yukun Ma. Yafeng Chen, Hui Wang, Jiaqing Liu, Xiangang Li, Jieping Ye Tongyi Fun Team, Alibaba Group (tanchachong.ch) ch, tanqing.cqy w.
com
ABSTRACT
Recent studies on end-to-end (E2E) speech generation with large language models (LLMs) have attracted significant community attention, with multiple works extending text-based LLMs to generate discrete speech tokens. Existing E2E approaches primarily fall into two categories: (1) Methods that generate discrete speech tokens independently without incorporating them into the LLM's autoregressive process. resulting text generation being unaware of concurrent speech synthesis. (2) Models that generate interleaved or parallel speech- text tokens through joint autoregressive modeling, enabling mutual modality awareness during generation. This paper presents Drvoice, parallel speech- text voice conversation model based on joint autoregressive modeling, featuring dual-resolution speech representations. Notably, while current methods utilize mainly 12.5Hz input audio representation, our proposed dual duso-uusol mechanism reduces the input frequency for the LLM 5Hz, significantly reducing computational cost and alleviating the frequency discrepancy between speech and text tokens and in turn better exploiting LLMSs capabilities. Experimental results demonstrate that DRVOICE-7B establishes new state-of-the-art (SOTA) on prominent speech benchmarks including OpenAudioBench. VoiceBench, UltraEval- Audio and Big Bench Audio, making it leading open-source speech foundation model in 7B models.
INTRODUCTION
Developments in spoken dialogue systems are critical to human-computer interaction, as natural human communication inherently relies on verbal exchanges. Recently, Large Language Model (LLM) based spoken dialogue systems, exemplified by systems like GPT-4o (OpenAl, 2024b),
demonstrate great potential for seamless and natural interactions with users. LLM- spoken dialogue systems be generally categorized into cascaded systems and end-to-end (E2E) systems, with the distinction lying in whether the backbone LLM can directly comprehend speech representations and generate speech outputs. Early cascaded systems integrate separately trained Automatic Speech Recognition (ASR), L.M,, and Text-to-Speedt (TTS) modules. Such systems inherently suffer from error accumulation, loss of acoustic details (e.g., emotion), and significant latency. Alternatively. the ASR module could be eliminated by introducing audio understanding foundation models (Chu et 2023 2024). E2E systems have emerged to further eliminate ASR and TTS modules, and establish direct connections between speech representations and LLMs. However, training LLMS to generate highly intelligent speech outputs remains challenging. E2E systems struggle with the quality of speech token generation, primarily due to inefficient utilization of textual information during speech token generation.
All source code and the model checkpoint hased enhanced base model are available at https://gith com / FunAudi OLLM// Fun. Audi o- Chat

---

Published as a conference paper at ICLR 2026
Recent works E2E models to address these challenges have focused two primary direc- tions (Chen et al. 2024a): Text-Driven Speech Models (Yao et al. 2024 Li al. 2025) and Joint Speech-Text Models. Text-Driven Speech Models refer to systems in which LLMs process speech representations as input, produce textual responses, and utilize the L.M's representations as input to speech decoder for speech generation. In contrast, Joint Speech- Texech-Texttte Models involve L.MSs taking speech representations as input and generating both text tokens and speech tokens simultaneously. The key distinction lies in whether speech token generation can influence the subsequent generation of text tokens within the LLM: this feedback loop of speech tokens to LLM is present in Joint models but absent in Text-Driven models. Joint Speech-Text Models can be further categorized into interleaved speech-text modeling (Zeng et al. 2025; Zhang et al.) 2024a) and parallel speech-text
modeling (Déoossez et al. 2024 Chen et al. [024a]] KimiTeem et al. 2025). Interleaved speech-text modeling alternates speech and text representations as inputs to the L.M,, while parallel speech-text
modeling fuses speech and text representations before feeding them into the LLM.
While both Text-Driven Speech Models and Joint Speech-Texi Models explicitly incorporate textual guidance into speech token generation to leverage the LLM's capabilities, they have distinct limitations. The architecture of Text-Driven Speech Models creates unidirectional information flow where text generation is completed before speech synthesis begins. This prevents the LLM from conditioning its textual output on the generated speech tokens, thus limiting its ability explore fine-grained paralinguistic attributes like emotion and prosody. On the other hand, Joint Speech- Text Models degrade the original text generation capabilities due to speech token interference, making the preservation of text capabilities a critical challenge. Nevertheless, Joint Speech-Text Models enforce multimodal interaction and generation and empower greater potentials (Chen et al.) 2024a); hence. in this work, we focus on enhancing joint speech-text modeling. Recently, Kimi-Audio (Kimi Team jet al. 2025) sets new of-te-of-the-ar (SOTA) in joint speech-tex: modeling. However, this approach still suffers from notable limitations. It not only requires extensive training data but also incurs significant computational costs due to its 12.5Hz audio representation. Furthermore, the high token rate creates frequency mismatch with the much lower rate of text tokens (~3Hz) (Chen al., 2024a), which can dilute semantic information and consequently hinder the full utilization of the LLM's core capabilities.
In this work, we propose DRVOICE, a novel parallel speech-text voice conversation model with Dual-Resolution Speech Representations (DRSR). For speech comprehension, we introduce grouping mechanism that maps 25Hz discrete audio tokens to 5Hz speech representations, effectively alleviating the temporal resolution discrepancy between speech and text tokens. During generation, combined speech-text embeddings form the assistant's autoregressive input. The hidden states captured from shared LLM layer are then passed in parallel to a Text Head for text token prediction and a Speech Refined Head (SRH) to generate the corresponding ungrouped speech tokens.
To further enhance the model's reasoning and coherence of its output, we incorporate Chain-of- Modality (CoM) (Zhang et al. 2023a) strategy. CoM prompts the model to first generate a complete response in text, allowing it to structure its thoughts before engaging in parallel speech- -eeet text generation. This intermediate reasoning step improves modality alignment and reduces errors. We design system prompts to control the output modes, enabling text-only output, direct parallel speech-text output, or the CoM-enhanced parallel output. We also develop a Core-Cocktail training strategy to refine model optimization and LLM knowledge retention.
1) We propose DRVOICE, a novel parallel speech-text conversation model featuring Qual-Resolluton Speech Representations (DRSR). This core architectural innovation effectively alleviates the temporal resolution mismatch between speech and text tokens, reducing computational costs (empirically achieving a reduction of nearly 50% in GPU hours for training) and better preserving the LLM's semantic processing capabilities. 2) We introduce two new training strategy, including a CoM-Mixing training strategy acting as a curriculum, using the structured CoM reasoning to scaffold speech generation, and Core-Cocktail training strategy for retaining the knowledge of LLMs. 3) Our comprehensive experimental results reveal that DRVOICE- 7B achieves new SOTA performance on prominent benchmarks including OeenAudioBench (audio understanding), VoiceBench (benchmarking LLM--aseed voice assistants). UltraEval-Anne Audio (both speech understanding and generation) and Big Bench Audio (reasoning and understanding capabilities of cetffo-processingg models), solidifying its position as premier open-source speech foundation model among 7B models.
Our contributions are three-fold:
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Speech Tokenization. Two primary directions exist for converting continuous speech signal into processable sequences: continuous representations, e.g., Whisper (Radford al. 2022), and discrete representations. Since LLMs generate discrete tokens, continuous representations face challenges in direct integration with LLMs for speech generation. In contrast, discrete tokens enable LLMs to handle speech similar to text tokens. and are typically categorized into two categories: 1) acoustic tokens optimized for reconstructing high-quality audio through neural audio codecs (Défossez et al., 2023), and 2) semantic tokens typically derived from self-supervised pre-trained models with masked language modeling as training objective (Hassid et al.) (023) or supervised learning as S3Tokenizer using in Cosy Voice (Du et al.) 2024a by 2025). and prioritized for linguistic content (Hsu al. 202T). While acoustic tokens achieve superior acoustic fidelity in sound reconstruction, semantic tokens demonstrate stronger alignment with semantic representations (Zhang al., (023b). thereby facilitating more effective extension of MLLMS' capabilities in both speech comprehension and generation (Zeng al. 2025 Zhang al. 2024b 2023a| Borsos al. 2023). In this work, use S3Tokenizer as the speech tokenizer due to its solid semantic capabilitie and compatibility with the strong Speech Detokenizer from CosyVoice Voice for synthesis. S3Tokenizer is a supervised semantic speech tokenizer based on the pre-trained Sense SenseVoice- Voice-Large model (An et al. (024)) that enhances the semantic relationship of extracted tokens. S3Tokenizer is robust to data noise, and reduces the reliance on clean data.
E2E Speech Foundation Models. Text-Driven Speech Models (Fang et al. 2024 Wang et al. 2024, Fu et al.] 2025 Yao et al. 2024] Huang et al.) 2025: Chen et al., 2025) integrate speech encoder, adapter, LLM, and a streaming speech decoder, and can simultaneously generate text and speech. Qwen2.5-0mni (Xu et al. 2025) employs the Thinker-Talker architecture, with Thinker handling multimodal understanding and text generation and Talker handling speech token production. Since Thinker cannot receive speech tokens during generation, the framework inherently limits awareness of speech token generation states, constraining applications such as full-duplex voice conversation.
Joint Speech-Text Models explore two different architectures, Interleaved and Parallel. Interleaved models (Zhang et al.] 2024aa Zeng et al.] (2025] Li et al.) [025 Long et al.] [025] Wu et al.) 2025) adopt interleaved decoding parallel models (Défossez et al. [0024] Xie & Wu 2224a Chen et al.
support simultaneous generation of speech and text tokens, while
2024a KimiTeam et al. 2025) conduct parallel decoding. The parallel model Moshi (Défossez al.
2024) employs a compact Deep Transformer to predict & tokens while relying solely on current-step representations. To mitigate limitations of discrete tokens in audio understanding. Kimi-Audio (KimiTeam al. (202) introduces dual-tokenizer combining discrete semantic tokens with continuous Whisper (Radford et al. 2023) features, preserving both semantic and acoustic information. of both training and inference and alleviate the issue of granularity misalignment between the text and speech modalities, our approach leverages DRSR to reduce the LLM input frame rate to 5Hz, without sacrificing performance.
order to further enhance the efficiency
METHODOLOGY
Figure I iillssrates the architecture of DRVOICE. The system consists of three main components: (I) Speech Encoder and Speech Tokenizer process the speech wave into hidden representations for the user end and the assistant end respectively, (2) Multimodal Large Language Model (MLLM) consists of shared LLM layer, a Text Head, and a Speech Refined Head (SRH) for token generation,
and (3) Speech Detokenizer to generate wave from speech tokens. The system operates through multimodal input encoding and coordinated speech-text output generation. During inference, user inputs (text or speech) are first mapped a unified semantic space, processed by MLLM to produce parallel speech-tex text responses through SRH and text head. To effectively train this system, we introduce the CoM-Mixing Training and Core-Cocktail Training strategies.
3.1 SPEECH TOKENIZATION AND DETOKENIZATION
Aiming for enhanced audio understanding, we utilize Whisper-Large-v3 Speech Encoder to extract continuous audio representations at the user end. Subsequently, an Adapter is introduced to downsample the temporal resolution of these representations and align their hidden dimension
3

---

Published as a conference paper at ICLR 2026
Speech Detokenizer
Predicted Speech Token
Speech Refined Head
82_s
Assistont Speech Taken
Ungrouping Mixxed Hidden
Predicted Text Token
Ungrouping
Text Head
Mixed Hidden
Shared LLM Layer
Time
Assistont Text Token
Text promot
Grouping Speech Hidden
Adapter
Speech Taken
Speech Encoder
BL
B2
Overview of DRVOICE. User speech inputs are tokenized, grouped, and encoded by the MLLM for autoregressive text and speech token prediction. The MILM consists of Shared LLM Layer, a Text Head, and Speech Refined Head (SRH) for token generation. The generated speech tokens then converted to speech waveform by the speech detokenizer. Note that SRH generates speech tokens through & autoregressive forward passes, where & is the grouping factor.
Figure 1:
with that of the LLM. Semantic tokens have been widely used for speech tokenization (Zhang et al. 2023a: Borsos et al. 2023), due to their strong alignment with text (Zhang et al. 2023b); therefore, we employ S3Tokenizer (Du et al., 2024a by 2025) as the Speech Tokenizer to convert speech waveform to semantic speech token sequence S $1, ST-1 at the assistant end, where 7 is the speech token sequence length. For speech detokenization, conditioned on speaker embeddings capturing acoustic details such as timbre, the Flow Matching model (Lipman et al., 2023) converts speech tokens into the Mel spectrum for given speaker. Finally, a pre-trained vocoder HiFi-GAN (Kong al. 2020) transforms the Mel spectrum back into audio signal.
3.2
MULTIMODAL LARGE LANGUAGE MODEL (MLLM)
Built upon text-LLMs, the MILM learns joint speech-text modeling process speech or text inputs while producing parallel speech and text outputs.
Parallel Joint Speech-Texs Texe Model. Inspired by Moshi (Défossez et al. 2024). explicit text streaming is incorporated to aid speech generation as a common semantic scaffold. We focus on performing modality alignment exclusively at the assistant end. This design adheres to the asymmetric nature of human-maching interactions: while user inputs are typically unimodal (either text or speech), the assistant's 's responses can be a coordinated multimodal output.
Leveraging the autoregressive generation capability of LLMs, both generated speech tokens and text tokens 4, are iteratively fed back into the shared LLM layer at each timestep. Their embeddings added as model input, forming parallel speech-text architecture. Formally, the combined input embedding o at timestep is computed as:
C Expeceh (t) + (tex((s)
where Expeech and E denote the embeddings of speech and text tokens, respectively. Since the lengths of speech tokens and text tokens are typically mismatched, the shorter sequence is padded with special token <ISIL align them for each utterance. The autoregressive generation process is follows:
P(yl x) = P(y x))
(2)
4

---

Published as a conference paper at ICLR 2026
where is the input sequence and y (st, tz) denotes the joint speech-text output at timestep This unified formulation enables seamless integration of speech and text generation within single autoregressive framework. To preserve the intrinsic linguistic understanding and generation capabilitiis of pretrained text-ILMMs while enabling cross-modal interactions, three key methodological designs are introduced for intermodal coordination, including Speech Token Grouping and Ungrouping, and Speech Refined Head.
Speech Token Grouping. A crucial parameter for discrete tokens is the sampling rate, which determines the input/output sequence length. GLM-4-Voice (Zeng et al. (025) investigates sampling rates from 6.25Hz 50Hz on LibriSpeech, revealing minimal differences Word Error Rate (WER) between 50Hz and 25Hz but significant degradation at 12.5Hz and 6.25Hz. Hence, our work adopts 25Hz sampling rate. To resolve the temporal resolution mismatch between speech signals (25Hz) and text tokenization (~3Hzz) (Chen et al. 2024a), grouping mechanism is designed:
(i+1)--1
g: Linear
Sj
E
R
dase
j=ik
where 8j denotes speech tokens, || represents feature concatenation, and k is the grouping factor determined by the ratio between speech token rates and text token rates. This design compresses the speech token length from T to T/ k and the resulting grouped speech representations align better with text. Notably, different from Chen al. (2024a) that employs linear projection of audio logits into group-sized representations for parallel multi-token prediction, DRVOICE strategically designs an ungrouping mechanism and incorporates dedicated Speech Refined Head (SRH) enable autoregressive generation of individual speech tokens.
Speech Refined Head (SRH). SRH is proposed enhance speech generation capabilities. utilizes the last hidden state of the shared LLM layer (SLLM) as conditional input and incorporates contextual speech information to autoregressively generate speech tokens. While conventional speech grouping strategies-which cluster speech tokens into semantically meaningful units-have proven effective for speech recognition and understanding tasks (Chen et al., 2024a Zhang et al., 2024b), our experiments reveal their inherent limitations in generative scenarios, since speech token grouping inevitably loses some fine- mne-grained acoustic details. To address this limitation, DRVOICE performs an ungrouping process follows: the SLLM's final hidden state is mapped to group- sized embeddings via linear projection
hag
W Wh WyhiSUI
[SLLM]
where
Wp E Rdgxdh
and time splitting
H Split, (hag)
h h
ug
where hug E /dd The resulting H serves as the conditional input for SRH that autoregressively generates speech tokens. Following our practice on SLLM, representations of preceding speech tokens and H are aggregated. Speech token prediction is trained to maximize the conditional probability:
T
CSRH Σ
log
(6)
where 8; represents the i-th speech token. By optimizing this objective, SRH learns to predict subsequent speech tokens conditioned on both preceding speech tokens and the rich contextual embeddings H derived from SLLM. This design enables SRH to effectively leverage the semantic and acoustic information encoded the hidden representations of SLLMM, thereby producing more natural and coherent speech outputs compared to conventional grouping-based approaches.
The E2E training objective LMLLM integrates the two losses through multi-task learning:
LMLLM = XLTI + pLSRH.
T
LTH = log P(klo2i.g)
Σ
(8)
where Lrn is the autoregressive loss over the text head, À and μ are hyperparameters.
5

---

Published as a conference paper at ICLR 2026
Table 1: Multimodal Interaction Patterns.
Pattern Name
Abbr.
Modality Flow
Speech-to-Munn
Mo-Mulimdaa
S2M
Speech
Joint speech- response
Speech-to-Tese
S2T
Speech +) Text-only response
Text-to-Me
Text-to-Multimndll Multimodal
T2M
Text - Joint speech-text response
Text-to-Text
T2T
Text > Text-only response
Speech-Text Chain
STC
Speech + Text transcription Text response Multimodal response
apech-Assisted Chain
SAC
Speech - Text response (agent perspective) Multimodal response
Speoch- Caecc-Usserree Chain
SUC
Specch + Text transcription (user perspective) Multimodal response
3.3 TRAINING STRATEGY
Initialization. The Speech Encoder is initialized with the weights of Whisper-Large-v3, while the Shared LLM Layer is initialized using Qwen2.5. Additionally, the pre-trained Speech Tokenizer and Detokenizer from Cosy Voice employed and kept frozen throughout the entire training process. We initialize SRH with a pre-trained TTS model.
CoM-Mixing Training. The oifii--o--odaaiii (CoM) methodology (Zhang et al. 2023a) can enhance performance by leveraging intermediate textual transcriptions, which is particularly suitable for scenarios where real-time processing is not critical. Furthermore, practical applications involve scenarios where both speech and text output or text-only output are required, necessi- tating the system to dynamically generate different modalities based on specific needs. Seven interaction patterns for diverse input-ouiput requirements are summarized in Table [ System prompts are employed to regulate the model's output behavior. For example, prompts such as " "[System] You are helpful assistant and asked to generate both text and speech tokens at the same time. " explicitly guide the model to produce multimodal outputs. Detailed prompts are shown in Appendix B During inference, specifying these system prompts enables the generation of desired output results. For CoM-Mixing training, we construct data variants following the seven interaction patterns and obtain mixture of data for training the model. During inference, user can manually prepend the appropriate system prompt to the input sequence to meet the output requirement. This flexibility ensures adaptability to varying modality generation demands.
Core-Cocktail Training. We identify learning rate dilemma: employing high learning rate significantly compromises the performance of the MILMM, whereas using a low learning rate leads to training stagnation, with the loss decreasing quite slowly. To overcome this optimization challenge. we develop a specialized two-stage training strategy, termed "Core-Cocktail". The first stage involves full fine-tuning the entire MLLM with relatively high learning rate. The goal of this phase is to rapidly move the model's 's parameters into more favorable region of the loss landscape. However, counteract the potential performance degradation of the MLLM caused by this aggressive initial training, an intermediate merging step is introduced. Drawing inspiration from Xiao et al. (2024), the parameters of the MILLM trained in the first stage (M1) are merged with the parameters of the base, pre-trained LLM (Mo). This merging creates new, interpolated model, Mr, as defined by the following equation:
M, <- a.M + (1 Mo
where M,, denotes the merged model and a is the interpolation weight. The merging step effectively re-integrates the robust knowledge of the base LLM. A smaller corresponds greater preservation of the base LLM's capabilities. The second stage conducts full fine-tuning on the merged model M, using small learning rate. The core-cocktail training approach allows for careful and precise optimization, refining the model's performance without instability from a high learning rate.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. Approximately 100K hours of audio-text paired data for speech-text modality alignment is adopted for pre-training Speech Refined Head. For DRVOICE post- bost-training, we first synthesize speech for about 3B text tokens using Cosy Voice (Du et al. 2024a b 2025), then select about 26K hours for

---

Published as a conference paper at ICLR 2026
Table 2: Performance Comparison various benchmarks in terms of benchmark--seciffe metrics (the best and second-best results in each row are in bold and underlined, respectively). ). With the exception of GLMc-Voice, whose results cited from Xu et al. (2025) and Chen al. (2024b), and the BBA results, which are cited from Xiaomi (2025) except for MiniCPM-o 2.6 that we reproduce, all other
results were generated by running inference on the released checkpoints. FR(In/Out) denotes the
input speech frame rate and the output speech plus text frame rate for the LLM backbone. denotes average number of text tokens corresponding to one second of speech.
MiniCPM
Baichuan
Qwen2.5
Kimi
Step-Audio2 -iin
Goic-Voice
DRVOICE
2.6
-0mmi-1.5
-Omni
AAdio
FR (In/Out)
12.5/12.5+7
25/7
12.5/12.5+T
25/T
12.5/12.5
12.5/25+7
5/5
OpenAudioBench (S2T)
AlpacaEval
57.89
64.10
77.90
72.76
75.73
59.60
82.61
Llama Q.
76.00
78.00
78.50
75.33
79.33
75.00
83.00
Reasoning QA
47.43
38.60
50.00
63.76
58.02
46.04
59.90
TriviaQA
51.80
63.00
57.20
57.06
62.10
57.70
64.50
Web
55,40
69.20
59.10
62.80
70.20
65.10
70.20
Overall
57.70
62.58
64.54
66.34
69.08
60.69
72.04
VoiceBench (S2T)
AlpacaEval
3,97
4.42
4.50
4.33
4.46
4.17
4.74
CommonEval
3.42
4.15
4.05
3.84
3.97
3,00
4.08
SD-QA
36.98
50.72
43.40
57.41
63.12
56.06
64.30)
MMSU
39.75
54.78
57.25
56.38
62.17
52.18
67.27
OpenBookQA
53.41
78.02
74.51
79.12
83.52
64.18
82,20
IFEval
52.80
49.25
54.54
53.88
61.10
38.01
71.39
AdvBench
88.08
97.69
97.31
99.62
100.00
93.08
99.62
Overall
59.83
71.69
71.14
72.83
76.93
63.84
80.17
UlraEval-Audie (S2S)
AlpacaEval
51.00
51.00
58.69
56.10
44,20
51.72
$9.29
Llama
50.00
61.00
67.33
66.30
57.33
67.67
75.33
TriviaQA
36.40
40.20
30.57
40.52
35.71
33.50
46.09
Web Q
32.00
40.00
38.09
38.93
33.90
34.65
45.92
Overall
42.35
48.05
48.67
50.46
42.79
46.89
56.66
Big Bench Audio (S2T & S2S)
S2T
44.8
$6.2
47.1
54.2
59.4
50,9
76.9
S2S
42.7
55.4
44.6 45,8
53.6
51.0
71.2
Overall
43.8
55.8
53.9
55.2
49.2
74.0
Table 3: Speech quality performance on the collections of UltraEval-Audio, in terms of UTMOS for assessing overall speech quality and ASR- WER for assessing alignment between generated speech and text. FR(In/Out) indicates the input speech frame rate and the output (speech + tex() frame rate the LLM backbone, while 7 represents the average text tokens per second of speech.
Model
FR(In/Out)
UTMOST
ASR-WER.
MiniCPM-o 2.6 2025
25/T
4.18
13.17
Baichuan- -mmii 1.5 (2025
12.5/12.5+7
4.27
23.38
Qwen2.5-0mmi (2025)
25/T
4.28
3,48
Kimi-Audio 2025
12.5/12.5
3.06
21.06
Step-Audio2-minn 2025
12.5/25+T
4.53
9.50
DRVOICE
5/5
4.29
8.36
speech-to-speech conversation and about 20K hours user speech plus 1.3B assistant tokens for speech- to-text conversation, based on Word Error Rate (WER) of the synthesized speech. Furthermore, to enhance the model's comprehension of real-world speech, the training data was augmented with about 10K hours of English Automatic Speech Recognition (ASR) data, comprising several corpora, including Common Voice (Ardila et al. (202)), MELD (Poria al. 2019). LibriSpeech (Panayotov) jet al. 2013). SPGISpeech (O'Neill et al.
2021. and Voxpopuli (Wang et al., 202I). Following prior works (Ya et al. 2024 Kimi al.
2025 OpenAl 2024b). evaluate the performance the
7

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study of Continuous Speech Encoder (CSE), Speech Refined Head (SRH), SRH-
Pretraining and CoM-Mixing on Llama Questions. S2M, S2T, etc are defined in Table
Model
S2M (T/S)
S2T
T2M (T/S)
T2T
(T/S)
SAC
SUC (T/S)
DRVOICE- Saac-l w/o, CSE w/o. SRH-Pretruining
68,67 56.00
72.33 72.33/ 56.00
75.33 75.67 68.33
72.67 73.33/62.00
61.67 53.00
62.33 70.00/ 60.00
74.00
69.33/61.00
63.00/ 55.00 66.33/ 58.67
38,33 /0033 56.00
59.33/46.35 46.33
73.33
67.33/57.67
54.00/42.33
54.33/42.61
w/o. SRH
21.67 15.33 56.00
45,22/ 35.00
73.00 64.33/50.67
/42.33
40.33/27.67
w/o. CoM-Mixing
58.00/ 49.00 58.00
69.33/55.00
68.33
+
widely used benchmarks, VoiceBench (Chen ct al. 2024b) and OpenAudioBench for Speech- To- Text (S + 7) evaluation, UltraEval-Audig for Speech-to-Spect (S + S) evaluation, and Big Bench Audio P for both evaluations.
Evaluation Metrics. Evaluations adhere to the established protocols for each respective benchmark. Specifically, for the open-ended QA tasks AlpacaEval and CommonEval, G-Eval (Liu et al. (023) is used for scoring. For AdyBench, the Refusal Rate is reported, while performance on all other benchmarks is assessed with Accuracy. The generated speech is transcribed using Whisper-v3-lan model (Radford et al. 2022). then WER (denoted by ASR- ASR-WERR) of transcripts is computed against the generated text assess the alignment between generated speech and text. UTMOS (Saeki et al.) 2022 is used to evaluate the overall speech quality, following Zeng et al. (0255)
Baselines. Representive and competitive open-source audio language models are selected as baselines to cover diverse modeling paradigms: Text-Driven models including MiniCPM-o 2.6 (8B) YYao et al. 2024) and Qwen2.55-mmn (7B) (Xu et al. 2025). Among Joint Speech- -ext models, interleaved models including GLM-4-Voicc (9B) (Zeng et al. 2025),
(7B) (Li jtt al., (025). and -ee--Audi22-Min' (8B) (Wu et al. 2025). alongside the parallel model Kimi-
Audio (7B) (KimiTeam et al. 22225). This suite enables systematic comparisons across mainstream speech-text modeling strategies.
Implementation Details can be found in Appendix A
4.2 MAIN RESULTS
Overall Performance. Table compares DRVOICE (7B) with representative and competitive baselines on speech-to-text (S (S>T) and speech-to-speech (S-S) generation performance. As shown in the table, DRVOICE demonstrates exceptional capabilities across a wide range of tasks, achieving new state-of-the-art (SOTA) results all four major benchmarks. Specifically, DRVOICE secures the top position on ApdnAudioBench (audio understanding) with an overall score of 72.04, VoiceBench (benchmarking LLM-bassd voice assistants) with 80.17, on UltraEval- Audio (both speech understanding and generation) with 56.66, and on Big Bench Audio (reasoning and understanding) with 74.0. This consistently dominant and balanced performance across diverse benchmarks establishes DRVOICE as the leading open-source speech foundation model in the 7BB parameter class.
Computational Efficiency and Speech Quality. A key innovation of DRVOICE is its remarkable computational efficiency, highlighted in Table 2 and Table 3 As shown in the FR (In/Out) rows, DRVOICE operates at a frame rate of 5/5, indicating that the LLM backbone processes only 5 audio tokens per second for both input and output. This is a substantial reduction compared other models, which operate at frame rates of 12.5 or 25, thereby significantly lowering computational requirements and potential latency. Crucially, this efficiency does not come at the cost of speech quality. Table 3 shows that despite its low frame rate, DRVOICE, produces high-quality and well-aligned speech. It achieves UTMOS score of 4.29 for overall speech quality, which is competitive with top-performing models like Qwen2.5-Ommi (4.28) and superior to others like Kimi-Audio (3.06). With an ASR-WER of 8.36, the model shows robust speech-text alignment and surpasses several baselines. Still, underperforms compared to Qwen2.5-Omni A potential explanation lies in the architectural design:
8

---

Published as a conference paper at ICLR 2026
Qwen2.5-O 5-nni'ssss feeding text directly to its "Talker" module, whereas the proposed model only sends hidden states to SRH. This unique combination of SOTA performance, high-quality speech generation, and unparalleled efficiency makes DRVOICE a highly powerful and practical model for real-world applications.
4.3 ABLATION AND ANALYSES
We conduct extensive ablation study and analyses. For computational efficiency, some experiments are performed on DOTCE-SSmlll (1.5B). More analyses about the data quality and data scaling can be found in Appendix C
Core-Cocktail Cocktail Training Strategy. While Stage training causes a performance drop from text baseline of 81.77 70. Stage reverses this trend, recovering the performance 74.73. This result confirms that the strategy effectively counteracts the initial degradation and leads to an optimized model. Further details are available in Appendix C
Continuous Speech Encoder. As shown in Table 4 the Continuous Speech Encoder (CSE) is vital for tasks involving speech inputs. this setting, user audio input processed solely using the discrete speech tokenizer (same as the assistant's output tokenizer), without the Whisper features. Removing from DRVoICE-Small leads to a significant performance drop in speech understanding and speech generation. Specifically, the S2T score decreases from 72.33 to 62.33 (13.8% relatively), and the S2M (T) score falls from 68.67 to 61.67 (10.2% relatively). In contrast, the impact on text-only tasks minimal, with the T2T score only slightly decreasing from 75.33 to 74.00. This confirms the CSE's effectiveness in representing speech.
Dual-Resolution Speech Representations (DRSR). Our dual-resolution approach combines input grouping for efficiency and comprehension with a refinement head for high-quality generation. 1) Grouping Factor. Contrary to degrading generation, grouping substantially improves both speech understanding (S2T) and speech-to-speech generation (S2M). For instance, increasing the grouping factor from to 5 raises the S2T score from 55.67 63.33 13.7% relative gain). The S2M (T/S) score sees significant improvement, peaking at 37.67 28.00 with a grouping factor of 5. Furthermore, using a grouping factor of 5 instead of reduces nearly 50% GPU hours in each setting, proving the efficiency of grouping machinism. Detail results can be found in Appendix C 2) Speech Refinement Head. shown in Table the Speech Refinement Head (SRH) remarkably effective for speech token generation tasks. To handle the resolution mismatch when SRH is removed, the hidden state from the shared LLM is passed through a projection layer (dmodet + k x depeech depeech_token) and then split into k tokens. This setup simulates a standard parallel prediction approach without autoregressive refinement. By comparing the model with SRH (w/o. SRH-Pretraining) to one without (w/o. SRH), we observe that adding SRH achieves a 76.9% relative improvement in S2M (T) (from 21.67 to 38.33) and a 31.2% relative improvement in T2M (T) (from 45.22 to 59.33). The text- only performance (T2T) remains stable, confirming that SRH enhances speech generation without interfering with text processing pathways. Our dual-resolution architecture effectively combines these components, using grouping for efficient comprehension and SRH at the raw frame resolution for speech generation.
Srit--retraining, To examine the impact of pretraining, we remove the SRH pretraining step and retrain the model. As shown in Table 4 removing pretraining (comparing w/o. CSE with w/o. SRH- Pretraining) has the most significant impact on speech generation, causing S2M (T) performance to drop by 37.8% and T2M (T) by 15.2%. The effect on S2T is smaller (10.2% drop), and negligible for T2T. This underscores the critical importance of SRH pretraining for refining speech token generation.
Mixin Training Strategy. As shown in Table 4 where tasks guided by contextual system prompts (STC/SAC/SUC) demonstrate substantial improvements over direct S2M generation. For example, STC (T) achieves a score of 75.67, significantly surpassing the S2M (T) baseline of 68.67. This shows the model successfully learns to adopt the generation paradigm guided by system prompts. The data augmentation effect is confirmed by ablating the CoM-Mixine strategy entirely. Training without it (w/o. CoM MoM-Mixing Training) results in a 15.5% relative performance degradation in S2M (T) (from 68.67 58.00), confirming the value of our mixed-mode modality training approach.

---

Published as a conference paper at ICLR 2026
5 CONCLUSIONS
We introduce DRVOICE, a novel parallel speech-text voice conversation model that leverages joint autoregressive modeling with dual-resolution speech representations. Our experimental results demonstrated that DR establishes a new state-of-the-at (SOTA) on OpenAudioBench for audio understanding, VoiceBench for voice assistant tasks, SaraEval-Audio for both speech understanding and generation and on Big Bench Audio for reasoning capabilities, confirming its strong and versatile capabilities. Notably, its dual-resolution mechanism significantly improves inference speed and computational efficiency (empirically achieving reduction of nearly 50% in GPU hours for training) without sacrificing performance. We discuss limitations and future work in Appendix D
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we will make our resources publicly available. The complete source code for our model, training and evaluation scripts, and all pre-trained model checkpoints will be released upon publication. The speech data used in our experiments were synthesized using the publicly available CosyVoice model; will provide the necessary scripts and instructions to replicate the dataset. Furthermore, a comprehensive description of the implementation details, including architectural choices, hyperparameters, and training setup, is provided in Appendix [A] We believe these resources are sufficient for the community to reproduce our findings and build upon our work.
REFERENCES
Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, Shengpeng Ji, Yabin Li, Zerui Li, Heng Lu, Haoneng Luo, Xiang Lv, Bin Ma, Ziyang Ma, Chongjia Ni, Changhe Song, Jiaqi Shi, Xian Shi, Hao Wang. Wen Wang, Yuxuan Wang, Zhangyu Xiao, Zhijie Yan, Yexin Yang. Bin Zhang, Qinglin Zhang, Shiliang Zhang, Nan Zhao, and Siqi Zheng. Funaudiollm: Voice understanding and generation foundation models for natural interaction between humans and llms. CoRR, 0adp/4407.0055/ 2024. doi: 10.48550/ARX IV.2407.0405
//d::///oooooo .rg. 4ww.11..00000.......... .4055
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. Common voice: A massively- multilingual speech corpus. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerek, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pp. 4218-4222. European Language Resources Association, 2020. URL http:///ccaanhhoooy...o.
1rec-1. 1rec-1.520/
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matthew Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: A language modeling approach to audio generation. IEEE ACM Trans. Audio Speech Lang. Process 31:2523 2533, 2023. doi: 10. 1...10//SLL..223322222222 org/ 11199/1A1TTL 2023. 3288409
URLbttee //doi /doi
Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, Yabin Li, Xiang Lv, Jiaqing Liu, Haoneng Luo, Bin Ma, Chongjia Ni, Xian Shi, Jialong Tang, Hui Wang, Hao Wang, Wen Wang. Yuxuan Wang, Yunlan Xu, Fan Yu, Zhijie Yan, Yexin Yang, Baosong Yang, Xian Yang, Guanrou Yang, Tianyu Zhao, Qinglin Zhang. Shiliang Zhang, Nan Zhao, Pei Zhang, Chong Zhang, and Jinren Zhou. Minmo: multimodal large language model for seamless voice interaction. CoRR, abs/2501 002822 2025. doi: 10.48550/ARXIV 06682.. URL htllatts::/doo wro. 10. 48550/ wttp 06282
Wenxi Chen, Ziyang Ma, Ruiqi Yan, Yuzhe Liang, Xiquan Li, Ruiyang Xu, Zhikang Niu, Yanqiao Zhu, Yifan Yang, Zhanxun Liu, et al. Slam-on omni: combro-controllable voice interaction system with single-stage training. arXiv preprint a41 15649, 2024a.
10

---

Published as a conference paper at ICLR 2026
Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. Voicebench: Benchmarking Ilm- -lm-based voice assistants. CoRR, abs/2410.17196, 2024b. doi: 10.88550/ARXIV
.2410.17196. URL httpps:://....tt/////////
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang. Shiliang Zhang. Zhijie Yan, Chang Zhou, and Jingren Zhou. Owen-audio: Advancing universal audio understanding via unified large-scale audio-language models. CoRR, abs/231 1. 07919, 2023. doi: 10.48550/ARXIV.233 1..99999 URL
https::/dbf..rr//11 htt.://..orrg11....5///a/a// 48550//rXivv 2311 2311 201100001111
Yunfei Chu. Jin Xu, Qian Yang. Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report. CoRR, abs/2407.10759, 2024. doi: 10.48550/ARXIV 2407.10999, URL https:://oi..rrggt
8550/arXiv. 2407. 10759
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Trans. Mach. Learn. Res., 2023, 2023. URL
lum:iddivCd82221
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. CoRR, abs/2410.00051 2024. doi: (044550/ARXXIV2240000033 URL ptttps://do/: org/ 10
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. Technical report, 2024. URL ptteps://arrke wttp://r/r/abb//.../0/// https///110000..0
abs
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang. Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijie Yan. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. CoRR, abs/2407 .05407. 2024a. doi: 10.48550/ARXIV http:/AXXXXX.222..0.0
URL https: wrg/10ggg/..ooo 48550/ / 2407
05407
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, and Jingren Zhou. Cosyvoice 2: Scalable streaming speech synthesis with large language models. CoRR, abs/2412. 2024b. doi: (0.48550/ARXIV.2 241 2.10117. URL https:::ss.........t...ttt doi. or0/10.../.oooo......... 1.0...8.0
2412. 10117
Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lx, Hui Wang, Chongjia Ni, Xian Shi, Keyu An, Guanrou Yang. Yabin Li, Yanni Chen, Zhifu Gao, Qian Chen, Yue Gu, Mengzhe Chen, Yafeng Chen, Shiliang Zhang, Wen Wang. and Jieping Ye. Cosyvoice 3: Towards tetheewwld speech generation via scaling-up and post-training. CoRR, abs/2505.17589 2025. doi: 10.48550/ARXIV 250. URL /tt:::///oooiiee wwt./o..///000 http://.1...55//22222 505. 1789
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. CORR, 0wt./2499006666 2024. doi: 10.4 8550/ARXIV Www.RRXIVV.249.00O 24099066666
/ttttpp://ddo.oog/1100 01000..855//aiiii.
2409.0000000 06668
Chaoyou Fu, Haojia Lin, Xiong Wang, Yifan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, Long Ma, Xiawu Zheng, Rongrong Ji, Xing Sun, Caifeng Shan, and Ran He. VITA-1.5: towards gpt-4o level real-time vision and speech interaction. CoRR, abs/2501.01157, 2025. doi: 10.48550/ARXIV. www.AARIII..225...00 5599071977 URL htttts::/ddb.oor/11100 501 01957
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Défossez, Gabriel Synnaeve, Emmanuel Dupoux, Roy Schwartz, and Yossi Adi. Textually pretrained speech language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, IA, USA, December 10 16, 2023, 2023. URL http: httt::://ppaprs...m wna.snnippp nips ce/p

---

Published as a conference paper at ICLR 2026
aper oet.:///...0////haoo./0000107760
ract tact-Conference.hee html
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE ACM Trans. Audio Speech Lang. Process. 29:3451-3460) 2021. doi: 10.1 10.1109/TASLLP22211112222
URL http:::pp////mm... /ttepp::/doi..o org. 1ww.:1111991111 2021.3
122291
Ailin Huang, Boyong Wu, Bruce Wang. Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, Peng Liu, Ruihang Miao, Wang You, Xi Chen, Xuerui Yang, Yechang Huang, Yuxiang Zhang, Zheng Gong, Zixin Zhang, Brian Li, Changyi Wan, Hanpeng Hu, Ranchen Ming, Song Yuan, Xuelin Zhang, Yu Zhou, Bingxin Li, Buyun Ma. Kang An, Wei Ji, Wen Li, Xuan Wen, Yuankai Ma. Yuanwei Liang, Yun Mou, Bahtiyar Ahmidi, Bin Wang, Bo Li, Changxin Miao, Chen Xu, Chengting Feng, Chenrun Wang, Dapeng Shi, Deshan Sun, Dingyuan Hu, Dula Sai, Enle Liu, Guanzhe Huang, Gulin Yan, Heng Wang, Haonan Jia, Haoyang Zhang, Jiahao Gong, Jianchang Wu, Jiahong Liu, Jianjian Sun, Jiangjie Zhen, Jie Feng, Jie Wu, Jiaoren Wu, Jie Yang, Jinguo Wang, Jingyang Zhang, Junzhe Lin, Kaixiang Li, Lei Xia, Li Zhou, Longlong Gu, Mei Chen, Menglin Wu, Ming Li, Mingxiao Li, Mingyao Liang, Na Wang, Nie Hao, Qiling Wu, Qinyuan Tan, Shaoliang Pang, Shiliang Yang, Shuli Gao, Siqi Liu, Sitong Liu, Tiancheng Cao, Tianyu Wang, Wenjin Deng, Wenqing He, Wen Sun, Xin Han, Xiaomin Deng, Xiaojia Liu, Xu Zhao, Yanan Wei, Yanbo Yu, Yang Cao, Yangguang Li, Yangzhen Ma. Yanming Xu, Yaqiang Shi, Yilei Wang, Yinmin Zhong. Yu Luo, Yuanwei Lu, Yuhe Yin, Yuting Yan, Yuxiang Yang. Zhe Xie, Zheng Ge, Zheng Sun. Zhewei Huang, Zhichao Chang, Zidong Yang. Zili Zhang, Binxing Jiao, Daxin Jiang. Hcung- Yeung Shum, Jiansheng Chen, Jing Li. Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang, and Yibo Zhu. Step-audio: Unified understanding and generation in intelligent speech interaction, 2025.
/tt.ttsss///.tttleeeeeee www. abs/ / 25020 11946
Dhiraj D. Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A study of BFLOATI6 for deep learning training. CORR, Abs/1995.11322. 2019. URL httlhhht:://// /ttp:///aaaax org abs//a03
12322
KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng. Songxiang Liu, Tong Liu, Zeyu Shang. Kai Shen, Wei Song. Xu Tan, Heyi Tang. Zhengtao Wang, Chu Wei, Yifci Xin. Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, Jun Chen, Yanru Chen, Yulun Du, Weiran He, Zhenxing Hu, Guokun Lai, Qingcheng Li, Yangyang Liu, Weidong Sun, Jianzhou Wang, Yuzhi Wang, Yuefeng Wu, Yuxin Wu, Dongchao Yang, Hao Yang. Ying Yang. Zhilin Yang, Aoxiong Yin, Ruibin Yuan, Yutong Zhang, and Zaida Zhou. Kimi-audio technical report, 2025. URL https
7/arxiv. org. abs wtt.:rgggabba//e0/ /204 18425
Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL pttp:ttppp//ppcceednns.. neur 20tp://20///aaa// s. httpa//cc/...aaoooo.0.000300635. tm1
Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al. Baichuan-omni-1.5 technical report. arXiv preprint 1rX1v:2501.15368, 2025.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL net forum? werrumidd-Ppae Pa-.rddd-PVVDCSSSS
ht https:://ppnrewwe
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika
12

---

Published as a conference paper at ICLR 2026
Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10. 2023, pp. 2511-2522. Association for Computational Linguistics, 2023. doi:
110.553/112222.MMMMPPP... //doi.org/11 10..1..166610 011111663112000 emn p-maa-1mm11mm mw..manii...... 153
|
Zuwei Long, Yunhang Shen, Chaoyou Fu, Heting Gao, Lijiang Li, Peixian Chen, Mengdan Zhang, Hang Shao, Jian Li, Jinlong Peng, Haoyu Cao. Ke Li, Rongrong Ji, and Xing Sun. Vita-audio: Fast interleaved cross-modal token generation for efficient large speech-language model. CoRR, fbs/2505.007399 2025. doi: 48855///AXX.22555.0000 1550/arXiv.25055 03739
URL https:://6i..or//10...
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. http::://peeneeieeeeeeeee//ooomm
OpenBMB MiniCPM-o Team. Miniepm-o 2.6: A gpt-4o level milm for vision, speech, and multimodal live streaming on your phone, 2025. BRKLhhtttss:/:ppebbmitt not ion. site/
MiniCPM 2-6 A GPT- 40-T-4o-120 Level MM for -or--oiiiiooo www.ssiio--Seeeee Speech and Multim odal Live- Seve--Streamnn on Your www.e-aoooeeeeeeeeeeeeeeee 135cdelb/4550022255666333333
Patrick K. O'Neill, Vitaly Lavrukhin, Somshubra Majumdar, Vahid Noroozi, Yuckai Zhang, Oleksii Kuchaiev, Jagadeesh Balam, Yuliya Dovzhenko, Keenan Freyberg, Michael D. Shulman, Boris Ginsburg, Shinji Watanabe, and Georg Kuesko. Spgispeech: 5, 000 hours of transcribed financial audio for fully formatted end-to-end speech recognition. In Hynek Hermansky, Honza Cernocký, Lukás Burget, Lori Lamel, Odette Scharenborg, and Petr Motlícek (eds.), 22nd Annual Conference of the International Speech Communication Association, Interspeech 2021, Brno, Czechia, August 30 September 3, 2021, pp. 1434-1438. ISCA, 2021. doi: 10.1143/INTEERSPEEHH.OOO URL https: /ooo wttp://......11//eeeeeeeee.e....
OpenAl. Hello GPT-4o, 2024b. URL
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19- 24, 2015, pp. 5206-5210. IEFF, 2015. doi: 10. DB.109//CSSSPP221577779000 URL httpss 7/doi //do/.org//1 10. 1109/ ICASSP 2015. 001.577788 7178964
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 527-536. Association for Computational Linguistics, 2019. doi: 10.18653/V1/P19--10 1050. URL htt /ttp:::////oiooo wrw.:/o...rr....0 8653/v661vvvv
050
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URL
3/abs/ 04356
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. https:://proces http://eeee..anao.......sseaa//a 2ttps///w222/rraao2o3.3.....
PMLR, 2023. URL
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023. New Orleans, LA, USA. December 10- 16, 2023, 2023. URL http: httptt:::/ppppaeeeeeeeeee www.aaprrr..iipppopmmm http::/ppwppppppaaaaaaaaaa/a///p 74
html
13

---

Published as a conference paper at ICLR 2026
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. In Christine Cuicchi. Irene Qualters, and William T. Kramer (eds.), Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, pp. 20. IEEE/ACM, 2020. doi: 10.1109/SC41405.5 20..4000022200222 URL httpst //doi.org/// 1009/1141401..... 202C 2002 00024
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. UTMOS: utokyo-sarulab system for voicemos challenge 2022. Hanseok Ko and John H. L. Hansen (eds.), 23rd Anmual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022 pp. 4521 4525. ISCA, 2022. doi: (0.2437/NNERSSEEEHHOOO er erspeech. 2022-439
URL /ttp::///.....g//...... 21..0.277./...13371111
Changhan Wang, Morgane Rívière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Miguel Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation learning. semi-supervised learning and interpretation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joins Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 993-1003. Association for Computational Linguistics, 2021. doi: 10.18653/V 1TT////222122222 ACL-LLON
-aac--1on .aal-llogg.0 80
Qichao Wang. Ziqiao Meng, Wenqian Cui, Yifei Zhang, Pengcheng Wu, Bingzhe Wu, Zibin Zheng, Irwin King, Liang Chen, and Peilin Zhao. Parrot: Seamless spoken dialogue interaction with double-channel large language models, 2025. URI/htts:: /tllitttps::/peervvvieee wet.iiewwwwweeteeee.eeeeeee mortfforu??!
d=73EDGbG6mB
Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen LLM. CoRR, abs/2411.00774 2024. doi: 0044850/ARXXV.2411.2
URL https:://dot http///oi.org//...
8550 13500arxXiv. 2411. 00774
Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, Mingrui Chen, Peng Liu, Wang You, Xiangyu Tony Zhang, Xingyuan Li, Xuerui Yang, Yayue Deng, Yechang Huang, Yuxin Li, Yuxin Zhang, Zhao You, Brian Li, Changyi Wan, Hanpeng Hu, Jiangjie Zhen, Siyu Chen, Song Yuan, Xuelin Zhang, Yimin Jiang, Yu Zhou, Yuxiang Yang, Bingxin Li, Buyun Ma, Changhe Song, Dongqing Pang, Guoqiang Hu, Haiyang Sun, Kang An, Na Wang, Shuli Gao, Wei Ji, Wen Li, Wen Sun, Xuan Wen, Yong Ren, Yuankai Ma, Yufan Lu, Bin Wang, Bo Li, Changxin Miao, Che Liu, Chen Xu, Dapeng Shi, Dingyuan Hu, Donghang Wu, Enle Liu, Guanzhe Huang, Gulin Yan, Han Zhang, Nie Hao, Haonan Jia, Hongyu Zhou, Jianjian Sun, Jiaoren Wu, Jie Wu, Jie Yang. Jin Yang. Junzhe Lin, Kaixiang Li, Lei Yang, Liying Shi, Li Zhou, Longlong Gu, Ming Li, Mingliang Li, Mingxiao Li, Nan Wu. Qi Han, Qinyuan Tan, Shaoliang Pang, Shengjie Fan, Siqi Liu, Tiancheng Cao, Wanying Lu, Wenqing He, Wuxun Xie, Xu Zhao, Xueqi Li, Yanbo Yu, Yang Yang, Yi Liu, Yifan Lu, Yilei Wang, Yuanhao Ding, Yuanwei Liang, Yuanwei Lu, Yuchu Luo, Yuhe Yin, Yumeng Zhan, and Yuxiang Zhang Step-audio 2 technical report. CoRR, 0bbb/2507.116322 2025. doi: URL httph: /ww. /tt.://:///.....00..45555/////// h 2507. 16632
2557
Shitao Xiao, Zheng Liu, Peitian Zhang, and Xingrun Xing. Lm-cocktail: Resilient tuning of language models via model merging. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 2478. Association for Computational L Linguistics, 2024. doi: 10. 8653W11200A.FFIINNSSSSACC /ttp:///.......///............/.
URL htw.lttpss:.. ottp://://///.../0/0000....
2024. fo44ffindingg acl. 145
LLM-Core-Team Xiaomi. Mimo-audio: Audio language models are few-shot learners, 2025. URL https://g com x aomí MiMM/MM Mo-Audio
14

---

Published as a conference paper at ICLR 2026
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in
streaming. CoRR, abs/2408.16725. 2024a. doi: 1048500ARXXV22088.6222 URL https?
//doi.org//// /wt//555///iiii.. 1672 16725
Zhifei Xie and Changqiao Wu. Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities. CoRR, abs/2410.11190 2024b. doi: 10.48550/ARXIV224100 1410.111990 1900. URL
https://dol: wrg/000....g..... 48550 arxiv. 24110.1119
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report. CoRR, 021/2503.20215 2025. doi: (044850/ARXIV.2503..0 250..5003220 20215. URL httpss:/ddn www./ogg.o......oo.ooo.oooo..o.o 48550/5550000 htt.varxiivv.. 2003.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint a4Xiv:2408. 01000, 2024.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong, and Jie Tang. Scaling speech-text pre-training with synthetic interleaved data. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. net/ wttp://ommmmmddkuummaa
OpenReview.net, 2025. URL
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 15757-15773. Association for Computational Linguistics, 2023a. doi: 10.18665/V1/222333 1055. /attsp::: org 10. 18653/8633//// 20233 2023. f www.iiing-emmnn 1055
EMNLP.
Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang. Siqi Zheng, Jiaqing Liu, Hai Yu, and Chaohong Tan. Omniflatten: An end-to-end GPT model for seamless voice conversation. CoRR, abs/2410. 17799, 2024a. doi: 1144555AAXII.2211.11777 10.
URL_httss://doi.ooo
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. CoRR, abs/2308. 1692,, 2023b. doi: 10.48550/ARX
URL http::::p///.......t..lpt //doi. wrw. 4tw5:/10..45550/x111111 2308 16692
Xin Zhang, Xiang Lyu, Zhihao Du, Qian Chen, Dong Zhang, Hangrui Hu, Chaohong Tan, Tianyu Zhao, Yuxuan Wang, Bin Zhang, Heng Lu, Yaqian Zhou, and Xipeng Qiu. Intrinsicvoice: Empowering lims with intrinsic real-time voice interaction abilities. CoRR, abs/2410.08035 2024b. doi: 10.48550/ARXIV.. 241000005.. (ttps:t:s: .otg/1.. .2410. 08035
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization. CORR, abs/2507. 18071, 2025. doi: 10.48550/ARXIV. 22.50///XX...2....111 18071. //
15

---

Published as a conference paper at ICLR 2026
Appendices
A IMPLEMENTATION DETAILS
Qwen2.5-0.2 is trained following 72M paradigm with Speech-text Alignment data to initialize SRH (SRH-PT). The maximum sequence length duration of approximately 6.8 minutes. DRVOICE uses Qwen2.5-7B- ----tttruetttt as its base LLM, while DRVOICE-Sman utilizes Owen2.5-1..B-IInss
2K tokens, which corresponds an audio
as the base LLM. Grouping factor is Core-cocktail interpolated factor is set to a = 0 for extremely maintaining the base LLM capability. and 1. The warmup rate is set to 2% of the total training steps. For the two-stage training of DRVOICE. the learning rate is decayed from 1 x 10-1 to 1 X 10-5 in stage one, and subsequently from x 10- to 2 x 10-6 in stage two, with both stages utilizing cosine annealing schedule. In contrast, DRVOICE-Smie
Multiple loss hyperparameters are set
and SRH-PT are trained in single stage, which adopts the same learning rate schedule as the first stage of DRVOICE. The AdamW method (Loshchilov & Hutter 2019) is used for optimization. Experiments
64x
NVIDIA Tesla A800 80G GPUs with Brain floating-point format (BF16) (Kalamkar et al. 2019) accelerate training and decoding. For DRVOICE, DeepSpeed ZeRO-2 (Rajbhandari al. 2020)
is implemented to prevent GPU out-of-memory. It takes about 20 hours on SRH-PT, and about 45 hours on DRVOICE post-training.
B PROMPT TEMPLATES
System prompts for multimodal interaction patterns. As shown in Table there are five types of system prompts categorized based on the model's output partitioning. During inference, specifying these system prompts enables the generation of desired output results.
Table 5: System Prompts for Multimodal Interaction.
Pattern Abbr. System Prompts
S2M & T2M
You are helpful assistant and asked to generate both text and speech tokens at the time.
S2T & T2T
You are helpful assistant and asked generate text tokens.
STC
You are a helpful assistant. Let's think step by step. Convert speech to text if the query is speech, think of an appropriate text response, and then convert the response back to both text and speech tokens at the same time.
SAC
You are helpful assistant. Let's think step by step. Think of an appropriate text response, and then convert the response back to both text and speech tokens the same time.
SUC
You are helpful assistant. Let's think step by step. Convert speech text if the query is speech, and then think of both appropriate text and speech responses at the same time.
C MORE ABLATION AND ANALYSES
Core-Cocktall Training Strategy. Table 6 shows the precise trade-offs and benefits of our two- stage approach. For instance, under the comprehensive Inhouse v2, MagpiePro, InfGen dataset setting, the model's average performance after Stage 1 drops to 70.19, a significant decline of over 11 points compared to the text-only baseline (81.77). This observation perfectly aligns with our initial hypothesis: the aggressive full fine-tuning with high learning rate in Stage 1, while intended to rapidly move parameters into a more favorable region of the loss landscape, temporarily compromises the model's 's general capabilities, predicted. This performance degradation is precisely the problem that the subsequent steps are designed to rectify. Following the intermediate merging which "cocktails" the aggressively trained model with the base L.M to re-integrate its robust
16

---

Published as a conference paper at ICLR 2026
Table 6: Training Strategy Performance comparison the Voicebench benchmark with different training datascts.
Description
AlpacaEval
CommonEval
SD-QA
MMSU
OpenBookQA
IFEval
AdvBench
Avg
Qwen2.5-7B I-sttutttt (T2T)
4.67
4.34
76.13
69.97
82.20
65.45
99.04
81.86
Inhouse vl
OnlyText- Text Text-Training (T2T)
4.15
3.57
60.15
$4,50
66.50
50.22
99.40
69.31
Stage 2 (S2T)
4.08
3,54
55.88
52.80
69.45
38.48
99.23
66.89
Inhouse v2
OnlyText- Text- Training (T2T)
4.22
3.60
68.35
58.10
68.13
65.69
99.62
73.76
Stage (S2T)
4.10
3.15
64.01
58.30
79.56
55.25
98.27
71.48
Inhouse v2, MagpiePro, InfGen
OnlyText- Text- Training (T2T)
4.64
4.26
71.79
69.97
84.40
68.84
99.42
81.77
Stage (S2T)
4.25
3.09
57.32
54.29
75.82
$8.07
99.04
70.19
Stage 2 (S2T)
4.54
3.35
64.56
61.61
80.44
59.83
98.85
74.73
Stage ASR datu (S2T)
4.52
3.77
68.54
60.31
79.56
59.30
98.65
76.02
3500
Model Configuration
1.5B, Continuous Audio Encoder: Enabled
3000
1.5B, Continuous Audio Encoder: Disabled
78, Continuous Audio Encoder: Enabled
2400
78, Continuous Audio Encoder: Disabled
GPU GPU Hours
2000
1820
1520
1424
1248
1000
1120
1088
100B
1056
592
560
562
500
Group
Figure 2: Computational Resources under 17K hours training data across different Grouping Factor.
knowledge-Stage 2 proceeds with careful fine-tuning using small learning rate. As shown in the table, this second stage successfully recovers and refines the model, boosting the average score significantly to 74.73. This substantial improvement from Stage 1 demonstrates that our strategy effectively mitigates the instability caused by the initial high-learning-rate training. This two-stage process confirms that the Core-Cocktail strategy provides powerful solution to the learning rate dilemma, enabling rapid adaptation without sacrificing final performance.
Data Quality. The OnlyText-Tn -raining (T2T) results in Table|o]serve maximum potential embedded within the textual content of our datasets. There is a clear correlation between data quality and the final model's performance: Inhouse v (69.31) to our most comprehensive dataset (81.77), the final Stage (S2T) model performance also sees a substantial lift. Notably, our final dataset mixture enables the T2T model to reach an average score of 81.77, almost perfectly matching the text-only S- backbone. This confirms the exceptional quality of our instructional data and establishes high- performance ceiling. Our final S2T model (74.73) successfully translates significant portion of this potential into the multimodal domain.
powerful indicator of the
the T2T average score improves from
Furthermore, the results underscore the critical importance of data composition for specific tasks. The CommonEyal benchmark, which evaluates models on real human speech, a case in point. Our of 3.35 on this challenging benchmark. However,
standard Stage (S2T) model achieves by augmenting the training data with ASR derived pairs (Stage 2 w/. ASR data). the CommonEval score noticeably increases 3.77. This gain is because ASR data exposes the model the natural
17

---

Published as a conference paper at ICLR 2026
S3M (5)
T2M (9)
82T
Purforanno
60
50
45
40
3.K
AK
17.8K
Speech to Sp-echhh Dataset Siza
Figure 3: Performance Scaling of DRVOICE-Sman (w/o. Continuous Speech Encoder) on LLaMA Question Benchmark.
disfluencies, varied intonations, and ambient noise present in real-world speech, enhancing its robustness.
SRII-Pretraining Policy. Two pre-training strategies were employed for the Speech Refined Head (SRH). The first strategy involves independently training Text-to-Speech (TTS) task the Qwen2. 5-0.50 5B model in T2M format with 25Hz token rate; the resulting trained weights are then loaded onto the SRH. In contrast, the second strategy freezes the Speech Encoder. Adaptor, Shared L.LM Layer, and the Text Head, while exclusively training the SRH with streaming TTS objective. Experiments demonstrated that the second strategy yields a substantial improvement in Speech-to-Speech (S2S) performance on the U1 t raEva l benchmark, with the score increasing from 47.66 to 56.66. This significant gain is attributed to the closer alignment between the pre-training task in the second strategy and the format of the downstream task.
Reinforcement learning. To further enhance the model's capabilities in real-world speech comprehension, instruction following, and reasoning, reinforcement learning (RL) methods were employed. Specifically, to address the scarcity of authentic speech data, a preference dataset was constructed using open-source ASR datasets. A text L.M used to identify samples suitable for user queries, for which canonical responses were then generated by another text LLM. Subsequently. the DrVoice- SFT model performed inference the original speech queries establish the preference data. For the eestruction-follovvii capability, the open-source preference dataset was utilized. Training for these first objectives was performed using the DPO (Rafailov al. algorithm. Finally, to improve reasoning ability, the model underwent GSPO (Zheng et al. training on mathematical problems
2023)
2025
Data Scaling, To investigate the impact of data scaling, we conduct experiments by progressively expanding the S2S training data on DRVOICE-Small (w/o. Continous Speech Encoder). Each $2S instance can be augmented into 7 distinct multimodal interaction patterns. As illustrated in Figure 3 expanding the dataset from 3.6K to 17.8K samples yields consistent improvements. The performance of T2M(S) exhibits nearly linear growth, while other patterns, despite showing slower growth rates, still demonstrate upward trends the figure, suggesting potential for further performance enhancement through additional data.
Grouping Factor. Table 7 and Figure 2 demonstrate that our grouping strategy significantly enhances both performance and computational efficiency. Contrary to degrading generation,
/tu wtw.uuuuuueeeeee www------e----aaaoeeee instruct
on Fol FFlllwwwr cwing
thub. http://///tttttceco////o////agaa
18

---

Published as a conference paper at ICLR 2026
Table 7: Impact of the Grouping Factor Llama Q. S2T and S2M performance, using DRVOICE- Small (1.5B) w/o CSE and data subsets for faster experiments. The S2T model is trained exclusively on mixed S2T and T2T data, while the S2M model is trained on mixed S2M and T2T data.
Grouping Factor
S2T
S2M (T/S)
1
55.67
4.00 2.67
3
64.67
15.67 5.00
5
63.33
37.67 28.00
7
62.67
36.00 16.67
Table 8: Benchmark Results: VoiceBench (T2T)
VoiceBench
Data
AlpacaEval
CommonEvva
SD-QA
MMSU
OBQA
Overall
Qwen2.5-7B-Instte
4.67
4.34
76.13
69.97
82.20
65.45
99.04
81.86
Inf7M
3.83
3.50
69.98
55.43
71.87
48.79
94.42
69.58
InfGen
4.51
4.13
74.14
62.46
76.04
58.35
77.46
openorcu__gee
4.33
4.03
73.96
61
73.85
54,05
96.35
75.20
openorcu__ps gpt__5
4.35
4.17
71.07
$7.84
66.59
49,39
95.00
72.90
Evol Instruct__et Code
Code
69.26
EvoL_Instruc
4,09
3.91
71.61
60.38
72.31
54,02
98.27
73,80
magpie_pmi Ilama3 1 300k
4.42
4.08
72.69
58.33
74.95
56.08
86.73
74.11
magpie_pro prop_poo_mm___ hrro_mt_iig 3tt_lllma311_____11 1_300k 300k
4.46
4.05
73.60
58.07
60.00
60.05
86.35
72.61
numinamath col OpenHermes__2 v2_
4.11
3.78
70.16
$5.43
68.35
52.00
97.69
71.63
4.05
3.71
71.61
74.07
71.08
Syathia_vI_ v1_3
4.30
3.96
72.33
63.63
76.70
53.32
97.69
75.55
ull, wlu_3_sft
4.04
3.73
65.64
61.74
79.78
61.78
100.00
74.91
magpie_ pro prrp_poo_laaaaaaa Ilama3
500k filtered
4.48
4.17
73.42
64.31
78.53
grouping substantially improves both speech understanding (S2T) and speech-to-specch generation (S2M). For instance, increasing the grouping factor from 1 to 5 raises the S2T score from 55.67 to 63.33 (a 13.7% relative gain). The S2M (T/S) score sees a significant improvement, peaking at 37.67 / 28.00 with grouping factor of 5. Furthermore, shown Figure 2
using a grouping factor of 5 instead of reduces nearly 50% GPU hours in each setting, proving the efficiency of grouping machinism.
Retention vs Forgetting, An attempt is made to select suitable open-source datasets for speech synthesis. All results are from fine-tuning Qwen2.5-7B-Insteee
low learning rate (Ie-5), so they primarily indicate retention vs. forgetting rather than absolute capability gains. The gap to the baseline quantifies forgetting: magpie pro llama3 3 500k filtered and InfGen exhibit the smallest overall deltas (about 3-4 points on both suites), while very large, noisier corpora like Inf7M induce substantial forgetting despite their size. Quality and filtering matter more than scale: openorca_gpte forgets less than openorca_ _____p33__55555 with fewer tokens, and the magpie_pro family maintains instruction-followinn and factual QA best per token. Forgetting is selective; some submetrics improve (c.g., LlamaQ rises for Ini7M/tulu sft; AdvBench reaches 100 for tulu (_sf), but these gains often trade off against instruction and QA scores. Overall, compact, well-filtered, model-proximal data minimizes destructive drift 1e-5, whereas bulk token count tends to amplify forgetting, the most stable recipes preserve the base model while making targeted adjustments.
D LIMITATIONS AND FUTURE WORK
Our future work will address limitations of this work and advance two key areas: (1) Enabling Full-Duplex Interaction: A crucial direction is to enable full-duplex interaction for more natural conversations. Inspired by recent advancements like Parrot (Wang et al. (025). future work will investigate the use of a time-division multiplexing (TDM) input stream. This would permit DRVOICE to accept user speech inputs during its own speech generation phase, thereby efficiently utilizing idle time slots and allowing for responsive, interruptible dialogue. (2) Expanding to General Audio and Multimodality: Finally, a promising avenue is to extend the model's 's capabilities beyond speech-centric tasks broader audio comprehension and synthesis,
19

---

Published as a conference paper at ICLR 2026
Table 9: Benchmark Results: UltraEval-Audio (T2T) & OpenAudioBench (T2T)
Tokens (M)
Eltra-vval-uute
OpenAuuto
Total
AlpacaEval
LlamaQ
TriviaQA
WebQ
Overall
Reasoning QA
Avg
Qwen2.5--7 1oan2.5-7BInsttrae
81.41
82.33
53.91
$2.56
67.55
58,00
75.10
55.35
84.67
48.51
64.13
InfGen
1022.4
74.29
82.33
55.96
51.18
65.94
57.43
71.95
openorca_gpo gpt4
361.8
66.21
83.00
54.39
49.56
63.29
54.46
69.50
openorca_gpp spt3_5
1101
55.91
83.00
54.69
48.47
60.52
48.51
66.74
Evol Instric__oode
28.3
$2.32
81.33
52.64
44.24
$7.63
52.48
67,39
Evol Instruct
56.62
53.81
47.15
59.81
53.47
67.44
magpie_peo [wo_Ilamaa__ 1amm3_1_330 300k
213.9
76.16
83.67
57.91
55.02
68.19
53,96
70.46
jagple_pro_mt, pwt__m____ieeeeeeee.
_ww_a____laaaa____________ Ilaaa 1330kk
323.2
75.35
83.00
54.00
54,53
66.72
57.43
69.38
numinamath_com cot
463.2
64.04
80.67
54.88
50.64
62.56
50,00
66.81
OpenHermos__22 v2_5
375.9
58.54
83.67
54,49
49.85
61.64
54.46
66.55
Synthia v1_
61.67
83.67
47.88
61.76
55.45
69.28
iulu___i
570.2
55.91
84.33
53.52
49.41
60.79
55.94
68.62
magpie pro__llmaa 3_50kkk Hitered
78.64
57.32
54.18
72.53
including the recognition and generation of music and environmental sounds. The subsequent integration of the visual modality will be critical step toward developing a more comprehensive, multimodal conversational AI.
E THE USE OF LARGE LANGUAGE MODELS
During the preparation this work, the authors used Large Language Models (LLMs) improve grammar, clarity, and overall readability. The LLM was used as writing aid and for language polishing purposes. The authors reviewed, edited, and take full responsibility for the final content and all claims presented in this manuscript.
20


Reviews:
[{"review_id": "dHUj1uB8pZ", "reviewer": "Reviewer_6YdV", "summary": "This paper introduces DrVoice, a parallel speech-text voice conversation model built upon a large language model (LLM). The core contribution is a novel Dual-Resolution Speech Representation (DRSR) mechanism designed to address the temporal resolution mismatch between speech and text tokens in end-to-end (E2E) spoken dialogue systems. Specifically, the model groups higher-frequency (25Hz) input speech tokens into lower-frequency (5Hz) representations before feeding them to the LLM, reducing computational cost and aligning the speech processing rate more closely with that of text. The authors also propose two training strategies: a CoM-Mixing strategy to train the model on various interaction patterns (e.g., text-only, speech+text), and a Core-Cocktail training strategy to mitigate catastrophic forgetting of the base LLM's capabilities. Experimental results show that DrVoice-7B achieves good performance on OpenAudioBench and Big Bench Audio, and is competitive on other benchmarks like VoiceBench and UltraEval-Audio, establishing it as a strong open-source model in its size class.", "questions": "1. Could you clarify the training process for the Speech Refined Head (SRH)? Is the SRH jointly pre-trained with the LLM backbone, allowing gradients from the SRH loss to propagate back to the LLM? Or is the SRH trained separately, merely using the frozen LLM's hidden states as input? Have you explored both strategies, and what is their comparative impact on performance?\n\n2. In the ablation study presented in Table 4, you analyze the impact of removing the Continuous Speech Encoder (CSE) and the Speech Refined Head (SRH). Could you please specify the alternative mechanisms used in these ablated models? For the \"w/o. CSE\" setting, how are the raw speech inputs processed and fed into the LLM? For the \"w/o. SRH\" setting, how are the speech output tokens generated from the shared LLM layer's hidden states? Is a simpler projection layer used instead?\n\n3. Table 3 shows that DrVoice has a relatively high ASR-WER of 11.2, which is significantly worse than top-performing models like Qwen2.5-Omni (3.48). This suggests a potential weakness in speech-text alignment or intelligibility. How do you interpret this result, and what specific architectural or training factors do you believe contribute to this performance gap?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The central idea of using dual-resolution speech representations is well-motivated and addresses a key, practical problem in joint speech-text modeling: the significant discrepancy in token rates between speech and text. The proposed grouping/ungrouping mechanism is an elegant solution that directly tackles this issue, leading to significant computational efficiency gains (processing at 5Hz within the LLM) without sacrificing output quality, thanks to the Speech Refined Head.\n\n2. The paper presents a comprehensive set of experiments on multiple challenging benchmarks. DrVoice achieves SOTA results on two benchmarks (OpenAudioBench, Big Bench Audio) and demonstrates performance on par with other leading models on VoiceBench and UltraEval-Audio. This strong and balanced performance across tasks involving both understanding and generation solidifies the effectiveness of the proposed architecture.\n\n3. The reduction of the LLM's operating frame rate from the common 12.5Hz or 25Hz to 5Hz is a major practical advantage. As shown in the ablation studies (Figure 2 in appendix), this leads to a nearly 50% reduction in GPU hours for training. This makes training and deploying such powerful spoken language models more feasible and is a valuable contribution to the field.\n\n4. The proposed training strategies, CoM-Mixing and Core-Cocktail, are thoughtful additions. CoM-Mixing provides a structured curriculum for learning complex, multi-stage reasoning (think-then-speak), while the Core-Cocktail method offers a pragmatic approach to balance new learning with knowledge retention from the base LLM. The ablation studies effectively demonstrate the positive impact of these techniques.", "weaknesses": "1. The paper notes that DrVoice's ASR-WER (11.2) is higher than that of Qwen2.5-Omni (3.48), suggesting weaker text-speech alignment. The authors hypothesize this is because Qwen2.5-Omni feeds text directly into its \"Talker\" module, while DrVoice only uses hidden states. This is a crucial architectural trade-off. While the proposed solution (adding text as input to SRH) is mentioned as future work, the current limitation is significant. High ASR-WER can indicate issues with intelligibility or word-level alignment, which is a critical aspect of a voice conversation model.\n\n2. A large portion of the training data (3B assistant tokens) is synthesized using CosyVoice. While the paper uses WER filtering, training on synthetic data can introduce biases from the teacher model and may not fully capture the diversity and nuances of real human speech. The paper could benefit from a discussion on the potential limitations of this approach and how it might affect the model's performance on in-the-wild, real-world conversations.\n\n3. While the dual-resolution mechanism is a clever and effective engineering contribution, the overall architectural design of DrVoice bears a strong resemblance to existing models, particularly the Qwen-Omni series. The core \"Thinker-Talker\" paradigm, where a central LLM (\"Thinker\") generates intermediate representations that are then passed to a separate speech synthesis module (\"Talker\"), is conceptually very similar to DrVoice's structure of a shared LLM layer feeding into a dedicated Speech Refined Head (SRH).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XgIxA4irAW", "reviewer": "Reviewer_EyEh", "summary": "The author proposed a 7B speech text voice conversion model, DrVoice, that leverages joint autoregressive modeling and dual speech representations. \n\nThe paper contributed on both model architectures and the training strategies. The model introduces a mechanism to alleviates the temporal resolution mismatch between text and speech tokens and reduce the computational cost. Specifically, it applied several grouping/ungrouping techniques to map 25 Hz audio tokens to 5Hz speech tokens and combined it with text tokens. Also, it uses a speech refined head(SRH) to preserve the intermediate semantic and acoustic representation during token generation. On the training side, two training strategies are applied: chain-of-modality(CoM) mix training for generating structured thoughts being used as language model text prompt, and a two-stage training method called Core-Cocktail Training for refining model's performance during training with large learning rate.\n\nThe model was trained on 100k hours of audio-tex paried data and evaluated with comprehensive benchmark metrics comparing to multiple baselines including previous SOTA. The author claimed their model achieved state of the art on OpenAudioBench and Big Bench Audio benchmarks, and performance comparable to SOTA on VoiceBench and UltraEval-audio benchmarks.", "questions": "1. Does the ungrouping(linear + split) and SRH cause any increased computational cost? How does it compare to the effect of reducing temporal resolution?\n2. From the ablation study, the ungrouping and SRH contributes more to S2M and T2M, while it doesn't help much with T2T. Is SRH making redundancy for T2T tasks?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The model extended existing parallel joint speech-text model by the new methods below:\n1. The author contributed an idea of reducing the temporal resolution of extracted audio representations from 25Hz to 5Hz to match the 3Hz text representation using speech token grouping. This benefits to reduce the computational costs and alleviate the frequency discrepancy between different representations. Leveraging the fine-grained acoustic information during generative scenarios is intuitive. The evaluation shows the speech quality is not compromised. \n2. The author proposed to use both CoM-mixing and Core-Cocktail Training for model performance during training phase. The chain-of-modality mixing training allows the model to summarize intermediate textual transcriptions before feeding into LLM. The Core-Cocktail Training contributes to model training with large learning rate while preserving performance.\n3. The SRH is proved useful for speech generation in the ablation study.", "weaknesses": "1. Although the grouping mechanism helps with reducing the temporal resolution and alleviate two representations, the ungrouping and the Speech Refined Head is making the model architecture more complicated and may worsen the model's speed due to its auto-regressive nature.\n2. In the benchmark results, the model doesn't beat all the baselines in OpenAudioBench. The author claimed that the average score is the highest, but it is also true that it is not showing comparable perforamance on some tasks such as Reasoning QA(57.92 vs 63.76 Qwen2.5). There are not enough evidence showing that DrVoice is a new SOTA on openAudioBench. \n3. Similarly, for VoiceBench and UltraEval-Audio, DrVoice only shows best perforamance on 1 or 2 tasks. While Kimi-Audio as a former SOTA still dominates on 5 out of 8 tasks. It could be better to discuss the performance comparison more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GkBaKpNDGh", "reviewer": "Reviewer_mDyG", "summary": "This paper proposes the use of dual speech representations (both continuous and discrete representations) and their grouping (to reduce the token rate and reduce the computation) as input along with the text tokens to an LLM and then at the output first a text head produces the text tokens which helps guide the speech head that produces the ungrouped sequence of speech tokens (referred to as Chain of Modalities). The output speech tokens are then converted into mel features via a flow matching model and a HiFiGAN vocoder is used to convert the mel features into the wave files. Hence, the system can take both speech and text as input and can produce output in both text and speech format (or a combination of the two). Based on the input output types, the paper describes 7 types of modes for the model which can be switched with a predetermined prompt. Another training detail that the paper emphasizes is that the importance of linearly mixing the base LLM's weights with that of the multimodal tuned model. Experiments on various speech+text benchmarks competitive performance to the several other multimodal LLMs including Qwen2.5-Omni  and Kimi-Audio. In addition, the paper claims to achieve the SOTA numbers on the Big Bench Audio tasks among the LLMs tested. In terms of generated speech quality, UTMOS score suggest that it is high quality audio even though the WER evaluation indicates some potential issues with the generated speech.", "questions": "1. Even though one of the major claims of the paper is the computational cost reduction by grouping of audio tokens to reduce the token rate, the presented experiments do not provide the details of that improvement other than the token rate. What would be the computational savings in terms of FLOPs of RTF (whenever applicable)? \n\n2. Although the appendices contain some details, the main text seems to be glossing over the details. Do the authors think that the manuscripts would require a longer paper? \n\n3. Based on the results in Table 2, the proposed DrVoice performs similarly to the other baselines in the table for all benchmarks except the Big Bench tasks. That is the relative differences are rather small between models. But for the Big Bench tasks we see a very large improvement (about 20% as compared to the Kimi Audio (the 2nd best)). Do the authors have any explanation of why DrVoice performs this well on these two tasks? Please comment on that.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "originality\n* Use of both continuous and discrete representations of audio simultaneously is relatively less investigated in multimodal LLMs. Chain of modality and model weight mixing are presented as novelties which are somewhat novel. Especially, model averaging by linear combination is not particularly new but its application to LLMs during training might be an experimental design novelty.\n\nquality\n* The experimental results show competitive performance and the paper claims the SOTA results on the Big Bench Audio speech-to-text and speech-to-speech tasks.\n\nclarity\n* Mostly clearly written. However, some details are left to the Appendices.\n\nsignificance: \n* Successful implementation of multimodal LLMs is a relatively popular topic nowadays and this paper would be relevant to that community.", "weaknesses": "* The novelty is limited but sufficient (see the comments in the strengths section) Most of the individual components are well known (audio encoders, audio decoders, the LLM backbone, etc.) \n\n* The main text skipped many details and provided those details in the Appendices. Which makes the reviewer questioning whether the paper needed more number of pages and a conference setting is not suitable for this paper.\n\n* Even though one of the major claims of the paper is the computational cost reduction by grouping of audio tokens to reduce the token rate, the presented experiments do not provide the details of that improvement other than the token rate. What would be the computational savings in terms of FLOPs of RTF (whenever applicable)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)