Towards True Speech-to-Speech Models Without Text Guidance
Authors: Xingjian Zhao
Zhe Xu
Luozhijie Jin
Yang Wang
Hanfu Chen
Yaozhou Jiang
Ke Chen
Ruixiao Li
Mingshu Chen
Ruiming Wang
Wenbo Zhang
Qinyuan Cheng
Zhaoye Fei
Shimin Li
Xipeng Qiu
Venue: ICLR 2026 Poster (2026)
Keywords: multimodal large language model
large language model
speech language model
Primary area: foundation or frontier models, including LLMs
TL;DR: We present a true speech-to-speech LLM that understands and generates speech directly, without text intermediates, achieving state-of-the-art spoken QA.

Abstract: Spoken dialogue systems often rely on cascaded pipelines that transcribe, process, and resynthesize speech. While effective, this design discards paralinguistic cues and limits expressivity. Recent end-to-end methods reduce latency and better preserve these cues, yet still rely on text intermediates, creating a fundamental bottleneck. We present a true speech-to-speech large language model that directly understands and generates speech without relying on text guidance. Our approach combines a modality-based layer-splitting architecture with a frozen pre-training strategy, preserving the reasoning and knowledge of pretrained text LLMs while adding native speech capabilities. Experiments show that our model achieves state-of-the-art results in spoken question answering and delivers comparable speech-to-speech performance relative to existing text-guided systems, while still maintaining competitive text performance. By narrowing the gap between text-guided and direct speech generation, our work establishes a new paradigm for expressive and efficient end-to-end speech interaction. We will release our code and models to support further research in true speech-to-speech foundation models.

Published as a conference paper at ICLR 2026
TOWARDS TRUE SPEECH-TO-SPEECH MODELS WITH- OUT TEXT GUIDANCE
Xingjian Zhao 133- Zhe Xu Luozhijie Jin 1223 Yang Wang! 1.3
Hanfu Chen Yaozhou Jiang 3. Ke Chen 1,2,3 Ruixiao Li
Mingshu Chen
Ruiming Wangl Wenbo Zhangl.2.3
Qinyuan Chengl
Zhaoye Fef2.3 Shimin Li Xipeng Q231.2.31
Fudan University
2 Shanghai Innovation Institute
MOSLAI
ABSTRACT
Spoken dialogue systems often rely on cascaded pipelines that transcribe, pro- cess, and resynthesize speech. While effective, this design discards paralinguistic cues and limits expressivity. Recent end-to-end methods reduce latency and bet- ter preserve these cues, yet still rely on text intermediates, creating fundamental bottleneck. We present true rectly understands and generates speech without relying text guidance. Our approach combines modality-based layer-splittngg architecture with frozen pre-training strategy, preserving the reasoning and knowledge of pretrained text LLMs while adding native speech capabilities. Experiments show that our model achieves state-of -f-the--rt results in spoken question answering and delivers compa- rable speech-to-speede performance relative to existing text-guided systems, while still maintaining competitive text performance. By narrowing the gap between text-guided and direct speech generation, our work establishes new paradigm for expressive and efficient end-to-end speech interaction. We will release our code and models to support further research in true speech-to-speech foundation models.
large language model that di-
INTRODUCTION
Speech is one of the most natural and intuitive modalities for human-computer interaction, mak- ing spoken dialogue systems central focus of contemporary AI research. Traditional systems for spoken interaction are typically implemented using a cascaded pipeline: speech input is first tran- scribed into text, a text-based large language model (LLM) generates a response, and the output is subsequently converted into audio through text-to-speech (TTS) module (Figure [T].. While this architecture leverages the full reasoning capacity of text-based L.LMs, it inevitably discards infor- mation encoded in the original speech signal and constrains the system to produce only responses that can be faithfully represented in text.
Early end-to-end attempts such GSLM (Lakhotia et al., 2021 and AudioLM (Borsos et al. 2023)
demonstrated that speech could be modeled directly, but these works remained largely confined to experimental dialogue continuation tasks and faced challenges in scaling into full-featured assis- tants. Later work shifted toward text-guided generation as a compromise: SpeechGPT (Zhang et al., 2023a) used chain-of-modie moddaliyy design, while Moshi (Défossez et al.) (022)) and PSLM (Mitsui et al.] 2024) achieved low-latency streaming through parallel speech-tex: generation. GLM-4-Voice (Zeng et al. 2024b) advanced this further by interleaving text and speech in chunk-based generation (Figure [b), reaching near-text-level performance in streaming dialogue. Importantly, while GLM- 4- Vovce primarily relies text-guided responses, also supports direct speech generation-but its direct mode remains noticeably weaker than its text-guided counterpart.
"Equal contribution.
1 'Correspondng author. Email: zhaoxj2420 24mm. fudan.edu en, chenggy2 1 @ém. fudan. edu. er, xpqi u Fudan. edu. en.

---

Published as a conference paper at ICLR 2026
Decoder
Decoder
am helpful assistant.
I am
helpful assistant.
nin da
GODEa Andro
LLX
Text Text-guided Speech LLM
LLM
Hello! you?
Addo Toles
DOBOO Aedo Ideass
ASR
Tokenize
Tokenize
1/tx Hnoif
(a) Cascaded pipeline
(b) Text-guided speech models
(c) True
models
Figure 1: Paradigms for spoken dialogue modeling. (a) Cascaded pipelines rely ASR - LLM - TTS, discarding paralinguistic cues. (b) Text-guided speech models incorporate speech input but still depend text as intermediate during generation. (c) True speech-to-speech language models directly comprehend and produce speech, avoiding the text bottleneck.
By accepting speech directly as input, these approaches preserve paralinguistic cues such prosody, emphasis, and emotion. Yet their reliance on intermediate text during generation creates a fundamen- tal bottleneck: it introduces latency, reduces efficiency, and restricts expressivity, since non-verbal vocalizations (e.g.. laughter, hesitation) lack natural text equivalents. In addition, because of the in- herent gap between speech and text modalities, current methods often introduce speech capability at the expense of text ability, leading to measurable degradation in the backbone's 's text performance. For instance, SpiritLM(Nguuee 36.9 after incorporating speech modeling. Closing the gap between text-guided and direct speech generation is therefore critical for realizing true speech-to-spese interaction.
et al. 2024) shows notable drop in MMLU accuracy from 45.3 to
In this work, we introduce novel approach that enables large language models to natively model speech while largely retaining their text-based capabilities. Our method builds on pretrained text LLM backbone but diverges from prior approaches through modality-specifie layer-splitting scheme and a frozen pretraining strategy. This design preserves the backbone's linguistic knowledge while equipping the model with native speech understanding and generation abilities comparable to existing text-guided systems. As a result, our model can directly produce high-quality speech with- out relying intermediate text representations, establishing new paradigm for end-to-end speech- to-speech generation. Importantly, because the majority of knowledge remains in the pretrained text model, our approach avoids dependence large-scale, knowledge-intensive speech datasets. Instead, alignment transfers reasoning, world knowledge, and generalization abilities from the text backbone to the speech modality.
The main contributions of this paper are as follows:
large language model that achieves state-of-the: We present a true sreech-t--speech benchmarks without relying on any intermediate text art performance on speech-to-spee guidance. At the same time, the model natively supports both text and speech as input and output modalities, thereby narrowing the gap between spoken and written interaction.
We introduce modality-based layer-splitting and frozen pre-training that improves alignment between speech and text while mitigating the degradation of reasoning ability and world knowledge typically observed when extending LLMs to new modalities.
We conduct extensive experiments and ablation studies validate the effectiveness of our approach, demonstrating advanced speech-text cross-modal alignment and textual ability preservation.
2 MODEL ARCHITECTURE
For advancing toward true speech-to-spest large language model, add modality- bassed layer- splitting to an autoregressive Transformer, enabling deep fusion of heterogeneous modalities and modality-specific generation.
2

---

Published as a conference paper at ICLR 2026
353
(a) Layer 0
(b) Layer 10
(c) Layer 24
(d) Layer 27
0.70
Samplet
Similarity Score
0.65
Sample2
Sumple3
Samplet
0.60
Sumples
0.55
0.50
0.45
5
10
15 Layer Index
20
23
(c) Similarity score across layers
Figure 2: Visualization of the layer-wise similarity between speech and text representations. (a)-d) Cosine similarity heatmaps at representative layers (0, 10, 24 and 27) reveal how cross- modal alignment evolves across the model depth. The yellow dots are the points selected by DTW sampling based on similarity. It can be seen that the points selected by our evaluation method largely coincide with the points of high similarity. The whole cosine similarity figure of 28 layers will be posted in Appendix D.I (e) Similarity score across all layers on five samples shows progressive increase up to around layer 10, then there are slight fluctuations the subsequent 14 layers, followed by noticeable decline in the final layers. This trend indicates that speech and text representations become gradually fused in the lower-to-middae layers but diverge again at the top layers. Content of samples provided in Appendix D.2 More analyses across models provided in the Appendix D33
2.1 MODALITY-BASED LAYER SPLIT
For the Transformer backbone, design goal is preserve the original text capabilities of LLMs while augmenting LLMs with speech understanding and generation. Existing approaches typically rely on the Depth Transformer (Défossez al., 2024) that generates multiple VQ tokens sin- gle input, or alternatively, expand the vocabulary to directly encode speech tokens into the input sequence (Zeng al. 2024b). However, our preliminary study on ppeechgnt2-rreieew(Oeeee (OreriiewwOee--sss 2025), revealed that the hidden-see state alignment between a sentence and its corresponding speech se- quence gradually deteriorates in deeper layers: while strong diagonal similarity emerges in lower layers, it vanishes in later layers.
As shown in the Figure by examining the hidden-state similarity between the same spoken ut- terance and its corresponding text across different layers, we observe that in a 28-layer model, the similarity steadily increases in the first 11 layers, fluctuates and gradually stabilizes in the following 14 layers, and then decreases in the final 3 layers. This finding suggests that as the model is trained, the representations text and speech become increasingly fused within the first 25 Transformer blocks, but gradually diverge in the last four layers.
Motivated by this, we introduce a modality- based layer split at the 32nd block our 36-layer Trans- former. At this point, the shared hidden state is routed into mppelit- branches: branch continues through the final four layers to predict text tokens, while the other routes into a parallel four-layer stack to predict speech tokens.
This
design allows the model to leverage the first N layers for joint multimodal- ity fusion, while reserving the final layers for modality-specific generation. As result, the archi-
3

---

Published as a conference paper at ICLR 2026
Stage
Text LM Head
A Speech LM Head
Stage II
Text Head
Speech Head
& Text
A Speech
A Text
Speech
Transformer Layers
Transformer Layers
S Shared
Shared
/ Text Embedding
A Speech Embedding
Text Embedding
Speech
Embedding
Figure 3: Model architecture and training strategy. We split the trailing Transformer layers based on modality, and freeze the text backbone during Stage I pre-training Both branches are initialized from the same pretrained text model backbone.
tecture enhances cross-modality transfer, enabling the system inherit the capabilities of textually- pretrained LLMs and express it natively in the speech modality.
2.2 SPEECH TOKENIZATION
Our speech tokenizer is designed with four key objectives: (1) to achieve a single-codebook, low- bitrate representation for efficient autoregressive generation and simplified context management; (2) to maximize semantic content in order to facilitate knowledge transfer from text to speech; (3) to preserve sufficient paralinguistic detail to enable faithful reconstruction of human speech; and (4) to support full streaming operation for low-latency processing.
Encoder Discrete speech tokenizers commonly trained with reconstruction objectives (Gong et al. 2025 Zhang et al.] 2023c) or self-supervised discovery methods (Shon et al.) [0224] Liu et al, 2024 However, prior work has observed that tokens optimized primarily for reconstruction are often suboptimal for LLM learning (Défossez et al., 2024). To address this, and following the design of Cosy Voice 2 (Du al., 2024), we adopt automatic speech recognition (ASR) as the sole training objective for our tokenizer encoder. Our encoder is further trained based on the GLM-4- Voice Tokenizer (Zeng et al. 2024b), but we modify it to be fully causal rather than block-causal, thereby ensuring true streaming support.
Decoder For decoding, we adopt the flow-matching architecture introduced CosyVoice 2 (ip- man et al. 2022 Du et al. 2024). While Cosy Voice 2 employs chunk-attention to improve efficiency, this mechanism introduces undesirable time delays. To mitigate this issue, we compress the chunk size, which significantly reduces latency while maintaining reconstruction quality. This modifica- tion makes our tokenizer particularly well-suited for streaming dialogue systems that demand both high fidelity and low response delay.
TRAINING STRATEGY
3.1 PRE-TRAINING
The objective of pre-training is to introduce a speech modality into a pretrained text-based LLM while preserving its original text capabilities. To this end, we initialize our model from Qwen-3- 8B Yang et al. 2025) and adopt a two-stage pre-training strategy using a large-scale, high-quality speech corpus. The procedure is outlined below.
3.1.1 DATA COLLECTION AND PROCESSING
We begin with approximately 9 million hours of real-world audio data collected from the internet. To remove non-speech content, we apply custom voice activity detection (VAD) pipeline based on pyannote (Plaquet & Bredin, 2023 Bredin 2023), resulting in roughly 4 million hours of speech.
4

---

Published as a conference paper at ICLR 2026
Table 1: Statistics of pre-training data. Hours are shown in thousands (k).
Dataset
Total (h)
Real (h)
Synthetic (h)
English Interleaved
690k
624k
66k
Chinese Interleaved
952k
876k
76k
Unsupervised
2,303k
2,303k
0
These data are organized into two categories according to source type: (1) interleaved speech-text pre-training, drawn primarily from podcasts, and (2) unsupervised speech pre-training. drawn pri- marily from video content. Podcasts are chosen for interleaved pre-training because they typically provide cleaner recordings and clearer speech, enabling automatic speech recognition (ASR) sys- tems to generate more reliable transcripts. In contrast, video sources, while more diverse and noisier, are better suited for the unsupervised pre-training setting, where robustness to challenging acoustic conditions is essential.
For the interleaved task, first apply automatic speech recognition (ASR) to obtain text transcripts. Connectionist Temporal Classification (CTC) word alignment is then used to segment the audio into random-length chunks of 3-6 seconds. Each chunk contains either the corresponding audio segment or its transcribed text, and sequences are constructed by interleaving the two modalities. For unsupervised speech pre-training, we simply use full-length audio segments without transcript.
To mitigate the low knowledge density inherent in natural speech corpora, we also synthesize addi- tional interleaved data from high-quality text corpora. Following the approach of of et al. (2024c). we use Web--uu (Lozhkov et al.) (024) for English and Chinese FineWeb-Eod
( et al., 2025 for Chinese. These texts are converted into audio using the Voiye TTS system (Du et al.
2024), producing large-scale synthetic speech-tex pairs that enrich the training corpus.
summary of dataset statisties provided in Table
3.1.2
TWO-STAGE TRAINNNG
We initialize our model from the Qwen3-8B backbone and employ two-stage pre-training pipeline designed to introduce the speech modality while preserving the model's 's text capabilities.
Stage 1: Speech Alignment with Frozen Text Backbone In the first stage, we freeze all param-
eters of the Qwen-3-8B backbone and train only the newly introduced speech-related components, including the speech token embeddings, speech-specific transformer layers, and the speech language modeling (LM) head. This stage serves to initialize speech parameters and establish stable align- ment with the pretrained text representations. Training is conducted for approximately one epoch using the AdamW optimizer with cosine learning rate scheduling. The initial learning rate set to 4 x 10-4, with batch size of 2.2M tokens, weight decay of 0.1, context length of 14,336 tokens.
Stage 2: Joint Training with Text Knowledge Preservation In the second stage, we unfreeze larger portion of the model allow cross-modal adaptation. We experiment with three configu- rations: (1) unfreezing the entire model and training all parameters jointly, (2) unfreezing only the shared transformer layers while keeping the text embeddings, text-specific layers, and text LM head frozen, and (3) gradually unfreezing the shared layers in reverse order (from last to first). Since unfreezing text parameters risks degradation of textual abilities, we incorporate additional text-only pre-training data to preserve the model's linguistic competence. Specifically, we include FineWehr Edu (Lozhkov et al. 2024) for English and Chinese FineWeb-Fdu V2.1 Yuu at.. 2025) for Chinese, filtering entries with quality scores 2 3.
Stage 2 training is conducted for two epochs on the same speech dataset used in Stage 1, combined with 0.1 epoch of text-only pre-training data. Hyperparameters are largely consistent with Stage I, except that the learning rate is reduced (decaying from 6 x 10 to 6 x 10-6) and the batch size is increased to 2.8M tokens to account for the additional text data. In practice, the three configurations achieve comparable results. For simplicity, we adopt configuration (1) as the default initialization for subsequent supervised fine-tuning. A detailed ablation study is provided in section 5
5

---

Published as a conference paper at ICLR 2026
3.2
SUPERVISED FINE-TUNING
3.2.1 DATA ADAPTATION AND CONSTRUCTION
Because high-quality supervised fine-tuning data for speech assistants are scarce in natural settings, we construct such data synthetically. Our process begins with existing open-source text-based su- pervised fine-tuning datasets listed in Appendix B
Text Adaptation We employ the GPT-5 API to transform question-answer pairs into formats suitable for speech representation. This process involves (i) converting non-vocal content such as mathematical expressions, tables, or Markdown into TTS-compatible forms, and (ii) filtering out instances that cannot be effectively rendered speech (e.g.. long code dumps dense IЯТЕ pas- sages). Adaptation also improves data quality by shortening excessively long responses to make them more appropriate for spoken delivery, correcting obvious factual errors, and suggesting suitable emotional tones for TTS synthesis. Prompt for the adaptation process is available in Appendix C
Speech Synthesis The adapted text then synthesized into audio using multiple TTS systems. We primarily employ Seed-TTS (Anastassiou et al. (024) from VolcEngine. For the user role, we generate speech with diverse set of speaker voices improve robustness. For the assistant role, always use single consistent speaker to establish stable and recognizable system identity. To further enhance voice diversity, naturalness, and stylistic control, we additionally employ MOSS- TTSD Team 2025) to synthesize conversational datasets. By assigning different system prompts, we can vary the assistant's 's speaking style and role in a controllable manner.
Quality Filtering Although LLM-based TTS systems produce highly natural speech, they sus- ceptible to synthesis errors. To mitigate this, we apply automatic quality filtering using Sense Voice- Small ASR (An et al. 2024). Specifically, discard entries whose ASR transcripts exhibit word error rate (WER) 2 0.2 relative to the original text.
Statisties In total, we end up with over 1500k question-answer pairs for supervised fine-tuning, comprising approximately 650k English pairs and 860k Chinese pairs.
3.2.2 TRAINING DETAILS
Building on the pretrained model, we conduct supervised fine-tuning on the constructed multimodal dataset for epochs. Training is performed with the AdamW optimizer, using a cosine learning rate schedule that decays from x 10-5 to x 10-6, We use a batch size of 8, apply a weight decay of 1, and set the maximum context length 10,240 tokens with sequence packing.
To further strengthen cross-modal alignment between speech and text, fine-tuning incorporates four input-output modality configurations: speech question + speech answer, speech question - text answer, text question - speech answer, and text question - text answer. The modality pairing is controlled by system prompts, while the underlying content remains identical across configurations. This design ensures that the model learns to handle both unimodal and mooa interactions, enabling to accept text or speech as input and generate either text or speech as output within unified framework.
4 EVALUATION
4.1 TOKENIZER
In this section, we present the experimental evaluation of our encoder and decoder components. For the encoder, a crucial aspect is the preservation of semantic information. To assess this, we fine-tuned a Qwen3-0.66 6B model (Yang et al. 2025) for ASR. Distinct from embedding- proaches (Yang et al.) 2021), our method directly leverages discrete codebook IDs generated by various encoders as input, better aligning with the Large Language Model (LLM) paradigm. This ASR model was trained on the 960-hour Librispeech training dataset (Panayotov al. 2015). We then evaluated the corresponding Word Error Rate (WER) on the test sets (test-clean, test- and dev-clean). Each model was trained for 100k steps using a batch size of 128 and learning rate of
ap-

---

Published as a conference paper at ICLR 2026
le-4, and report the lowest WER achieved. Our baselines include codecs designed capture semantic information, such as Mimi (Défossez et al. 2024) and XCodec 2.0 (Ye et al. 2025). as
well ASR-trained codecs like GLM-4- Voice (Zeng et al. 2024a), Cosy Voice (Du al.) 2024)
and Cosy Voice (Du et al. 2024). Ours represent our proposed streaming model, which is further fine-tuned from 4
Table 2: Evaluation results of our speech encoder
Frame Rate (Hz)
WER (%)
Model
BPS
Streaming
test-clean
dev-clean
overall
Mimi-8
12.5
1100
X
9.65
9.67
14.45
XCodec2.0
50
800
X
14.17
13.82
20.07
Cosyvoice
25
300
10.15
9.64
14.21
Cosyvoice2
25
325
x
9.45
9.42
13.78
GLM-4-Voice
12.5
175
Chunk(2s)
6.59
6.07
9.17
Ours
12.5
175
7.89
7.29
10.80
To evaluate our decoder, we utilize the Seed-TTS-Ea. benchmark (Anastassiou et al. 2024 em- ploying its standard English and Chinese test datasets. We assess intelligibility (WER), speaker similarity (SIM), and speech quality (DNSMOS). Speaker similarity (SIM) is computed as the co- sine similarity between WavLM-TDNN embeddings (Chen et al.] 2022) of the prompt and gen- erated speech. WER is measured using whisper-large-v3 (Radford et al. 2023) for non-Chinese languages and paraformer- zh for Chinese (Gao et al. 2022). Additionally. we incorporate the DNS- MOS (Reddy et al. 2022) metric to assess the perceived quality of the generated speech. Since decoder fine-tuned from Cosy Voice 2, we compare directly with the Cosy Voice series.
Table 3: Evaluation results of our speech decoder
Seed-TTS-Eval
Seed-TTe
Model
Frame rate
WER 4
SIM f
DNSMOS 1
WER
SIM
DNSMOS f
Cosyvoice
25hz
10.53
0.66
3.07
11.29
0.74
3.21
Cosyvoice2
25hz
4.63
0.68
3.09
3.11
0.75
3.22
Ours 12.5hz
4.14
0.67
3.10
2.86
0.73
3.24
Our experimental evaluation demonstrates the robust performance of Our codec across both encoder and decoder components. For the encoder, Ours achieve an overall Word Error Rate (WER) of 10.80% While this is slightly higher than the 9.1 9.17% of GL.M-4- it is important note that GLM-4-Voice operates with 2-second processing blocks rather than pure streaming. While being full streaming architecture, Our model achieves competitive WER. Furthermore, Our encoder significantly surpass other non-streaming codecs like Mimi-8 (14.45%) and Cosy Voice 2 (13.78%) despite its lower BPS and frame rate. Consequently, our decoder, fine-tuned from Cosy Voice 2, benefits from this enhanced capture of semantic information. Even at lower frame rate, Our de- coder achieves better intelligibility (lower WER) and perceived speech quality on both English and Chinese benchmarks compared to CosyVoice 2, with only a marginal trade-off in speaker similarity.
4.2 PRE-TRAINING
To assess the effectiveness of our pre pre-training strategy, evaluate the resulting speech-enablee model on both speech modeling and text understanding benchmarks.
For speech modeling ability, Chinese counterpart, ah-StoorClloe. These benchmarks test the model 's capacity to reason over and generate coherent speech continuations.
StoryCloze (Hassid al. 2023) together with our in-house
For textual capability preservation, we evaluate on MMLU (Hendrycks et al., 2021b a) and
CMMLU el. 2024), which measure knowledge and reasoning across diverse subject domains.

---

Published as a conference paper at ICLR 2026
Table 4: Evaluation result of our pre-trained model. the table, "S.C." refers to "StoryCloze" "s" refers to "Spoken", "t" refers to "Topic". SpiritLM results are takens from Nguyen et al. (202)) The tS.C. and sS.C. results for GLM-4- Voice are taken from Zeng et al. (2024b), and the tS. and sS.C. C. results for Moshi are taken from Défossez et al. (2024). Chinese language evaluations are not performed on models trained only in English.
Speech
Text CMMLU
Model
ts.c. sS.C. zh-tS.C. zh-sS.C.
MMLU
Moshi
83.60 62.70
49.8
Voice
82.90 62.40
83.27
69.10
57.49
54.39
SpiritLM
82.90 61.00
36.90
Ours
84.87 63.17
90.32
71.94
67.19
69.53
Table 5: Spoken question answering evaluation results & speech quality. L.T./W. QA refer to LlamaQA, TriviaQA, and WebQA, respectively. Except for our model and GLM-4- results for other models in the table are taken from Zeng et al. (2024b) and Défossez et al. (002)). We follow KimiTeeam al. (2025) to normalize the answer before judging.
L. QA
QA - T
W.
QA
UTMOS
Model
S S S
s + T S S
Pre-trained Model
GLic--Voice
64.70
50.70
39.10
26.50
32.20
15.90
TWIST
4.00
1.50
Supervised Fine-tuned Model
SpeechGPT
21.60
14.80
6.50
4.00
Moshi
21.00
7.30
9.20
2.81
Moshi*
62.30
22.80
26.60
GLM-4-Voice
74.33
65.67
45.90
43.20
39.22
38.34
4.25
Ours
77.33
63.67
45.20
28.80
45.90
36.71
4.37
S - results obtained with text guide
This dual evaluation allows us to verify that (i) the model acquires robust speech modeling abilities, while (ii) maintaining the original linguistic competence of the pretrained text backbone.
4.3 SUPERVISED FINE-TUNING
To comprehensively evaluate the capabilities of SFT model, LLLMAAQuestion,, Trivia QA, and Web Questions (Nachmani et al. 2024 Joshi et al. 2017] Chang et al. (2022). And the quality of generated speech is evaluated with UTMOS (MOS style evaluation) (Sacki et al. [2022).nnd And we additionally conducted subjective evaluation experiment, relevant details are provided in the Appendix H
QA ability using
ABLATION STUDY
We study the effect of two key components in our pre-training pipeline: Modality-based Layer Split and Frozen Pre Pre-training.
We first compare a naive baseline without either strategy (NF-NoSplit) against a variant that in- troduces Modality-based Layer Splir but trains all parameters directly (NF), isolating the benefit of modality separation. Next, we evaluate the effect of Frozen Pre-training by comparing NF with FP-Full, where text parameters frozen during pre-training and then unfrozen.
8

---

Published as a conference paper at ICLR 2026
Table 6: Ablation study on pre-training strategy. FP: Frozen Pretrain (text parameters frozen during pretrain). FP-Full: all parameters unfrozen after Frozen Pretrain. FP-Layerwise: shared layers gradually unfrozen from last first. FP-Shared: only speech- text shared layers unfrozen, text-specific remain frozen. NF: No Frozen Pretrain (all parameters trained directly). NF-NoSplit: without Modality- Bassed Layer Split, i.e., speech tokens added directly into text vocab without modality-speciffe layers. All models are trained for around 2 epochs on the pre-training dataset.
Speech zh-tS.C.
Text MMLU
Model
Split Layers
ts.c.
sS.C.
zh-sS.C.
CMMLU
FP-Full
85.20
63.12
90.21
72.10
66.50
69.15
FP-Layerwise
84.77
62.64
90.11
71.51
68.82
69.26
FP-Shared
83.27
63.50
90.11
72.69
67.26
69.27
NF
78.73
56.33
88.88
69.16
62.92
63.84
NF
79.05
56.49
89.10
67.93
63.27
63.79
NF
4
77.66
56.60
88.51
67.56
62.11
64.11
NF
2
78.09
56.87
88.62
68.31
62.92
63.74
NF-NoSplit
77.12
55.80
88.72
67.02
60.97
63.73
Qwen3-8B
76.60
77.35
We further ablate different unfreezing strategies after Frozen Pre-training: (i) FP-Full, unfreezing all parameters at once; (ii) FP-Shared, unfreezing only speech-text shared layers while keeping text-specific parameters frozen; (iii) FP-Layerwise, gradually unfreezing shared layers from last to first. The learning rate schedule for FP FP-Layerwise is described in Appendix
Results in table highlight three main findings: (1) Modality-based Layer Split improves both speech modeling and textual ability preservation; (2) Frozen Pre-training provides substantial addi- tional gains; (3) unfreezing strategies yield relatively small differences.
Overall, the ablation confirms that modality separation and freezing text parameters during pre- training are both critical to balancing speech learning with text knowledge preservation. While different unfreezing schedules provide slight trade-offs, their impact is minor compared to the gains from Modality-based Layer Split and Frozen Pre-training themselves.
RELATED WORKS
A detailed discussion of related work is provided in Appendix G
CONCLUSION
We introduced large language model capable of true speech-to- to-speech interaction without interme- diate text, advancing the state of spoken dialogue systems beyond cascaded and text-guided frame- works. Our modality-based layer-splitting and frozen pre-training strategies enable the transfer of linguistic and reasoning knowledge to speech modality from pretrained text LLMs while preserving text abilities, avoiding the degradation often observed multimodal adaptation. Our model achieves state-of-th afrt results in spoken question answering, while supporting both text and speech native input and output modalities. This work demonstrates that end-to-end speech modeling can reach near parity with text-guided methods while overcoming their inherent limitations in latency and expres- sivity. Looking forward, we envision speech-native models as the foundation of future human-Al interaction, supporting seamless, multimodal dialogue across diverse languages and contexts.
ETHICS STATEMENT
This work introduces a speech-enabled large language model. While the model has potential benefits for accessibility and natural human-compinn interaction, it also carries risks. Despite incorporating

---

Published as a conference paper at ICLR 2026
alignment datasets during fine-tuning, the model may still produce unsafe or biased content. Addi- tionally, the speech decoder could be misused for voice cloning or impersonation. We do not release tools or data optimized for such misuse and recommend responsible deployment with safeguards to mitigate these risks.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (No. U24B20181), Shanghai Pilot Program for Basic Research Fudan University 21TQ1400100 (22TQ018), and Fu- dan Kunpeng & Ascend Center of Cultivation.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Keyu An. Qian Chen, Chong Deng, Zhihao Du, Changleng Gao, Zhifu Gao, Yue Gu, Ting Hc, Hangrui Hu, Kai Hu, Shengpeng Ji, Yabin Li, Zerui Li, Heng Lu, Haoneng Luo, Xiang Lv, Bin Ma, Ziyang Ma, Chongjia Ni, Changhe Song, Jiaqi Shi, Xian Shi, Hao Wang, Wen Wang, Yuxuan Wang, Zhangyu Xiao, Zhijie Yan, Yexin Yang, Bin Zhang, Qinglin Zhang, Shiliang Zhang, Nan Zhao, and Siqi Zheng. Funaudiollm: Voice understanding and generation foundation models for natural interaction between humans and llms, 2024. URL https::/aaxii.org///aaat
04051
Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.008 404.02200,, 2024.
Yuelin Bai, Xeron Du, Yiming Liang, Leo Jin, Junting Zhou, Ziqiang Liu, Feiteng Fang, Mingshan Chang, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Moore Wang, Ruibin Yuan, Haihong Wu, Hongquan Lin, Wenhao Huang, Jiajun Zhang, Chenghua Lin, Jie Fu, Min Yang. Shiwen Ni, and Ge Zhang. COIG-CQIA: Quality is all you need for Chinese instruction fine-tuning. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.). Findings of the Association for Computational Linguistics: NAACI. 2025, pp. 8190-8200 (2058200,400 Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 977-8-89176-19999 195-7, doi: 10. 106188611/2225.. 2tt.:////222......nn-ss-a-a findinnss-naccl 457. URL https: /atttps://acaaaaeeee WBLL:ntteee//cc..nnoooooo.. 2025 fir.iiiinsssss naceo-aaccl... .1a.cc1.444 457/
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghi- dour. Audiolm: language modeling approach to audio generation, 2023. URL https: 7/
03143
Hervé Bredin. pyannote. audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. In Proc. INTERSPEECH 2023, 2023.
Yichen Chang, Yiming Chen, Ray Ng. Andrew Gallagher, Alexander Schwing, Dhruv Batra, Noah Snavely, Yannis Kalantidis, et al. Webqa: Multihop and multimodal Computer Vision and Pat-
qa. In Proceedings of the IEEE/CVF Conference tern Recognition (CVPR), pp. 16474-16483, 2022. doi: 10.1109/CVPR526888 2022.
01600. URL Chang WebOA Multihov and Multimodal CVPR 2022 1022_paper..tar
wtrrcceeessstteeeeeeeeee thecvf. com. wttpt/t//ttttttee//eeec html/
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self supe-supe pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, (666):1505- 1518, 2022.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2- technical report. arXiv preprint arXiv:2407.10755 2024.
Liu Cong, Wang Zhong, Shen Sheng Yu, Peng Jialiang, Zhang Xiaoli, Du ZhenDong, and Wang YaFang. The chinese dataset distilled from decpseck-r1-671b httttp:::/huqqinnn
co/
Cheeppppeeecsss---eeeeee----- -at.aeeeeeeeeeeeeeee------------ saattillldatt-lae 110k 2025.
Nilaksh Das, Saket Dingliwal. Srikanth Ronanki, Rohit Paturi, Zhaocheng Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, Xilai Li, Karel Mundnich, Monica Sunkara, Sravan Bodapati, Sundararajan Srinivasan, Kyu Han, and Katrin Kirchhoff. Speech- verse: A large-scale generalizable audio language model, 2025. BRLhhttss://xxxi... abs/2405 08295
Alexandre Défossez, Laurent Mazaré, Manu Orsini. Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dia- logue. arXiv preprint arXiv:2410.000:: 2024.
11

---

Published as a conference paper at ICLR 2026
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based supervised semantic tokens. arXiv preprint urXiv:2407.0540) 2024.
Adrien Défossez, Jakob Uszkoreit, David Lalanne, and Dzmitry Bahdanau. Encodec: A neural audio codec. In NeurlPS, 2022.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélic Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dia- logue, 2024.
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models, 2025. URL Utthttss: httttps://arxiv.oot
abs/2400 06666
Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Yan. Paraformer: Fast and accu- rate parallel transformer for non-autoregressive end-to-end speech recognition. arXiv preprint arXiv:2206.0831 08 2022.
Yitian Gong, Luozhijie Jin, Ruifan Deng, Dong Zhang, Xin Zhang, Qinyuan Cheng, Zhaoye Fei, Shimin Li, and Xipeng Qiu. Xy-tokenizer: Mitigating the semantic-acoustic conflict in low-bitrate speech codecs. arXiv preprint arXiv:2506 2332335500..... 2025.
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, et al. Textually pretrained speech language models. arXiv preprint arXiv:2305. 13009, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representutions (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021b.
Jiaming Ji, Jiayi Zhou, Hantao Lou, Boyuan Chen, Donghai Hong, Xuyao Wang, Wenqi Chen, Kaile Wang, Rui Pan, Jiahao Li, Mohan Wang, Josef Dai, Tianyi Qiu, Hua Xu, Dong Li. Weipeng Chen, Jun Song, Bo Zheng, and Yaodong Yang. Align anything: Training all-modality models to follow instructions with language feedback. 2024. URL /tttttsss//aattt// org/ abs/2412. 15838
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale dis- tantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Pupers), pp. 1601-1611, Vancouver, Canada, 2017. Association for Computational Linguistics. doi: D0.18653/v1/P17.1 1tt.//1771,711 147.
http:::://acllaahologoyy wwwp/oggg/ll1i
KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang. Xinyu Zhou, Y. Charles, Jun Chen, Yanru Chen, Yulun Du, Weiran He, Zhenxing Hu, Guokun Lai, Qingcheng Li, Yangyang Liu, Weidong Sun, Jianzhou Wang, Yuzhi Wang, Yuefeng Wu, Yuxin Wu, Dongchao Yang, Hao Yang. Ying Yang. Zhilin Yang. Aoxiong Yin, Ruibin Yuan, Yutong Zhang, and Zaida Zhou. Kimi-audio technical report, 2025. http:::::
//arxiv.orge 25004. 18425
Siddharth Kumar, Neil Zeghidour, Abhinav Katiyar, Krzysztof Choromanski, and John Schulman. High-fidelity audio compression with improved rvggan. NeurlPS (Poster Extended), 2023.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. On generative spoken language modeling from raw audio. Transactions of the Association for doi: 10. 162/tall a 00430. URL httpp:://
Computational Linguistics, 9:1336-1354 1354, 2021.
acc allannho www.an/ooo....////22222.. wtt.:......ll.11..
12

---

Published as a conference paper at ICLR 2026
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Tim- othy Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese.
In Lun- Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics: ACI. 2024, pp. 11260-1 1285, Bangkok, Thailand, August 2024. URL
Association for Computational Linguistics. doi: 10. 16653//1/222.fnnninns-aaoo http://wwlaatttloo..rorr////// wwwp//24..//222.... ftng///22....nigggss..... acl. 6717
Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. Semanti- codec: An ultra low bitrate semantic audio codec for general sound. IEEE Journal of Selected Topics in Signal Processing, 2024.
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest catdaasstt.
collection of educational content, 2024. URL
HuggingF www.a/eeeeece..... www.w///winwwwbeeee eduu
Kentaro Mitsui, Koh Mitsuda, Toshiaki Wakatsuki, Yukiya Hono, and Kei Sawada. PSLM: Paral- lel generation of text and speech with LLMs for low-latency spoken dialogue systems. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Com-
putational Linguisties: EMNLP 2024, pp. 2692-2700, Miami, Florida, USA, November 2024.
Association for Computational Linguistics. doi: 10. 18.18653/ 1/2024. 1/t0://............giiseeeeeee findings-emmll.1511 URL Bttps://aclanthoole
2024. findings emnlp............. 151/
Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan and Michelle Tadmor-Raan Ramanovich. Spoken question answering and speech continuation using spectrogrampoowent
1lm. In International Confer- ence on Learning Representations (ICIR), 2024. URL /aatps::/arx/. org/ abs /2305. arXiv preprint arXiv:2305.15255.
15255
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoit Sagot, and Emmanuel Dupoux. Spirit Im: Interlcaved spoken and written language model, 2024. URL https::/ org/abs/
2402. 05755
Open-Moss. Speechgpt 2.0-preview. preview 2025.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-52 5210. IEEE, 2015.
Alexis Plaquet and Hervé Bredin. Powerset multi-clase cross entropy loss for neural speaker diariza- tion. In Proc. INTERSPEECH 2023, 2023.
Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi,, Alexei Baevski, Yossi Adi, Xi- aohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Scaling speech technology to 1,000+ languages, 2023.
org/ /ttp://b../335....///
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large- scale weak supervision. In International conference on ma- chine learning, pp. 184492288511 PMLR, 2023.
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2022-2022 IEEE inter- national conference on acoustics, speech and signal processing (ICASSP), pp. 886-890, IEEE, 2022.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Uumos: Utokyo-sarulal sarulab system for voicemos challenge 2022. arXiv preprint, 2022.
URL URL httphhtt/////ettee wwwpxxivvvoorvvrroo www.g/a////aaae 02152
13

---

Published as a conference paper at ICLR 2026
Suwon Shon, Kwangyoun Kim, Yi-Te Hsu, Prashant Sridhar, Shinji Watanabe, and Karen Livescu. Discreteslu: A large language model with self-super discrete speech units for spoken lan- guage understanding. arXiv preprint 24X.::2400009345, 2024.
OpenMOSS Team. Text to spoken dialogue generation. 2025.
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist Ilm assistants, 2023.
URL htttp:://hugginggggttt
um www.peneeeeppppeeeeeeeeeeeeeeee rme.- -2.5
Xidong Wang, Jianquan Li, Shunian Chen, Yuxuan Zhu, Xiangbo Wu, Zhiyi Zhang, Xiaolong Xu, Junying Chen, Jie Fu, Xiang Wan, Anningzhe Gao, and Benyou Wang. Huatuo-26M, a large- scale Chinese medical QA dataset. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 3828-3843 Albuquerque, New Mexico, April 2025a. Association for Computational Linguistics. ISBN 979-8-89176-199 7. doi: 10.1 (0.18653//1/200 1/2033/v12222...// 22n2://////2..nnnn---aaaadaaa findiingg--mmall.1.... oog. Findings waaaa...2111....
Xiong Wang, Yangze Li, Chaoyou Fu,, Yike Zhang, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm. ICML, 2025b.
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in peattps://axxiv.rrttt OLL.
streaming, 2024.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing. 2024. URL https: /ectps://arxiv...o arwiv. org/ abs////////222/ 08464
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang. Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.094 093889, 2025.
Dongjie Yang, Ruifeng Yuan, Yuantao Fan, Yifei Yang, Zili Wang, Shusen Wang, and Hai Zhao. RefGPT: Dialogue generation of GPT, by GPT, and for GPT. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 2511-2535. Singapore, December 2023, Association for Computational Linguistics. doi: 10. 10.18653/v1/2023, (tt.:////11/222...nnnnnnnnnn findings-emnll..6.. URL https:://aclattthllooo hotgo/oooocoolloorr/223333 org/2033. firdings-emmlp.1e
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, al. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:210S.01051 2021.
Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi Dai, et al. Llasa: Scaling train-time and inference-ting compute for llama-based speech synthesis. arXiv preprint arXiv:2502.044 04128, 2025.
Yijiong Yu, Ziyun Dai, Zekun Wang, Wei Wang, Ran Chen, and Ji Pei. Openesg chinese corpus: A series of high-que high-quality chinese datasets for Ilm training, 2025. uttssttp://axxiv.oo abs/2501 08197
Neil Zeghidour, Rémi Dossa, Abhinay Katiyar, Jakob Uszkoreit, Siddharth Kumar, Krzysztof Choromanski, Samuel Gershman, and John Schulman. Soundstream: End-to-end neural audio codec. In NeurIPS, 2021.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glice-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02212 2024a.
14

---

Published as a conference paper at ICLR 2026
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glice-voice: Towards intelligent and human-like end-to-end spoken chatbot, 2024b. Ut//atpss//acxiv.orrr abs 2412. 02612
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong, and Jie Tang. Scaling speech-text pre-training with synthetic interleaved data, 2024c. URL https:
// //tpiii.ro//as///111..1...
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abil- ities. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15757-15773, Singapore, December 2023a. As- sociation for Computational Linguistics. doi: 10.18653/v 1/2023./112222/// 1/2n23..nnnnggss Sindings-emmll.1055 URL https://ack antho . org 2023. f ndi ngs-emm p. 1055/
Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong, Yu Shu, Zhaoqun Li, Zekun Wang, Chenghua Lin, Wenhao Huang, and Jie Fu. Chinese open instruction generalist: A prelim- inary release, 2023b.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308. 16692, 2023c.
15

---

Published as a conference paper at ICLR 2026
A LLM USAGE DISCLOSURE
ChatGPT's 5 was used to refine the writing style of certain paragraphs and as supplementary tool suggest related works. was not the sole or primary source for the related work section; all references were independently identified, reviewed, and selected by the authors. The LLM did not contribute to research design, experiments, analysis, or results. The authors assume full responsibil- ity for the content of this paper.
B SUPERVISED FINE-TUNING DATASETS
Table lissts the supervised fine-tuning datasets used in our work. We report only the number of examples actually used for training.
Dataset
Language
Used Samples
OpenHermes-22. 2.5
EN
200k
OpenHermes-22 2.5 (Chinese translated)
200k
Magpie-L.lama-3, 1-Prr-MMT-00KK Filtered
300k
Magpie-Qwen2-Pro-e Proae2-PPoooO--Chhiiiii
ZH
200k
BAAI OL-CC
ZH
11.7k
ReiGPT-Fact
ZHI
50k
COIG-CQIA
ZHI
45k
Ruozhiba
ZHI
1.4k
Huatuo26M-Lite
ZH
30k
Align-Anything- Instruction-
-onn-nnyttiing--stuuttooo------ 100K
EN&ZH
100k
Chinese- DeepSeck-F --istill-SFF
ZH
110k
Bhain-of-Thought-She
7.14k
Table 7: Supervised fine-tuning datasets used in our experiments.
C PROMPT FOR SUPERVISED FINE-TUNING TEXT ADAPTATION
You are converting a supervised fine-tuning dataset (Q&A) into a format that can be read naturally by a TTS system for speech language model. For each question and answer pair, you must determine whether the text is suitable for TTS according to the following rules.
Possible states are: require no changes (PASSTHROUGH), requires adaptation (ADAPT), or unsuitable (REJECT).
CORE RULES
1) Preserve Meaning, Adapt for Speech Preserve content verbatim where possible, but always rephrase rigid or written phrasing into natural, flowing speech transcript. Adapted transcript should net include parentheses, brackets, symbols, Markdown,
formatting that only works paper, or structured written formatting like "Method:... Result:...".
Do NOT add meta-statements such as "the spoken versien is",
2) Convert Non-Vocal Elements
Math Formulas: simple inline LaTeX is supported by TTS. Convert all math to inline LaTex, e.g. &\cos 30"Veire =Xfrac ('sqrt (2))2 Complex or block LaTeX (derivatives, integrals, etc.) is not supported, read out if feasible, otherwise REJECT. Note that inline LaTeX command involving summatios//inita. Integralss/erriattiie
sets/intervals, vectors/matrices. Greek
letters (except \pi), \approx, \text, etc. are not supported. Use English letters instead of Greek letters for variables. [IMPORTANT] single variables (e.g., 5x$) MUSI be wrapped in LaTeX.
Tables: content summary, e.g., "First, two CPUs at two hundred thousand dollara".
dol lars each, then one
16

---

Published as a conference paper at ICLR 2026
URLs: rephrase remove. Only simple URLs may be read ("google dot com").
Short Code: narrate, e.g., "this. parsepptiooo. "this dot parso options".
Other nen-vocal elements: narrate or rephrase.
Do NOT explicitly read punctuation.
3) Length Constraint
If the content exceeds 200 words, shorten it to "200 words while preserving factual correctness and Logical ilow. the question explicitly requests detail, allow up 400 words.
Prioritize clarity over brevity it is acceptable to keep light redundancy or slightly exceed the word 1imit if it improves understanding.
If simplification cannot be dene without major distortion, REJECT.
4) Language Policy
Supported languages: English and Simplified Chinese.
Mixed English-Chinese allowed. not translate. Unless specified, avoid Classical Chinese or 0ld English.
If other languages appear: REJECT.
5) Formatting-Specific Prompts
If the question requires fermat impossible for speech (tables, LaTex \boxed()), REJECT if inseparable from meaning. Otherwise, keep the question unchanged and begin the answer with a polite clarification: "Sorry, I cannot provide the answer in table. However..."
Do NOT add neta-statements like "this is the adapted answer."
6) Pause & Rhythm Control
Adjust punctuation sentence boundaries to guide natural pauses and raythr.
Break long or complex sentences into shorter ones.
Insert natural discourse markers when helpful ("so," "in other words," "fer example").
7) Oral Smoothness
Always ensure the adapted text sounds like something a human would naturally say aloud.
Favor conversational flow, short clauses, and rhythm. over rigid literalism.
Add light redundancy for clarity if needed ("That means..., "In short...").
Prioritize spoken fluency ever textual fidelity.
8) Correction
If the answer contains incorrect or irrelevant content, correct it. Do not reject unsafe or inappropriate content. Instead, rewrite the answer to make it appropriate.
Unless specifically instructed, the answer should be neutral and objective. Rewrite answers that are not. Keep corrections minimal and faithful to the question. Do NOT add meta statements like "the corrected answer is".
9) Chain of Thought
When answering a complex question from the user: if the assistant gives the answer first, then reasons, reorder to reasoning first, then answer. (Uniess explicitly instructed otherwise.)
10) Style
Recommend a IIS style for each content.
Available options: neutral, happy, sad, angry, surprised, fear, hate, excited, coldness.
17

---

Published as a conference paper at ICLR 2026
11) Rejection Policy
REJECT if content contains leng code dumps, complex math, dense LaTeX, giant tables, etc.
REJECT if any of the rules above are violated.
OUTPUT RULES
Return a JSON that follows this schema:
"questicn";
"state": "PASSTHROUGH ADAPT REJECT", // adapted text if ADAPT, else empty "style": "", // recommended TTS style
"text": ""
"simplified"; false
"answer":
"state": "PASSTHROUGH ADAPT REJECT",
"text": "",
"style"; ""
"corrected"; false, "simplified"; false
"quality": 1-5
// 5 excellent, 1 unusable
D SIMILARITY DETAILS
D.1 HEATMAPS
Figure shows the similarity maps of all layers for sample O. It can be observed that the similarity diagonal begins to appear layer 7, becomes clearly noticeable by layer 11, and after layer 24, the similarity of other tokens gradually increases at layers 25 and 26, while at layer 27, all similarities drop significantly. This pattern is consistent with the observations in our Similarity Score figure.
D.2 SAMPLES
Below the five random samples used calculate the similarity score, with all text content aligned and presented English.
Sample0:
Speech-to-text alignment is a core problem in the field of speech processing, and it becones especially critical during the training of large-scale speech models. "Alignment" refers to accurately matching acoustie segnents the speech signal with characters, words, or subword units the text sequence along the temporal dimension, enabling the model to learn the mapping between speech and language. If the al innmen is inacourate, the model struggles to capture the correspondence between speech and text effectively, speech recognition, speech
which in turn affects the performance synthesis, and multimodal tasks. Therefore, building a high-quality speech-text alignment nechanism is not only a fundamental step in training large speech models but also prerequisite for enhancing model generalization and practical performance.
Samplel:
In the previous lesson, learned the concepts of the greatest
divisor (GCD) and the least common multiple (LCM), , well how calculate them. Today, we will continue to learn about the concept of prime numbers and use prime factorization to find the common divisors of two numbers.
First, 1et us review what we 1earned in the previcus lesson. We mainly atudied several methods find common divisors. The first
18

---

Published as a conference paper at ICLR 2026
Embedding
Layer
Layer 1
Layer
Layer
10%
Layer 4
Layer
Layer 6
Layer
Layer
Layer 9
Layer 10
Layer 11
Layer 12
Layer 13
Layer 14
Layer 15
Layer 16
Layer 17
Layer 18
Layer 19
Layer
Layer 21
Layer
Layer 23
Layer 24
Layer 25
Layer 26
Layer 27
Figure 4: Heatmaps for embedding and layers 0-27.
19

---

Published as a conference paper at ICLR 2026
method is the listing method. The listing method is general approach that can be applied to any two numbers
Samplc2:
A team of energing scientists has developed novel artificial intelligence algorithn that, by observing and analyzing human brain activity, successfully replicates the thinking processes of the human brain. This enables AI to simulate human thought while performing computations and processing information more efficiently, generating significant response in the global scientific community. According to reliable sources, the research findings have al already passed review by international academic institutions and will be presented and announced this month at the world's most prestigious scientific conference. It is expected to have profound impact the development of artificial intelligence.
Samplc3:
What about large language medels? How do they work? They learn from enormous corpus of text using natural language processing to understand the relationships between all the sentences in that corpus. For example, if one person says a sentence and another person responds, there is often cartain delationshiy between the two sentences. For instance, it the first person says, "I'm hungry, the second person might respond, "I make something for you." Once the model learns these relationships, perform tasks such as translation or generating appropriate responses in other contexts.
can use them to
Sample4:
The harsh way of survival. Some species rely on complex social structures or specialized physical traits to ensure their survival and reproduction. Some species can oven assist humans in locating resources. In the tropical regions of the Pacific, certain species live on the water surface, while others inhabit the seafloor. There are also species whose range apans both the surface and the seaflcor, forming what is known as miree-dimensiona ecosystem. Due to the vast expanse of the Pacific, these species must adapt to survive.
D.3 LAYER-WER WISE SIMILARITY SCORE OF DIFFERENT MODELS.
We conducted layer-wise similarity analyses on more models and larger set of samples. For the two models shown in the Figure 5 due to implementation constraints that prevent us from mapping text tokens back to the original text, we used a global Daw-based metric measure similarity score. In contrast, for the Figure 6 we adopted the fine- ane-grained similarity computation method described in Appendix F both cases, we used the same set 1K long-form speech samples and averaged the scores to obtain each model's layer-wise similarity profile.
From the results, we observe consistent trend across all speech models: similarity increases from shallow middle layers, and then decreases toward the deeper layers. In the global-DTW setting, the last few layers show a slight increase because stopwords and punctuation often lead to artificially higher similarity. However, this does not alter the overall trend of "rising then falling."
Across all models in the figure, a unified pattern can be observed: similarity begins to decline at approximately one-third of the total depth. The above analysis result guided our model design choice.
E LAYER- WISE SIMILARITY SCORE.
To evaluate cross cross-modal alignment inside a Transformer backbone, we compute a similarity score at each layer i (1,..., T.). Let the text tokens be (4,...,,4u) .n)) and the speech tokens (81,-..5 8m)-
20

---

Published as a conference paper at ICLR 2026
Similarity Score (Gobal)
Kimi-Aucio
1.3
1.6
144
10
Layer Index
Figure 5: Similarity Score of GLM-4-Voice and Kimi-Audio. Due to implementation constraints that prevent us from mapping text tokens back to the original text, we used global DTW-based metric to measure similarity score in this figure.
0.7
SpecchGPT 2.0 preview
MiMo-Aucio
Similarity Score
0.5
50
Layer Index
Figure 6: Similarity Score of SPe 2.0-preview and MiMo-Audio. We adopted the fine-
grained similarity computation method described in Appendix
Using forced alignment((ratap et al. 2023), we obtain J, alignment pairs
1, S,,), (T Sr,2,,
where Tú is the set of aligned text tokens and S is the corresponding set of speech tokens (the alignment is fixed, but hidden states depend on the Transformer layer i).
For each pair (i, construct cosine similarity matrix
Mi [u, v] cos (hi,eu hi,.w):
where h4 and ha,, denote hidden states of text token t and speech token at layer i.
The DTW-based similarity is defined as
DTW, J.
Σ
Mrslu, v],
P..
(M.
where P.y is the optimal Dynamic Time Warping path through Myj. i.c., the alignment trajectory maximizing similarity under temporal constraints.
As background normalization, compute
BG = [St. (n l
Σ
Σ
cos (hi,e his))
BESIS LE (t
Finally, the layer-wise similarity score layer is
DTW BG.j
SS, = Σ
A=
DTW...
Σk=1 Jk k=1j=1
21

---

Published as a conference paper at ICLR 2026
Here, DTW,, f.j measures the mean similarity along the DTW-optimal path for pair at layer i, BG normalizes against similarities with non-aligned text tokens, and SS, means Similarity Score of layer i quantifies the relative strength of text-speech alignment at Transformer layer with serving as global coefficient averaged over all pairs and layers.
F LAYER-WISE UNFREEZE LEARNING RATE SCHEDULE
To implement gradual layer-wise unfreezing, we assign each transformer layer its own learning rate with a delayed warmup-cosine schedule. Let denote the global training step, the model contain N 32 layers to be unfrozen indexed by E {O....,N -1 (with i layer), and define:
denoting the final
d= = (N 1- - i)k: K, D, = d.
where i is the per-layer delay (in steps). T the global step which all layers have reached Muinn Tmax the peak learning rate, and r = 0.1 the final decay ratio.
For each layer define
d, and set its learning rate
0,
u < 0,
Mmax 1
0 u < w,
(s)s)
Manax
quin 1 cos
10 )) क Dj
Mmin
w w + D;:
7min
u u> w D;
Variable definitions
s: global training step.
t layer index (N - 1 = last layer).
N: total number of layers (32).
k: per-layer delay (5000 steps).
warmup duration (2000 steps).
T: global step when all layers reach Mmmnn-
Mmax: maximum learning rate.
Mmin = 0.1 7/aax: minimum learning rate.
d; = (N 1 - i) k: start delay for layer i.
D. = T d: cosine decay duration for layer i.
This schedule ensures that higher (later) layers are unfrozen earlier, while lower (earlier) layers remain frozen longer, enabling controlled and stable adaptation of the pretrained text backbone.
G RELATED WORKS
Codec Speech codecs are crucial for speech large language models (SLMs) and can be grouped into two categories. Neural acoustic codecs based on (R)VQ-GAN optimize reconstruction loss to preserve fine-grained acoustic details (Zeghidour et al. 2021 Défossez et al.) 2022 Kumar et al.) 2023), but their tokens often lack semantic coherence when used for language modeling. In contrast, semantic-oriented codecs adopt a single-layer VQ to encode linguistic content and recover timbre with generative modules such as conditional flow matching (Du et al., 2024 Zeng et al. 2024b). While trading off perfect fidelity, this design yields tokens better suited for semantic modeling. We therefore follow the latter approach and further enhance with streaming encoder- decoder modules for real-time interaction.
22

---

Published as a conference paper at ICLR 2026
Speech-to-Speech Interaction Models Existing models mostly depend on text guidance for speech generation. For instance, SpeechGPT (Zhang et al. 2023a) integrates large language mod- els with discrete speech representations but requires text-based prompts to guide speech genera- tion. Similarly, Moshi (Détossez et al. 2024) employs a full-duplex spoken dialogue framework, generating speech tokens from neural audio codec, yet it still necessitates text instructions for generating speech responses. Qwen-Audio (Chu et al. (024) accepts diverse audio inputs and out- puts text, relying on textual prompts for speech understanding. LLaMA-Omni (Fang et al. 2025) and Freeze-Omni (Wang et al. 20256) extend LLMs to process speech inputs and generate speech outputs directly, but they continue to depend on text prompts guide the interaction. Mini-Ome (Xie & Wu, 2024) fine-tunes language models to generate text and speech responses simultane- ously using instruction datasets, yet the quality of both text and speech responses is limited without prior speech pre-training. GI.M-4-Voice (Zeng et al. (024b) further advances toward speech-to- speech interaction by integrating speech input and output with large language models, but still fundamentally relies textual supervision for alignment and instruction following. These models demonstrate progress toward speech-to-speee interaction but still require text guidance for effective performance.
Frozen and Progressive Training Recent work on integrating speech into decoder-only LLMs has emphasized retaining text capabilities while extending to new modalities. A common strat- egy is to freeze most LLM parameters and train lightweight adapters. For instance, [Wng et al.) (2025b) proposed Freeze-Ommi, which augments a frozen LLM backbone with speech encoder and decoder modules, while Das et al. (2255) introduced SpeechVerse. combining frozen speech and text backbones with adapters to enable zero-shot speech processing from text instructions. Beyond such frozen-backbone
frozen- backbone designs, other works adopt progressive staged adaptation. Xie & Wu 2024) pre-
sented Mini- Omni which proceeds in phases: first learning speech adapters with the LLM frozen, then performing L.M-olly fine-tuning align modalities, and finally unfreezing all but the audio coder for joint multimodal training. Together, these studies show that freezing L.M backbone helps preserve language modeling ability, while progressive unfreezing provides pathway for more flex- ible and effective multimodal integration.
H DOUBLE-BIINND HUMAN EVALUATION
We conducted an additional double-blind blind human evaluation specifically targeting non-verbal speech generation.
H.1 HUMAN EVALUATION SETUP
We recruited nine independent anonymous raters, all graduate students with bachelor's 's degrees and CET-6 English proficiency certification, ensuring sufficient linguistic competence and evaluation reliability. None of the raters were involved in this project.
The evaluation focused on accuracy and naturalness across three controlled non-verbal behaviors:
Silence: Models paused for randomly sampled durations (1-10 seconds) before respond- ing.
Vocal fillers: Models produced paralinguistic cues (e.g., light laughter or sighs) prior to verbal responses.
Response style: Models replied in designated affective states (e.g., hesitant or confident).
H.2 EVALUATION PROTOCOL
Each condition included five distinct prompts. Raters evaluated every model response using a 5- point Likert scale along three dimensions:
Speech Naturalness
5: completely natural (humaa-indissinguushabll))))
4: natural with minor flaws
23

---

Published as a conference paper at ICLR 2026
3: acceptable with noticeable defects
2: unnatural with clear issues
I: severely unnatural unintelligible
Instruction Adherence
5: perfect compliance
4: minor deviations
3: partial compliance
2: minimal compliance
1: non-compliant
Response Quality
5: high relevance / accuracy
4: trivial errors
3: moderately acceptable
2: poor but marginally reasonable
1: irrational strongly non-human
Final MOS values were computed by averaging across all raters and dimensions.
H.3
RESULTS
Our model demonstrates substantial improvements, particularly in fine-grained prosodic behaviors such as pauses and paralinguistic vocalizations. Table summarizes the MOS results.
Non- Verbal Behavior
Ours
MIMO
GLM-4
Kimi
Qwen3
Gemini
GPT--o
Silence
4.17
2.40
2.35
1.93
2.54
2.73
2.81
Vocal fillers
4.15
3.85
3,04
3.56
3,04
2.85
3.11
Response style
4.25
3,60
3.81
3.85
3,44
3.22
3.59
Table 8: MOS results for non-verbal behavior generation across models.
These results highlight clear differences in non-verbal expressiveness across models. Our sys- tem consistently excels across all three cnoggrics-especiine in fine-grained prosodic con- trol trol-indicating strong capacity for natural, interpretable, and controllable non-verbal behavior. We believe this human evaluation provides concrete empirical evidence that directly addresses concerns that UTMOS and WER alone do not capture prosodic quality.
24


Reviews:
[{"review_id": "lH0IWKUDyg", "reviewer": "Reviewer_bYkG", "summary": "This paper tackles the problem of building a speech-to-speech model (speechLM). \nLike most previous works, it uses a single-codebook speech tokenizer (finetunes the GLM tokenizer), and finetunes a causal language model with an expanded vocabulary for speech tokens on text and audio data (paired, interleaved and unsupervised).\nTheir main contribution claims two: (1) layer splitting - \"duplicating\" the K deepest layers of the LLM that would specialize on producing speech tokens. (2) two stage training: first only trains randomly initilized parameters, where the second traines all parameters.\n\nIm my view, both ideas have already been proposed and investigated. Having speech/text prediction heads for example has been done in SpeechT5 (speech/text post net), lately with HiggsAudio (Audio adapter).\nWhat they denote as frozen pretraining is also a common practice, e.g. projector training in speech/vision-aware LLMs. It ensures your backbone won't get noisy gradients from the randomly initilized modules.\nAn interesting direction this paper explored is how to select the layer to start layer splitting, they do so by investigating an existing speechLM, and visulizing the similarity between the text embeddings, and the audio embeddings of the matching audio. They see the highest similarity within the ~80% of the depth, which they use to perform splitting in that point.", "questions": "You claim that there’s a text “bottleneck” in text-guided speechLMs. But the audio is still within the context in those models, and the non-verbal cues can be utilized implicitly. Do you have an indication that the input audio tokens are not used in text-guided speechLMs?\nLooks like there are high-similarity results on Figure 4 (appendix) layer 26. Any explanation on why?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "comprehensive model training, with large-scale data collection. \nInteresting experimental results ablation on model pretraining (no new conclusions, but it validates the known approach of \"frozen pretraining\")\nInteresting analysis on speech/text embedding similarity in speechLMs across layers (but only done on one model)", "weaknesses": "- Most claimed novelties are not new ideas. A solid tech report, but not a major contribution.\n\n- Layer splitting also increases the parameter count, hard to say if the improvements comes from the increase in model size of not.\n- We're missing an ablation on what happens when you split in different locations. e.g. split on K first/middle/last layers - same parameter count but different locations. \n- The layer similarity experiment is best done on multiple speech-text LLMs (e.g. GLM?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G1dgHB2JJ1", "reviewer": "Reviewer_eUke", "summary": "This study presents a high-quality speech–text multimodal large language model (LLM) developed through techniques such as layer-wise analysis and unfreezing strategies aimed at improving speech–text interleaving and information sharing. The paper demonstrates the model’s performance across various benchmarks, including continuation and conversation tasks.", "questions": "- It would be helpful to analyze whether the layer-based behavioral patterns are commonly observed in other SLMs as well. \n\n- As for CosyVoice 2, if my memory is correct: (1) the decoder’s chunk size is quite large; (2) there is lookahead; and (3) it requires reference speech at inference. For MOSS-Speech-Codec, I’m curious about the flow-matching chunk size, whether lookahead is used, whether reference speech is required, and the vocoder’s chunk size (or whether it is fully causal).\n\n- The layers were split 32/4, but this analysis would likely vary depending on the criteria, the size of the model examined, and other variables. I’m also curious about what changes are observed (or expected) when adjusting that split point and experimenting with different branch locations.\n\n- You mentioned that the encoder supports full-streaming, but as far as I remember, both glm-4-voice and WhisperFeatureExtractor used a lookahead mechanism and also processed a full 30-second segment for computation. How did you handle this part?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Basing the approach on branching to differentiate the model’s behavior and architecture is novel.\n- The effectiveness of the proposed method is demonstrated through multiple experiments.", "weaknesses": "Rather than listing separate weaknesses, I noted a few brief questions in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FqQxdk0CZJ", "reviewer": "Reviewer_24Lx", "summary": "The paper proposed a speech to speech model that can directly understands and generates speech without using text. The proposed approach use modality-based layer-splitting architecture and freeze some of the parameters which preserve the reasoning and the world knowledge of the LLM. \n\nExperiments on the proposed model show that it gets state of the art results on spoken question answering with the comperable performance to text-guided algorithms.", "questions": "1. How the results change if you use real audio for training ? Or can you tell something about the results from using the TTS/ASR generated data?\n\n2. What happen if you use LoRA instead of the proposed approach? \n\n3. How the model behave with background noise and non-verbal sounds ?\n\n4. Regarding the latency - How does the quality degradate when using more compute/layers?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.The proposed architecture is elegant with shared lower layers and speech and text specialized upper layers, which balancing between the speech generation and the text capabilities of the LLM\n\n2. The proposed ides of frozen pertaining and unfreezing upper layers help with the problem of catastrophic forgetting\n\n3. The proposed architecture which use streaming encoder and a flow-matching decoder give good latency for interactive usage \n\n4. The evaluation of the proposed model is well designed - all IO directions S2S, S2T, T2S, T2T", "weaknesses": "1. The training of the proposed model use TTS generated speech and ASR data with can give compound error to the model\n\n2. The metrics of UTMOS and WER is not enough to evaluate prosody such as hesitations or laughter.\n\n3. Using single codebook may limit the results of the proposed model using richer multi codebook may improve the results", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ntrVYDc3II", "reviewer": "Reviewer_XJ6K", "summary": "This paper presents a speech-to-speech large language model that directly processes and generates speech without relying on intermediate text representations. The key contributions are: (1) a modality-based layer-splitting architecture that separates text and speech generation paths in the final transformer layers, (2) a frozen pre-training strategy that preserves text capabilities while adding speech modality, and (3) demonstration of competitive performance on speech-to-speech benchmarks while maintaining text performance.", "questions": "check weakness, particularly if you can provide some discussion on Weakness 1, 2, 3 and 4.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Novel Architecture Design: The modality-based layer splitting is well-motivated by the empirical analysis in Figure 2, showing that speech-text representations fuse in lower/middle layers but diverge in final layers. This observation drives a principled architectural choice rather than an arbitrary design decision.\n2. Comprehensive Evaluation: The paper evaluates multiple dimensions - speech modeling, text modeling preservation, speech quality (UTMOS), and spoken QA across multiple languages.\n3. Thoughtful Training Strategy: The two-stage frozen pre-training approach with progressive unfreezing is well-designed and the ablation study (Table 6) validates its effectiveness.\n4. Open sourcing code and models", "weaknesses": "1.  The streaming-capable tokenizer with ASR-based encoder training shows good performance, though at slightly higher WER than non-streaming baselines. However, there is no discussion of how fast the streaming codec is. Some discussion on RTF of different models is definitely needed.\n2. Incomplete Comparison: Include stronger speech-in speech-out baselines like Qwen2.5-OMNI and Kimi Audio. Also, more discussion about comparison with closed API models like GEMINI or GPT-4o audio preview to understand performance gap.\n3. Text Performance Degradation: Compared to Qwen3-8B used as this model's LLM backbone, MMLU drops from 76.60% → 67.19% (12% relative decline), CMMLU drops from 77.35% → 69.53% (10% relative decline), this is still substantial degradation. The paper doesn't adequately discuss whether this trade-off is acceptable or how to further mitigate it.\n4. Incomplete Comparison with Text-Guided Generation: Table 5 shows mixed results: the model underperforms GLM-4-Voice* (text-guided) on TriviaQA and WebQA S→S tasks. Missing comparison: What if you added text guidance to YOUR model? This would isolate whether architectural improvements matter beyond the guidance mechanism.\n5. Incomplete Evaluation of All Claims: While the evaluation covers reasoning and QA well, some claims about expressivity and paralinguistic capabilities are not deeply evaluated. Some examples or analysis showing the model can generate non-verbal vocalizations (laughter, hesitation) that text cannot represent would strengthen this work.\n6. Limited Scope of Evaluation: No evaluation with real human speech. No evaluation on multi-turn dialogue scenarios, or interactive human in the loop conversations\n7. Ablation Study Limitations: It would be interesting to ablate the layer split position (currently at layer 32 of 36)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)