Closing the Gap Between Text and Speech Understanding in LLMs
Authors: Santiago Cuervo
Skyler Seto
Maureen de Seyssel
Richard He Bai
Zijin Gu
Tatiana Likhomanenko
Navdeep Jaitly
Zakaria Aldeneh
Venue: ICLR 2026 Poster (2026)
Keywords: Speech language models
large language models
multimodal language models
modality alignment
cross-modal alignment
cross-modal transfer
cross-modal distillation
modality gap
speech processing
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large Language Models (LLMs) can be adapted to extend their text capabilities to speech inputs. However, these speech-adapted LLMs consistently underperform their text-based counterparts—and even cascaded pipelines—on language understanding tasks. We term this shortfall the text–speech understanding gap: the performance drop observed when a speech-adapted LLM processes spoken inputs relative to when the original text-based LLM processes the equivalent text. Recent approaches to narrowing this gap either rely on large-scale speech synthesis of text corpora, which is costly and heavily dependent on synthetic data, or on large-scale proprietary speech datasets, which are not reproducible. As a result, there remains a need for more data-efficient alternatives for closing the text-speech understanding gap. In this work, we analyze the gap as driven by two factors: (i) forgetting of text capabilities during adaptation, and (ii) cross-modal misalignment between speech and text. Based on this analysis, we introduce SALAD—Sample-efficient Alignment with Learning through Active selection and cross-modal Distillation—which combines cross-modal distillation with targeted synthetic data to improve alignment while mitigating forgetting. Applied to 3B and 7B LLMs, SALAD achieves competitive performance with a strong open-weight model across broad-domain benchmarks in knowledge, language understanding, and reasoning, while training on over an order of magnitude less speech data from publicly available corpora.

Published as a conference paper at ICLR 2026
CLOSING THE GAP BETWEEN TEXT AND SPEECH UNDERSTANDING IN LLMs
Santiago Cuervot; Skyler Seto', Maureen de Seyssel', Richard He Bai', Zijin Gul, Tatiana Likhomanenko', Navdeep Jaitly Zakaria Aldeneh*
Université de Toulon, Aix Marseille Université, CNRS, LIS Apple
ABSTRACT
Large Language Models (LLMs) can adapted to extend their text capabilittes speech inputs. However, these speech-adapted LLMs consistently underperform their text-based counterparts-ann even cascaded pipelines-on language under- standing tasks. We term this shortfall the lext-ypeech understanding gap: the performance drop observed when speech-adapter LLM processes spoken inputs relative when the original text-based LLM processes the equivalent text. Recent approaches to narrowing this gap either rely on large-scale speech synthesis of text corpora, which is costly and heavily dependent on synthetic data, proprietary speech datasets, which are not reproducible. As a result, there remains need for more data-efficient alternatives for closing the text-speech understand- ing gap. In this work, we analyze the gap as driven by two factors: (i) forgetting of text capabilities during adaptation, and (ii) cross-modal modal misaligmment between speech and text. Based on this analysis, we introduce SALAD Samplc-efficient A Alignment with Learning through Active selection and cross-modal Distillation- which combines cross-modal distillation with targeted synthetic data to improve alignment while mitigating forgetting. Applied to 3B and 7B LLMs, SALAD achieves competitive performance with strong open-weight model across broad- domain benchmarks in knowledge, language understanding. and reasoning, while training over order of magnitude less speech data from public corpora.
large-scale
Speech Iadapted
(original
Thxt sprtosprech unde erstanding gap (%6)
DIVA Lláma BB
100
25
Volco--9B
Qwen2 Audia 70
Accuracy
15(
10.
3B
AD-78
20
7B
SALAD-3B
Qwen2.5 Omni 70
T0
10"
10
CtoryClo
MMSU
OBQA
ARC-C
HellaSwag
PIQA
Training speech data size in hours scale)
Qwea2 Qwea2-Audio-75 Audio:7
DiVA-Llama-9i
GLM NB
Qwen2.5 500.5-0mmi-----
SALAD -38
SALAD-70
Figure I: SALAD reduces the text-speech understanding gap on general reasoning and language understanding benchmarks while requiring over order of magnitude less training data than com- peting speech-adapter LLMs. Left: The gap between the performance of the base LLM given text inputs and the performance of the speech- aadettdd LLM given speech imputs as a function of training dataset sizd"] Right: The performance of the base L.MM given text inputs and the performance of aneech-adapted L.LM given speech inputs across several language understanding tasks.
"Work done during internship Apple.

---

Published as a conference paper at ICLR 2026
1 INTRODUCTION
Large language models (LLMs) have demonstrated impressive capabilities in general knowledge and reasoning. often surpassing specialized systems across a wide range of tasks. This success has motivated efforts to extend LLMs to the speech domain, opening new possibilities for spoken nat- ural interaction. A straightforward approach to extend LLMs to the speech domain is the cascaded pipeline, where automatic speech recognition (ASR) models map speech to text and the LLM is applied to the transcribed text. While effective in preserving text capabilities, cascaded pipelines largely remove speaker and paralinguistic cues essential for natural spoken interaction (Maimon it al. 2025b). To address this limitation, recent work has explored end-to-end approaches, adapting
lext-based LLMs to directly process speech inputs (Tang et al. 2024 Chu et al. 2024 Xie & Wu
2024 Fang et al. 2024, Nguyen et al. 2024 Défossez et al. 2024). Despite their promise, speech- adapted LLMs struggle with the core requirement of language understanding, consistently under- performing text-based LLMs and even cascaded systems on language understanding tasks (Cuervo) & Marxer, 2024] Chen et al., 2024 Cui et al., 2025). We refer to this shortfall as the text-speech understanding gap: the performance drop when specch-adapted LLM performs a language under- standing task in the speech domain compared to the original LLM performing the same task in the text domain. Closing this gap is crucial step toward building AI systems capable of truly natural spoken interaction.
Prior work has proposed several strategies reduce this gap. A common direction is cross-modal alignment, achieved either by optimizing the fusion between speech encoders and text LLMs to promote modality-agnostic representations (Tang al. 2024 Deng al. 2025 Held et al.) 2025 Tseng et al. 2025) or by explicitly training for consistent outputs across modalities given equivalent inputs (Fathullah et al. 2024; Wang al. 2024 Held al. 2025). Another direction is data-driven methods, which synthesize large-scale speech from text corpora to narrow the distribution gap be- tween speech and text training domains (Zeng et al. 2025). While these methods show performance improvements, they focused on narrow-domann benchmarks and several did not evaluate perfor- mance relative to the original text-based LLMs. When evaluated on broader benchmarks, these approaches showed substantial drops compared to their text-based LLM backbones (Chen et al.) 2024). Despite these efforts, the text-speech understanding gap persists. More recent methods (Xul et al. 2025 Team et al. 2025) demonstrated notable progress, but remain irreproducible due to missing training details and their reliance on massive proprictary speech datasets spanning millions over 100B text tokens?] i.e., budget comparable full pretraining budgets in the text domain. Given the scarcity of publicly available speech and parallel speech-text data relative to text data, more sample-efficient methods are needed.
of hours-equivaled
In this work, we seek to understand the text-speech understanding gap in greater depth to bet- ter guide the design of efficient remedies. Bridging the text-speech understanding gap requires speech-adapted LLM to retain the knowledge of its text-based counterpart (i.e., avoid forgetting) and produce consistent outputs for equivalent speech and text inputs (i.e., avoid cross-modal mis- alignment). Forgetting refers to the loss of pretrained text capabilities during adaptation to speech, a well-documented effect of domain shift between pretraining and fine-tuning in L.MSs (Béthune) ee al. 2025). Cross-modal misalignment is observed when semantically equivalent speech and text inputs give divergent outputs. We quantify forgetting and cross- cross-modal misalignment, and study these measures under different training objectíves and data regimes. The gained insights lead us to pro- pose SALAD: Sample-efficien Alignment with Learning through Active selection and cross- modal Distillation, -ample-efficient method to address the performance gap. Our main contributions are:
We quantify forgetting and cross-modaa misalignment as the statistical distances between the out- LLM and its text- based LLM backbone on matched text sext-speech inputs
puts of a from -omadn pretraining corpus, and show that these measures are highly predictive of the text-speech understanding gap on broad-domain benchmarks.
We find that training on karrow-domaii speech corpora with standard objectives used in prior work worsens both forgetting and broad-domain misalignment as the amount of training data increases. In contrast, using a cross-modal knowledge distillation objective-wher the text-
For Qwen2.5-Ome mmmm, we approximate the dataset size hours hased on the reported number of pretraining audio tokens. However, the proportion of this data that is actual speech is not reported. Esttmatedd from the average duration of text tokens (~~320 ms) in our data.
2

---

Published as a conference paper at ICLR 2026
Deastty
0.2
Fine Webe Wdu-EEu
LibriHleavy
Emilia
0.0
Mesallll But..
Edgannnnn s...
Medical Gond.
Giimate Chan..
Melecalar
Nutritoonl
Medical Sert..
Literary Sut.
Webatta Cont.
Figure 2: Distribution of samples across 64 automatically annotated domains (x-axis). Text corpora Web-Fuu Penedo al. 2024)
wide range of domains, whereas speech datasets (Lib- riHeavy: Kang et al. 2024 and Emilia: He et al. (2024) are concentrated in only a few. The ten least represented domains in speech are labeled in grey.
based LLM backbone serves as the teacher-improve alignment and mitigates forgetting, even when trained on narrow-domain data.
when trained only on narrow-domain data. To further address this misalignment, introduce active learning algo- rithm that targets domain mismatches between natural speech and text corpora.
We show that cross-modal distillation alone leaves residual misalignment
We propose method that builds on insights from analyses, apply it 3B and 7B LLMs, and benchmark them against recent speech-adapted LLMs the 3B-9B range on spoken versions of six broad-domain knowledge and reasoning benchmarks. Our models outperform most baselines in spoken language understanding and perform competitively with the strongest, while training on over order of magnitude less data.
PRELIMINARIES
intermediate representation conditioned on speech inputs, and then generate speech conditioned on that text (Xie & Wu 2024 Défossez al.J [024]] Fang et al. 2024| Xu al., 2025 Kimi et al. 2025). In this work, we focus on generating intermediate text conditioned since it directly reflects language capability, leaving the task of generating speech for future work. Specifically, we aim to build speech-adapted language model Pa. parameterized by weights 0, that defines a probability distribution over the next text token given a multimodal context. Let e denote such a context, which may contain subsequences of text tokens and/or speech representations a. For each position i, the model predicts the distribution over the next text token W conditioned on Cgsi Po (1 C.s).
Recent end-to-end methods for speech-adapted LLMs typically generate
speech input as our primary task
We seek to train Po so that its predictions match the distribution of (w, c) pairs drawn from the natural language distribution Q. Most prior works approximate Q by minimizing the negative log-
likelihood on a speech dataset (Zhang et al.) 2023 Nguyen et al.) 2024 Défossez et al. 2024 Chu
al. 2024
Zeng et al. 2025):
Σ (W.C
log (o egi).
(1)
For this approach to succeed, the dataset must be a sufficiently representative sample of Q. How- available speech datasets narrow in scope and fail to capture the full general language distribution. This limitation is evident in the domain distribution shown Figure
Transfer learning is used to alleviate this limitation. with Po initialized from pretrained text-only language model @e-which better approximates Q-in the hope of enabling general knowledge transfer when fine-tuned narrow speech data. In practice, however, models trained this way under-perform on spoken language understanding tasks relative to Qs (Chen et al. |024 Cui et al. 2025). One contributor this under-performance is cross-modal misalignment, i.e., inconsistent predictions across modalities, which we formalize as
M = Σ Pra(Po w || Po(ws-1 cgs)),
(2)
(w.e)~Q
3

---

Published as a conference paper at ICLR 2026
where D-I(-I-) denotes the Kullback-Leible divergence between two distributions, denotes text sequence, с denotes semantically equivalent multimodal context sequence, and W;41 denotes the next text token. This quantity measures how differently the model predicts the next token when conditioned on text versus equivalent multimodal context.
Another contributor this performnee spoken language understanding tasks relative to Qo is forgetting, which measures the loss of the original text behavior:
F
E w~
DRI(Qo
wgg) || Po Po (wg)).
This quantity measures how much the speech-adapted model Pe diverges from the text-hased L.LM Qo on text inputs, indicating loss of text knowledge and reduced ability to transfer capabilities to the speech domain.
ANALYZING THE TEXT-SPEECH GAP
In this section, we study how cross-modal misalignment (Bquation 2 and forgetting (Equation 3 in speech-adapted LLMs affect downstream language understanding performance, and how differ- ent design decisions impact these two metrics. Specifically, we train multiple models while vary- ing training objectives, datasets, and training budgets. Then, for each trained model, measure cross-modal alignment, forgetting, and the performance on broad-domain language understanding benchmarks.
3.1 OBJECTIVE
We train models in pretraining setup, modeling general sequences without the data templates used instruction-tuie tutteed-tteed models. This choice reflects our focus on broad-domain alignment, avoid- ing restriction dialogue data and acknowledging the scarcity of speech instruction data. Our data therefore consist of multimodal sequences e composed of interleaved speech a and text w inputs, as in Nguyen al. (2024). For our training objective, we introduce variable a [0, 1] that inter- polates between standard maximum likelihood objective, and a cross-modal distillation objective, similar that used by Wang et al. (2024); Held al. (025)::
L(D,0) a LDIST(D, (1 a) (D,
(4)
where
LDIST (I3,
Σ
Σ feet at i+1) DrI(Qs(wiz) wsi)) || Po egi)),
(5)
(w.C)ED
( =
Σ
Σ 1 ftext at i+1) log Po (g().
(6)
(w.o)E
Here, 1 (text L+1) is an indicator function equal to the (i+1)-th element in the interleaved context is text token, and otherwise.
3.2 DATA
We consider the two natural English speech corpora LibriHeavy (Kang et al. 2024) (read speech) and the YODAS-EN subset of the Emilia dataset (He et al.) (024) (conversational, both among the largest and most semantically diverse publicly available speech datasets. However, shown in Figure 2 they still lack domain coverage relative to text pretraining corpora. Since forgetting is driven by domain shift, also synthesize spoken version of 10B-text-tols subset of Web- Edu (Penedo et al. 2024) a high-quality broad-domain text pretraining corpus-44 to study the im- pact of aligning text and speech training domains, following the approach of [Zeng al. (0255)

---

Published as a conference paper at ICLR 2026
R 0.750
R = 0.744
Avg. speech speechperf. perf. (%)
Avg. text perf: (%)
79
66
77
64
75
Suno
(2
73
60
71
58
3.2
10
9.70
0.52
69
0.04
0.12 Forgetting (F)
0.08
0.16
Cross-modal Misalignment (log(M))
Figure 3: Relationship between speech performance and cross-modal misalignment (Equation 2
(left); relation between text performance and forgetting (Equation (right).
We use the Kokoro-TTS mode with the af-heart voice, which provides the highest quality generation to synthesize the data.
To produce interleaved speech-text sequences, segment each utterance into subsequences and interleave spans random length at runtime: 10-30 words for text segments and 5-15 words for speech segments. For LibriHeavy and Emilia, word-level timestamps of the corresponding tran- scriptions are obtained using the forced aligner from Pratap et al. (2024). For synthesized speech, use the built-in functionality of Kokoro TTS to get word-level timestamps.
3.3 BENCHMARKS
We evaluate on broad-domain benchmarks of general knowledge, reasoning, and language under- standing commonly used for LLMs, considering both their text and spoken versions: StoryCloze (Hassid et al. 2023), MMSU and OpenBookQA (OBQA) from VoiceBench (Chen et al. 2024). HellaSwag (Zellers et al. 2019). ARC-Challenge (Clark et al. 2018)) and PIQA (Bisk et al. 2020
We adopt a feo-shot prompting approach for evaluation across all tasks, with accuracy as the metric. For further details on the benchmarks, Appendix A.5 A.5
3.4 ARCHITECTURE
We follow the standard design of speech-adapted LLMs (Arora et al. 2025), consisting of a speech encoder that extracts representations from waveforms, an adapter that maps them into the input space of the language model, and the language model itself. For these experiments we initialize Po from the text LLM Qwen2.5-3B (Qwen et al.) (025) base model, and use the same text LLM as teacher Qo in Equation 5 allowing us to measure how much original text capability is maintained when processing speech and how much is lost during speech training.
While prior work has paid great attention to speech encoder and adapter stpiially to pro- mote cross-modal alignment by making speech representations more text-like (Tang et al.] [222]]
Deng et al.) 2025) Held et al., 2025 Tseng et al., 2025 we we adopt simple architecture: the lightweight Mimi speech tokenizer (Délossez al. 2024 as encoder and 1 122M-parameter stack of transformer decoder layers as adapter. We make this choice because current representation align- ment methods rely large non-causal encoders and complex modules unsuited for low-latency streaming, which is essential for downstream conversational speech-adapted LLMs (Défossez et al.) 2024). Accordingly,
models with low-level, non-text-like repre- sentations "worst-case" input alignment scenario, expecting our findings to generalize to more aligned text-speech representations while being directly applicable lowenat architectures. The encoder remains frozen during training, while the adapter and language model are optimized. For further details on the model architecture, see Appendix A.I
causal, streaming-frientt
http:::/hugginggfce..oo/// htt.://gg.......e/////////////oa http://kwooooooo/obbbbbbb ht i WW..CCSS....
5

---

Published as a conference paper at ICLR 2026
3.5 RESULTS
We train our models by selecting a from {0, 0.25, 0.5, 0.75, 1}, selecting training data D from (Emilia+LibriHaary Hineeieeaaa..ieeeebeeeeeee FineWeb-Edu), and adjusting training budget D within the range (2B-12B tokens). Each model is trained using the hyperparameters described in Appendix A.2 For each trained model, we measure cross-modal misalignment (Hquation ( and forgetting (Equation B on test subset of our text-speech version of FineWeb-Edu and calculate the average accuracy on the broad-domain benchmarks. We present the results below.
How do cross-modal misalignment and forgetting relate to broad-domain performance? Figure shows scatter plots with ordinary least squares fits for models trained on barrow-domain speech data (LibriHeavy + Emilia): (i) average speech perfor- mance (%) vs. (igisaaliig,mnt)) and (ii) average text performance (%) vs. forgetting. The solid line indicates the fitted regression; the shaded band is the 95% confidence interval for the mean prediction. Each panel also reports the Leave-One-Our Cross-
Table I: Univariate and partial R2 explaining the variance speech and text performance, attributed to forgetting and misalignment.
Predictor
Speech perf. R²
Text pert. R²
Forgetiing
0.455
0.744
0.750
0.614
Misalignment Forgetting
0.563
-0.001
Forgetting | Misalignment 0.049
0.323
Validation (LOOCV) R² of the univariate fit. The results show that speech performance declines with increasing misalignment (LOOCV R² 0.75), and text performance declines with increas- ing forgetting (LOOCV R2 = 0.74). Table reports both univariate and partial It² (i.e., variance explained when adding the other factor). Misalignment uniquely explains a large share of speech variance given forgetting (partial R x 0.56), while forgetting uniquely explains text variance given misalignment (partial R² R5 0.32). These patterns hold when controlling for a and training budget, with unique cross-valiin validated R2 of ~ 0.34 for speech (misalignment) and ~ 0.23 for text (forgetting). For more details on the least squares analysis, see Appendix A.6
How does the training objective interact with cross-modal alignment and forgetting? Figure reports scaling curves for cross-modal misalignment, forgetting, and average speech performance. Training with NLL (i.e., a = 0 in Equation on narrow-domain speech data (LibriHeavy + Emilia) leads increasing cross-modal misalignment with scale. Given the strong relation between mis- alignment and speech performance shown in Figure 3 NLL training on narrow-domain data also yields the weakest results. NLL training leads to greater forgetting of the pretrained text behavior compared to models trained with nonzero a values. This greater forgetting, however, has limited impact on the average text performance (see Appendix A.7).
Higher values of a yield better alignment, and for > 0, training on narrow-domain data generalizes to reduced broad-domain misalignment with scale. Misalignment is well described by a typical log- linear neural scaling law (Kaplan al. 2020 Hoff- mann et al. 2022). We fit scaling laws of misalign- ment with respect to the training budget D of the form M = E + B D-3 where E denotes the irre- ducible misalignment and B and 3 capture scaling efficiency. The fitted laws are reported in Table 2 along with LOOCV R² and the estimated number of tokens required to reach within 5% of E. The irre- ducible misalignment E decreases with a, and for all 0 < < 1, misalignment saturates early in training. Distillation is therefore the most scalable approach with respect to misalignment. Although forgetting increases slightly with scale regardless of this effect has limited impact on performance.
Table 2: Scaling laws of misalignment with training tokens (D) for runs with a > 0: M E+B D 3 Reported are the LOOCV R² and needed to be within 5% of F.
E
B
8
Reooe
Tokens@05%
LibrilHeav + Emilia
0.25 0.32
45.1
0.46
0.81
4.94B
0.50
0.21
3.2e10
1.34
0.75
2.07B
0.75 0.16
1.2e8
1.04
0.85
6.00B
1.00 0.13
3806
0.54
0.91
47.58B
Fine Web-Edu
1.00
0.04
5.4
0.56
0.96
268.4B
Does a better data domain match between speech and text training alleviate the issues of NLL training? Figure 4 shows that training on broad-domain data (FineWeb-Bel Web-Edu) with = 0 reduces misalignment relative to narrow-doman data, although misalignment still grows with scale. This in- dicates that domain matching alone does not resolve cross-modal misalignment. Counterintuitively,

---

Published as a conference paper at ICLR 2026
forgetting is slightly worse than with narrow-domain training. but the difference is small. Speech performance is not tied to misalignment, with the model outperforming others despite higher mis- alignment. However, while a > 0 yields consistent improvements with scale, NLL training broad-domain data shows no meaningful scaling gains.
Does better data domain match between speech and text training yield gains for cross-modal distillation? Figure 4 presents the results with a = and broad-domain (FineWeb-Eduu -duuer-nn-- training data. In this setting, misalignment is essentially the quantity being optimized- distilatiinn directly targets cross-modal consistency- and, since the evaluation distribution (Fine (eeb-Edu-tesst matches the training distribution, the metric aligns with the objective. Accordingly, domain-matchn distillation yields the lowest misalignment and the strongest specch-understanding performance.
Takeaways. Our analyses highlight two central insights: (i) cross-modal knowledge distillation objective is more effective than maximum likelihood for mitigating misalignment and forgetting (Figure 4 Table 2 and (ii) better matching the domain of speech training data to that of text pre- training yields further gains when combined with cross-modal distillation. We use these insights to design learning strategy address the text-speech understanding gap in the next section.
4 CLOSING THE TEXT-SPEECH GAP
Natural speech corpora are narrower in terms of domain coverage compared to text pretraining corpora (Figure 2). Large-scale synthetic speech (i.e., same size as text pretraining corpora), while useful for improving the domain coverage, is both costly and lacking in the paralinguistic richness essential for natural spoken interaction (Debnath al. 2024 Minixhofer al. 2025). is therefore desirable to reduce reliance on large-scale synthetic speech data. To this end, we propose SALAD: Sample-efficient Alignment with Learning through Active selection and modsl Distillation. SALAD is designed directly around our two key insights: distillation ensures robust alignment and mitigates forgetting, while active selection enables closer domain matching through minimal, model-guided infusion of synthetic speech data rather than costly large-scale synthesis.
4.1 METHOD
We structure SALAD as a two-stage process. Stage (Distillation on Natural Speech) trains Pa on natural speech by minimizing LDIST between Pe and Qo on Dapesch (Equation 5) leveraging the strong scaling behavior of distillation until alignment plateaus at the irreducible misalignment E, which, shown in Table 2 occurs within practical training budget. Stage II (Active Se- lection for Domain Expansion) then addresses the residual misalignment by introducing a small but strategically chosen amount of synthetic speech through active selection, guided by the model's own misalignment signals. This targeted augmentation complements Stage I by expanding domain
72
0.20
Misalignment
Forgetting
Avg. speech perf
69
0.15
66
0.4
0.10
63
0.2
0.05
0.00
10"
10
10
10
10"
10"
Tokens
Tokens
Tokens
LibriHeavy Emilia
Fine
0.25
0.5
0.755
Figure 4: Impact of the training objective (controlled by in Equation 4 the number of training tokens (shown on the x-axis), and the dataset choice on cross-modal misalignment (Equation 2) forgetting (Equation B) and average speech performance.
7

---

Published as a conference paper at ICLR 2026
coverage while keeping reliance synthetic data minimal, consistent with emphasis sample efficiency and the use natural speech.
First, aim to develop sampling strategy for choosing which text samples to synthesize. We draw inspiration from CRISP (Grangier et al.) 2025), adopting a clustered importance-sampling strategy that derives target-domain dataset from broad-domain corpus by reweighting clusters. Concretely, let Deb be a large broad-domain text corpus. We partition into K clusters K (c))== using sentence embeddings and k-means, and define
K
[D(c) wehl Dewh
Pweh (c) =
D(e) web
{w E IDweh : w E K(c)) J.
We compute per-cluster importance weights as w(c) = (una(o) (a)(c) In CRISP, Plrger(C) comes from the
distribution of a small target-domain dataset across the clusters. In our case, no such dataset exists, so we let the model itself define Pusget. Specifically, we treat the divergence between Po and Q within each cluster proxy for how much that cluster belongs to the "missing" domain. We define
Parger (c) = Pweb(c) (M(()))
[((m) = m
Ed=1 (web(c') (,(((())
(e) where Mcc DIST (D )robe) is the misalignment at cluster c, measured on a small probe set ((e) prohe C ID(c) for which we pre-synthesize speech. Clusters with higher misalignment-where Po diverges most from Qø are therefore upweighted. The resulting importance weight
P Puuge((C) Pweb
w.,(c) =
& (x(M(c)).
The parameter 7 2 controls the trade-off: y reduces sampling purely from Pweb, while larger values focus more heavily on clusters with greater misalignment.
We sample clusters in proportion their importance rather than reweighting per-example losses, thereby avoiding the need to synthesize and train on the entire D Given fixed synthesis budget, repeatedly draw cluster according to Purress select text sequence wo uniformly from [D(e] synthesize its speech a, and form an interlcaved context e. Each sample is added to the active dataset Dactive until the budget exhausted, after which Detive used continue training Ps. To prevent forgetting of Stage training, we combine Davive with ID speech and minimize LDIST spppecti UlDeeivee 0).
web
4.2 SETUP
We apply our method to the Qwen2.5 3B and 7B base LLMs (Qwen al. 2025 yielding the SALAD-3B and SALAD-7B models. We follow the experimental setup of Section for architec- ture, data, and evaluation tasks. We use Emilia and LibriHeavy (141, 612 hours) as Dapoosh and use a 10B-token Fine Web-Edu subset as our We For Stage II, we train clustering model on Dweb using balanced k-means with K 128 over BAAI/bge large een--n--oi en-vl. embeddings with synthesis budget of 1% of ID speech and 5 (Equation 8) We train SALAD models for 24B tokens during Stage and additional 1.9B tokens during Stage II. Further training details are provided in Appendix A.3
and additional ablations are reported in Appendix A.8
We benchmark SALAD models against the following speech-adapted LLMSs from the literature: Qwen2-Audio (Chu et al. 2024), DiVA (Held et al.) 2025), GLiM-4-oice (Zeng et al. 2025). and Qwen2.5-One 1-en2.5-Omni (Xu et al. 2025). We also evaluate against a cascaded pipeline that pairs Whisper- Large-v3 ASR with the Qwen2.5 LLMs used as the backbones of Qwen2.5- and our SALAD models. The cascade pipeline serves as our topline reference for spoken language understanding. For each model, report the per-task accuracy well the eext-speech gap, defined as the difference between the performance of speech-adapted LLM given speech input and the performance of the original bast- LLM given the corresponding text input.
8

---

Published as a conference paper at ICLR 2026
Table SALAD outperforms most basclines and is competitive with the strongest speech-adapted LLMs and cascaded pipelines. "Acc." denotes the accuracy of the speech-adapted LLM given speech imput; "Gap" denotes the difference between the accuracy of the text-based LLM given text
input and the "Ace." column. Gap cells are color-coded from best (green) to worst (red) for each task, where lower is better.
StoryCloze
MMSU
OBQA
HellaSwag
ARC-C
PIQA
Avg.
Acc.
Gap
Acc.
Gap
Acc.
Gap
Acc.
Gap
Acc.
Gap
Acc.
Gap
Acc.
Gap
Random
50.0
25.0
25.0
25.0
25.0
50.0
33.3
Cascaded toplines: ASR (Whisper-v3-Large) LLM
Qwen2.5-3B Qwen2.5-7B
82.7
0.2
58.1
3.8
76.9
4.9
78.5
0.1
73.8
2.5
ASR
84.2
0..8
67.1
3.7
84.0
5.0
74.7
2.0
86.5
79.9
0.0
79.4
2.2
and systems
Owen2-Audio-P
71.9
9.0
29.5
18.7
39.6
37.1
64.1
7.9
43.5
28.5
73.4
5.4
53.7
17.8
RVVA-Llama311-4 1-8B
68.6
19.7
36.1
25.8
40.9
42.4
54.2
21.3
45.9
36.0
70.0
11.4
52.6
26.1
GI.M-4 Vnice--999
20.6
38.6
28.7
63.4
Qwen2.5-0mni-77
80.1
4.9
61.0
-9.8
85.5
3.5
68.4
8.3
87.1
78.0
1.9
76.7
5.0
SALAD-38
Stage I
75.5
7.4
47.3
14.6
65.5
16.3
68.8
2.2
75.6
6.2
78.3
0.3
68.5
8.0
Stuge
75.8
7.1
52.5
9.4
76.7
68.7
79.9
78.1
0.5
72.0
4.6
SALAD-7B
Stuge I
81.5
3,5
55.3
15.5
69.7
19.3
74.2
2.5
82,3
6.1
80.3
0.4
73.9
7.9
Stage
81.5
3.5
57.5
13.3
75.1
13.9
74.0
84.0
80.3
0.4
75.4
6.2
4.3 RESULTS
Table summarizes the performance of approach compared existing baselines and the cas- caded pipeline topline. Overall, SALAD achieves performance competitive with the strongest model. Qwen2.5-Omni, while using over an order of magnitude less speech data (Figure [T. In particular, SALAD-3B outperforms all larger end-to-end baselines except Qwen2.5-Omnt show- ing that strong text-speech alignment can achieved with much smaller models when combined with our training strategy. Relative to cascaded toplines, the strongest SALAD models are compet- itive, underperforming them only slightly while retaining the advantages of end-to-ent modeling.
We next ask whether targeting the misaligned domains responsible for the performance gains we observe. Table 4 shows the effect of Stage training when using our active data selection strategy compared to uniform sam- pling. Active data selection shows greater gains in MMSU, OpenBookQA, and ARC-C. These tasks involve scientific questions and more tech- nical terminology than the others, making them more likely to fall outside the natural speech distribution on which the model is trained in Stage I (Figure 2). For more analyses on the effect of active selection see Appendix A.8
Table 4: Performance (Accuracy, %) of the SALAD-3B model after Stage II training with active selection vs. random (uniform) selection.
MMSU OBQA HellaSwag ARC-C PIQA
Uniform 75.0
49.5
71.9
68.6
78.9
78.1
Active Sel. 75.8
52.5
76.7
68.7
79.9
78.1
Finally, we ask how well our approach preserves the text capabilities of the original text-based LLM backbone compared to the baselines. Table shows that, unlike other speech-adapted models, which exhibit substantial forgetting, SALAD maintains the closest performance to the original text LLM. This result highlights the effectiveness of the distillation objective constraining the model to remain faithful to its teacher while learning achieve cross-modal alignment.
use
DIVA model available
https: httpss///uuggLaaaaccc..///...
117A-llama-33 82) While Held et al. (2025) report freezing a Llama-3-8B backbone, the re-
leased version on HuggingFace appears to be based on Llama-3.1 3.1-BB Moreover, we found the released weights to differ from the Jlama-3.1-89 checkpoint, which explains the differences in text performance
reported in T

---

Published as a conference paper at ICLR 2026
Table 5: SALAD best preserves the original text capabilities of the text-based LLM backbone after speech training compared to the baselines. "Acc." denotes the accuracy of the sreech-adapted LLM given text imput; "Gap" denotes the difference between the text-based LLM given text input and the "Acc." column. Gap cells are color-coded from best (green) to worst (red) for each task, where lower better.
StoryCloze
MMSU
OBQA
HellaSwag
ARC-C
PIQA
Avg.
Acc.
Gap
Acc.
Gap
Acc.
Gap
Acc.
Gap
Acc.
Gap
Acc.
Gap
Acc.
Gap
Owen2-Audio-711
81.1
-0.2
46.9
1.3
73.4
3.3
69.4
2.6
68.6
3.4
76.0
2.8
69.2
2.2
DiVA-Llamar 1-8B
80.9
7,4
60.9
1.0
82.0
1.3
67.8
7.7
81.7
0.2
80.8
0.6
75.7
3.0
GLM-4-Voice-9B
81.6
17.2
48.4
17.8
69.9
17.8
70.6
9.9
70.9
22.4
77.9
-3.4
69.9
13.6
Qwen2.5-0mni-77
80.5
4.5
66.3
4.5
87.3
1.7
69.8
6.9
87.6
0.8
78.8
1.1
78.4
3.3
SALAD-3B
82.7
0.2
62.5
-0.6
83.7
-1.9
70.5
0.5
83.1
-1.2
78.6
0.0
76.9
-0.5
SALAD-7B
84.9
-2.0
71.6
-0.8
90.1
-1.1
76.9
0.2
89.2
-0.8
80.2
-0.3
82.2
-0.9
RELATED WORK
Cross-modal transfer in speech-adapted LLMs. A large body of work has focused on trans- ferring the general language capabilities of text LLMs to the speech domain. Deng et al. (2025);
Wang et al. (2023); Tang al. (022)); Chu et al. (2023 2024); [uu al. (2024) adapted text LLMs to process speech through multi-task supervised learning, combining tasks like speech recognition with speech instruction-followe following data to promote general-domain transfer. Other works, such as
Fathullah et al. (2024); Wang et al. (2024); Held et al. (0025). addressed the scarcity of broad- domain supervised data by deriving rich supervision from unsupervised paired text-speech datasets via cross-modal distillation. Related to our method with a = 1.0 in Equation 4 Wang et al. (2024);
cross- Held et al. (2025) showed that dense supervision mpaervision-mmtccingg the full output distribution
modal distil lation can outperform prior sispervised-eeamingg -based approaches. Several works (Wang ct al. 2021 Held al. 2025 Tseng et al. 2025) further promoted transfer through specialized rep- resentation alignment modules, typically transformer encoders with downsampling and query-based cross-attention, and WWag et al. (2024); Tseng et al. (2025) additionally matched sampling rates across modalities. While these works demonstrated varying degrees of cross-modal knowledge transfer, none matched the performance of text models of equivalent capacity on reasoning- and knowledge-ing nhowledgeinttnsive intensive LLM benchmarks. Among open- coan-ttaining-eete text-speech understanding gap by 11.7% over the next-best model and rivals the strongest closed- recipe system-withe 1.2%- using over an order of magnitude less training data and main- taining expressive, streaming-friendly speech representations. Our analysis of forgetting and mis- alignment also provides actionable insights into the factors driving the text-speech gap and the training dynamics of such models.
models, SALAD reduces the
Domain matching text-speech pretraining. Cuervo & Marxer 2024
demonstrated the bene- fits for speech LM training of using TTS-syythesized spoken version of a small pretraining corpus designed to promote essential language capabilities. ZZng et al. (2025) synthesized an LLM pretrain- ing corpus (FineWeb-BBae
corresponding to Dweh our study) for interleaved text-speech language modeling. Our active selection approach enables the of far less synthetic speech by employ- ing adaptively to cover domain gaps in natural speech corpora. Moreover, we show that domain matching should be paired with cross-modal distillation achieve general-domann alignment.
6 CONCLUSION
In this work, we studied the text-speech understanding gap, identifying forgetting of text capabilities and modal misalignment as the two factors behind the underperformance of speech-adapted LLMSs compared to their text-based counterparts. Building on this analysis, we introduced S SALAD, a sample-efficient approach that combines cross-modal distillation with active data selection to tar- get these challenges directly. Our experiments on 3B and 7B models demonstrated that SALAD achieves competitive performance with strong open-weight baselines, while using over an order of magnitude less data. These results suggest that carefully designed objectives and data selection strategies can substantially reduce reliance on costly, massive synthetic data or proprietary speech resources, paving the way for data-efficient methods to close the text-speech understanding gap.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlfčck, Agustín Piqueres Lajarin, Vaibhav Srivastav, Joshua Lochner, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Hao- jun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric training of small language model, 2025. URL https:://axivv http////www.orraas///52.2.02.0
Siddhant Arora, Kai- Wei Chang, Chung-Ming Chien, Yifan Peng, Haibin Wu, Yossi Adi, Emmanuel Dupoux, Hung- Yi Lee, Karen Livescu, and Shinji Watanabe. On the landscape of spoken language models: A comprehensive survey, 2025, Ittsstt::://aax
http:/bb//200000 08528
Louis Béthune, David Grangier, Dan Busbridge, Eleonora Gualdoni, Marco Cuturi, and Pierre Ablin. Scaling laws for forgetting during finetuning with pretraining data injection. In Pro- ceedings of the 42nd International Conference on Machine Learning (ICML), 2025. URL
Stttps://ppereeee wtt.l/ewweeeeeeeeeeeeeeeeeeeee.e fod 133mm
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. Voicebench: Benchmarking LLM-based voice assistants, 2024.
17196
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models, 2023. URL htttt::
abs/ http//b///2131113
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report, 2024.
URL https: httpps://axxvv.oo org/ abs 10759
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning chal- lenge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- ties (ACL), 2018
Santiago Cuervo and Ricard Marxer. Scaling properties of speech language models. In Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 351-361, Miami, Florida, USA, nov 2024. Association for Computational Linguis- tics. doi: 10. 22tp://1.2222..mmmmaaa. emrlp ma in. 217
URL Utthhttss:// hhtttppss//el.annoooooooooo///// org/2024.
Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, and Irwin King. Voxeval: Benchmarking the knowledge understanding capabilities of end-to-end spoken language models. Proceedings of the 63rd An- nual Meeting of the Association for Computational Linguistics (ACL), pp. 16735-16753, Vienna, Austria, jul 2025. Association for Computational Linguistics. doi: 2118866511222555 lon-..o... 818. htLhtt::: http:::::////aaaaaaoooo....//// org/ 202500020002 2025.accliong; 818/
Diptasree Debnath, Asad Ullah, Helard Becerra Martinez, and Andrew Hines. Naturalness and the utility of synthetic speech in model pre-training. Proceedings of the Workshop on Synthetic Data's 's Transformative Role in Foundational Speech Models (SynDatt4GenA/ 2024), pp. 101- 105, Kos, Greece, 2024. ISCA. doi: 10221477/SyDDaaa44GGnAla... htgp:/hh22___yaddaaaaaeeeeaaa
htt
isca-archive.coo http://iiiie...r/y//ttttataaaaaa
Alexandre Défossez, Laurent Mazaré, Manu Orsini. Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dia- logue, 2024. URL http////tss////aa//o//r/////////

---

Published as a conference paper at ICLR 2026
Keqi Deng, Guangzhi Sun, and Phil Woodland. Wav2prompt: End-to-end speech prompt learn- ing and task-based fine-tuning for text-based LLMs. In Proceedings of NAACL-HLT 2025, pp. 6940-6956, Albuquerque, New Mexico, apr 2025. Association for Computational Linguis- tics. doi: 10. 1..18865/11222555 http//2022..11111....55555 URL ht https:////....... http:::s//aaaaaaahhllcccc org/2025. naacl- 3547
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023. [RR[aatss//axxiii www.aaiwww....aaaaaaaaaaao abs 2tabb//2305 2305.
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni:
Seamless speech interaction with large language models, 2024. htttps://aaxii>
abs/2409.0663
Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Ma- hadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. Audiochatllama: Towards general- purpose speech abilitics for LLMs. In Proceedings of NAACL-HLI 2024, pp. 5522-5532, Mex- ico City, Mexico, jun 2024. Association for Computational Linguisties. doi: 10.18653/v//200 1/2024 naacl-long.309. URL aattps:// aal/accannn http:s://aaaanthoooogg
David Grangier, Simin Fan, Skyler Seto, and Pierre Ablin. Task-adaptive pretrained language mod- cls via coustered-importanne samplinn In International Conference on Learning Representations (ICLR), 2025. URL http:::://tppeeerraaea/
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL htttps::/pppeeeeiiiiii
net / Forum?id: Eorum?Iddy11gSET]
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Défossez, Gabriel Synnaeve, Emmanuel Dupoux, Roy Schwartz, and Yossi Adi. Tex- tually pretrained speech language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL /RRp:tttp://ppeerriiii
net wttpm//rrmmmmm/////e//mm ideuiHueeUUUUU.
Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation, 2024. UtLhtts:://aaat
.rgg abs/ 2407 05361
William Held, Yanzhe Zhang, Minzhi Li, Weiyan Shi, Michael J. Ryan, and Diyi Yang. Distilling an end-to-end voice assistant without instruction training data. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 7876-7891, Vienna, Austria, URL
jul 2025. Association for Computational Linguisties. doi: 10. 1/265///1/2055.cl1llloo
nitts://aaclanie wwwnnhhhooyy..oo wow.ooog../ooooooo..oo..oo.oo..o 2025. long. 3887
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL https:: /ttps:///axivv..a orq/ abs/ 2203. 15556)
Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xun- ying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, and Furu Wei. WavLLM: Towards ro- bust and adaptive speech large language model. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp- 1552-4572, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 111138/61/02/4/ /
URL. https: //acl//ccll manthelooy..rgg.
Wei Kang, Xiaoyu Yang, Zengwei Yao, Fangjun Kuang, Yifan Yang, Liyong Guo, Long Lin, and Daniel Povey. Libriheavy: A 50,000 hours ASR corpus with punctuation casing and context. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 10991-10995. IEFF, 2024. doi: 10.1 D0.1099/CASSP44455..0 10447120
12

---

Published as a conference paper at ICLR 2026
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020. URL (RLLtttt://axatt /ttppps://aaxv..../ org/ abs 2001. 08361
KimiTeam, Ding Ding, Zeqian Ju. Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, Jun Chen, Yanru Chen, Yulun Du, Weiran He, Zhenxing Hu, Guokun Lai, Qingcheng Li, Yangyang Liu, Weidong Sun, Jianzhou Wang. Yuzhi Wang, Yuefeng Wu, Yuxin Wu, Dongchao Yang. Hao Yang, Ying Yang, Zhilin Yang, Aoxiong Yin, Ruibin Yuan, Yutong Zhang, and Zaida Zhou. Kimi-audio technical report, 2025. URL https:
//arxiv. Taa2xi..orr/ass/255011111 18425
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9: 11336-155,, 2021.
doi: 10.1 a uRI.https://
aclanthelogy org. wac.ll.tacl.....
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Con- ference on Learning Representations (ICLR), 2019. URL httpp:://pperreiiii .et/
forum?id=Bkg652
Gallil Maimon, Michael Hassid, Amit Roth, and Yossi Adi. Scaling analysis of interleaved speech- text language models. In Second Conference on Language Modeling, 2025a. URL https:
//openzeview.
Gallil Maimon, Amit Roth, and Yossi Adi. Salmon: A suite for acoustic language model evalua- tion. In ICASSP 2025 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025b. doi: 10.1 (0.1099/CASSP4996000200 10888561.
Christoph
Minixhofer, Ondrej Klejch, and Peter Bell. Scaling laws for synthetic speech for model training. In Proceedings of Interspeech 2025, ISCA, 2025. URL httpp:://www //ww..
wwt.eerrrriineepeeee..eo 2025/miniiii httm/mmnixxoofffeee interspeech.por
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino. Benoit Sagot, and Emmanuel Dupoux. Spirit LM: Interlcaved spoken and written language model, 2024. URL htttp:://axii.. E2C2.05755
/
Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In NeurIPS Datasets and Benchmarks Track, 2024. URL https:
//openreview.
Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi,, Alexei Baevski, Yossi Adi, Xi- aohui Zhang. Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Scaling speech technol- ogy to 1,000+ languages. Journal of Machine Learning Research, 25(97):1-52, 2024. URL http: /ttt:://mmx.... http:://mlx..oog//ee..mem http://oorpppapee////222222 .hml.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu. Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia. Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhennu Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL
www.org///./.o/.oooooo..oo.....t /245//444 2412 15115
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language mod- els. In International Conference on Learning Representations (ICLR), 2024, URL https: //openreviess httpr//wwwwe.etetooooooon//oaaee
13

---

Published as a conference paper at ICLR 2026
Liang- Hasuan Tseng, Yi- Yi-Chang Chen, Kuan- Yi Lee, Da-Shan Shiu, and Hung- Yi Lee. Taste: Text- aligned speech tokenization and embedding for spoken language modeling, 2025. URL https:
wwexxv..rrg/abs/25500 07053
Changhan Wang, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Ann Lee, Peng-Jen Chen, Jiatao Gu, and Juan Pino. fairseq s22: A scalable and integrable speech synthesis toolkit. In Proceedings of EMNLP 2021: System Demonstrations, pp. 143-152, Online and Punta Cana, Dominican Repub- lic, nov 2021. Association for Computational Linguistics. doi: 10.
17. /tLL:tttt////alltttooooo wwwpoogy.org.eoe... wrg//02g11 emn demo.
Chen Wang. Minpeng Liao, Zhongqiang Huang, and Jiajun Zhang, Bisp-kd: Bootstrapping language-specch pre-training via knowledge distillation, 2024. URL httts::/aaxii..ot abs/2405.19041
Minggiu Wang, Wei Han, Izhak Shafran, Zelin Wu, Chung-Cheng Chiu, Yuan Cao, Nanxin Chen, Yu Zhang, Hagen Soltau, Paul K. Rubenstein, Lukas Zilka, Dian Yu, Golan Pundak, Nikhil Sid- dhartha, Johan Schalkwyk, and Yonghui Wu. Slm: Bridge the thin gap between speech and text foundation models. 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8, 2023. doi: 10.1 109/ASRUS7994.20000 10389703.
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming, 2024. URL arxiiiiiiiv../... org/ abs / 2408 16725
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang. Bin Zhang, Xiong Wang. Yunfei Chu, and Junyang Lin. Qwen2.5-omm technical report, 2025. Btthttts::/ aariivv orq/ abs / 2503/5533.. 20215
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong, and Jie Tang. Scaling speech-text pre-training with synthetic interleaved data. In International Conference on Learning Representations (ICLR), 2025. URL
eww net forum? ? d=
jtvyE
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abil- ities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15757- 15773, Singapore, dec 2023, Association for Computational Linguistics. doi: 10.18653/v1/2023. findings-emnlp.109 1055. URL settps//accanthoon 20t.a//rcoooooooooo22000...... wttp//22........oo fi -enddiing--eppp....
1055
A APPENDIX
A.1 MODEL DESCRIPTION
Speech encoder. The speech encoder transforms 4a speech audio frames a E 18, into a sequence of d,-dimensional latent speech representations Z Rl of length 1, < la- We use the Mimi speech tokenizer (Défossez et al.) 2024), causal lightweight speech encoder. Mimi produces multi- codebook representations Z E R xqxd, where q is the number of codebooks. We add the repre- sentations across codebooks, resulting in Σ Z... l.: E xd, The speech encoder remains frozen.
Adapter. Z encodes low-level phonetic and acoustic information. The role of the adapter is to transform Z into Z' E Ri.xd, a higher-level, more text-like representation. We implement the adapter as a stack of decoder-only transformer layers, preserving causality and streamability. If d, # d, linear projection is applied at the output to obtain d-dimensional representations. We evaluated several adapter sizes and found performance to saturate at around 122M parameters; this configuration was therefore adopted for all reported experiments. The adapter consists of 12 Llama- style decoder layers with residual dimension 960, MLP dimension 2560, and 15 attention heads with 5 KV heads.
14

---

Published as a conference paper at ICLR 2026
Language model. The language model processes multimodal sequences of text and speech rep- resentations H E Rixd and outputs a probability distribution over a vocabulary Vi of text to- kens. Subsequences of H may correspond either to adapter-output speech sequences Z' or to quences of text embeddings
E RR xd obtained by applying an embedding function to text tokens (Wi,-.... 10,,)) with m, V.. H is processed by stack of transformer decoder layers, and the output logits at position i are linearly predicted from the corresponding hidden representation, yielding / dildennmsioo logit vector that is Softmax-normalize
into the probability distribution P(w++1| H.v). We initialize the language model from pretrained text language models from the Qwen2.5 family LLMs (Qwen et al. 2025).
A.2 TRAINING DETAILS: ANALYZING THE TEXT-SPEECH GAP
The models are trained with the Adam optimizer (Loshchilov & Hutter 2019) with a weight decay
of 0.1. We adopt a warmup-stable-decay learning-rate schedule (Hägele et al. 2024), consisting of linear warmup of 500 steps followed by linear decay to zero over the final 20% of training. The peak learning rate is tuned separately for each model size, with distinct values for the language- model backbone and the adapter. We use a learning rate of 10-3 for the adapter and 5 x 10 for the LLM. All models are trained with a batch size of approximately 1 M tokens and a context window of 2048 tokens.
A.3 TRAINING DETAILS: CLOSING THE
GAP
All models follow the hyperparameters and setup in Appendix A.2, except that SALAD-3B uses learning rate of 10-3 for the adapter and 5 x 10-5 for the LLM; SALAD-7B uses 10 for the adapter and 5 x 10-6 for the LLM.
In Stage each batch is sampled with probability 2/3 from speech and 1/3 from the SmolLM corpus, following the common practice of mixing in pretraining data to mitigate forgetting (Béthune jet al., 2025] Nguyen et al.] 2024, Zeng et al.) 2025). In Stage II, batches are drawn with equal probability from Dspeechh Dactive and the SmolLM corpus (Allal et al. 2025).
For Stage II training of SALAD models, we resume from the last checkpoint before learning-rate decay and continue for 1.9B tokens with a linear decay of the learning rate. An exception is the ex- Table (Appendix (..8), where training was resumed from the checkpoint
periments in Figure after decay and continued for 950M tokens with constant learning rate fixed to the post-decay value. These runs were conducted before we identified the setup that yielded stronger results for SALAD models.
During initial Stage II experiments, we observed that clusters with high misalignment often corre- sponded to domains such as academic citations (including DOIs and URLs) lonllsh content, which produced hard-to-parse TTS artifacts. These clusters were inherently difficult to align not necessarily because of domain gaps, but because of mismatches between the -TS-generated speech and the ground-truth text. Consequently, we filtered out the top 15% of clusters (19 clusters for our 128-cluster setup) with the highest character error rate (CER), measured by transcribing the gener- ated speech with Whisper-v3-larye and comparing it to the ground-truth text. After removing these TTS-adverse clusters, we observed no meaningful correlation between CER and misalignment, and therefore attribute the remaining misalignment to domain gaps.
A.4 CLUSTER ANNOTATION PIPELINE
We generate the domain annotations Figures 2 and 9 by annotating embedding clusters with an LLM (Claude 3.7 Sonnet). Below, we outline the annotation and validation procedures. For a clustering model with 64 clusters, the judge validated the annotations with an accuracy of 56% (random chance is 1.6%).
Phase A: Explain. For each cluster, sample À positives (closest to the centroid) and negatives (from the M nearest neighbor clusters). An LLM is prompted with these examples to propose a short title, descriptive label, and inclusion/exclusion criteria.
15

---

Published as a conference paper at ICLR 2026
Listing 1: Explain prompt.
System:
You expert explaining clusters short texts.
Return STRICT JSON with fields: short_title, label, inclusion_criteria, exclusion_eriteria, confidence
User:
"cluster_id"; int,
ceottancc_metric"" "cosine" "12",
"positives": ("id": int, "distance"; flcat, "text": stri, I "negatives": ("id": int, "Erom_cluster"" int, "di "distance"; float, "text": str), ... 1, "instructions": "Name and describe ONLY the positive cluster concept. Use distances as prototypicality cues."
Listing Judge prompt.
User:
"categories"; ("id": int, "title": str, "description": stri, I, "sample": "text": str }
"instructions": "pick exactly one best-matching category id."
Phase B: Judge. We construct catalog of all cluster labels and ask the LLM to classify holdout texts into exactly one cluster. This produces multiclass accuracy estimates per cluster and overall. The pipeline supports resumability and incremental judging, enabling efficient large-scale evalua- tion.
Prompt format. The LLM receives a ISON payload and is instructed to return JSON only. List- [ shows the structure of the EXPLAIN prompt.
The JUDGE prompt follows the same format, with a catalog of labeled clusters and a single sample to classify. Listing 2 shows its structure.
A.5 EVALUATION PROTOCOL
We StoryCloze Hassid al. 2023), MMSU and OpenBookQA from VoiceBench (Chen al.
2024). HellaSwag (Zellers et al. 2019). ARC-Challenge (Clark et al., 2018), and PIQA (Bisk et al, 2020 for evaluation. In all cases, we synthesize spoken versions from the corresponding text data using the Kokoro TTS tokenizer with the heart speaker-including for benchmarks such as StoryCloze and SoiceBench--o are multiple-choice benchmarks. For each task, the model estimates the normalized log probability of each answer given the context, and accuracy computed by checking whether the scchiig- option matches the gold answer.
maintain control over prompt formats, as described below. All
Table presents the prompt templates used for each task. the VoiceBench versions of MMSU and OpenBookQA, the model receives the list of answer options as part of the prompt context. However, we observed that smaller models, as well as models from the literature (c.g., DiVA), performed close to random under this setup. This behavior is consistent with prior findings that small language models often struggle with multi-choie formats when answer options are embedded in the input prompt (Allal et al. 2025). To address this limitation, synthesized our versions of MMSU and OpenBookQA with controlled prompt formats, including a continuation variant in which the model predicts the correct answer directly. For each model, report performance under the prompt
16

---

Published as a conference paper at ICLR 2026
format that yields the best results. We also observed differences between predicting the answer as the full option or just the letter label of that option. We also evaluate both variants.
Since most of basclines are instruction-tunee models, adopt when needed each model's native chat template: the "Answer:" " segment is assigned to the assistant role, while the remaining context is presented user input. To mitigate performance differences due to prompting, we further condition models on the task format using few-shot demonstrations. For each task, we include as many demonstrations can fit in the audio context window of 2048 tokens, up to maximum of five. The number of shots used per task is: StoryCloze (5), MMSU (4), OpenBookQA (5), HellaSwap (3), ARC- CRC-Challenge (1), and PIQA (5). For each model, we report results with the pest-performing prompt format.
17

---

Published as a conference paper at ICLR 2026
Table 6: Evaluation task prompts. Placeholders shown in monospace. Blue highlights denote text/audio inputs, and orange highlights denote continuations where likelihood is evaluated.
StoryCloze
<story prefix>
<answer>
MMSU (continuation)
The following are multiple choice questions (with answers) about <topic>
Question: <question>
Answer: <answer>
MMSU (multiple choice)
The following are multiple choice questions (with answers) about <topic>
Question:
<question>
A. <A>
B. <B>
C. <C>
D. <D>
Answer:
<answer>
OpenBookQA (continuation)
Question: <question?
Answer: <answer>
OpenBook QA (multiple choice)
Question:
<question>
A. <>>
B. <B>
C. <C>
D. <D>
Answer: <answer>
HellaSwag
<context> <answer>
ARC-Challenge (continuation)
Question:
<question>
Answer: <answer>
ARC-Challenge (multiple choice)
Question: <question>
A. <A>
B. <B>
C. <C>
D. <D>
Answer: <answer>
PIQA
<question>
<answer>
18

---

Published as a conference paper at ICLR 2026
Table 7: Type-II Analysis of Covariance (ANCOVA) results controlling for (fixed effects) and log( tokens). Left: dependent variable is average speech performance (%). Right: dependent variable average text performance (%). Reported are degrees of freedom (df). sum of squares (SS), mean
squares (MS), F-statistics (F), and corresponding p-values (p). Partial 82 values quantify the unique variance explained by each factor.
Speech (8)
Text (%)
Term
df
SS
MS
F
Term
df
SS
MS
F
log (or(misalignment)
2.789
2,789
11.51
0.0025
forgetting
1
0.2588
0.2588
7.945
0.0097
(FE)
2.255
0.5638
2.327
0.0867
(FE)
4
0,4121
0.1030
3.163
0.0329
log [tokeen))
38.39
38.39
158.4
8.5 x 10
log (tokens)
1
0.0661
0.0661
2.031
0.168
Residual
0.2423
Residual
0.749
Partial RP: log(mssalignnmett) 0.33,
0.29, log( (okens) 0.87
Partial R2: lorgetiin 0.26,
0.35, log(tokens) 0.08
A.6 ORDINARY LEAST SQUARES ANALYSIS
We fit ordinary least-squares models with (i) average speech performance (%) as
Vspeech Bo 81 log(misalignment) %o + 3r log(tokens) + E,
and (ii) average text performance (%) as
Blext = Bo + 31 forgetting + Ta + Br log(tokens) + E,
where Va are fixed effects for a E {0, 0.25, 0.5, 0.75, 1}.
We report Type-II ANCOVA in Table 7 for each term, compare the full model against the reduced model with that term removed (while keeping the other terms), yielding S.eem F, and p. We also report partial R "SSeem(SSSuu + SSessia).
Both analyses indicate that, after controlling for a and training budget, misaligmment is strongly associated with speech performance, and forgetting is strongly associated with text performance. To quantify out-of-sample explanatory power of the focal predictor beyond controls, we also compute the partial LOOCV R²: R Re versus R² Royoroossss We obtain 0.34 for misalignment (speech) and > 0.23 for forgetting (text), consistent with the ANCOVA results.
A.7 EFFECT OF DISTILLATION ON TEXT PERFORMANCE
Figure shows how the average text performance varies with the choice of a Equa- tion where (Emiiia+ibrrHeen Hin.iiieaay..ieeeeeee..ee FineWWe-Edu with training budgets ranging from 2B to 12B tokens. Overall, average text performance is less sensitive to these changes compared average speech performance. Nonetheless, we observe consistent, but small, improvement when training with the distillation ob- jective (a relative to the NLL objective (a = 0).
{0, 0.25,, 0.5, 0.75, 1}. The models are trained on data drawn from D e
75.9
Avg. text perf. (%)
75.6
75.3
75.0
74.7
74.4
10"
10
Tokens
Librilleavy
Emilia
dFine
0.25
0.5
0.75
Figure 5: Impact of the training objective (controlled by a Equation 4). the number of training tokens (shown the æ-axis), and the dataset choice on the average text performance.
19

---

Published as a conference paper at ICLR 2026
A.8 ACTIVE SELECTION ANALYSES
SALAD makes use of an active selection algorithm in Stage to identify and cover gaps between natural speech and broad-domain text datasets. We analyze the role of this stage and study the impact of its hyperparameters on
73.0
72.5
10
72.0
30
the overall performance. For these experiments, we ap-
Ave. spoedh
71.5
ply Stage II training to the model trained with a = 1.0
in Section 3] During Stage II, the model is trained for additional 950M tokens.
21.0-
70.5
70.0
Does two-stage training scale better than just first- stage training? An important question is whether the improvements from the active learning stage stem from its design, or merely from the extra training steps added after Stage I. Figure g shows that Stage II yields consis- tent gains across tasks over Stage I for given budget, with the exception of StoryCloze, where performance de- teriorates slightly.
0.
5%
10%
Syn. data bodget % of natural speech datzsot size
Figure 6: Stage performance for S SALAD-3B with different synthetic data budgets and selectivity ? values (K 128 clusters).
Is active selection sensitive to the number of clusters used? We present in Table the perfor- mance for different values of the number of clusters K E (64, 128, 256), using 7 5 and a Stage synthetic data budget of 5% of the natural speech dataset size (LibriHeavy + Emilia). We find that varying has only a negligible effect on the overall average performance.
Table 8: Performance (Accuracy, %) of the SALAD-3B model after Stage II training with ac- tive selection (synthetic data budget of 5% of the natural speech data budget, 7 5.0) across dif- ferent number of clusters K E {64, 128, 256).
How important is targeting domain gaps as more synthetic data is allowed? While Ta- ble B and Figure [ show meaningful Stage II gains-and Table A corroborates them-the role of domain-gap targeting remains unclear. Is it necessary to explicitly target domain gaps, or would tuning on a small amount of target, general-domain data suffice improve align- ment and performance? How does this trade-off evolve as we increase the selected data?
K
SC
MMSU
OBQA HellaSwag ARC-C PIQA
Avg
64
74.8
52.4
74.3
68.9
78.4
78.1
71.1
128
74.5
52.5
75.8
68.7
78.4
78.2
71.4
256
74.1
51.5
72.1
68.6
77.8
78.5
70.4
To investigate these questions, we vary in Equation which controls selectivity, and sweep the Stage II syn- thetic budget from 0.1% to 10% of the natural-speech dataset size (LibriHeavy + Emilia). We evaluate 7 e {0, 5, 10, 30), where y = 0 corresponds to uniform sam- pling from the target domain. Figure 6 reports average performance across spoken tasks as a function of bud- get. Active selection with 7 = 5 consistently outper- forms all other settings across budgets, underscoring the importance of targeting domain gaps. By contrast, over- focusing on gaps (7 5) yields gains only at very small budgets (0.1%) and falls behind even uniform sampling larger budgets, suggesting that while selective target- ing is beneficial, maintaining exploration and diversity is equally crucial.
g
Sugelt
13.9%
McNemar two skded, p 0.05
43%
Bedlgge
Jow
chansiery
oodor
fustioral
Amatd
Figure Improvements in accuracy be- tween Stage and Stage II for SALAD- 3B across MMSU categories. We high- light the categories with statistically sig- nificant improvements.
Does the active learning stage identify meaningful domain gaps? Figure 7 breaks down the accuracy improvements across MMSU categories from Stage I to Stage II, showing that the largest statistically significant gains occur in categories such as biology and chemistry. Figure g shows the density of samples per cluster for the top-10 clusters selected by the active learning algorithm. Using the LLM-assisted annotation procedure described in Appendix A.4, we assign domain label to each cluster and find that the most heavily sampled domain is Molecular Biology. These findings support our interpretation that Stage II boosts performance on these benchmarks by targeting more
20

---

Published as a conference paper at ICLR 2026
StoryCloze
OpenBookQA
HellaSwag
PIQA
ARC-C
80
50
73
83
Accuracy (%)
70
75
63
70
10
40
10
10
10
10
66
10
Tokens
Tokens
Tokens
Tokens
Tokens
Tokens
Stage (distil))
Stage 11 (active learning)
Figure 8: Impact of the two-stage training process on SALAD-3B, shown a function of the number of training tokens and reported across spoken language understanding tasks.
Table 9:
Impact of mismatched Stage II and evaluation speakers for SALAD-3B VoiceBench MMSU and OpenBookQA.
Evaluation speaker
MMSU
OBQA
Acc.
Gap (%)
Acc.
Gap (%)
Kokoro TTS; af ffeeteart voice (Original)
52.5
9.4
76.7
5.1
Kokoro TTS; fffbeella voice
53.0
8,9
76.0
5.8
VoiceBench Google TTS; male voice
52.4
9.5
74.3
7.5
technical domains. Moreover, they indicate that our active learning algorithm effectively identifies and addresses meaningful domain gaps, leading measurable performance improvements.
Do the gains of Stage II generalize across speak- ers? Both synthetic data from Stage II and the evaluations are generated with the same speaker, se- lected the highest-uuality voice available in text- to-speech system. A natural concern is that the ob- served gains from Stage II might generalize other speakers. To address this concern, we evaluated on the original VoiceBench samples from MMSU and Open- BookQA-the tasks showing the largest improvements in Stage II-which use a different synthesizer and a speaker of the opposite gender from the one used in our Stage II training. Table reports the results. While performance decreases slightly. large fraction of the Stage II gain preserved, indicating that the improvements are robust and not tied single speaker's characteristics syn- thesizers.
15.0%
12.5%
Moloodar Bealngr
The botical Sterum.
Bescarcc Periima saidd
Higgriitien theiadee of
Compttag Dee seevddastion
Itstors Polscrad...
Ecdunna aosal Guadaacee
Literatote
Biabical Feetl
Library Satternea
Figure 9: Distribution of Stage II- selected samples across top- most sampled domains.
A.9 OPEN-ENDED GENERATION EVALUATIONS
SALAD is pre-training method, and the standard way evaluate pre-trained LLMs is through multiple-choice benchmarks, where likelihood-baser accuracies can be computed-site such as the benchmarks used throughout the main body of this paper. These evaluations directly align with the pre-training objective and are therefore well suited for base models. In contrast, the open-ended generation benchmarks commonly used for instruction-turee L.LMs rely on controlled, instruction- following behavior, and not applicable to base model, which does not reliably follow for- matting or prompting constraints.
Nevertheless, to obtain coarse sense of the generation quality of SALAD-trained models, we fol-
low prior work on evaluating pre-trained LMs (Lakhotia et al. 2021 Maimon et al. 2025a Eldan
& Li 2023 and compute three ceneration-oocseel metrics samples generated from 300 Lib-
niHaayy Lest-cled prefixes. For all generations, we use top-k sampling with = 250 and temperature 0.7.
21

---

Published as a conference paper at ICLR 2026
Table 10: Generation-quality evaluation for Qwen2.5-3B and S3LAD-3B. Metrics include GenPPL., AutoBLEU, and LLM-as---judge scores (1-10 scale). For GenPPL and AutoBLEU we report the median (Q1-Q3), and
adjusted Mann- Mann-Whitney p-values against the Qwen2.5-3B baseline. Speech prompts for SALAD-3B use natural LibriHeavy speech syn- thetic speech generated with Kokoro TTS -eere.
GenPPL
AutoBLEU
LLMMaasa-judge
Median (QI-Q3)
p-value
Median (QI-Q3)
p-value
Median (QI-Q3)
p-value
Qwen2,5-3B (buseline)
3.519 (1.608-11.540)
0.257 (0.195-0.302)
7.30 (6.75-8.25)
SALADD-3B
Given
3.105 (1.504-7.660)
0.269 (0.205-0.33) 00339)
0.126
7.50 (7.00-8.25)
Given natural speech
4.103 (1.793-9. 293)3333
1,000
0.247 (0.190-0.323))
1,000
7.50 (7.00-8.25)
1.000
Given synthetic speech
3.855 (1 (1,960-96 (960-9.028)
1,000
0.245 (0.205 00.32)
1.000
7.50 (7.00-8.25)
1.000
GenPPL [Lakhotia et al.,
2025a perplexity of generated text under independent reference LM (SmoILM-1.7BB) assessing fluency;
2021 Maimon et al.
AutoBLEU (Lakhotia et al. 2021): D: self relf-repetition metric quantifying diversity and pe-
nalizing over-repetitive behavior typical of base models; and
LLM--as---judee scores (Eldan Li 2023), evaluating coherence, relevance, creativity, and fluency on 1-10 scale. The prompt is shown in Listing We use Claude 3.7 Sonnet with greedy sampling as judge.
To determine whether SALAD training affects generation quality relative to the text-only backbone, report these metrics for SALAD-3B and its Qwen2.5-3B backbone model. For SALAD-30 we evaluate generations conditioned on (i) text prompts, (ii) natural speech prompts, and (iii) syn- thetic speech prompts produced using the same setup in our main Koakuations-Koorro TTS with the af af-heart voice. All results, together with Welch's 's unpaired two-sample p-values computed against the Qwen2.5-31 backbone, are shown in Table I0 Across prompting modalities no metric shows any statistically significant difference relative to the Qwen2.5-333 backbone (Holm-adjusted Mann-Whitme Whitney tests), and we conclude no clear degradation generation quality.
22

---

Published as a conference paper at ICLR 2026
Listing 3: LLM-as-a-judge prompt.
System:
You are an expert literary critic and linguist, specialising in the evaluation of Al-generated creative text.
You MUST return ONLY a valid JSON object with this exact schema:
"coheronce"; <int 1-10>, "relevance"; <int 1-10>, "creativity"; <int 1-10>,
"fluency": <int 1-10>,
"coherance_just": "<brief ixplanation?",
"relevance_justt"; "<brief explanat iopllnaaiio>>> oon>", "ereativity_just:: "chrief explanation>", "aluunno_jjutt"" "<brief explanation>", "summary": *<one paragraph synthesis>"
Do NOT wrap the JSON in markdown backticks. Do NOT add any extra keys.
User:
INSTRUCTIONS FOR EVALDATION SCOPE:
The completiens
from base LLM truncated at fixed length given small book snippet as prefix, therefore:
1. Ident if the core completion pertion of the generated completion. R.g. ignore any trailing incoplete sentences or clearly off-topic text product of the I.I.M effectively switching to fifferent document.
2. Your analysis must stop at the end of the last complete sentence of this core completion.
3. Ignore any trailing incomplete sentences, loops, meta-commentary tokens like </endoftext
SPECIAL CONSIDERATIONS FOR THIS EVALUATION (included ONLY for audio_bin tts_mimi :
Punctuation Leniency: The prompt arrived via audio; if the generation continues the last sentence of the prompt, not penalise sentence fusion.
Name Leniency: Accept minor variations in proper nouns if the character 's zole and gender stays consistent.
Story Prompt: (STEM_PROMPT}
Generated Completion: (GENERATION)
Please provide the JSON rubric now.
23


Reviews:
[{"review_id": "VaU8OclD0v", "reviewer": "Reviewer_nz78", "summary": "This paper identifies and analyzes the \"text-speech understanding gap\" in speech-adapted Large Language Models (LLMs), attributing it to two quantifiable factors: forgetting of pre-trained text capabilities and cross-modal misalignment. To bridge this gap, the authors propose SALAD, a two-stage method comprising: (1) Cross-modal distillation from the text-based LLM teacher to the speech-adapted student model, and (2) Active selection of a minimal amount of synthetic speech data to target residual domain misalignment. Experiments on 3B and 7B models show that SALAD achieves competitive performance with state-of-the-art models while using significantly less speech data.", "questions": "- Given that  BLSP-KD already demonstrated the power of cross-modal distillation for this problem, what is the marginal contribution of the active learning component? \n- Please provide an ablation on the cluster count K and the exponent γ.\n- The paper choses a \"worst-case\" encoder to make a strong claim. However, how does SALAD compare against state-of-the-art non-causal encoders with built-in alignment mechanisms (e.g., TASTE)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Formally defining and quantifying the text-speech gap via forgetting and misalignment.\n- The paper is well-written and easy to follow.\n- The paper shows that high performance can be achieved with significantly less data.", "weaknesses": "- The paper does not meet the standard for a thorough related work section, failing to properly situate itself within the current literature and justify its novelty.\n- The paper fails to discuss and contrast its approach with highly relevant work, such as BLSP-KD and TASTE.\n- The paper lacks of an ablation study on the hyperparameters, such as K and γ.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fIooGt8Sdn", "reviewer": "Reviewer_TPWY", "summary": "This paper addresses the persistent performance gap between large language models (LLMs) adapted for speech input and their original text-based counterparts. While LLMs excel at text-based language understanding, adapting them to process speech directly results in a notable drop in performance - which is termed as \"text - speech understanding gap\".  The key contributions of this paper include:\n1. it analyzes the text–speech understanding gap in a quantifiable way and diagnoses 2 main factors that may cause this gap: forgetting and cross-modal misalignment.\n2. it proposes a 2-stage training strategy called SALAD that first does cross-modal distillation on natural speech data to improve alignment and mitigate forgetting, followed by an active synthetic speech sample selection to address domain coverage issue.\n\nEmpirical results show that SALAD achieves competitive performance compared to strong open-source speech-adapted LLMs, with much less speech data.", "questions": "1. In Table 3, How to interpret that SALAD-3B often achieves better/smaller text-speech gap than SALAD-7B? Doesn't larger model require more training steps?\n2. Could there any negative impacts on model's capabilities in understanding audio/paralinguistic inputs when adopting SALAD training?\n3. In Table 4, does the fact that SALAD-7B gets boosted on text task after SALAD training indicate that the training sets employed is giving SALAD models advantage in these tasks? (i.e. in-domain vs out-of-domain?)\n\nI am giving a rating of 6 but am open to reconsider when authors answer these questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\nThis paper formalize the common phenomenon in speech-adapted LLMs where the LLMs exhibits significant gap between text and speech understanding. The paper also proposes a quantifiable metric to measure this phenomenon and provides statistical measure for both forgetting and cross-modal misalignment.\n\nQuality:\n1. The paper presents a thorough empirical evaluation, benchmarking SALAD against a wide range of open-source speech-adapted LLMs.\n2. The analysis appears to be rigorous, accompanied with clear quantification of the factors contributing to the \"text - speech understanding gap\". \n\nClarity:\nThe paper is well organized and easy to follow. \n\nSignificance:\n1. This work attempts to address a challenge when training a speech-LLM: it requires massive amount of speech data to achieve on-par speech understanding capabilities which is often not accessible. SALAD exhibits comparable performance while being less data-hungry.\n2. The insights into the root cause of the text-speech understanding gap could be influential for future research on speech-LLMs.", "weaknesses": "1. The experiments focus on English speech and text, which couldn't address the generalizability to other languages or domains. Similarly, the synthetic data only contains 1 voice, which raises question about how the model can adapt to speaker variance.\n2. Besides the quantitative results, it would be beneficial to provide some qualitative analysis to intuitively demonstrate how SALAD training minimize the text speech gap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lOagNIXLJs", "reviewer": "Reviewer_VbQm", "summary": "This paper addresses the text-speech understanding gap—the performance drop when speech-adapted LLMs process spoken inputs versus text inputs to text-based LLMs. \nThe authors decompose this gap into two factors: (i) catastrophic forgetting of text capabilities during speech adaptation, and (ii) cross-modal misalignment between speech and text representations.\nBased on this analysis, they propose SALAD (Sample-efficient Alignment with Learning through Active selection and cross-modal Distillation), which combines cross-modal knowledge distillation with active data selection to improve alignment while mitigating forgetting. Applied to Qwen2.5 3B and 7B models, SALAD achieves competitive performance with recent speech-adapted LLMs while reportedly using much less training data.", "questions": "Check weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Well-motivated problem decomposition: The two-factor analysis separating forgetting from cross-modal misalignment is conceptually clear and provides a useful framework. The mathematical formalization using KL divergence (Equations 2-3) enables quantitative measurement of both factors.\n2. Rigorous empirical analysis (Section 3): The systematic study of how different training objectives (α parameter) affect forgetting and misalignment is valuable. The scaling law analysis with fitted curves (Table 2) and cross-validation provides insights into training dynamics.\n3. Clear presentation and organization: The paper is well-written with logical flow from problem analysis to solution. Mathematical formulations are precise and figures effectively communicate key results.", "weaknesses": "1. Missing validation of core motivation: The paper motivates end-to-end approaches over cascaded systems by citing ability to capture \"paralinguistic richness essential for natural spoken interaction\" (Introduction). Yet no experiments evaluate paralinguistic understanding (emotion, prosody, speaker characteristics). The distillation objective enforcing identical text-speech distributions may actually suppress these cues, contradicting the motivation. This is very important because if content is the only thing that you want to model, then you can simply do cascaded system and in fact ASR+LLM backbone (Qwen2.5) is still significantly better as shown in Table 3.\n2. Limited experimental scope: Only multiple-choice QA tasks—no open-ended generation. Would be curious to see if findings still hold in open-ended generation.\n3. Entirely synthetic evaluation undermines validity: All benchmarks evaluate TTS-generated speech rather than natural speech. This raises questions about whether improvements generalize to real speech with acoustic variability, accents, and spontaneous characteristics. Table 8's limited test on VoiceBench shows performance degradation with different TTS speakers, but more analysis is needed.\n4. Active selection assumes high-misalignment clusters represent domain gaps, but they could equally represent intrinsically difficult content or TTS artifacts. Some discussion around this would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)