Scaling Analysis of Interleaved Speech-Text Language Models
Authors: Gallil Maimon
Michael Hassid
Amit Roth
Yossi Adi
Venue: COLM 2025 (2025)
Keywords: Speech Language Models
Scaling Analysis
Speech-Text Interleaving
TL;DR: We perform the first scaling analysis of Interleaved SpeechLMs showing they scale more efficiently and differently than existing SpeechLM scaling.

Abstract: Existing Speech Language Model (SLM) scaling analysis paints a bleak picture. It predicts that SLMs require much more compute and data compared to text, leading some to question the feasibility of training high-quality SLMs. However, modern SLMs are often initialised from pre-trained TextLMs using speech-text interleaving to allow knowledge transfer. This raises the question - "Do interleaved SLMs scale more efficiently than textless-SLMs?" In this paper we answer a resounding yes! We conduct scaling analysis of interleaved SLMs by training several dozen and analysing the scaling trends. We see that under this setup SLMs scale more efficiently with compute. Additionally, our results indicate that the scaling dynamics significantly differ from textless-SLMs, suggesting one should allocate notably more of the compute budget to increasing model size over training tokens. We also study the role of synthetic data and TextLM model families in unlocking this potential. Results suggest that our scaled up model achieves comparable semantic speech performance to leading models, while using less compute and data. We open source models, samples, and data - https://pages.cs.huji.ac.il/adiyoss-lab/sims/ .

Published as a conference paper at COLM 2025
Scaling Analysis of Interleaved Speech-Text Language Models
Gallil Maimon, Michael Hassid, Amit Roth, Department of Computer Science and Engineering Hebrew University of Jerusalem galli wwa.mmmmmmi...luiuuul..... ac.
Yossi Adi
Abstract
Existing Speech Language Model (SLM) scaling analysis paints a bleak picture. It predicts that SLMs require much more compute and data com- pared to text, leading some to question the feasibility of training high- quality SLMs. However, modern SLMs are often initialised from pre-trained TextLMs using speech-text interleaving to allow knowledge transfer. This raises the question Do interleaved SLMs scale more efficiently than lexiless- SLMs? In this paper we answer a resounding yes! We conduct scaling analysis of interleaved SLMs by training several dozen and analysing the scaling trends. We see that under this setup SLMs scale more efficiently with compute. Additionally, our results indicate that the scaling dynam- ics significantly differ from textless-SLMss, suggesting one should allocate notably more of the compute budget increasing model size over train- ing tokens. We also study the role of synthetic data and TextLM model families in unlocking this potential. Results suggest that our scaled up model achieves comparable semantic speech performance to leading mod- els, while using less compute and data. We open source models, samples, culj.al.a.lll adiyoss-lab/ haib//ssm///../
and data https:/
1 Introduction
Speech Language Models (SLMs) have gained much attention from the research community (Cui et al., 2024a; Huang et al., 2024; Maimon et al., 2025b) showing impressive results in reasoning over speech and audio (Chu et al., 2023; Tang et al., 2023), and as a basis for spoken chat systems (Défossez et al., 2024; Ji et al., 2024). Early efforts in constructing SLMs were focused on training with speech data only (Lakhotia et al., 2021; Borsos et al., 2023), often known textless-SLMs.. Such approaches showed potential in speech modelling and generation, yet, such modelling approach was found to be limited in semantic abilities.
Recent methods for training SLMs integrate text alongside speech, either by processing both modalities concurrently (Défossez et al., 2024; Fang et al., 2024) or by interleaving between them (Nguyen et al., 2024; Zeng et al., 2024). These models are often referred to as joint speech-text SLMs. Such modelling approach resulted in significant improvement in SLM performance, specifically considering semantic content understanding and generation. Despite the great results, the scaling properties of such training paradigms remain unclear.
Following the scaling analysis of TextLMs (Hoffmann et al., 2022; Kaplan al., 2020), Cuervo & Marxer (2024) proposed the first scaling analysis for textless-SLMs focusing on the Generative Spoken Language Models (GSLM) approach (Lakhotia et al., 2021). The authors trained models of different sizes and token counts to fit parametric function which describes the validation loss as a function of the model parameters, and training tokens. Their analysis suggests that the semantic abilities of SLMs scale notably slower than TextLMs and would require significantly larger datasets for training (~ 3X more than text). While thorough and valuable to the speech community, the analysis presented by Cuervo & Marxer (2024) centres on textless-SL. -Latlss--LMMss and overlooks the role of the text modality and its impact on the scaling behaviour of joint speech-text SLMS.
1

---

Published as a conference paper at COLM 2025
Multi-Speakkr SSC
Suelk-Speaker t5C
S5C
15c
33
00
47
59
56
57-
Inter SUM 0.37HCh0
0.281C00.033
0.28nc8..22
1ear 5.23CC"..020
SUMI 0.35
5.07%C
1LM:0.15"C78.09
300
10h
300
300
163h
109
139
1001
(FLORS)
Compute
Figure 1: Analysing scaling of interleaved SIMs, considering the best model size per- compute, across semantic speech metrics Topic StoryCloze and Semantic StoryCloze (Hassid et al., 2023). We compare scaling trends to textless SLMs (denoted SLM), and TsxtLMs (TLM), as presented by Cuervo & Marxer (2024).
In this work, we explore the scaling behaviour of joint speech-text SLMs within inter- leaved setup. We train multiple interleaved SLMs across a range of sizes, compute budgets, and model families to derive practical insights for optimising SLM performance. Specifically after identifying well-performing model families and data mix- we fix several compute budgets and compare models of different sizes trained with the same compute budget.
Compared to textless-SLMe evaluating the scaling properties of interleaved SLMs poses further challenges. For instance, initialisation from TextLM could have a key impact on results (Hassid et al., 2023), but not explicitly modelled in scaling. Furthermore, since there are different modalities training is unclear with regards to which tokens one should analyse the scaling. While we do not solve all in this paper, we shed light on these challenges, and propose practical analysis methods. These provide insights into how to best allocate compute between model parameters and tokens, well as the potential benefits of scaling, as shown Figure 1.
The insights found directly translate into better SLMs, which far out-perform the expected results when considering speech-only scaling laws (Cuervo & Marxer, 2024). This in turn gives optimistic tone to the feasibility of training high-quality SLMs with existing compute and datasets. We apply all insights and train a 7B interleaved speech-text LM which achieves performance comparable to leading SLMs considering semantic speech metrics. We open source the above models and training code, thus encouraging the community to further analyse such scaling properties, and text-speech knowledge transfer.
2 Background
2.1 Speech Language Models
The speech research community has explored various types of SLMs (Cui et al., 2024b). In this work, we focus on models trained using both speech and text tokens, commonly referred to as joint speech-text SLMs. This approach typically involves three key components: (i) converting speech to discrete units, (ii) a joint speech-text SLM, and (iii) converting units back to speech, with each module trained independently. While previous studies introduced different methods for jointly modelling speech and text, we focus on modality interleaving,
Speech-to-unit modules encode the raw speech signal into discrete sequence. The common approach first encodes speech into a continuous represen and then quantises to get sequence of discrete units. Denote the domain of audio samples by X C I.. The representation for signal therefore x (xI,...,TT) where x; E X' for all S < T.
Let consider an encoder network, f, that takes a speech utterance input and produces a sequence of fowrr-fryyencc representations: f(x) (vy,..., UT ). While previous studies have examined various speech encoders, f, our focus is on HuBERT (Hsu et al., 2021) as it the most widely used speech encoder for SLMs. Because the outputs this model are continuous, k-means clustering is used to convert the f(x) into discrete units, denoted as
2

---

Published as a conference paper at COLM 2025
(Zi,..., 2T. Each represents discrete category, with values in the range zi I,...,K K for 1 i T where the total number of discrete units.
Interleaved speech-text SLMs are trained and text tokens by mixing them together in sequence. Given time-aligned transcriptions (W,, End;) that specify when each word begins and ends time-stamps, we assign a modality token M (either speech or text) to each segment. By grouping consecutive words with the same modality, we create modality-speciife time spans within an utterance, such as: [(Text, 0, 1.3), (Speech, 1.3.3.5)..... Each segment is then tokenized separately, producing an interleaved sequence like (Text, h,..., Speech, .k,,.., Text, tj....). For example, an interleaved phrase might look like: "[TEXT] the cat [SPEECH] [Hu3] [Hu7] [Hu200] [TEXT] the mat".
combination of discrete speech units
Different approaches assigning modalities to time-spans words have been suggested. We adopt strategy similar to that of Zeng et al. (2024), where speech segments are chosen by sampling their lengths (in words) from Poisson distribution with 10, continuing until these segments make up = 0.3 of the total word count. A language model (often initialised from Lex Hassid et al. 2023) is then trained on the resulting sequences following the standard next-token prediction objective. Previous works have also trained models on individual modalities - iee separate speech-only and text-only datasets.
Unit-to-speech module converts the speech discrete units to a raw waveform. We follow, Polyak et al. (2021); Lakhotia et al. (2021) in using a unit-based vocoder based on the HiFi-GAN architecture to convert units to speech directly.
2.2 Scaling Analysis
Scaling analysis in LMs examines how performance scales with increased compute re- sources (mainly training FLOPS, denoted C). This is mainly characterised by model size (N) and training data size (D), and is commonly approximated as C z 6ND (Kaplan et al., 2020). Several works indicate that final model loss (L) follows an approximate power-law relationship with respect to N, D and consequently C (Clark et al., 2022; Kaplan et al., 2020):
L(N) No; L(D) & D&; L(C) & CY.
Building on this, Hoffmann et al. (2022) proposed three approaches for scaling analysis of language models. The first fixes model sizes and varies the number of training tokens. The second aims to fit a parametric function to estimate the loss as a function of and D:
A
B
L(N,D) F.
Na
DF
where E, A, B, a, and B are parameters empirically estimated using data from multiple trained models. The third approach varies the model size for a fixed set of different training FLOPs. It then identifies the optimal model size for each training FLOPs budget. Based on the optimal models, it fits a function to predict the optimal number of training tokens or model size given training FLOPs budget. This approach was termed ISO-FLOP curves.
Experimental Setup
We start by describing our experimental setup considering: (i) dataset used for model training; (ii) optimisation and implementation details; and (iii) model evaluation.
Data. We follow Cuervo Marxer (2024) using collection of diverse, English datasets, namely LibriSpeech (Panayotov et al., 2015), LibriLight (Kahn et al., 2020), VoxPopuli (Wang et al., 2021), Tedlium (Hernandez et al., 2018), People Speech (Galvez et al., 2021) and SWC (Baumann et al., 2018) for training. We also utilise single speaker synthetic dataset stoniStories, which was shown to improve semantic abilities in non-interleaved SLMs (Cuervo & Marxer, 2024; Maimon et al., 2025a). Time aligned transcriptions for the datasets are estimated using Whisper v3-large-ture turbo (Radford et al., 2022). This collection of datasets was used both as speech-only and interleaved. Similarly to prior work (Nguyen
3

---

Published as a conference paper at COLM 2025
et al., 2024; Défossez et al., 2024), we additionally include text dataset during interleaved SLM optimisation. For that we use a subset from RedPajama (Weber et al., 2024), filtered by Gopher filtering rules (Rae et al., 2021). We provide full dataset statistics in Appendix A.1.
This corresponds to a total of 84K hours of real speech and 30k synthetic (5.8B and 2.2B tokens, respectively). We train all SLMs for epoch on the speech data, which corresponds to S 2 epochs of the interleaved data (though the interleaving pattern, i.e,, which words are speech, and which are text vary between epochs). Overall, the dataset mixed at equal ratios gets to = 20B tokens equally split between speech only, text only and interleaved data.
Optimisation. We follow the efficient SLM training recipe presented by Maimon et al. (2025a), and use their provided toolkit: SLM (Hassid et al., 2023), based on different Ms, using cosine scheduler and the datasets described above (including the synthetic s sTinyStories dataset). We provide full training sypper-araamterr, in Appendix A.2.
SlamKit'. Specifically, we train a TWIST initialised
Evaluation in scaling analysis plays an important role, while some studies address loss or perplexity, others use benchmark metrics as well. Furthermore, for interleaved models one could focus on speech only losses and metrics, cross-modal abilities, etc. For the majority of the scaling analysis, we focus speech-only abilities as evidenced in likelihood of the speech-only validation, as well as evaluation metrics. Specifically, we evaluate both grammatical abilities with sBLIMP (Yang et al., 2021) and semantic abilities with Spoken StoryCloze (sSC) and Topic-StoryCloze (ISC) (Hassid et al., 2023).
Notice, both sTinyStories, sSC, and ISC were synthesised using a single speaker dataset, specifically using i.J (Ito & Johnson, 2017) voice. This process is aligned with prior work (Has- sid et al., 2023; Cuervo Marxer, 2024). Hence, assess how well models trained on single-speaker synthetic datasets generalise to other speakers, we construct a multi-speaker version of the spoken StoryCloze benchmark. Specifically, we use Kokoro text-to-speech (TTS) (Hexgrad, 2025) to synthesise the same texts from the original Spoken-StoryClozz and Topic-StoryCin StoreCCoorr datasets with four distinct speakers male and female, British and American accents. Throughout this study we focus on the average performance across speakers, but provide per-speaker analysis in Appendix B.2.
This setup also allows us to separately generate audio prefixes and suffixes, enabling cross- modal evaluation, following the evaluation protocol of Nguyen et al. (2024); Cuervo et al. (2025); Zeng et al. (2024). Essentially, this is much like the uni-modal StoryCloze where we wish to see whether the "real" continuation is assigned a higher likelihood by the SLM, compared to distractor. However, for the cross-modal setting the prompt is one modality (e.g. text), and the continuation is a different modality (speech). In other words, given specific text prompt the SLM needs to select the more likely speech continuation. We publicly release these benchmarks.
4 Building the Scaffolding for Scaling
To conduct a scaling analysis of SLMs (Section 5), we start by analysing the impact of key design choices which could affect scaling experiments namely, the use of synthetic data, model family selection, and differences arising from interleaving and TWIST initialisation.
4.1 The Effect of Synthetic Data
We begin by analysing the impact of incorporating synthetic data. Following the approach of Cuervo & Marxer (2024); Maimon et al. (2025a), we train interleaved SLMs on both synthetic sTiiyStories and real speech data. To assess how synthetic data affects the model's ability to generalise across different speakers, we compare models trained solely on synthetic data, solely on real data, and on a mix of both. All models are initialised from a Qwen2.5-0.5B TextLM (Yang et al., 2024) and trained for 20k steps.
4

---

Published as a conference paper at COLM 2025
Training Data
Metric
Real
Syn.
SBLIMP1
sSCt
MS sSCt
tSCt
MS tSCt
Val. CE
x
56.77
54.94
52.66
72.15
78.93
1.96129
/
52.98
61.30
54.84
81.24
72.35
3,44569
V
/
56.98
59.81
54.85
81.51
81.67
1.98267
Table 1: Analysing impact of training with/ without synthetic cata on SLM performance, by training an interleaved Qwen2.5-0.5B based model.
Results provided in Table 1 indicate that models trained solely on synthetii data perform well on metrics involving the same single speaker (LJ)-such sSC and tSC-but show weaker performance on evaluations involving other speakers, like sBLIMP and multi- speaker ISC. This is also evident from the validation cross-entropy on real, speech-only data. While prior research indicates that phonetic speech tokens carry limited speaker information (Sicherman & Adi, 2023), our results suggest that evaluating on a single speaker seen during training may give misleading impression of model performance. Nevertheless, we see benefit across diverse speakers, by training on a mixture of synthetic and real data, compared to real data only or synthetic data only. This supports the choice of using mixture of real and synthetic data. In the context of scaling laws, partially replacing real data with synthetic data reduces the real speech data collection needed to train strong Ss. One could investigate synthetic data with diverse speakers, but we focus on the same single speaker as previous work and specifically Cuervo & Marxer (2024) for consistency.
When considering models trained on real-speech only vs. a mixture real and synthetic speech data, note that the validation loss real speech is lower for the former. We hypothesise that this is due to the validation set, which primarily consists of audiobooks, perhaps not fully representing the model's semantic capabilities. Therefore one should also evaluate using task metrics together with validation loss.
We therefore suggest leveraging TTS-generated data from text corpora, providing more semantically coherent inputalongside real speech data to enhance SLM training, In the case of interleaved SLMs, it's important to recognise that validation loss may not always reflect semantic performance accurately. Therefore, evaluation should include a range of metrics, particularly those involving varied and previously unseen speakers.
4.2 Model Family Impact
Scaling laws were shown to not always generalise between model families (Choshen et al., 2024a). However, in SLMs initialised from TextLMs, this effect may be more pronounced, as the choice of model family influences not just the architecture but also the weight initialisation--hiiee can play crucial role. Additionally, the quality of the TextLM used for initialisation can significantly affect interleaving training, since it also involves text tokens.
We conduct an empirical analysis leveraging leading TextLMs (< 7B) a base for inter- leaved SLMs. Specifically, we use LLama3.2 (Grattafiori et al., 2024), Qwen2.5 (Yang et al., 2024), Gemma2 (Team et al., 2024), SmolLM2 (Allal al., 2025), Pythia (Biderman et al., 2023), OPT (Zhang et al., 2022), and Bloom (Workshop et al., 2022). Results, provided in Figure 2, suggest that not all model families are born equal. For instance, Pythial60M underperforms OPT125M despite its larger size. In addition, Qwen2.5-0 0-.5BB outperforms other, notably larger, models like SmolL.M2-1.7 not always correlate with performance of the base TextLMs. For instance SmolLM2-1.7B outperforms Qwen2.5-1.5B and LLaMa3.2-1 2-BB on standard text evaluations (Allal et al., 2025), yet, in our interleaved SLM setup, even the smaller Qwen2.5-0.55 model performs better.
Bloom-1.78 and OPT1.3B We note that these results do
We therefore suggest selecting a SLM which shows good performance in smaller scale models (0.5-1B), then using larger versions of the same model. We found Qwen2.5 and LLama3.2 to work well in our tests, and continue most scaling analysis with Qwen2.5 as there more available model sizes.

---

Published as a conference paper at COLM 2025
LaMk332-31 2-38
Qween2.-11
0.50-255-0.51
Gemma3-22 -2
Pythie-1 14
Pyttie-180M OPT-125M
SmeeLLM2-1.78
OPT-1.38
Validation CE (Lower is Better)
2.3
Accuracy (Higher Better)
2.2
2.1
2,00
60
367
562
2.0
1955
877
1.990
55
12.
6L.
50
1.9
Mult-Speakes
15C
Multi-Speaker sSC
SBLIMP
Validation Cross Entropy
Figure 2: Comparing SLMs trained with the same recipe for 20k steps, from different TextLM initialisations. Models are sorted by parameter count from large small.
2.15
Validation Speech Crass Entropy
80-
80
2.10
78-
2.05
MuitiSpp
76
W 76
74
72-
Intericawing TWIST
72
Speech TWIST
GSLM
70
500020000
40000
100000
500020000
40000
100000
5000
400000
100000
Training Steps
Training Steps
Steps
Figure 3: Comparing SLMs based on Qwen2.5-0.5B with interleaving, without interleaving and without TWIST initialisation (denoted GSLM). This helps analyse the impact of these choices on performance and thus on scaling analysis. See Appendix 6 for other metrics.
4.3
When Does Text Benefit Speech Modelling?
One might wonder whether not interleaving could be useful for improving speech only abilities. Nguyen et al. (2024) demonstrated that it does models trained with interleaving outperformed those trained solely on speech. We follow up this question by asking "When is interleaving better than speech only?"
Following the above observation that different model families (and sizes) perform very differently, we focus on a model which showed good performance Qwen2.5-0.5B We compare interleaved SLMs trained with the same recipe and initialisation, but varying number of steps. Our results, described in Figure 3, show that for semantic abilities the interleaved model outperformed the speech only model from as little 5k training steps (~ 720M tokens). This impact seems to increase with more compute. Other results show that this not necessarily the case for weaker smaller TextLMs, e.g. OPT125M, which could require notably more steps for interleaving benefits, see Appendix B.1.
Hassid et (2023) showed that initialising SLMs from pre-trained TextLM could have diminishing returns with regards to training steps tokkns.. Intuitively, the longer the model trained, the less impact the good initialisation has, though not necessarily decaying to zero. We wish to investigate this in the context of interleaved SLMs. We hypothesise that this effect could have an impact on the functional form between D, N and performance. While in existing SLM scaling analysis (Cuervo & Marxer, 2024) the loss in assumed to have

---

Published as a conference paper at COLM 2025
2.15
5028
Hudget
Speech val CE Bettto))
esC Better)
2.05
82
2.00
80
3.90
76
500M
500M
70
Model Size (Parameters)
Model Size (Parameters)
Figure 4: Comparing the loss on speech only of interleaved SLMs of different model sizes trained for specific compute budgets.
an exponential relationship with D, it is not clear this assumption holds our setup. This training process resembles the addition of new language to an existing LLM, hence its scaling behaviour might differ significantly. For example, the influence of tokens may not diminish exponentially due to factors like initialisation effects and potential knowledge transfer.
We investigate this by comparing an interleaved SLM initialised from scratch to the common text-initialised version. Results are depicted in Figure We observe that the gap in loss does seem to decrease with more training steps, however the gap in tasks metric is less clear cut (though TWIST initialisation is clearly beneficial). This could mean that the impact DD model performance might decay faster, at least for some ranges and initialisations. This could induce a different scaling functional form, but intuitively this would mean that one should spend more of the compute on parameters.
We therefore suggest always training with interleaving, initialise from a pre-trained TextLM- even when interested in speech-only abilities, given that the base TextLM is good enough and there is reasonable compute budget (e.g 5k steps, and 720M tokens).
Scaling Analysis
As a functional form is challenging fit for the complex training dynamics of interleaved SLMs, we follow Hoffmann et al. (2022) in directly fixing the compute budget and training different model sizes. This follows a similar approach to the ISO-flop curve.
While working with pre-trained TextLMs restricts us to fixed model sizes- limitini the granularity of our scaling analysis and making harder assess low-compute regimes- found that analysis based on smaller but lower-quality models (e.g., Pythia) do not gener- alise well to real-world applications. Therefore, we opt for Qwen2.5 family due to its strong performance (see Section 4.2) and the availability of reasonably diverse set of model sizes. We name this Speech Interleaved Model Scaling suite SIMS for short.
We fix the compute budget to certain FLOP count, and then calculate the appropriate number of tokens D for each model by using C(FLOPs) = 6ND (Kaplan et al., 2020). We use model sizes 0.5B, 1.5B, 3B, 7B (omitting irrelevant models where applicable), and compute budgets- 2.1e18, 5e18, 2e19, 6e19, 2e20. We inspect both speech-only validation loss, and semantic speech metric multi-speaker (SC.
The results described in Figure show that compute allocated to model parameters should be large. While we lack fidelity to identify the exact ideal model size per-curve, can identify trends. For instance, when considering semantic metrics for compute < 5e18 the
2On could consider the compute used for training the TextLM,, however, these models are widely available thus there is not likely need to train them from scratch.
7

---

Published as a conference paper at COLM 2025
Model
Params
Spelt-Speakkr sSC
Mulli-Speaker ISC
S-T
T-S
T
Orig.
S
S-I
T-S
T
Orig.
SpiritLM (L22)
7.5e21
62.80
68.49
62.63
72.67
90.01
94.74
79.77
97.33
SIMS (L3)
3.28
5.9e19
58,23
66.10
60.36
73.01
74.40
86.22
90.37
76.31
96.26
95.88
SIMS (Q)
1.5B
2e20
55,80
60.73
58.28
66.92
71.78
83,87
76.91
71.73
92.73
95.88
SIMS (Q)
3B
2c20
57.00
61.38
58.23
65.47
73.86
85.36
75.16
71.11
88
97.22
SIMS (Q)
7.6B
2c20
58.73
66.23
60.10
72.85
75.09
87.6
91.35
76.11
96.26
97.92
SIMS (Q)
7.6B
6.86e20
60.33
67.73
60.65
72.47
75.09
88.27
93.29
75.34
95.67
97.92
Table 2: Analysing the cross-modal semantic abilities of different interleaved SLMs. We also report the performance of the original TextLM before training under Orig. L- indicates LLaMa TextLM initialisation, while Q - is Qwen2.5. *We use the official model weights, which were trained for 75% longer than originally reported by Nguyen et al. (2024).
smallest model is best, but beyond that we see that larger models are better - indicating that they are nearer the optimal. For instance, for 2e20 FLOP our best model, by large margin, is 7B parameters on 4.2B tokens vs. 463M parameters on 72B tokens suggested by Cuervo & Marxer (2024). This is also in stark contrast to large, common models like SpiritLM which are trained on much larger ratio of tokens (e.g 7B model for 2 100B tokens). We also note that only 50 60% of our tokens are speech, making the trend with regards to speech tokens even more pronounced.
While these results are hard to extrapolate directly to larger compute budgets, they do suggest that much larger part of the compute budget should be dedicated to model parameters when considering interleaving with good base models. This could also match the intuition from Section 4.3, that the benefit of more tokens decays faster because of TWIST initialisation. We also note that using a larger, better, TextLM not only means more efficient optimisation on the speech data (i.e. the architectural impact), but also better initialisation (due to a better TextLM).
Next, we analyse the relation between the downstream metric performance of the SLMs in relation to the training compute. We provide the results in Figure 1 and compare them to the estimates of textless-SI.Ms by Cuervo & Marxer (2024). We also fit an equation for the scaling of each, following the approach by Cuervo & Marxer (2024). However, we note that in our case these might be slightly more noisy, as we are limited with model sizes existing TextLM sizes, our best model could be further from the actual optimal. Nevertheless, the results clearly show improved performance per-compute as well as more optimistic trend (the slope for sSC is higher, for iSC they are similar, but results are likely near the saturation limit). These results suggest the feasibility of high semantic abilities in SLMs with existing data resources.
We therefore suggest using the largest possible model within a high-quality model family for large scale training (given > 25k steps and > 4.5B tokens). For smaller scales, e.g 6e19, we suggest following the results here as a rough guideline.
5.1 Comparison with Existing Models
Armed with the above insights we scale up training of our interleaved SLM training and show that it reach comparable performance to leading SLMs while using significantly less compute. Specifically, we train model based on Qwen2.5-7B for 15B tokens (both speech and text), which is equivalent to a compute budget of ~ 6.86e20. We note that, while this is the largest compute budget, it is not the largest amount of tokens we trained on, and it is still under the regime of less than one epoch on the speech-only data.
We begin by evaluating the cross-modal capabilities of several of our models in Table 2, and compare their performance to SpiritLM, which operates with more than an order of magnitude more computational resources. Even though most of our analysis focused on speech only evaluation, we observe a similar trend for cross-modal evaluation tasks. Unsurprisingly, performance is the best for text-only. However, perhaps surprisingly the speech only performance sometimes out-performs cross croal abilities. We hypothesise that the text modelling abilities are better than speech, yet-cross modality poses challenge.
8

---

Published as a conference paper at COLM 2025
Model
Params.
Tokens
sBLIMP1
sSC1
ISCT
GenPPL.
Self-BLEU
GSLM M (Lakhotia et al., 2021)
100M
1B
54.2
53.3
66.6
Ø
? 3.06
TWIST-7B (Hassid ct al., 2023)
7B
36B
59.0
55.3
74.1
93.74
TWIS1-13B (Hassid et al., 2023)
76.4
SyllableLM (Baade al., 2024)
300M
16B
63.7
75.4
Ø
-DPO (Maimon et al., 2025a)
358M
16.7B
58.5
58.2
80.7
67.3
3.25
Slam (Maimon et al, 2025a)
358M
16.7B
61.1
61.3
84.2
46.6
3.75
AlignSIM (Lin et al., 2024)
7B
36 158B
62.3
61.1
86.8
Ø
Ø
Cuervo Marxer (2024)
823M
78.0
Zeng et al. (2024)
9B
~1
62.4
82.9
Ø
Moshi (Défossez al., 2024)
7B
720B
58.8
60.8
83.0
02
SpiritLM (Nguyen et al., 2024)
7B
100B
58.3
61.0
82.9
SIMS (ours)
7B
15B
59.8
66.7
88.3
37.6
4.15
Table 3: Comparing SIMS to existing SLMs. Results on the upper block of the table corre- spond textless-SLMs, while results on the bottom correspond to joint speech-text SLMs.
Thus for complex enough tasks, e.g. sSC, the better text performance is enough to out-weigh the challenge of cross modality. However, for tSC this not always the case.
We finally compare our model with leading existing SLMs. We provide the results in Table We observe that our model outperforms existing methods on semantic metrics sSC, tSC and GenPPL often by large margin, even models that have gone through preference optimisation phase (e.g., Slam, AlignSLM). Regarding sBLIMP which measures grammatical abilities that SIMS outperforms existing interleaved SLMs. However, some textless SLMs trained with DPO, a different tokeniser, or notably more tokens reach better results.
Related Work
6.1 Speech Language Models
Various methods for learning speech and audio discrete representations have been studied. The authors of Hsu et al. (2021) use masked language modelling to construct phonetic representations, while Zeghidour al. (2021); Défossez et al. (2022); Kumar al. (2023)
trained auto-encoders coupled with vector-quantisationn for acoustic representations.
The task of Generative Spoken Language Modelling over such speech representations was first introduced by Lakhotia et al. (2021). This was extended by Borsos et al. (2023), which incorporated both semantic and acoustic tokens generate more expressive speech. Building on these foundations, many other SLMs have been proposed, each aiming to improve performance or address different aspects (Hassid et al., 2023; Baade et al., 2024; Maimon et al., 2025a; Cuervo & Marxer, 2024). Beyond the stream of SLMs deal with, speech-aware TextLMs such as (Chu et al., 2023; Tang et al., 2023; Kong et al., 2024) have been proposed, where they train audio encoder SLMs were also used to solve various traditional speech tasks (Wang et al., 2023a;b; Popuri et al., 2022; Inaguma et al., 2022; Kreuk et al., 2021; Maimon & Adi, 2023; Roth et al., 2024). Furthermore, many recent approaches incorporate speech-text in input and output (Défossez et al., 2024; Zhang et al., 2023; Cuervo et al., 2025; Zeng al., 2024).
adapter to pre-trained TextLM.
6.2 Scaling Analysis
Scaling analysis in TextLMs explores how performance improves with more computational resources, mainly model size and training data (Hestness et al., 2017; Clark et al., 2022; Kaplan et al., 2020). Hoffmann et al. (2022) introduced three scaling methods: varying training tokens with fixed model sizes, adjusting model size for fixed training FLOPs and fitting a parametric loss function (scaling law). Muennighoff et al. (2023) extended this work by examining how scaling laws apply to TextLMs in multi-epoch training scenarios. Additionally, Choshen et al. (2024b) analysed over 1, 000 scaling laws, providing practical recommendations for conducting scaling research.

---

Published as a conference paper at COLM 2025
In the field of speech, several studies explored scaling approaches, with primary emphasis on scaling model size and data for multilingual speech recognition (ASR) (Pratap et al., 2020; Xiao et al., 2021; Pratap et al., 2024). Recently, Chen et al. (2025) investigated scaling laws in ASR, revealing that scaling enhances performance on low-resource languages. Most relevant to work is Cuervo & Marxer (2024), which investigated scaling laws for textless SLMs.
Conclusion
In this paper we present the first scaling analysis of Interleaved SLMs and discover that using high-quality TextLMs with partly synthetic data can lead to optimistic scaling capabilities. As part of this we discover that under interleaved training more of the compute budget should be assigned to model parameters (and subsequent initialisation quality) at the expense of training tokens. We also provide practical insights for researchers by highlighting the impact of model families and synthetic data. We encourage the community to further investigate SLM scaling under various settings, and improve training efficiency further.
Acknowledgements. This research work was supported in by ISF grant number 2049/22 and by the NVIDIA Academic Grant Program providing computing resources, A100 and H100 GPUs hours on NVIDIA cloud computing cluster.
References
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček,, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. Smollm2: When smol goes big data-centric training of small language model. arXiv preprint ar arXio:2502.0275; 2025.
Alan Baade, Puyuan Peng, and David Harwath. Syllablelm: Learning coarse semantic units for speech language models. arxiv preprint ar arXie:2410.0402, 2024.
Timo Baumann, Arne Köhn, and Felix Hennig. The The spoken wikipedia corpus collection: Har- vesting, alignment and an application to hyperlistening. Language Resources and Evaluation, 53:303 329, 2018. URL https: //api. semanticscholar, org/ Parg/CCrrpusD:: 52825870.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430.
PMLR, 2023.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on 2023.
audio, speech, and language processing,
William Chen, Jinchuan Tian, Yifan Peng, Brian Yan, Chao-Han Huck Yang, and Shinji Watanabe. Owls: Scaling laws for multilingual speech recognition and translation models. Xrx preprint ar XrXiv:2502.1083 10373,, 2025.
Leshem Choshen, Yang Zhang, and Jacob Andreas. A hitchhiker's guide to scaling law estimation. arXio preprint ar xrXio:2410.11840, 2024a.
Leshem Choshen, Yang Zhang, and Jacob Andreas. A hitchhiker's guide to scaling law 11840.
estimation, 2024b. URL
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint ar axXi::2311.07919 2023.
10

---

Published as a conference paper at COLM 2025
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057-4086 PMLR, 2022.
Santiago Cuervo and Ricard Marxer. Scaling properties of speech language models. arvio preprint ar kie::2404.00655, 2024.
Santiago Cuervo, Adel Moumen, Yanis Labrak, Sameer Khurana, Antoine Laurent, Mickael Rouvier, and Ricard Marxer. Text-speech language models with improved cross-modal transfer by aligning abstraction levels. arviv preprint ar drXi::2503.06211, 2025.
Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Yiwen Guo, and Irwin King. Recent advances in speech language models: A survey. ArXiv, abs/2410.03751, 2024a. URL httpp://appl semanticscholar o org//orruusD::27318888
Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Yiwen Guo, and Irwin King. Recent advances in speech language models: A survey. arXiv preprint ar XrXi::2410.03751, 2024b.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. aiv preprint ar arXiv:2210.13438, 2022.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. Xrv preprint ar arXiv:2410.00037, 2024.
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-ommi: Seamless speech interaction with large language models. arviv preprint ar drXiv:2409006666 2024.
Daniel Galvez, Greg Diamos, Juan Manuel Ciro Torres, Juan Felipe Cerón, Keith Achorn, Anjali Gopi, David Kanter, Max Lam, Mark Mazumder, and Vijay Janapa Reddi. The people's speech: A large-scale diverse english speech recognition dataset for commercial usage. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https::/ wttpeewwwwwweeeeeeeenieeeeeeeeee htt.:///oorrmm..ddooooo.o.g/g/
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. ar Xiv preprint ar arXiv22407.21883 2024.
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, et al. Textually pretrained speech language models. arXiv preprint ar arXiv:2305.13009 2023.
François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Esteve. Ted-lium 3: Twice much data and corpus repartition for experiments on speaker adaptation. In Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18-22, 2018, Proceedings 20, pp. 198-208. Springer, 2018.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017. URL https: / https////aaxxiooogga///////// 00409.
Hexgrad. Kokoro-82m (revision d8b4fc7), 2025. URL https::/huuginggtteeee htt.:/faccccooo/ogrereee co/hexgrad/ Kokoro-82M.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. Xrv preprint ar arXin:2203.15556,
2022.
11

---

Published as a conference paper at COLM 2025
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhut- dinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451-3460, 2021.
Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang, Andy T Liu, Chen-An Li, Yu-Xiang Lin, Wei-Cheng Iseng, Anuj Diwan, Yi-jen Shih, Jiatong Shi, et al. Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks. Xrviv preprint arXiv:2411.05330 2024.
Hirofumi Inaguma, Sravya Popuri, llia Kulikov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. Unity: Two-pass direct speech-to-speech translation with discrete units. ar preprint arXiv:2212.08055 2022.
Keith Ito and Linda Johnson. UJ-Speech-Dataset/ Dat2
The speech dataset.
https://keithito... com/
Shengpeng Ji, Yifu Chen, Minghui Pang, Jialong Zuo, Jingyu Lu, Hanting Wang, Ziyue Jiang, Long Zhou, Shujie Liu, Xize Cheng, et al. Wavchat: A survey of spoken dialogue models. arxiv preprint ar arXto:2411.1357, 2024.
Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre- Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri-light: A benchmark for asr with limited or no supervision. In TCASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673. IEEE, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. ar preprint ar Xitx20001008361 2020.
Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. Xrv preprint ar arXiv:2402.01831 2024.
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. Textless speech emotion conversion using discrete and decomposed representations. Xiv preprint ar Xxx::2111.07402 2021.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36:27980-27999 -79993, 2023.
Kushal Lakhotia et al. On generative spoken language modeling from raw audio. Transac- fions of the Association for Computational Linguistics, 9:1336-1354, 2021.
Guan-Ting Lin, Prashanth Gurunath Shivakumar, Aditya Gourav, Yile Gu, Ankur Gandhe, Hung-yi Lee, and Ivan Bulyko. Align-slm: Textless spoken language models with rein- forcement learning from feedback. Xiv preprint arXiv:2411.01833 2024.
Gallil Maimon and Yossi Adi. Speaking style conversion in the waveform domain using discrete self-supervised units. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8048-8061, 2023.
Gallil Maimon, Avishai Elmakies, and Yossi Adi. Slamming: Training speech language model on one gpu in a day. ar Xio preprint ar 2xXi:22502.1584, 2025a.
Gallil Maimon, Amit Roth, and Yossi Adi. Salmon: A suite for acoustic language model evaluation. In ICASSP 2025 2025 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2025b. doi: 10.1109 2AAL///CCASSP4990020555.0000 108888666...
12

---

Published as a conference paper at COLM 2025
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01696 2016.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksan- dra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-50376 2023.
Tu Anh Nguyen et al. Spirit-Im: Interleaved spoken and written language model. Xiv
preprint Xx1::2402.05755, 2024.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015. doi: 10.1109/
ICASSP2015.7178994.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXit preprint arXie:2104.0033 2021.
Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. Enhanced direct speech-to-speede translation using self-supervised pre-training and data augmentation. In Proc. Interspeech 2022, pp. 5195-5199, 2022.
Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Hannun, Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert. Massively multilingual asr: 50 languages, 1 model, billion parameters. Xrviv preprint arXim:2007.03001, 2020.
Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(97):1-522 2024. URL http:///mml...org/ppprs///////// 1018.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.043366, 2022.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pp. 28492-28518, PMLR, 2023.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arxio preprint 1rX14:2112.11446, 2021.
Amit Roth, Arnon Turetzky, and Yossi Adi. A language modeling approach to diacritic-free hebrew tts. arXiv preprint Xix::2477.12206 2024.
Amitay Sicherman and Yossi Adi. Analysing discrete self supervised speech representation for spoken language modeling. In ÍCASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. Xiv preprint ar Xit::3331 13289, 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma Improving open language models at a practical size. arXio preprint Xir:::2408.0 00118, 2024.
13

---

Published as a conference paper at COLM 2025
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haz- iza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Chengging Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 993-1003, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ /v/5553/11 /1/2021.aallloong/. 80. URL ptttps//aade https///allatthhooggg. .rg. wttgg2021..0 acl-longg long 80/.
any
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huamingg Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. Xiv preprint ar Xrio22301.00111
2023a.
Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. Viola: Unified codec language models for speech recognition, synthesis, and translation. arXiv preprint ar arXiv:2305.16107, 2023b.
Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexan- drov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Trina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models. NeurlPS Datasets and Benchmarks Track, 2024.
BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-parand parameterr open-access multilingual language model. arXiv preprint arXiv:2211.05100 2022.
Alex Xiao, Weiyi Zheng, Gil Keren, Duc Le, Frank Zhang, Christian Fuegen, Ozlem Kalinli, Yatharth Saraf, and Abdelrahman Mohamed. Scaling asr improves zero and few shot learning. arv preprint arX::22111.0588
2021.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arviv preprint ar arXiv:2412.15115, 2024.
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, al. Superb: Speech processing universal performance benchmark. arXiv preprint ar xrX::22105.01011, 2021.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong, and Jie Tang. Scaling speech-text pre-training with synthetic interleaved data. ar Xiv preprint ar 2xX1::2411.17007 2024.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXtv:2305.11000 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. Xrv preprint arii:22005001068, 2022.
14

---

Published as a conference paper at COLM 2025
A Detailed Setup
A.1 Datasets
We use several speech datasets for training. Specifically, we use Libri-Light (Kahn et al., 2020), LibriSpeech (Panayotov et al., 2015), SWC (Baumann et al., 2018), Tedlium (Hernandez et al., 2018), PeopleSpeech (Galvez et al., 2021) and VoxPopuli (Wang et al., 2021). We only take English subsets for all datasets. We use full training splits where provided, otherwise splitting 99% for training. Full dataset sizes are in Table 4.
Dataset
Number of Hours
Number of Tokens
Libri-Light (Kahn et al., 2020)
50K
3.5B
LibriSpeech (Panayotov al., 2015)
960
67M
SWC (Baumann et al., 2018)
750
19M
Tedlium (Hernandez et al., 2018)
1.6K
110M
PeopleSpeech (Galvez et al., 2021)
7K
480M
Voxl'opuli (Wang et al., 2021)
24K
1.64B
sTinyStories (Maimon et al., 2025a)
30K
2.2B
Table 4: Dataset train set sizes that we use.
A.2 Hyper-Paramese
We train all interleaved SLMs with the same settings - context length of 2048, AdamW optimiser, with a Cosine decay learning rate scheduler with a warm-up of 1% of training steps and final minimal learning rate of 5e - 5. We use bfloat16 precision, with flash- attention2 and data packing. We preform gradient normalisation with norm of 0.5. We balance the modalities text-only, speech-only and interleaved by mixing samples based on length distribution each dataset, and not on exact tokens per batch. We use a maximum learning rate of 5e 4 for most models, but lower this to 3e 4 for training of compute 2 2e20 as the longer step count meant that the learning rate "shoulder" was high for too long and induced instability in training. For the compute budget 6e19 considered both 3e - 4 and 5e for all 4 models, and used the best for each by validation loss. Specifically this was 3e 4 for Qwen2.5-7B, and 5e 4 for all others (albeit not by big difference).
A.3 Evaluation
For generative perplexity we use the first 1000 samples of sSC (correct samples only), and take the first 3 seconds as prompt. We then generate maximum of 150 tokens with top K = 25, and temperature of 0.8, which matches Slam (Maimon et al., 2025a). We transcribe the samples using Whisper-large-v--uue measure Perplexity using the Llama-3.2-11 model (Grattafiori et al., 2024).
model (Radford al., 2023) and
We created multi-speaker versions of the same texts as the original versions introduced by Hassid et al. (2023), based on the text dataset (Mostafazadeh et al., 2016), and generated the speech using Kokoro TTS (Hexgrad, 2025) with 4 different voices: af bella, bf_emma, am-puck, and bm-george.
and Sopic-StoryCloze. We used
B Additional Results
B.1 When Does Text Benefit Speech Modelling?
We provide here additional results in analysing SLM performance. Specifically, in Figure 5 we show the impact of interleaving and TWIST initialisation for OPT125, which is smaller model compared to Qwen2.5-0.5B used in Figure 3. We note that for this model interleaving under--prforms compared to speech-only for all but the largest budget (where they are comparable). We also note that the non-TWIST initialised model (GSLM) closes the gap
15

---

Published as a conference paper at COLM 2025
78
Validation Speech Cross Entropy
2.30
Multt-Spealen 150
74
74
2.25
2.15
70
2.10
68-
Interleaiig CSLM
5000
20000
49000
5000
20000
60000
5000
20000
40000
Training Steps
Training Steps
Training Steps
Figure 5: Comparing SLMs based on OPT125M with interleaving, without interleaving and without TWIST initialisation. Comparing to the Figure 3, we can see that OPT125 benefits less from interleaving and TWIST initialisation compared to Qwen2.5-0.5B.
quite substantially, unlike with Qwen. This could further the support the need to use strong TextLM model families achieve good scaling performance.
B.2 Diverse Speaker Analysis
In order to analyse how much results vary per-speaker in our suggested multi-speaker benchmark, we report per-speaker metrics for select models. We provide the results in Table Results suggest slight harm performance for the British accent, which demonstrates the important of evaluating with diverse speakers. However, the difference in performance across speakers limited, which shows the robustness of our models.
For evaluation efficiency, we only selected four speakers for the official multi-speaker benchmark. However, in order to further establish to what extent these four speakers represent expected performance provide results in Table 6, which show similar trends and scores to the original speakers.
other speakers select four more speakers. We
sSC/tSC
af bella
puck
bf emma
bm george
SIMS (Q1.5)
56.23/84.45
57.35/83.50
54.30/82.66
55.32/84.77
SIMS (L3)
59.27 /6.48
59.49/86.88
57.19/88 /85.03
56.97 /6.57 59.01 39.01/88.78
SIMS (Q7)
61.41 61.41/88.35
28.77/8..2
58.52/87 87.28
Table 5: Model performance (sSC and tSC) for the four diverse speakers used for the multi- speaker versions.
sSC/tCC
heart
am am michaal
bf.ibblll
bm fabel
SIMS (Q1.5)
56.81 /881/
56.76/ /85003
55.64/81.24
55.53/81.22 .24
SIMS (L3)
58.90/87.23
58.15/87 65
56.97/83.86
58.20/ 28.20/84.61 84.61
SIMS (Q7)
61.52/90.22
61.57/90.00
58.52/85.94
59.27 885.5
Table 6: Model performance (sSC and tSC) for four additional speakers, not used in the multi-speaker version. We select male and female as well as British and American accents. The similarity of results to Table demonstrates that the original four speakers represent wider population well.
B.3 Scaling Analysis
We also provide results for additional metrics, corresponding to Figures and 4. Specifically, Figure 6 provides further metrics for analysis of the impact of interleaving and TWIST
16

---

Published as a conference paper at COLM 2025
58
54.5
57-
MuntSSpeakr
59
S8LIMP
56
53.5
0
53.0
55
Interleaving TWIST
52.0
Interfeaving + CSLM
500020000
40000
100000
5000
40000 Training Steps
100000
500020000
40000
100000
Training Steps
Traniing Steps
Figure Comparing SLMs based on Qwen2.5-0.5B with interleaving, without interleaving and without TWIST initialisation. This compliments Figure 3, yet results are a bit more noisy, perhaps because they are nearer random.
59
2e18
Se18
58
2e19
6e19
MS sSC (Higher is Better)
57-
2e20
56
54
53
500M
18
38
7B
Model Size (Parameters)
Figure 7: Analysing the scaling properties of interleaved SLMs regarding multi-speaker sSC. This compliments Figure 4, yet results are bit more noisy, perhaps because they are nearer to random.
initialisation, beyond those provided in Figure 3. Generally, the sSC versions show similar trends to tSC, yet there is much more noise making them harder to interpret clearly. We suspect this could do with the results being fairly low thus more prone to slight noise. Interestingly for sBLIMP the results correlate more with the loss and show better perfor- mance without interleaving. This could be in part to the Interleaved model seeing less speech tokens per-step (as some are interleaved or text), which for low training setup can be detrimental to performance on diverse datasets.
Figure 7 provides scaling analysis like in Figure 4, but for sSC. While showing similar trends to loss and other metrics, there seems to be slightly more variance.
17


Reviews:
[{"review_id": "ktbKeYOdUY", "reviewer": "Reviewer_LQ6D", "summary": "This paper presents a scaling analysis for Speech-Text Language models. Specifically, the paper pushes beyond the state of the art in the following aspects:\n\n1. The paper observes that under an interleaved setting, SLM scales more efficiently with compute than textless setting, and that more compute should be spent on scaling parameters instead of increasing token counts.\n2. The paper examines the impact of synthetic data and TextLM model families in the better scaling of interleaved SLM models. Specifically, it shows that TTS-generated synthetic data and interleaved training with initializations from a pre-trained TextLM are both beneficial for scaling parameters.\n3. Taking the insight above, the paper was able to train an interleaved 7B SLM that out-performs previous models with less tokens.\n\nIn general, I think this is a potentially impactful empirical work that can facilitate interleaved SLM training in the future, mainly because of the importance of the problem it is trying to tackle. In terms of advancing from the state-of-the-art, the paper picked out some good angles to investigate (synthetic data, model initialization), which I believe a lot of the practitioners in this field are also faced with. The arguments presented in this paper are mostly well-supported with experiments, and presented with good clarity.\n\nMy main reservation for this paper is in evaluation. Coming from LLM literature, it is surprising to see that the model is only evaluated in two aspects (sBLIMP and story cloze). I would like to understand better if this is constrained by the availability of the dataset or something else, and how does the authors think this would translate to end-task performance after SFT (e.g. ASR, translation, speech QA etc.)\n\nAnother minor comment is that the authors may want to normalize the use of hyphens in L145-153 for \"single-speaker\" and \"multi-speaker\". This might have occurred elsewhere in the paper.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This is an important problem to investigate in the field -- one should not expect textless SLM scaling law to apply to interleaved training because of the significant difference in data\n2. The statements are well-supported with empirical evidence.\n3. The presentation quality is high.", "reasons_to_reject": "I think there's a risk that the evaluation conducted in this paper does not cover enough downstream tasks and may not generalize well to downstream performance of common speech-text LLM applications.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GeisP2qBaB", "reviewer": "Reviewer_5iR4", "summary": "This work examines the question of whether interleaved speech-text LLMs initialized from pre-trained text LLMs show better scaling behavior than textless LLMs. The authors conduct various experiments to identify the association between (a) model family and scaling laws, (b) interleaved versus textless SLMs, and the impact of initializing from a pre-trained text model.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well-written and easy to understand. \n- This paper appears technically sound, and has useful advice on scaling LMs. Experimental results appear to corroborate the author's premise.", "reasons_to_reject": "- The impact of choice of acoustic units on SLMs is not explored, which could be crucial to understanding scaling laws in SLMs. \n- It is not clear why a single speaker TTS must be used to build SLMs. As the paper itself mentions, there are multiple multi-speaker TTS models that are available for public use, and hence that part of the analysis is not particularly well motivated. Why would a single speaker TTS trained on LJSpeech be preferred over a multi-speaker TTS model while building SLMs ?\n- 4 speakers may not be sufficient on the test set to accurately measure speaker independence for a general purpose SLM.", "questions_to_authors": "- Why does your conclusion from Line 234 follow as a logical consequence of the preceding lines ? That is not very clear. \n- How does your model perform on real world speech tasks like speech recognition, translation and language understanding ? How does this compare to other models ?", "extra_scores": {}}, {"review_id": "xnTFG538NK", "reviewer": "Reviewer_T9if", "summary": "This paper examines how language models that combine speech and text (SLMs) grow in capability compared to speech-only models. The key findings show that:\n\nModels trained on both speech and text improve more efficiently as you add computing power than models trained only on speech.\nUsing a combination of TTS and real speech data helps models better understand different speakers' voices.\nStarting with text-trained models before adding speech capabilities (called TWIST initialization) significantly improves understanding of meaning, with greater benefits as more computing resources are used.\nThese combined speech-text models can match the performance of top speech understanding systems while using fewer resources and less training data than other methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper thoroughly examines the landscapes of Interleaved SpeechLMs and provides useful observations for building a scalable model.\n\nThe empirical results provide several key insights for building a scalable SpeechLLM efficiently.", "reasons_to_reject": "This work may be relevant to a small sub-community.", "questions_to_authors": "1. What might cause inaccuracies in your mathematical formulas about how AI models improve with size? Since you only studied a few model sizes and your models might not be ideal, how could these limitations affect your results, and how can we reduce these problems?\n\n2. You found that mixing speech and text during training weakened grammar skills when compared to text-only training. Can you explain how the amount of speech included in training affects performance on different tasks when training resources are limited? What strategies could improve this situation?", "extra_scores": {}}, {"review_id": "fXvFBEgm3f", "reviewer": "Reviewer_sDgi", "summary": "This paper presents a scaling analysis of interleaved speech LMs and suggests that more of the compute budget should be assigned to model parameters than training tokens. Along the way, it also shows the effectiveness of using synthetic data. This paper provides focused insights that are useful to the community.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The scaling analysis of interleaved speech LMs and the proposed training receipt are useful to the community.\n- This paper presents comprehensive experiments to justify the analysis setting and conduct the scaling analysis.", "reasons_to_reject": "- Some concepts are not well explained, and some conclusions are not convincing. See my comments/question bellow.", "questions_to_authors": "- Figure 1. (1) Putting this figure in Introduction before explaining the setup is confusing. (2) From sub-figure sSC and tSC, we cannot conclude that interleaved SLM has \"a more optimistic trend.\"\n- L136. Why only the synthetic sTinyStories dataset is used?\n- L152. The \"cross-modal evaluation\" is unclear to me. What does it mean, and how does it work?\n- L174. Why \"replacing real data with synthetic data reduces the speech data needed to train strong SLMs\"? Real data is harder to obtain? You mean \"reduce the real speech data needed\"?\n- Figure 3. More training steps are needed to show longer-term trends. Current data points are not sufficient to conclude that (1) the interleaved model outperformed the speech only model and his impact seems to increase with more compute; (2) comparing interleaved SLM initialised from scratch to the text-initialised version, the gap in loss decreases with more training steps.\n- Table 3. What makes SIMS better than other joint speech-text SLMs?", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary and significance:

This paper investigates the scaling trends for speech language models (SLMs). Previous work has discouragingly shown that SLMs have more unfavorable scaling behavior than text LLMs. This work takes a different approach by interleaving text and speech tokens together, demonstrating that doing so improves the scaling performance of the models. The paper also shows that high quality synthetic data (TTS) can be used to help scale these SLMs.

The main significance of the paper is to highlight a promising new training strategy, interleaved text/speech LMs, as a way to improve the scaling behavior of SLMs.

Pros:

The main discovery of the improved scaling behavior of interleaved text-speech LMs will be impactful for the research community (LQ6D, 5iR4, T9if, sDgi)

The arguments made by the paper are clear and well-supported by experiment (LQ6D, 5iR4)

Cons:

Compared to text-only LLM research, the evaluation tasks used in this paper are limited to only sBLIMP and story cloze (LQ6D)

5iR4 pointed out that the paper's experiments only used one speech tokenizer and a small set of speakers, both of which may change the scaling trends of SLMs.

T9if believes that the work may be relevant more to speech researchers and less to the broader LLM community

sDgi thought several technical details were unclear.

Cons addressed by discussion:

Regarding the limited evaluation criticisms (LQ6D), the authors point out that the sBLIMP and story cloze tasks are the main evaluations used by prior literature on this problem, and that their focus is on the scaling behavior of base models rather than instruction-tuned models. LQ6D acknowledged this rationale makes sense but still was left wanting to see how these base model improvements translated to real world performance.

The authors replied to 5iR4's criticisms about varying the tokenizer and speaker generalization by pointing out that most LLM scaling papers use only one tokenizer because it is too expensive to exhaustively compare many tokenizers. They acknowledge that future work should look at training SLMs with wider speaker variability, but do present additional results measuring the performance of the model on 4 new speakers and show that it is similar to the performance on the training speakers, which mostly alleviates this criticism.

The authors also clarified sDgi's questions and committed to adding these clarifications to the camera ready paper.
Decision: Accept