On The Landscape of Spoken Language Models: A Comprehensive Survey
Authors: Siddhant Arora
Kai-Wei Chang
Chung-Ming Chien
Yifan Peng
Haibin Wu
Yossi Adi
Emmanuel Dupoux
Hung-yi Lee
Karen Livescu
Shinji Watanabe
Venue: Accepted by TMLR ()

Abstract: The field of spoken language processing is undergoing a shift from training custom-built, task-specific models toward using and optimizing spoken language models (SLMs) which act as universal speech processing systems. This trend is similar to the progression toward universal language models that has taken place in the field of (text) natural language processing.  SLMs include both "pure" language models of speech---models of the distribution of tokenized speech sequences---and models that combine speech encoders with text language models, often including both spoken and written input or output. Work in this area is very diverse, with a range of terminology and evaluation settings.  This paper aims to contribute an improved understanding of SLMs via a unifying literature survey of recent work in the context of the evolution of the field. Our survey categorizes the work in this area by model architecture, training, and evaluation choices, and describes some key challenges and directions for future work.

Published in Transactions on Machine Learning Research (10/2025)
On The Landscape of Spoken Language Models: A Comprehensive Survey
Siddhant Arora' Kai-We Chang Chung-Ming Chien Yifan Peng Haibin Wu?*#
Yossi Adi Emmanuel Dupoux*t Hung-Yt Lee Karen Livescu Shinji Watanabe
Carnegie Mellon University, USA
National Taiwan University, Taiwan
Toyota Technological Institute at Chicago, USA
Hebrew University of Jerusalem, Isracl
ENS PSL, EHESS, CNRS, France
Reviewed on OpenReview: httpe://ppareviii..eef//ouuuuttp
Abstract
The field of spoken language processing is undergoing a shift from training custom-built, task-specific models toward using and optimizing spoken language models (SLMs) which act universal speech processing systems. This trend is similar to the progression toward universal language models that has (aken place in the field of (lext) natural language pro- cessing. SLMs include both "pure" language models of speech models of the distribution of tokenized specch soquencos-aid and models that combine speech encoders with text lan- guage models, often inchuding both spoken and written input or output. Work in this area very diverse, with range of terminology and evaluation settings. This paper aims to contribute an improved understanding of SLMs via unifying literature survey of recent work in the context of the evolution of the field. Our survey categorizes the work in this area by model architecture, training, and evaluation choices, and describes some key challenges and dircctions for future work.
1 Introduction
In the last lew years, the field of natural language processing (NLP) has evolved from (1) training many task-specific models from scratch, to (2) combining pre-trained multi-purpose contextual representation models (such as BERT (Devlin et al. 2019) with small number of task-specific parameters, to (3) training generative universal large language models (LLMs) [Brown et al., 2020 OpenAI et al., 2024] that perform arbitrary text tasks given natural language instructions (prompts) and can generalize to unseen domains and tasks (Wei et al. 2022a Liu et al. 2023). and finally to (4) dialogue (chatbot) systems that function assistants and perform tasks while directly interacting with the user.
The field of speech processing has been undergoing similar evolution, although with some lag, and has mainly focussed on stages (1) and (2). The current state of the art (SOTA) for common specialized speech lasks including automatic speech recognition (ASR), speech translation (ST), spoken language under- standing (SLU) tasks, and speaker identification (SID) involves combining a pre-trained self-supervised encoder (Mohamed et al. 2022) with a task-specific prodiction "head". For some very common tasks namely, ASR and ST in relatively high-resource languages, large supervised models (Radford et. al. 2023 Peng et al. 2023b also have consistently good (if not. SOTA) performance.
Recent work has begun to develop spoken language models (SLMs), analogous to text LLMs, which are in principle capable of performing arbitrary speech tasks given natural language instructions. However, the
Throughout the paper, we use the terms L.MSs and LMSs interchangeably refer modern language models.

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Typology of text and spoken LMs. We use a loose notation here, where speech and teat are to be interpreted in context; for example, p(text]text) in post-trained text LMs corresponds to modeling some desired output text given an input text instruction or prompt. "Post-training" refers to any form of intrructionttuunng and/or seteerene-basee optimization. Please see the sections below for details and references for the example models.
Type of LM
Training Strategy
Model distribution
Examples
pure text LM pure text LM
pre-training
pltext)
GPT, I.aMA
post-training
p(tertteent)
ChatGPT, LLaMA-Instruet
pure speech LM
pre-training
p(speech)
GSLM, AudioLM, TWIST Align-SLM
pure speech LM
post-training
Mspecch|sspechl
speech+text LM
pre-training
pltext, speeck)
SpiRit-LM, Moshi (pre-trained) Moshi (post-trained), Mini-Ommi
speech+text LM
post-training
pttent, speech]tert speech)
speech-aware text LM
post-training
p[text] p(text|speech, speech, text)
SALMONN, Aann-Audio--hha
term "SLM" has not been standardized in the literature and has been used to refer to a wider range of model types than text. language models. Several common classes of models have emerged, all referred to as SLMs: (1) pure SLMs: models of the distribution of speech, p(speech), typically trained on unlabeled tokenized speech data only with a next-token prediction objective (similarly to the pre-training phase in text LLMs) Lakhotia et al. 2021); (2) steech+tezt SLMs: models of the joint distribution of speech and the corresponding text p(tert, speech), typically trained using paired (speech, text transcription) data, which can be viewed as a direct extension of class (1) (Nguyen et al.) 2025); and (3) speech- text LMs: models that combine text. LLMs with pre-trained speech encoders to retain the saaruuttion-folooinng (IF) characteristies of the text LLMs and reason about the input speech audio recordings (Tang et al.
2024 Chu et al. 2023).
Approaches in category (3) model a conditional distribution, speech,, text), of desired output text in response to the input speech and text instruction.
Following the text LLMs analogy, models in categories (1) and (2) can be viewed as analogues to pre-trained LLMs. Like text LLMs, these models can be post-trained- -va instructions, preforences, or other mcans to learn the distributions of desired output speech (for category (1)) or output speech+texx (category (2)) given desired inputs (speech speech+text, respectively). Models in category (3) typically start with fine-tuned LLMs, but involve some additional post-training to both align the speech and text representations and enable the model to perform new speech-specific tasks. Table [ provides a typology of these categories along with example models from the recent. literature. Figure presents a timeline of additional models and their development, and Tables and in the Appendix provide additional model details.
Although existing models have been applied to different tasks using different priors and modeling techniques, all of the three categories of SLMs mentioned above form steps on the way universal speech processing systems. For the purposes of this paper, we define a universal specch processing system model that satisfies the following criteria:
1. It has both spoken input and spoken output with optional text input and/or output. The spoken input may serve either an instruction context.
2. It is intended to be "universal"; that is, it should in principle be able to address arbitrary spoken language tasks, including both traditional tasks and more complex reasoning about spoken data.
3. It takes instructions or prompts in the form of natural language (either speech or text), and not, for example, only task specifiers (Radford et al. (023) or soft prompts (Chang et al. 2024).
This is a functional definition, and does not restrict the form of the model. We also note that most of the models in the current literature, and therefore in this survey, do not satisfy all of these criteria; for example, many do not have speech as both input and output, and many are trained and evaluated on a fairly limited set of tasks. However, the models we include can all be seen as steps toward the same goal of eventually
2

---

Published in Transactions on Machine Learning Research (10/2025)
Pure Speech LM
Speech-Aware Text LM
Speech Text LM
Speech Encoder
Text LM
Commercial SLM
Publicly available
Closed-source
Amazon
Qwen2.5. Omni
Audio Qwen3- Omni
Flow-SLM
Audio Flamingo
DeSTA2.5- Audio
Qwen
Aligned
LLaMA- 0mni2
Nova Sonic
EVI
2025
CSM
GPT-realtime Gemini 2.5 STITCH Step-Audio
PAST OWSM v4
Omni-R1
Kimi-Audlo
TASTE
SyllableLM Align-SLM
SLAM- Omni
Balchuan- Omni-1.5
MiniCPM-o
multimodal
2025
Freeze-Omni
GLM-4-Voice
VITA-1.5
LUCY
OpusLM
ESPnet-
LLaMA-
Mini-Omni2
SNAC Codec
EVI2
Mini-Omni
Qwen 2
D+0
VolceTextBlender
LAST
Moshi
DeSTA2
X-Codec
LLaMA
Mid. 2024
WavLLM
EVI
GPT-4o
DeSTA
Gemini 1.5
SpeechVerse
DiscreteSLU
AnyGPT
SpeechTokenizer
Qwen-Audio
2024
SpiRit-LM
COSMIC
Speech- LLaMA
LLaMA
LLaSM
NEXT-GPT
LTU-AS
OWSM
SALM
SUTLM
A
VoxtLM
SALMONN
Mid.
2023
VioLA
TWIST
SoundStorm
X-LLM
Spectron
SpeechGPT
PaLM 2
Whisper
2023
2022
PaLM
WavPrompt
BEST-RQ
WavLM
PGSLM
W2V-BERT SoundStream
2022
dGSLM
Midd.
GPT-2
GPT-3
wav2vec
2.0
HuBERT
GSLM
2019
2020
On The Landscape Spoken Language Modele A Comprohensiie Sarvey
Figure 1: Development timeline of spoken language models. "Publiely available" refers to models with publicly released weights (but not necessarily codc, data, or other artifacts). "Commercial SLMs" include some systems that handle additional (non-linguistic) modalities, such as images. For more details and references, the main article and Tables and
3

---

Published in Transactions on Machine Learning Research (10/2025)
developing SLMs that can serve as universal speech processing systems, in the same way that pre-trained and post-trained text LLMs have served as steps toward universal written language processing systems.
One may wonder whether a better path to universal speech processing is to combine speech recognition, text LLMs, and speech synthesis in series (often referred to as a "cascade"). This is indeed a strong baseline approach for many tasks (Huang et al.] 2025 Yang et al., 2024b). However, some tasks require access to aspects of the audio signal beyond the word string, such as properties of the speaker, their emotional state, prosody, or other acoustic propertics. For example, detecting sarcasm often involves aspects of prosody that indicate the speaker's 's meaning may be the opposite of the naive interpretation of the uttered word sequence. In addition, even for tasks that are in principle addressable using the word string alone, an end-to-end SLM approach can avoid the compounding of errors that can occur when combining ASR, text LLMs, and TTS systems in series. Similar observations have been made in earlier work, in the context of spoken language understanding via cascaded ASR, and NL. models versus end-to-end models (Haghani et al.) 2018 Chen ct al., 2018). Finally, even when cascade approach performs well, it can introduce substantial latency, because it requires several systems to be run in series although only the output of the final one is needed.
The past few years have seen major acceleration in work on SLMs. However, there have been limited efforts to survey the design and modeling choices made in this work. Additionally, these models are often evaluated on very different tasks and datasets, making it difficult to assess the relative performance of different approaches. As part of this survey, we collect and organize many of the existing evaluations (though standardized evaluation is still one of the remaining challenges in this research area). Lastly, as SLMs have been viewed and defined differently depending on the intended applications and modeling choices, we provide unified formulation and terminology for describing SLMs (Section 2
This survey is intended to serve as a snapshot of the current moment in the evolution of the field, providing a review of the recent literature and unified definition of SLMs and their components. While new SLMs are being proposed at a stcady pace, we hope that this survey of the research landscape will help readers more easily place new models in context. We aim to provide an improved understanding of the successes and remaining limitations of SLMSs developed thus far, along the path to SLMs as universal speech processing systems.
Scope of this survey Although the ultimate goal is lask-indeenndent models that can be instructed with natural language, many AiMinsspredd approaches thus far have been task-specific (for example, special cases of conditional p(text)) speech) models for speech recognition and translation such as Whisper (Radford et al., 2023) and OWSM (Peng al. 2023b), and conditional p(speech (eet) models for text-to-speech synthesis such as VALL-E (Chen et al., 2025b and VoiceBox (Le et al., 2024)), and some have been more general but rely on task tokens or soft prompts (e.g,, Qwen-Audio (Chu et al. (023) and SpeechPrompt (Chang) ct al. (024) respectively). In this survey, we focus on models that are at least in principle task bask-independent (although they may have been tuned on a relatively small set of tasks) and that take natural language as input rather than task tokens or soft prompts. We will, however, discuss some of the important task-specific models as they relate to the more general models.
In addition, as discussed in Section 2. SLMs often comprise several components including a speech encoder, speech decoder, specch-text alignment module (when applicable), and sequence model. In this survey, we provide relatively brief descriptions of speech encoding and decoding; these have been covered well in previous surveys (e.g., Wu et al. (2024a)), and our main focus is on the aspects that are specific to SLMs, such as sequence modeling and speech-text alignment.
Finally, music and general audio share many properties with speech. A number of language modeling
approaches have been developed specifically for music and general audio (e.g., (Copet et al. 2023 Agostinelli
ct al. 2023)) ), and some SLM rescarch combines speech and other andio (c.g., (Gong et al. 2023b In this
survey we include non-speech audio only to the extent that it is used in the context of SLMs.
Related surveys To place this paper in context, we note that several other recent surveys have addressod aspects of SLMs. Zhang et al.) (2024a) and Chen et al.] (2024a) review multi-modal LLMs, including a limited subset. of SLMs. By focusing on SLMSs as our main subject, we survey a substantially larger set of
4

---

Published in Transactions on Machine Learning Research (10/2025)
Output Text yu
Output Waveform yM
Pure speech LM
Speech-text LM
Speech Decoder De
Section 3.4
Speech-aware LM
Ourput Speech Representation
Sequence Model Seg
Section 3.3
Modality Adapter Adp
Modality Adapter Ad
Section 3.2
Training Section
Iniptt Spesch Represenlation HPP
Speech Encoder E
Section 3.1
Input Text X
Input Waveform X
Figure 2:
Overview of SLM architecture. See Sections and for more detailed descriptions of the components and training methods, respectively.
modols in greater detail and explore speech-spceific issues. Latif et al. 2023 provide a survey on large audio models. This previous survey covers a much larger space of audio language models (including environmental sounds, music, and other andio) and therefore does not present. ST.Ms as much detail, nor include recent instruction-follouvyy (IF) models from the past year. Another recent survey (Wu al. 2024aa provides overview of neural audio codecs and codec-based (Section 3] models, which mainly focuses speech tokenization rather than SLMs more broadly. Guo et al.] (2025) and Mousavi et al. tokenization methods. Ji et al. (2024) provide a survey of SLMs, but their main focus is on spoken dialogue systems, rather than the full range of SLMs covered here. Another related SLM survey paper (Peng et al. 2024a) focuses on speech-aware aware text LMs but leaves out other types of SLMs. Finally, concurrent work by Cui et al. (2025) surveys a similar range of SLMs; we provide a complementary view of this research area, covering certain aspects in greater detail and summarizing others more briefly. Our main goal to provide unifying view that shows the connections and distinctions between various SLM approaches and helps the reader navigate the changing research landscape.
(2025) also survey
Outline In the remaining sections, we begin by outlining general formulation of SLMs (Section 2 followed by discussion of various design choices (Section 3 including speech encoders and decoders, modality adapters, and sequence modeling. We then describe the multiple optimization pipelines used for training SLMs (Section 4 Tn Section we review and categorize some of the notable recent. models. In Section we discuss how SLMs have been adapted for dialogue ("full-duplex") mode. In Section 7 we present existing approaches for evaluating SLMs. Finally, we conclude by discussing the limitations of current approaches and provide some suggestions for future research (Section
2 Overall architecture
We start by providing a general formulation (shown in Figure 2 of SLMs, which takes either /both speech and text input and generates either/both speech and text (where at least of the input output
5

---

Published in Transactions on Machine Learning Research (10/2025)
includes speech). Although SLMs differ in many of their design choices, this unifying description subsumes all of the models we cover
Let Xtxt E V and X E I= denote the input text and speech, respectively. That is,
x {ti, (ti.t2......
(1)
is a sequence of text tokens t, of arbitrary length N drawn from vocabulary V. The speech input is a waveform, that is a sequence
X {51, ,2,
(2)
of realevalued samples of arbitrary length T, where typically T >> N. We first map XPP using the speech encoder Pn to a sequence of speech representations H
HPP Enc (
33)
The resulting speech representations of arbitrary length L, HP = {hi, , hL}, can take one of two forms, cither a sequence d-dimensional continuous vectors hy E Rd or a sequence of discrete tokens h; E D drawn from a learned codebook D.
The speech representations 4!D are further transformed through modality adapter
AHP* E L'x''
(4)
where typically L' S L and d' is the dimension of the transformed representation. This transformation is intended to improve the alignment of the speech representations with the sequence model, which may have been pre-trained as a text model. In addition, the length disparity between speech and text sequences can be addressed at this stage by applying temporal adjustments in Adp* The text token sequence Xixt is also transformed into a sequence of vector representations, with the same dimensionality d', via an adapter
Adl"xt(X"t))) E N'xd"
The text adapter typically simply an embedding table, and therefore N' N After the adapters, therefore, the text and speech are mapped to the same representation space with dimension d', but the text and speech representation sequences are still not necessarily the same length (i.e., L' and N' need not be identical).
Next, given the adapted input text and/or /or speech representations, a sequence model, denoted Seq(), generates outputs, typically in an autoregressive manner. Here, we assume that each invocation of Seq() corresponds to one generation step. For the model types described in Table I the formulations are as follows:
Pure speech LMs
The output of Seq() is speech representation
(6) The input to Seq() is a sequence of representations the generated speech so far, with shape L' d', and the output speech representation W/ is cither a continuous vector (in R&)) or a discrete token (in D). Following the usual autoregressive generation formulation, h' then appended Seq() then generates the next output, and so on. All of the generated outputs h together form sequence T {M1 h/, h1r=} with sequence length L". Finally, the speech decoder Dec converts the output speech representations ASP to a waveform ysp;
h' Seq(Adp")
HaP increasing its length by one,
y Dec (APP).
We do make some ussumptions, for example, that the sequence model autorcgrossive, which in principle need not hold but in practice do hold for current modcls.
Wee not aware any SLMs that use any adapters besides the typical embedding table, nor models where N' * but allow this possibility the definition for maximal generality.

---

Published in Transactions on Machine Learning Research (10/2025)
5
77
Byte Pair Encoding (BPE)
Temporal Compression (Section 3.1.3) )
Deduplicate
Discrete Tokens (Section 3.1.2)
Quantization
Speech Encoder
(Section 3.1)
Continuous Features (Section 3.1.1)
Speech Representation Model
Figure 3: general pipeline for speech encoders. Note that different encoders use different components of the pipeline. See Section 3.1 for more details.
Specch-aware text LMs this type of model, the sequence model Seq() generates a text token t' according to
t' = S Adp (xtx))))
(8)
Here, the input to Seq() a concatenation of the adapted speech representation Adp* and text representation Adp (Xtex) forming a tensor with shape (L' (J++N') d'. The output is text token t' E V. This generated token is then appended to X and the process repeated to generate subsequent tokens. Finally, the sequence of all generated tokens forms the oulput text sequence yix:
Steech+text LMs In this more complex scenario, speech and text are modeled jointly:
W',t' (eq(Ad Adp (x(tt)).
(9)
Here both the inputs and outputs are "hybrid" representations consisting of a combination of speech and text representations. There are multiple approaches for generating such hybrid speech+ representations, which will be discussed in Section 8.3.2
The encoder, decoder, and sequence model Seq() are often pre-trained separately. The modality adapter is lypically trained from scratch, sometimes along with fine-tuning of the sequence model. The encoder and decoder parameters are usually fixed after pre-training. Section 3 provides more detailed information about the model components, while Section describes typical training methods.
3
SLM components
3.1
Speech Encoder
In text LMs, text is typically tokenized into subword units, generated through methods like byte pair encoding (BPE) Gage 1994), and these tokens are then represented as vector embeddings. In contrast, speech is a continuous waveform with no obvious tokenization and no separation between linguistic information and other acoustic aspects. The task of the speech encoder, shown in Figure g is to extract meaningful representations from the waveform.
The first part of the encoder a speech representation model, which transforms the speech signal into continuous features (vectors). These continuous features either be used directly as input to the speech

---

Published in Transactions on Machine Learning Research (10/2025)
modality adapter (-) quantized into discrete units (using, e.g., k-means VQ- VQ-VAE (van den Oord )) In general, pure speech LMs and speech+text LMs tend to use discrete speech tokens, whereas aware text LMs tend to use contimuous representations (with some exceptions; e.g., Flow-Omni (Yuan is a text LM that generates contimuous speech ropresentations, and DiscreteSLU (Shon 2024 is a speech-aware lext LM that uses discrete speech loken input). Finally, temporal compression (e.g-:
et
al.
speech-
ict al.
2024
et
al.
deduplication or BPE) is optionally applied to reduce the sequence length.
3.1.1
Continuous Features
To extract informative representations from raw waveforms, a speech representation model either a learned encoder or a digital signal processing (DSP) feature extractor converts speech into continuous features. These continuous features may include:
1. Traditional spectrogram features, such as mel filter bank features (Huang al. 2001
Hidden representations from self-supervised learning-basee (SSL) speech encoders, such as wav2vec 2.0 [Baevski et al. 2020), HuBERT (Hsu el al. 2021), WavLM (Chen el al. 2022). or XEUS (Chen et al.) 2024b).
3. Hidden representations from supervised pre-trained models, such as Whisper Radford et al. 2023 or USM Zhang et al. 2023b).
4. Hidden representations from neural audio codec models, such as SoundStream Zeghidour et al. 2022) or EnCodee [Défosscz et al. 2023).
3.1.2 Discrete Tokens
We divide discrete speech tokens into two main categorics, "phonetic tokens" and "audio codec tokens". based on how they are learned and their perceived properties
Phonetic tokens Phonetic tokens are most commonly derived by quantizing self-supervised speech rep- resentations (Mohamed et al. 2022) or, occasionally, supervised encoders from a pre-trained ASR model (as in Cosy (Du et al. 2024). GLM-4-Voice (Zeng et al.) 2024), and WhisperSpeech (WhisperSpeechi 2024 ). These quantized representations tend to have strong similarity to phonetic units (Sicherman & Adi 2023 The pioneering pure speech LM GSLM (Lakhotia al. 2021 treated these tokens as "pseudo text," allowing the model to function similarly to text-based LMs and to generate phonetically coherent speech. These tokens, learned directly from raw speech, formed a foundation for subsequent research, often described
as "textless NLP" (Polyak et al. 2021 Lakhotia et al. 2021 Kharitonov et al. 2022b Nguyen et al. 2023
Hassid et al. 2023
The process of deriving phonetic tokens involves several design decisions, such as selecting a pre-trained representation model, which layer's representations to quantize, and the number of clusters. Different lay- ers in SSL speech models tend encode different types of information (Hsu et al., 2021 Mousavi et al., 2024b Pasad al. 2023). Representations from layers rich in phonetic information are typically chosen for extracting phonetic tokens. The number of clusters is also a key hyperparameter, and is often optimized based on downstream task performance or proxy task (e.g., the ABX task (Schatz et al.) 2013 Too few clusters may result in a loss of fino-gran aoo-grained phonetic information, while too many clusters may risk encoding undesirable speaker-specifi features [Lakhotia et al. 2021 Kharitonov et al. 2022a
Audio codec tokens In contrast phonetic tokens, audio codec tokens are intended to capture more detailed acoustic characteristics. These tokens are derived from neural codec models, which were originally designed for audio compression and therefore for faithful reconstruction of audio from its encoding (Zeghidour
"In some literature, phonctic tokens reforred to "semantic tokens" [Borsos al.) 2023a). However, it's important akin
to clarify that "scmantic" in this context does not imply the traditional linguistic meaning, these tokens phonetic units (Sicherman Adi 2023) and not typleally earry semantic content. Therefore, we use the "phonetic tokens" throughout this paper.

---

Published in Transactions on Machine Learning Research (10/2025)
et al. 2022 Défossez et al. 2023). Although audio codecs were initially intended for compression, their intermediate discrete representations have proven valuable as tokens in SLMs (Borsos et al. 2023a).
Neural codecs comprise three components: an encoder that converts raw audio into frame-level features, a vector quantization (VQ) module that converts these features into discrete tokens, and a decoder that reconstructs audio from the codec tokens. Typically, the main encoder-decoder backbone is a convolution- dominant architecture (e.g., Encodec (Défossez et al.] (023)) ), but some recent work has introduced trans- formers intermediate layers (c.g., Mimi (Défosscz et al. 2024 or transformer-only architectures to reduce computation (e.g.. TS3-Codec (Wu et al. 2025b[).
For the VQ module, a commonly used approach is residual vector quantization (RVQ) (Gryy
1984
Zeghi-
dour et. al. 2022), which gemerales several discrete tokens for each time step, corresponding lo multiple (hierarchical) levels of gramularity (with the first level encoding most of the information, and the remaining levels encoding residual information). Decoding such multi-codebook speech tokens typically requires ad- ditional design considerations for the SLM architecture and decoding strategy (Borsos al., 2023a Chen
et al. 2025b Yang et al., 2024a). Alternatively, several single-codebook quantization codecs (Wu et al., 2025b
Xin et al. 2024) have been developed to simplify decoding in SLMs. Section 3.3 describes decoding
strategies in detail.
Comparison of token types Early SLMs, such as GSLM, typically used phonctic tokens, which reduce speaker-specific information (Polyak et al. 2021). This reduction allows language models to focus primarily on spoken content, making them more effective for understanding tasks like detecting words vs. nonwords or syntactic correctness (Lakhotia et al.) 2021 Hassid et al. 2023 and for applications such as speech-to- speech translation (Lee et al. 2022a b). In contrast, audio codec tokens are frequently used in tasks where preserving speaker identity and acoustic details is crucial (Chen et al., (025b)).
Recently, several tokenization approaches have explored hybrid strategies that combine phonetic and acoustic tokenization. Some examples include distilling pre-trained SSL representations into RVQ codebooks (Zhang 2024b Défossex et al. 2024). integrating phonetic or textual information during the tokenization process (Tenn et al.) 2025 Har-Tur et al. 2025), and incorporating LM-based objectives into tokenization
et al.
learning (Turetzky & Adi 2024).
In SL.Ss, two important factors in the choice of tokenization are the token bit rate, which impacts efficiency, and the token quality, which related to generation quality and suitability for downstream tasks. Several benchmarks have been established to evaluate different types of tokens (Shi et al. 2024 Wu et al. 2024c Mousavi et al., 2024a Wu et al.] 2024b). Codec-SUPERB (Wu et al.) 2024ch, the first neural audio codec benchmark, evaluates the quality of resynthesized audio, using subjective metrics and pre-trained downstream models for comparison. DASB (Mousavi et al. 2024a) evaluates tokenization methods by using the extracted tokens for various downstream tasks. DiSPnet-Coder (Shi et al.] 2024 is an open-source framework that functions as both a toolkit for neural codec training and a platform for evaluation in a controlled setting.
Designing efficient and effective tokens for SLMs remains an active area of research; for more detailed surveys, see Guo al. 2025 Mousavi et al. 2025)
3.1.3 Temporal Compression
Quantized speech feature sequences are often very long due to their high frame rates. To mitigate the challenges this poses for language modeling, several techniques are typically applied within the encoder to reduce the sequence length. One such technique is "deduplication", which merges consecutive identical tokens into a single token (as shown in Figure 3. However, this approach loses information about the duration of individual tokens. To address this issue, some approaches use multi-stream modeling to capture token duration in a separate stream (Nguyen et al. 2023), while others introduce duration prediction when generating output speech (Kreuk et al.) 2022 Maimon & Adi 2023 Lee et al., (022a).. Additionally, byte pair encoding (BPE) (Gage. 1994) also sometimes applied to the discrete token sequences to capture recurring
patterns [Wu et al.] 2023a Shen et al. 2024).
9

---

Published in Transactions on Machine Learning Research (10/2025)
3.2 Speech Modality Adapter
In many SLMs (especially speech-aware text LMs), the speech encoder (Section 3.1 and the sequence model (Section 3.3) are initially developed separately and then combined. It is therefore necessary to somehow align the output of the speech encoder with the expectations of the sequence model, and this the role of the modality adapter. The modality adapter typically trained on downstream tasks or, in the case of speech+text. LMs, as part of pre-training (see Section for more details on training).
the output sequence of the speech encoder not very long, the modality adapter can be as simple linear layer, which transforms the encoder output into the embedding space of the sequence model. This typically occurs when the encoder produces discrete tokens with temporal compression. On the other hand, if the output of the encoder is too long, the adapter is typically also responsible for shortening the sequence. This usually happens when the encoder produces continuous representations without discretization or temporal compression. Shortening the imput sequence simplifies both training and inference for the sequence model. Furthermore, since the sequence model often processes both text and speech inputs, it is helpful for the loken rale of the speech sequence roughly match that of the text sequence.
Common adapters include:
Linear transformation / vocabulary expansion A straightforward way to integrate speech into pre- trained sequence model is by applying a linear transformation to the speech representation H*P In this approach, the adapter Adp**() is defined as a linear transformation. This method commonly used when the speoch encoder produces discrete token outputs. This approach can be interpreted as vocabulary expan- sion, where the speech tokens are treated as additional tokens in the sequence model's vocabulary. Their embeddings then learned during subsequent task-oriented training. For example, in SpeechGPT (Zhang et al. 2023a), the vocabulary of the sequence model LLaMA Touvron et al. 2023a) is expanded to in- clude HuBERT-basee phonetic tokens. Similarly, in Mini-Omai (Xie & Wu 2024b), the sequence model Bai et al. 2023) incorporates eight. layers of acoustic codec tokens alongside standard text tokens.
Qwen
CNN with strides: Convolution layers with strides reduce the sequence length while preserving temporal information (Lu al. 2024), which is essential for tasks that require such information, like ASR (Wu al., 2023b). A special case of this adapter type pooling layers with strides (Chu et al. 2023).
Connectionist temporal classification (CTC)-based compression: This method compresses HPP
(Fq. 3) according to the posterior distribution from CTC-based speech recognizer (Gaido et al. 2021 CTC (Graves et al. 2006), commonly used approach for ASR, assigns each time step a probability distri- bution over a set of label tokens, including a blank ("none of the above") symbol. The time steps with high non-blank probabilities indicate segments that are likely carry important. linguistic information. CTC compression aggregates the frame-level labels, specifically by merging repeated non-blank labels and remov- ing blanks. This approach produces compressed representation intended to retain the relevant content of the original sequence while significantly reducing its length (Wu et al. 2023b Tsunoo et al. 2024h.
Q-Former: The Q-Former [Li et al. 2023 is an adapter that produces fixed-length representation by encoding a representation sequence of arbitrary length into M embedding vectors, where M is hyperpa- rameter (Lu et al. 2025a).
Let the input spocch representation sequence bc:
HaP h2, h by E E
(10)
where L' is the sequence length and d' is the dimension of the embeddings. To achieve a length representation, Q- introduces M trainable query embeddings:
Q {q1 2...... ) 9m Rd
(11)
These queries interact with SP via cross-attention mechanism:
Attn(Q,HP) = softmax (QWo(H?WW>))) HrW,,
(12)
Vd
10

---

Published in Transactions on Machine Learning Research (10/2025)
LM
Temporal Transformer
Head
Prediction Head
Coarse Token
Fine Fine-Grain Token
Finer Token
1
3
1
X
3
2
1
1
1
2
2
K
A
+
4
(a) Coarse first, then fine-grained
(b) Interleaved coarse and fine tokens
3
3
Head
Head
Head
Head
Head
Head
Head
Head
Head
Head
LM
LM
(c) Temporal generation plus depth generation
(d) Delay pattern
Figure 4: Hierarchical generation strategies (see Section 3.3.1
where Wo, WK. and Wv are learnable projection matrices. The result sequence of M embeddings.
In some approaches, instead of directly encoding the entire utterance into M vectors, window-level Q- Former applied (Yu et al. 2024 Pan et al. 2024 Taang et al. 2024) to retain temporal information. In the window-level Q-Former, the input embedding sequence is segmented, and the Q-Former is applied to each segment.
Lu et al. 2024) compare the Q-Former with CNN-based modality adapters in a speech-aware text LM, finding that the Q-Foreer produces better performance on the Dynamic-SUPEiR benchmark (Huang et al. 2024 (see Section for more on this and other SLM benchmarks).
AlignFormer: AlignFormer [aan al. 2025 combines CTC compressor Gaido al. 2021 with Q- Former [i et al. 2.23). When the LLM backbone is frozen, Fan et al. 2025) find that AlignFormer enables capabilities for speech QA tasks using only ASR training data. Additionally, AlignFormer surpasses Q-Former in instruction -owettiiinnooooiiinggg.oo performance across multiple datasets and tasks, including SQA and speech translation.
zero-shot instruction-follovii
Others: Several other methods have been developed for the modality adapter. For example, LTU-AS uses time and layer-wise transformers (TLTR) (Gong et al.) (2023a). WavLLM (Hu et al.) (2024) uses bottleneck adapter (Houlsby et al. 2019). and other approaches multi-layer perceptron (MLP) adapters (Fang et al.
2025at
Microsolt et al. 2025
3.3 Sequence Model
In principle, generating speech tokens similar to generating text tokens. However, there are some important differences. Unlike text tokens, audio tokens may include mix of coarse and fine tokens - for example, pho- netic tokens coarse tokens and audio codec tokens fine tokens. For TTS models, using phonetic tokens as an intermediate representation rather than directly mapping text to audio codec tokens provides better supervision and reduces modelling complexity. For example, SPEAR-TTS (Kharitonov el. al. 2023), which uses phonetic tokens, has similar performance to VALL-E (Chen et al. 2025b) while requiring significantly less parallel text-to-spoech training data. The same concept applics SLMs, as described below. In addition to generating multiple speech token types, some SLMs combine text and speech token generation to enhance the linguistic quality of the generated speech, described in Section 3.3.2
Soction 3.3.1
11

---

Published in Transactions on Machine Learning Research (10/2025)
3.3.1 Hierarchical Token Generation
There are several decoding strategies for hierarchical modeling of multiple lypes of tokens of different gran- ularity (see the discussion in Section 3.1.2). shown in Figure 4 Phonetic tokens and first-layer codec tokens (Zhang al. 2024b) ) can be regarded as coarse tokens in Figure while the remaining layers of codec tokens can be regarded "fine-grained" and "finer" tokens
(o.g.: Mimi (Défossez et al., 2024) and Spoech
We categorize common decoding strategies into the following types:
Coarse first, then fine-grained (Figure 4 (a)): In this approach, coarse tokens for all time steps are generated first, followed by fine-grained tokens conditioned the coarse tokens, and finally the finer tokens. For example, AudioLM (Borsos et al. 2023a] uses such three-stage autoregressive process (which can be modeled by different LMs), first predicting phonetic tokens (the coarse tokens in Figure 4 (a)), then first- layer audio codec tokens (the fine-grained tokens in Figure 4 (a)) conditioned on the phonetic tokens, and finally the remaining-laye ett al. 2023b) replaces the second and third stages of AudioLM with a non-auuoreegesssive prediction approach inspired by MaskGIT (Chang et al., 2022), which begins with masked tokens and then oou--atoorerrssively predicts subsets of tokens in rounds based on confidence scores. Similarly, VALL-E (Chen et al. 2025b) uses autoregressive model to predict the first-layer audio codec tokens based on the corresponding phoneme transcriptions, and then uses non-autoregressive model to generate the remaining codec token sequences.
codec tokens (the finer audio codec tokens in Figure (a)) SoundStorm (Borsos
Interleaved coarse and fine tokens (Figure 4 (b)): In this decoding strategy, the three types of tokens- coarse, fine-grained, and finer finer-are aligned along the time axis and interleaved for generation. When predicting tokens for time step i, the coarse, fine-grained, and finer tokens corresponding to t are gen- erated sequentially. SpiRit-LM (Nguyen et al. 2025) applies this interleaved structure (interleaved phonetic, pitch and style tokens) to enhance speech expressiveness, and also includes text tokens addition to the multiple types of speech tokens.
Temporal generation plus depth generation (Figure 4 (c)): This strategy employs a large trans- former LM (indicated in blue) to model inter-frame information along the time axis and small transformer head (indicated in green) capture intra-frame correlations. The small transformer head predicts multi-layer ceetens-wherre fino-grained and finer tokens correspond to different layers of audio codec tokens lakens-wwthhn the samne time step. For example, UniAudio (Yang et al.] 2024a] adopts this approach, inspired by RQ- Transformer (Lee et al. 2022ch and the MegaByte transformer model (Yu et al. 2023). Moshi DDéfossez
et al. 2024 also adopts a similar strategy.
Delay pattern (Figure 4 (d)): In this approach, there are time delays (of 1 or more time steps, depending on the design) between coarse and fine tokens. Such time delays allow the model to utilize look-ahead, that
is to use future coarse tokens as conditions when generating fine tokens. Figure (d) shows an example where the delay is 1. The large blue LM in Figure (d) autoregressively outputs the temporal embeddings
and feeds them into the green prediction head, which then predicts the coarse token and the delayed fine token in parallel. For example, pGSLM (Kharitonov et al. 2022b) introduces a delay between the phonetic tokens and the "expressive" (pitch and duration) tokens.
Using multiple stages of prediction can achieve both high audio quality and long-term consistency [Borsos et al. 2023a). However, this approach makes the decoding strategy complex and increases latency, making it less suitable for real-time applications like spoken dialogue generation. To address this drawback, there is a growing literature on tokenization methods that reduce the number of tokenization layers. One example is to combine the generation of phonetic and audio codec tokens into a single tokenizer by distilling the information from phonetic tokens directly into the audio codec Défossez et al. 2024 Zhang et. al., 2024b).
12

---

Published in Transactions on Machine Learning Research (10/2025)
3
77
23
12
71
34
3
23
(a) Mini-Omni
how
are
you
77
12
71
(b) LLaMA-Omni
how
are
3
77
71
34
3
23
(c) Moshi
how
you
(d) SpiRit-LM
how
12
71
you
16
88
15
Figure 5:
Text and speech hybrid generation, exemplified by four representative models (a)-(d) (see Sec-
tion 3.3.2 Green and yellow boxes represent text and audio tokens, respectively.
3.3.2 Text and speech hybrid generation
Some sequence generation models use text tokens the first-layer tokens, followed by speech tokens (either phonetic or audio codec tokens). This approach allows the SLM to leverage the knowledge from pretrained text LM, which is typically trained on much larger-scale data than the available speech data, to improve the factuality and linguistic quality of generated speech. As shown by Défossez et al. (2024), this approach can improve language modeling quality, as measured by negative ligelikelihood scores measured by an external text L Another advantage to this approach is that during the process of developing the SLM, it is straightforward to separately evaluate both the correctness of the output content and the speech generation quality.
Text and speech tokens operate on different scales: Speech has fixed sampling rate (unless tokenized into tokens of different durations), whereas text tokens do not. A text token sequence is also usually shorter than the corresponding speech token sequence. Hybrid decoding needs to address the issue of differing lengths and, ideally, also to achieve temporal aligument (synchronization) There are four main types of text-speech hybrid generation used in recent SLMs, shown in Figure We use four representative models to illustratc the ideas. The first type (Figure (a)) addresses the mismatch in sequence lengths by adding padding tokens after the text token sequence. In this setup, the text sequence ends first, and the generated text then guides the generation of the speech token sequence, making the process similar to text-to-speech (TTS). A representative example is Mini-Omni
(Xie & Wul 2024a), which adopts a delayed decoding approach (Copet et al., 2023
The second type (Figure 5 (b)) involves adding fixed padding tokens after each text token to extend the text token sequence. Padding tokens are also added between speech tokens so that both sequences have the same length. An example of this approach is LLaMA--mni (Fang et al., 2025a): After predicting the text tokens (padded with fixed-length padding tokens), it then predicts the phonetic token sequence based on the padded text (where the phonetic loken sequence is shorter than the padded text token sequence), and finally the CTC loss is applied to the phonetic token sequence to guide training.
The third type (Figure 5 (c)) dynamically adjusts the number of padding tokens between text tokens. The generative model learns to insert these dynamic padding tokens make the text and speech sequences
9In Figure 4 and in running examples, assume 3 granularity levels. In practice, sequence model can include smaller larger number of levels, and this number also important design choice. 6n this case, LateIllma-460M-17 httpp:// hww.gg/ggaaccccoo////ooooo////// wtw.t/cccccoocc.//e/c/oooocco.et httpa/////////////aaaaaaaaa Iamteaaaammm-0000--
13

---

Published in Transactions on Machine Learning Research (10/2025)
the same length. During training, time-aligned text-speech paired data is used to construct padded text sequences that match the lengths of the speech sequences. Moshi Délossez et al. 2024) is an example of this model type.
The fourth type (Figure M (d)) interleaves speech and text tokens in a single sequence, with text-speech
token alignments derived in advance. SpiRit-LLM (Nguyen et al. 2025) an example of this approach, using time-aligned text-speech paired data for training. GLM-4-Voice (Zeng et al., 2024 2025) uses pre-trained
text-to-token model to generate audio tokens from text, and interleaves the generated audio with text tokens.
3.4 Speech Decoder
The speech decoder converts speech representations- wwetther continuous features, phonetic tokens, or audio codec tokens back into waveforms. The speech decoder can take various forms:
1. Vocoder [ong et al. 2020) for continuous features, similar to those used in traditional synthosis sys- lems. For instance, Spectron (Nachmann al.] (024)), a generated Mel spectrogram is synthesized into audio using the WavFit vocoder (Koizumi al. 2023).
2. Unit-based vocoder (Polyak al., 2021 based on HiFi-GAN (Kong al. 2020 for phonetic to- kens. These vocoders take phonetic tokens inputs and optionally combine them with additional information to improve synthesis quality. For example, when phonetic tokens are deduplicated, a duration modeling module is often included in the vocoder (Lakhotia et al. 2021 ).
3. Codec decoder (Guo et al. 2025 When the SLM generates audio codec tokens, these tokens can be input directly into the corresponding pre-trained codec decoder (without additional training) to generate the waveform.
4 Training Strategies
We divide the training phases of SLMs into pre-training and post-training, analogously (but not identically) to the typical division in text LLM training. For text LMs, pre-training refers to training a model of p(tezt) with next-token prediction objective, whereas post-training is task-oriented and directly facilitates alignment and improved performance on tasks. Since many SLMs start with post-trained text LLM, we consider this to be a part of SLM pre-training. More preciscly, we define pre-training and post-training of SLMs as follows:
Pro-training: We define pro-training as any training strategy that does not have the explicit gool of enabling the model to perform wide variely of downstream speech lasks. In particular, pre-training does not directly aim to make a model universal speech processing system (but, as described below, it can include training for specific tasks). In the context of SLMs, which handle multiple modalities, pre-training often involves a multi-stage process. This may include initial next-token-prediction training on text, followed by continual training on speech contimuation tasks using large unlabeled speech corpora. Continual training may enhance the capabilities of SLMs for specific tasks, but the resulting pre-trained SLMs remain limited in scope and are still far from being universal models. Text LLM post-training before adding the speech modality also considered pre-training of the SLM, since it does not target. universal speech processing.
Post-training: Post-training refers to training strategies focused on enabling a variety of dounstream lasks, with the goal of producing a more or less universal speech processing system. The tasks can be either predefined with specific task specifiers or dynamically defined through text or spoken language prompts. Instruction tuning (Tang et al. 2024 Pan et al.) 2024 Shu et al. 2023) and preference optimization Lin et al. 2025c) are examples of post-training.
4.1 Pre-training strategies
Here we focus on speech-specific pre-training strategies; that is, we do not discuss the training of any text LM that is included in SLM pre-training.
14

---

Published in Transactions on Machine Learning Research (10/2025)
4.1.1 Generative pre-training
Pure speech pre-training p(speech) Pure speech pre-training refers to training a model of p(speech) from unlabeled speech (a pure speech LM in Table D. typically by tokenizing the speech and modeling the token sequences with an autoregressive model trained to perform next speech token prediction. The success of SSL speech representation modcls facilitated the development of pure speech LMs, because SSL models provided high-quality representations that could easily be quanlized to produce lokenized speech. Pure speech LMs, such GSLM (Takhotia et al., 2021]. typically use discrete phonetic speech tokens as the basic language modeling units, but non-phonetic information such as duration and pitch can also be incorporated (Kharitonoy et al. 2022b Nguyen et al. 2023 to enhance prosody modeling. Section 5.1 expands on pure speech LMs.
Joint speech and text pre-training p(tezt, speech) This approach refers to pre-training SLMs on aligned speech and text to model p(tezt, speech) (speech+text LM in Table [ (Chou et al., 2023 Nguyen et al, Tian et al. 2025a). The speech and text sequences may be interleaved (Chou
2025
Défossez et al., 2024
et al. 2023 Nguyen et al.
2025 or treated as two channels in a dual-channel approach DDélossez el al.
2024
as detailed in Section
3.3.2
Continual pre-training following text pre-training p(tezl) Continual pre-training (Ke et al. 2023 refers to the process of further training a pre-trained model on additional data that is domain- or modality- specific, but still not task-oriented. This intermediate step allows the model to adapt to new domains or datasets while (hopefully) retaining its general-purpose purpose capabilities (Tang et al. 2024 Chu et al.) 2023 Hassid et al., 2023]. A common example of continual pre-training for SL.Ms is to start with a text L.M pre-trained with next token prediction objective (a pure text LM in Table [ and then further pre-train it
with speech data to improve its performance the speech continuation task (Hassid et al. 2023
4.1.2 Conditional pre-training p(tezt|speech)
While generative pre-training the most common form of pro-training, is also principle possible initialize SLM training from conditional model. For example, the pre-trained conditional model could be encoder-decoder speech recognizer (Chan et al.) (2016)
joint speech recognition and translation model like Whisper (Radford et al.] 2023) or OWSM (Peng et al.) 2023b 2024b 2025a) that uses task specifier tokens to indicate the desired task. Such models can be trained on massive amounts of transcribed speech, and therefore have already learned to align the speech and text modalities. Although these models are trained for very specific tasks, they have shown promise as an initialization for instruction-followe following models that perform a wider range of understanding tasks (Lai et al. 2023 Arora ct al., 2024).
4.1.3 Aligning speech and text modalities
For SLMs that combine pre-trained text LMs and speech encoders, another pre-training strategy is to align the speech and text modalities in a task-independent way.
Implicit alignment Speech and text modalities can be implicitly aligned through techniques such as the Smodal-invariance trick" (Fathullah et al. 2024 or behavior alignment (Wang et al. 2023a). The idea is that the model should produce identical responses regardless of the input modality, provided the input conveys the same meaning. This approach often utilizes ASR datasets. The text transcript is input to a text LLM to generate text response, while the corresponding speech recording is input into the SLM, which is trained to generale the same text response. Another useful idea for implicit alignment is training spoken LLMs for audio captioning, where a spoken LLM takes audio as input and outputs its description. It has been observed that training a spoken LLM solely through audio captioning can generalize to tasks it has never seen during training (Lu et al. 2024 2025a).
Explicit alignment Speech and text modalitios can also be explicitly aligned by matching spooch features to corresponding text embeddings, via optimization of appropriate aaltance/smmmiaaiii measures. For exam- ple, Wav2Prompt (Deng et al. 2025) and DiVA (Held et. al. 2025) align modalities by minimizing the Toz
15

---

Published in Transactions on Machine Learning Research (10/2025)
Table Examples of instructions for speech-related tasks used in SLM instruction tuning.
Task
Examples of Instructions
Recoguize the speech and give me the transcription. Tann et al.
2024
Speech ronognition
Repeat after me in English. (Grattafiori et al. 2024
Translate the following sentence into English. Grattafiori al.
2024
Speech translation
Revognize the speech, and translate into English (Clu et al.
2023
Speaker recognition
How many speakers you hear this audio? Who are they? Tang et al.
2024
Describe the emotion of the speaker. (Tang al. 2024
Emotion recognition
Can you identify the emotion? Categorise into: sad, angey, neutral, happy
Das et al.
2024
What happened to this person? (Wang et al.
2023b
Question answering
Generate factual answer to preceding question (Das al. 2024
What medicine mentioned? Briefly introduce that medicine. Peng et al. 2025b
distance between speech features and the token embeddings of their transcripts in a text LLM while keeping the text embeddings fixed.
4.2 Post-training strategies
The pre-training of LLMs and SLMSs in Section 4.1 enables the modeling of the general data distributions of text p(tezt) or speech p(speech), the joint distribution of speech and text p(teat, speech), or a conditional distribution vitezt|speech) basod on specific pre-training tasks. However, pro-trained modcls still often lack the capability solve a large range of downstream spoken language Lasks, to follow natural language instructions, or both. In the post-training phase, carefully curated datasots are used train SLMs generate desired outputs or perform tasks, typically specified using natural language instructions.
4.2.1 Task-specific training
While the eventual goal to handle arbitrary tasks within a single model via instructions, some SLM approaches begin with simpler post-training setting: multi-task training with task specifiers p(-lspeech, (lask specifier)). In this approach, the pre-trained SLM is fine-tuned for predefined set of target tasks. Swwn-Audis is example of such approach (Chu et al. 2023). Task-specific training then be followed by instruction tuning other post-training approaches, described below.
4.2.2 Instruction tuning p(-|speech, instruction)
Instruction tuning of SLMs has been inspired by, and closely follows, successful approaches for text LLM instruction tuning. Instruction-tuning data typically consists of a speech recording, an instruction that describes the speech task, and the ground-truth output. During instruction tuning, the instructions are appended to the speech recording as inputs to the SLM, which is trained to generate the corresponding ground-truth output. Depending on the model design, the ion can be in either text format (Tang ict al. 2024 Pan et al. 2024) or speech format (Shu et al.) 2023 In both cases, it has been found that SLMs trained on diverse instruction-Luning data can perform tasks unseen during the instruction-luning phase (Tang et al. 2024 Das et al. 20244 Peng et al. 2022b).. Table shows examples of instructions for various speech tasks taken from existing instruction tuning sets.
Instruction-luning data can be generated through various methods:
Conversion of task-specific data to instructions Task-specific data (Section 4.2. can be adapted to the instruction-tuning format by replacing task-specific tags with natural language instructions (Tang et al.
2024h. LLMs can be used to rephrase those instructions to increase diversity (Arora et al. 2024).
Speech-based based question answering (SQA) data Such data is typically generated using LLMs such ChatGPT. Tn this process, the transcript of speech recording is provided as input to LLM, which is
16

---

Published in Transactions on Machine Learning Research (10/2025)
instructed to generate question-ansee answer pair related to the speech content in text format Taag et al. 2024 Gong et al. 2024 Peng et al. 2025b). To incorporate additional context, supplementary textual descriptions about attributes such as the speaker, gender, age, emotion, and noise level may also be provided to the LLM Yang et al., 2024b.. During training, the speech recording and the corresponding LLM-generated question are provided as inpuls, and the model learns to predict the L.M-generated answer.
Synthesis of text instruction-tuning data Speech instruction-tuning data, can also be created by applying TTS to existing textual instruction-tuning or user-assistant conversation datasets (Peng et al, 2025b). In this approach, either a subset or the entirety of the user's input is converted to speech. This type of data encompasses wide variety of instruction types and response styles. The answers tend to be more descriptive than the ones in SQA and are presented in diverse textual formats, such as markdown.
Compositional instructions Instructions for individual tasks can be combined to form more complex instructions, which improves the performance on more challenging tasks. For example, an SLM can be instructed to first perform speech recognition and then speech translation conditioned on the ASR hypoth- esis Hu al. 2025).
4.2.3 Chat SLM training
In addition to the general post-training methods mentioned above, an important specific application setting that is gaining attention conversational or chat SLMs, which require carcfully curated training data and tailored training strategies. The development. of chat SLMs has involved two main directions: (1) building speech-aware text LM based on a text LM with chat capabilities, and (2) creating a pure speech LM or conversations. For (1), one approach is
ttet LM that can handle speecd-inpuut-oo-ppeed-ooouu to use more powerful text LLM to generate text-based conversations centered around speech recording and use this pseudo-conversation data to fine-tune the SL. (Chu et al. 2023). For (2), common approach is to use a TTS system to generate speech conversation data from text datasets (Zhang et al.) 2023a Défossez et al. 2024). available, real speech conversation data can be used to provide more natural, spontaneous behaviors- such as pauses and interruptions that occur in real conversations, though such data is often noisier and requires careful preprocessing (Défossez et al., (024).. Recent efforts have included post-training conversational SLMs with tool-usage capabilities to enhance response accuracy (Arora et al. 2025a). Section 6 addresses conversational SLMs in greater detail.
4.3 Other training details
In addition to the basic strategies discussed in Sections 4.1 and 1.2 several additional training methods have proven useful for building universal SLMs, including:
Parameter-efficient fine-tuning (PEFT) Since pre-training equips LMs with strong text and speech generation capabilities, it is possible to not update the entire LM during the fine-tuning phase. Common strategies include: (1) freezing both the sequence model Seq and the speech encoder Enc (see Figure 2 and updating only the parameters of the adaptor Adp (Wang et al. 2023b). and (2) adding to the sequence model a set of parameter-effi efficcien trainable adapter modules, which are trained alongside Adp" (Tang et al. 2024).
Progressive fine-tuning A key objective in training specch-aware LMs and speech textt. LMs is to align the hidden representations of speech and text, in order to leverage the generation and reasoning capabilities of the pre-trained LLM. To achieve this goal, it is common to kick off post-training with content-heavy tasks, such as ASR, and gradually progress to tasks that require more comprchensive understanding of additional information embedded in speech (e.g. emotion recognition) Ç (Tang et al. 2024h or tasks composed of multiple sub-tasks (Hu et al., 2024). I. To stabilize training, a small number of components is updated during the initial training stage, while more are updated later.
A similar strategy has been adopted in work on conversational SLMs Défosscz al. 2024). The intrinsic properties of human conversations present several challenges for building such SLMs. For instance, speech
17

---

Published in Transactions on Machine Learning Research (10/2025)
from different speakers often overlaps, rather than following strictly turn-based structure. Therefore, training on real human conversations is essential. However, collecting spoken human-human conversation datasets is difficult, and publicly available datasets are often limited in size. To address these challenges, these models may be first trained on multichannel audio data before being fine-tuned on real human conversation data. See Section f for more details about such models.
Experience replay of pre-training data during post-training To prevent the catastrophic forgetting phenomenon (Kirkpatrick et al. 2017), reusing pre-training data during the post-training stage can help the SLM retain important capabilities learned during pre-training, such as the rcasoning abilitics of LLMs associated with text. instruction-Luningg (Peng et al. 2025b). Another approach to prevent lorgetting is to use data generated by the original text LLM during post-training (Lu al.) 2025a).
Preference optimization Reinforcement learning from human feedback (RLHF) (Ouyang et al. 2022) has become a key method for aligning LLM generations with human preferences. This method was used in Qwen2-Audia (Chu et al., 2024) to improve the quality of responses generated by the speech-aware text LM. On the other hand, Align-STM (Lin et al., 2025e) was the first to apply this method to a pure SLM and used Al-generated feedback as a substitute for human evaluation to make the process more cost-efficient, and this strategy has also proven useful in speech+ LMs well Wu et al. (2025a). Recently, Maimon et al. (2025a) analyzed the performance of pure SLM fine-tuning using direct prefenrece optimization as function of fine-tuning examples.
5 Survey of representative SLMs
The previous sections have described the components, design choices, and training strategies for SLMs. In this section, we review how these choices have been instantiated in existing SLMs in the litcrature, categorized according to the classes in Table [ For each category, we provide a brief review of the model design and highlight example models in the literature. Note that, aside from the examples listed, there are many SLMs with similar configurations in each category which we are unable to introduce in detail; Tables and 5 provide a more extensive catalogue of models and references, and Figure provides a timeline to place them in historical context.
We also note that some of the multi-modal LLMs developed by industry groups, such as GPT-4o (OpenAl 2024) and Gemini (Gemini Team et al. 2024), can also be seen as generalizations of speech LMs or speech-aware text L.s. These models have conversation capabilities and include text, speech, general audio, and visual inputs and outputs. However, details of their model configurations and training stratogics have not been publicly released, and rigorous evaluations of these models have focused on limited speech lasks such ASR and ST. For these reasons, it is important that future work complements these high-impact industry models with open-source, reproducible approaches.
5.1 Pure Speech LM
Pure speech LMs are trained on unlabelled speech data to model p(speech). and are the spoken analogue of pre-trained generative text LMs like GPT (Radford et al. 2019 Brown et al. 2020). In order to apply the noxt-rookn-preciition objectives of LM training to continuous speech data, pure speech LMs typically use discrete speech represenlations (Section 3.1p as the units for sequence modeling. One of the earliest approaches in this category is the generative spoken language modeling (GSLM) work of Lakhotia et al. (2021). GSLM relies on pre-trained HuBERT model Enc*P() to convert speech signals into phonetic tokens Ha and uses an embedding layer Adp*P() to map token IDs to continuous vectors. The model is trained to autoregressively predict the ID of the next speech token, and the predicted tokens are converted back to audio by a unit-based vocoder (). GSLM's deme shows anecdotally that this initial model can generate speech continuations with locally acceptable syntax given a speech prompt of a few seconds, but does not in general produce coherent long-forn form speech.
"https://speeccboo.ggi/ htt.://www.cbo..tt...ooo.o./e/e io/gslm
18

---

Published in Transactions on Machine Learning Research (10/2025)
Since GSLM initiated this line of research, follow-up work has improved on it in several ways. For example, the prosody-aware generative spoken language model (pGSLM) (Kharitonov et al. 2022b) predicts prosodic information like pitch and duration jointly with phonetic tokens for better expressiveness. TWIST (Hassid
ct al. 2023) is similar to GSLM but the model is initialized with a pre-trained text LLM (OPT (Zhang et al. , which improves lexical and syntactic language modeling performance, as measured by the sWUGGY
2022
and sBIIMP metrics (sce Section 7.1 for more details of the evaluation). Another approach (Algayres et al.
2023 uses longer lexical units instead of phonetic tokens, significantly reducing memory consumption. Audi- 2023a) introduced hierarchical generation of multiple types of tokens for pure speech LMs, using a coarse-to-fine strategy that generates phonetic tokens and audio codec tokens in order (Section 3.3. SpeechSSM (Park et al.
oLM [Borsos et al.
2025 uses state-space model (Gu et al. 2022) rather than & Transformer, produc- ing more natural and semantically consistent. long-form (minutes-long long) speech generation. Flow-SLM (Chou et al. 2025) and CALM (Simon et al. 2025) generate continuous representations rather than or in addition to discrete tokens, to improve or maintain acoustic detail while reducing the computation involved in using hierarchical codecs.
5.2 Speech+Texx Text LM
Speech+text LMs (Nachmani et al. 2024 Nguyen et al. 2025 Défosscz et al. 2024) are SLMs that jointly model the distribution of speech and text speech) ) and can therefore understand and generate both modalities. Such models often start with generative speech+ +txt text pre-training (Section 4.1 and can then be
post-trained in variety of ways.
A notable example of this approach is the Moshi model (Défossez et al. 2024). which is the first open- source speech-in-speech-oott SLM-based dialogue system with real-time inference capability. Moshi takes time-aligned text and discrete speech representations as inputs and generates outputs in both text and speech forms. Moshi's text processing components-Ein Ence--Enexxiie (), Adp" and Dec (), and the LM backbone Seq()= are initialized from a pre-trained Helium text LLM (Déoossez et al.) 2024). After text pre-training, the LM is continually trained on joint prediction of the next speech and text tokens, followed by post-training in duplex mode (described in detail in Section 6) on conversation data. Moshi's training strategy produces model that can generate consistent speech across long (several-minute monologues and can engage in multi-turn conversations (Défossez et al. 2024).
Speech+text LMs vary widely in their decoding methods and approaches to constructing speech and lext iaputs/outputss For example, SpiRit-LM (Nguyen et al.) 2025) uses interleaved sequences of text and discrete speech tokens as both inputs and outputs, focusing solely on pre-training without doing any task-specifie post- training. Compared to pure speech LMs, SpiRiT-LM has improved semantic understanding, as measured by the StoryCloze test (see Section 7.1 for more details). A precursor of this idea, SUTLM (Chou et al. 2023), modeled interleaved speech and text units of various sizes, but did not include a waveform synthesizer and was evaluated mainly on spoken language understanding tasks, with the main goal of enabling cross- modality transfer. Zeng et al. (2025) scale up the idea of interleaved speech-text training by using a speech synthesizer to generate massive datasets of interleaved speech-text sequences. As alternatives to time- aligned or interleaved speech and text sequences, SpeechGPT (Zhang al. 2023a) outputs a sequence of text followed by corresponding speech, while Mini-Omni (Xie & Wu 2024a b) and LLaMA-Omni (Fang) et al. 2025a) generate text and speech using separate output channels with a delay pattern in the speech token prediction (sce Section 3.3.2). All three of these models (SpeechGPT, Mini-Omni, and LLaMA-Omnil unlike Moshi, focus on a traditional turn-taking structure and therefore have limited capability to model spontaneous turn-taking behavior (see Section G for more on conversation modeling).
5.3 Speech-aware text LM
Models in this calegory combine text LMs with speech encoders, and usually lake both speech and text as inputs and generate responses in text form. The text input t can include instructions asking the model to perform tasks related to the speech input XTP, possibly including tasks for which the model is not specifically trained. This type of SLM is typically initialized with a text LLM (which has often already been post-trained with instruction tuning or RLHF) so as to inherit its linguistic knowledge and instruction-
19

---

Published in Transactions on Machine Learning Research (10/2025)
following capabilities. After combining the text LLM with a speech encoder and modality adapter, it is common to apply training methods that encourage better alignment. between the speech and text modalities (Section 4.1.3 or to start post-training with an ASR task (Section 4.3) to help extract content information from speech inputs. The following post-training typically uses speech instruction-tuna datascts with diverse instructions (Section 4.2.2
WavPrompt (Gao et al. 2022) represents the first example of this line of research. WavPrompt combines wav2vee 2.0 (Baevski et al., 2020) SSL speech encoder with GPT-2 text LM and is trained for speech classification tasks, keeping the LM backbone frozen and updating only the speech encoder. Although WavPrompt can improve over +tRt model basclines on certain tasks, it was not evaluated on unseen tasks. Since then, many models in this category have been proposed.
SALMONN (Tang et al., 2024) is another early and impactful approach. The model is built on a pre-trained Vicuna LM backbone (Zheng ct al., 2023). During post-training, the model is first trained on ASR and audio captioning and later on more diverse set of speech understanding tasks. SALMONN's post-training keeps
most of the parameters frozen- only the speech modality adapter and LoRA (Hu et al. 2022) adapters for the LM are learned but the model is still able to generalize to unseen tasks with various natural language prompts.
Several other speech-aware text LMs were developed nearly simultaneously with SALMONN, facilitated by the release of open-source LLMs like LLaMA (Touvron et al. 2023a) and Vicuna (Zheng et al. 2023 often
post-trained and evaluated on different tasks [Gong et al. 2023b] Wang et al. 2023b Chu et al. 2023 Chen
et al.) 2024d]. For example, LTUUAS (Gong et al. 2023b) combines spoken content understanding with paralinguistic analysis tasks (c.g., emotion recognition). SLM (Wang et al. 2023b) generalizes to unseen tasks specified by natural language prompts even while the LM backbone and speech encoder are both frozen, and only the speech adapter is optimized during instruction- based training, but focuses only on content- heavy tasks. Qwen-Audio (Chu et al. 2023) shares similar post-training tasks to those of SALMONN, but includes learning of the speech encoder and LM backbone (sequence model) weights. During post-training, Qwen-Audio starts with a more diverse lask set instead of the single ASR task, including both content and paralinguistics, followed by a chat-based supervied fine-tuning stage that improves the model's robustness to variation in input prompts. The follow-up work, Qwen2-Audio (Chu et al., 2024) also applics direct preference optimization [Rfailov el al. 2023 after supervised fine-tuning. On the other hand, the more recent DeSTA (Lu et al, 2024) and DeSTA 2 (Lu et al.) 2025a propose a single descriptive speech-text alignment post-training task which requires the model to both recognize the spoken content and describe paralinguistic attributes.
Table 0 summarizes and compares a representative set of these models' training and evaluation choices. One key consideration is the tradeoff between preserving knowledge acquired through text pre-training and learning new speech tasks, especially those requiring non-textual information such speech emotion. An important design choice, therefore, is which of the pre-trained parameters to freeze or update, and this issue has not yet been thoroughly explored.
In addition to the multiple post-training approaches, speech- text LMs also include some differences in architecture choices. For example, WavLLM (Hu et al. 2024) combines both SSL and supervised pre- trained speech encoders, and BESTOW (Chen et al.] (024c) experiments with a cequenco-to-scquene model in addition the common decoder-only LM architecture. While most models in this category are initialized with pre-trained text LLMs, UniverSLU (Arora et al. 2024 starts from Whisper model for conditional pro-training (see Scction 4.1.2p and is then fine-tuned for various understanding tasks, first by prepending new task specifiers and then by adding natural language instructions to the input.
6 Duplex speech dialogue
As discussed in the previous sections, most SLMs to date assume that dialogue with a user follows a turn-by- by- turn structure, where a user provides an input and the SLM generates a response. However, natural speech dialogue duplez: Both speakers can simultancously send and receive information. Compared to rext-basch interaction, duplex dialogue presents unique challenges. Figure 0 (a) shows an example of full-duplex
20

---

Published in Transactions on Machine Learning Research (10/2025)
toll
story
ok
stop
don't
lkk
the story
User:
(a)
Oce upon a time
in smail v Ulage
SLM:
(b)
Spoken II M
00
(c)
Spoken LLM
Chunk
SLM generated
speak
sten
SLM generated (silence)
User's speech
User's speech (silence)
(d)
Spoken LLM
speak
listen
Figure 6: Full-duplex speech conversation. (a) An example of full-duplex speech conversation between a and an SLM. (b) Dual-channel approach. (c) Time-multiplexing approach (with equal chunks). (d) Time finee-mltiplexing approach (where the SLM controls the switching between listening and speaking modes).
dialogue. The SLM must determine whether the user has finished their utterance before starting to speak. Backchannel signals (e.g., saying "mm-hmm," "I see," or "okay") and non-verbal vocalization (e.g., laughter) are often used to facilitate smoother conversation. Additionally, the SLMM may be interrupted while talking and must respond appropriately to maintain the flow of the dialogue. None of these characteristics exist in text-based dialogue. Over the years, researchers have explored various methods to enable duplex dialogue in speech systems (Masumura et al. 2018 Roddy et al.] 2018 Skantze 2017 Meena et al. 2014 Ekstedt & Rather than revicwing all prior work on spoken duplex dialoguc, we focus specifically on end-lo-end SLM approaches.
Skantze 2020 8
Dual channel One way achieve full-duplex dialogue is by using a dual-channel approach (Nguyen et al. 2023: Délossez et. al. 2024: Ma al.) 2025 Meng et al., 2024), shown in Figure i (b). The SLM has two input channels: the listening channel (the sequence with red blocks) which continuously receives input, and the speaking channel (the sequence with blue blocks) where the spoken output from the SLM is directed, allowing the model to track what has said. The model produces output al each step, including tokens representing speech or silence (blocks with a dotted outline). In this way, an SLM can listen to the user's words while generating output simultaneously. An early representative model for this method is the dialogue GSLM (dGSLM) (Nguyen al., 2023); Moshi has also implemented this strategy (Défossez)
8rThere an extensive amount of older related literature, including nule-based dialogue modeling; here, only machine learning based approaches.
21

---

Published in Transactions on Machine Learning Research (10/2025)
et al. 2024). One challenge when using the dual-channel approach is that, instead of a typical autoregressive model, specialized architecture is required. dGSLM (gguyen et. al. 2023) uses dual-tower transformer architecture, where two transformers handle the two channels, using cross-attention to exchange information. Other models (Dfoosscz et al. 2024 Ma et al.) 2025 Meng et al. 2024 modify the input structure of the transformer to accommodate the dual -calnhen input.
Time multiplexing Figure 6 (c) and (d) illustrate the time multiplexing approach. In this approach, the SLM has only one channel, so it must switch between listening and speaking modes. During the listening mode, the SLM takes input from the user (red blocks) without generating output. During the speaking mode, the SLM generales speech representations (blue blocks) and takes its own output as input in an autoregressive manner. The strength of this approach is that the sequence model can be a typical decoder- only autoregressive model, and can therefore be initialized with text LLM.
The core challenge is determining how to switch between listening and speaking modes. One approach, shown in Figure [
(c), is to alternate between processing a fixed-duration time slice from the user's input and then switching to the speaking mode (Zhang et al. 2024c). This approach has been adopted in Synchronous
LLM (Veluri et al. 2024). OnniFlatten (Zhang el al.] 2025), ), and SCoT Arora et al. 2025c
Another approach allows the model to determine when to switch modes (Wang et al.) 2024b Xu et al. 2024; Wang et al., 2025b), as shown in Figure 0 (d). In listening mode, while processing the user's input, at cach time step the model predicts whether to imitiate response. This decision is signaled by the prediction of special token (Ispeak), as shown in the figure). The model processes its generated output autoregressively until it produces the [listen] token, which triggers a switch back to listening mode. Users can continue providing input during speaking mode, dynamically influencing the model's responses. In particular, user imput influences the [listen] token generation, enabling users to interrupt the model's output when desired. This interaction is implemented by interlcaving the model's responses with user input, using some special tokens or embeddings to distinguish between the two sources
The architectural designs described above, whether dual-channel or time multiplexing, share a common goal: to enable natural user interaction. Evaluating how well model has achieved this goal can be challenging, since most benchmarks involve oflline rather than interactive evaluation. Section 73 below describes the community's efforts so far to benchmark interactivity in SLMs for dialogue.
7 Benchmarking and Evaluating SLMs
Like text language models, SLMs may possess a broad array of capabilities. In addition (and in contrast to text TMs), they also involve variety of design choices that differ significantly across models. These properties can make it difficult to compare SLMs, and also make it more important to assess them from multiple angles and at multiple training stages. Below, we outline commonly used evaluation methods and benchmarks, categorized into four primary groups: (1) likelihood- based evaluation metrics; (2) generative metries; (3) evaluation of interactivity; and (4) trustworthiness evaluations.
Traditional lasks (e.g., ASR, TTS, ST) are also frequently used to assess SLM performance. However, we exclude them from this survey, as they are well-established and extensively covered in the existing literature. For more comprehensive survey of benchmarks for SLMs, please refer to a recent overview paper focusing on SLM benchmarks (Yang et al. 2025b).
7.1 Likelihood- -aaeed Evaluation
Likelihood-based evaluation metrics DDunarr et al. 2021 are designed to assess the sequence modeling capa- bilities of SLMs, typically for the pre-training stage of pure speech LMs and joint speech+text LMs (Lakhotia et al. 2021 Hassid et al., 2023 Borsos et al. 2023a Défossez et al. 2024). Such evaluation metrics can be considered analogues to perplexity (PPT.) for text LMs. Since there no standard speech tokenization,
Theoretically, there is always some input, even when no one speaking. Here, we assume that voice activity detection (VAD) system is in place, so only the tokens representing actual user speech are sent the SLM during speaking mode.
22

---

Published in Transactions on Machine Learning Research (10/2025)
the speech "vocabulary" may change between pure SLMs. Therefore, PPL cannot be directly applied. Like perplexity, likelihood-based metrics are viewed not tasks, but rather as proxies indicating the ability of the base SLM to represent the speech distribution it is modeling. The assumption is that good performance on these metrics will, after fine-tuning, be reflected in improved downstream task performance, and there are some indications from analyses of likelihood-based metrics that this assumption holds (Dunbar et al. 2022).
We emphasize that speech tokenization plays a significant role in shaping sequence modeling capabilities and therefore cannot be disregarded. For instance, utilizing tokens generated by self-supervised learning model such as HuBERT significantly outperforms the quantization of log mel spectrograms (Lakhotia et al. 2021). Likewise, employing speech tokenizers with higher compression rales (i.e., sampled at lower frequency in the latent space) yields better performance in speech modeling (Borsos et al.) 2023a Hassid et al.) 2023). While the likelihood-based metrics in this subsection focuses on the sequence model, in principle the two components should be evaluated together.
In likelihood- based evaluations, the pre-trained SLM is typically provided with two imput sequences: one representing natural speech sequence (positive example) and the other unnatural sequence relative to a specific property (negative example). We then calculate the percentage of instances where the SLM assigns a higher likelihood to the positive example than to the negative one. This type of evaluation has also been
popular in the NLP community until very recently (Zellers et al. 2019 Mostafazadeh et al. 2016 Touvron
iet al. 2023b Gemma Team et al. 2024 LCM team et al.) 2024).. Due to siguificant model improvements,
this type of NLP benchmark has saturated and significantly more complex ones have emerged (Jimenez
et al. 2024 Srivastava et al. 2023
Metrics in this category are designed to evaluate model's capacity to represent various speech characteris- tics. For example, sWUGGY (Dunbar et al. 2021) evaluates the lexical capabilities of models by presenting them with pairs of utterances, one consisting of a real word and the other a similar non-word (e.g.. 'brick' vs. "blick'), where nonwords were obtained from the text Wuggy task (Keuleers & Brysbaert 2010). Sim- ilarly, sBLIMP (Dunbar et. al. 2021) (which was adapted from the text-based BLiMP (Warstadt et al., 2020)) evaluates the ability of the model represent syntactic properties. The model is presented with a matched pair of grammatically correct and incorrect sentences. Hassid et al. (2023) expanded this method to evaluate semantic underslanding of spoken text. They generated two spoken adaptations of the StoryCloze text paragraph story completion task (Mostafazadeh et al.] 2016). In the first adaptation, they adhered to the original StoryCloze setup, while in the second, they randomly sampled negative examples, resulting in a simplified version primarily requiring an understanding of the general topic of the paragraph. ProsAudit (de Seyssel et al. 2023) constructs negative examples by inserting pauses unnatural positions in teranncss to evaluate SLMs sensitivity to speech prosody. Lastly, [Mimon al. (2025b) proposed the SALMon evalua- tion suite, which, unlike the previously mentioned benchmarks, focuses a variety of non-lexical acoustic and prosodic elements (e.g., speaker identity, speaker sentiment, background noise, and room acoustics) two levels of complexity: One mcasures the consistency across time of given acoustic element, whereas the other measures alignment between the acoustic details and the semantic, spoken content.
These likelihood braeee metrics evaluate complementary aspects of SLM performance, and performance across benchmarks not necessarily correlated. For example, even though some SLMs manage to model sentiment to some extent (as shown by the SALMon sentiment consistency measure) and have good text modeling capabilities (as demonstrated by their performance on sWUGGY), they achieve near-random performance the SALMon alignment measure (Maimon et al.) 2025b). This shows that jointly reasoning over both text and acoustic content is still challenging for SLMs. Similarly, some recent studies on newly proposed SLMs (Tseng et al. 2025: Chou et al. 2025) show that the models achieving the best performance on content modeling (e.g., on StoryCloze or sWUGGY) are not always the best at handling acoustic details (as indicated by their performance on SALMon).
7.2 Generative Metrics
Unlike likelihood-based metrics, which assess a model's ability to assign higher likelihoods to in-domain samples than out-of-domain ones, generative metrics focus on evaluating the model's ability to produce
23

---

Published in Transactions on Machine Learning Research (10/2025)
meaningful continuations responses based a given prompt or instruction. Such evaluation methods are suitable for all SLM types.
Intrinsic quality metrics For SLMs that generate spoken output (e.g,, speech+ LMs), the quality of the generated response can be evaluated along several axes:
Speech quality, using subjective lests or objective metrics such as MOSNet (Lo et al. 2019 While
this is primarily a speech synthesis metric that emphasizes signal quality, it can also be influenced to some degree by the intelligibility of the generated speech (Lakhotia al., 2021)
Speaker, acoustic, and/or sentiment consistency using human judges or pre-trained classi- fiers (Nguyen et al. 2025). This group of metries measures how consistent the generated speech is with the input speech prompt, in terms of different acoustic properties.
Quality of the spoken content, evaluated by either comparing to a target response or transcribing the speech and using a text LLM as a judge to score the content (Arora al., 2025c Zhang et al. 2025).
Like perplexity- based metrics, this set of evaluation measures is not tied to any specific downstream task. Instead, their primary purpose is to evaluate the generative quality of SLMs along various dimensions.
For pure SLMs and +tet SLMs, a common evaluation metric is the generative perplemily, in which we transcribe the generated speech and measure its perplexity (PPL) using a pre-trained text LLM (Lakhotia) et al. 2021). However, known problem with this metric is that text L.MS tend assign high probability
to repeated content (Holtzman et al. 2020). To mitigate this issue, Lakhotia et al. (2021) | propose the
VERT metric, which balances between quality and diversity of the generated response. VERT uses a linear
combination of the text PPL and the ALU metric of within-sentence diversity (Lakhotia et al. 2021
base- evaluation For joint speech-text LMs and speech-aware text LMs, a common approach is to evaluate the model's ability to answer questions and follow instructions. Nachmani et al. (2024)
proposed two spoken question answering benchmarks: LLaMA--uestions derived from a text LLM, and a
synthesized version of the WebQuestions (Berant et al. 2013) textual benchmark. Défossez et al. (2024) built upon this approach and developed a synthesized version of TriviaQA (Joshi et al.) 2017). To evaluate SLM responses on these question-answering benchmarks, the generated speech is transcribed and the accuracy is measured against the target answer. Chen et al. (2024c) proposed VoiceBench, an extended question- answering dataset consisting of (1) open-ended QA; (2) interence-assed QA; (3) multiple-choice QA; and (4) instruction
evaluations. VoiceBench also evaluates the model's ability to handle background noise and speaker variability. Fang et al. (2025a) proposed to evaluate the instruction-following capabilities of SLMSs using I.L.Max-u-jully methods. In this approach, an external text LLM rates the quality of the responses considering both content and style.
Recently, there have been several efforts to scale up evaluations of speech-aware text LMs to a broader range of tasks and instruction data. In these evaluations, each task consists of text instructions, speech utter- ances, and text labels. Huang et al. (0244) introduced the Dynamic-SUPERB evaluation suite, designed as a dynamic benchmark allowing for community contributions of tasks, analogously to BIG-bench (Srivastava) let al. 2023 for text LMs. The initial phase of Dynamic-SUPERJ focused on classification tasks related to content, speaker characteristics, semantics, degradation, paralinguistics, and non-speech audio information.
Phase 2 of Dynamic-SUPERB (Huang et al. 2025) significantly expanded the task set to include regression and sequence generation tasks, making it the largest benchmark for speech and audio evaluation. [Yang] et al. 2024bb proposed AlR-Bench, which includes both classification and open-ended chat-style questions. Both Dynamic-SUPERB Phase-2 and AIR-Bench use LLM-as-a-jugee evaluation. AudioBench (Wang et al. 2025a) includes eight speech and audio tasks based on 26 datasels, focusing on speech understanding (e.g.. ASR, spoken QA, speech instruction), audio scene comprehension (e.g., audio captioning, audio scene ques- tion answering), and voice analysis (o.g., accent recognition, gender rocognition, emotion recognition). [aakshi] et al. (225) introduced MMAU, benchmark for audio understanding and reasoning using natural language. MMAU includes both information extraction tasks (e.g., "Which word appears first?") and reasoning tasks
24

---

Published in Transactions on Machine Learning Research (10/2025)
(e.g.. "What are the roles of the first and second speakers in the conversation?"), formulated multiple- choice questions. When we compare model performance on MMAU and VoiceBench, an interesting trend emerges: Their results are negatively correlated Tt VoiceBencch emphasizes QA ability, which is primarily content-oriented, whereas MMAU covers a broader range of tasks and perspectives. This divergence under- scores the necessity of evaluating SLMs from multiple angles, rather than relying on a single benchmark, to obtain a comprchensive understanding of their capabilities.
Conversation-basee evaluation While the benchmarks above cover wide variety of tasks, some recent ones focus specifically whether SLMs can perceive acoustic details in conversations and respond appro- priately. For example, StyleTalk (Lin et al. 2024a) assesses SLMs' ability to generate text responses that align with specific speaking styles, for example in terms of intonation and emotion. The evaluation com- pares the generated output with a target response using standard text gencration metrics (BLEU, ROUGE, and others). Similarly, the E-chat.200 dataset (Xue et al.] 2024) also assesses a model's ability to generate responses that align with the speaker's emotion. The dataset was synthetically created using an expressive TTS, while the text-basod questions and responses were generated by an external LLM. SD-Eval {Ao et al., 2024) and VoxDialogue (Cheng al.) 2025) go beyond emotion alignment to inchude alignment with other speaker and style features (e.g., accent, age, prosody, timbre, volume) as well as environmental context. The recently introduced StressTeest benchmark (Yosha et al.) 2025) evaluates the ability of speech-aware SLMs to understand differences in meaning conveyed by sentence stress, finding that cxisting models tend to do poorly on this task. VoxRole (Wu et al. 2025d) is a benchmark designed for speech-based role-p aale-playing con- versational agents, which evaluates how well they maintain their personas. S2S-Arena (Jiang et al. 2025) evaluates SLMs' ability to respond appropriately to spoken instructions that involve paralinguistics. Unlike the benchmarks above that rely on automatic evaluation, S2S-Arena uses human judgments to evaluate SLMs.
Comparisons of cascaded vs. end-to-end SLMs When evaluating SLMs, researchers often include cascaded baselines for comparison with end-to-end models. For speech-aware text LMs, many studies contrast end-to-end approaches with ASR - LLM cascaded systems. Across benchmarks, cascaded systems typically excel at semantic tasks such as spoken language understanding but underperform on speaker-related and
paralinguistic tasks (Huang et al. 2024 2025 Sakshi et al. 2025 Liu ct al. 2025 Yan et al. 2025 Yang et al.,
2025a). While cascaded systems remain strong baselines, jointly optimizing the speech encoder with the LLM can yield additional benefits by better aligning speech representations with the backbone sequence model. 2025c) and paralinguistic understanding (Lu
These benefits include improved instruction following (Lu et al.
ct al. 2025b). However, such joint training also risks catastrophic forgetting when integrating speech encoders and pre-trained sequence models (Lu et al. 2025c Hsiao et al. 2025
When considering SLMs with full-duplex behaviors, trade-offs between cascaded and end-to-end models become even clearer. Cascaded approaches (typically ASR - LLM + TTS) provide modular flexibility
and strong semantic modeling (Wang et al.) 2025b Zeghidour et al., 2025). but they often incur higher latency and lose fine-grained conversational behavior modeling (Lin et al. 2005b Arora et al. 2025b; Chang eet al. 2025). End-to-end models can mit iggae some of these limitations by enabling smoother turn-laking and backchanneling behaviors (Arora et al. 2025b Lin et al. 2025b). However, challenges remain regarding their overall modeling capacity, as they often rely on fine-tuning a text-based LLM to directly model both the listening and speaking channels (Défossez et al. 2024). as discussed in Section 6 which is inherently difficult. Therefore, developing robust end-to-end architectures for full-duplex SLMs remains a major challenge {Tin) et al. 2025b). For broader discussion of the trade-offs between cascaded and end-to-end SLMs, readers may refer to the recent survey on SLM evaluation by Yang et al. (002bb).
Thiss comparison was conducted Oct 3rd, 2025. The results of VoiceBench are from http:r///gihhub..co//ht/teeeee VoiceBench] while the results of MMAU from https://sakahhi113 1...iithhb............. github.io/mmau www.ppagg//eaaddbbbrreeeeeeeeeee using MMAU-v05.11 We evaluated models on both benchmarks. The metrics are the overall score for VoiceBench and the average test score for MMAU. We found that the correlation coefficient between these scores negative.
25

---

Published in Transactions on Machine Learning Research (10/2025)
7.3 Evaluating Interactivity
When evaluating SLMs for spoken dialogue (described in Section 60 quality is in large part determined by inteructivity measures; that is, how well a model handles the dynamies of real conversation, such engaging in smooth turn-taking, providing timely backchannels, and responding graccfully to user interruptions (barge- in). Evaluating these complex dynamics has progressed from early statistical analyses to modern automated benchmarks and subjective user studies.
Pioncering work in the dialogue SLM area, such as dGSLM Nguyenn et al., 2023), introduced methods for evaluating dialogue by identifying basic conversational patterns and comparing model-generated dialogue with ground-truth human conversations. dGSLM's evaluation was based several key components of turn-taking derived from voice-activity based statistics (e.g., inter-pausal units, pauses, gaps, and overlaps) computed from Human-human interactions.
One of the most basic components of SLMs' interactive capabilities is latency, typically evaluated by the delay from when the user stops speaking to when the model begins its spoken response, often termed firsl-packel latency (Yan et al. 2025 Xu al.) 2025) or response latency (Wang et al., 2024b Lin al. 2025b). The average human conversational response latency has been estimated approximately 200 ms (Stivers et al, 2009). Latency reported for dialogue models depends greatly on hyperparameters, hardware specifications, and network conditions. Table 3 lists selection of models and their reported latencies from the recent literature.
Recent efforts to evaluate interactivity have moved beyond basic conversational statistics or latency toward automated, scenario-driven benchmarks, which provide more user-oriented measures of interactive perfor- mance. For example, Full-Duplex-Bente Bench (Lin et al., 20025 ap assesses full-duplex behaviors including pause management, backchannel appropriateness, smooth turn transitions, and interruption handling. The more recent Game-Time Benchmark (Chang et al testing their time awareness, adherence to tempo, and ability to engage in simultaneous speech with users.
2025) evaluates the temporal dynamics of SLMs, specifically
While automated benchmarks like Full-Duplex-Beree Bench and the Game-Time Benchmark provide useful quanti- tative and qualitative evaluation, user studies remain essential for capturing interaction quality. For instance, "Talking Turns" (Arora et al., (025b) compared the practical behaviors of VAD-based cascaded system against an end-to-end model, Moshi Défossez et al. (2024). This study found that while the VAD-basod system was better at yielding its turn when interrupted, the two systems shared some weaknesses such rarely producing backchannels and often failing to take a turn when user expected them to. A combination of automated benchmarks and human-in- inttan--o- evaluations will likely continue to be necessary to advance toward truly natural conversation flow.
7.4 Evaluating Trustworthiness
The benchmarks discussed thus far focus on the performance of SLMs. Next, we consider evaluations that measure their trustworthiness. Since SLMs generate word sequences (whether directly as text or within the spoken output), all trustworthiness considerations related to text-based LLMs also apply to SLMSs. However, in addition to this content information, non-content information in the generated speech creates additional aspects of trustworthiness that need to be considered.
7.4.1 Hallucination
Text-based LLMs often suffer from hallucinations, or outputs that are inconsistent with the given context or other knowledge considered to be fact. In addition to such content hallucinations, SLMs may also generate audio hallucinations. For example, consider an audio clip without dog barking. If you ask an SLM to describe it, it would most likely not mention anything related to a dog barking. However, if you directly ask the model, "Is there a dog barking?" it might answer, " "Yes," even though it can provide accurate audio captions when explicitly prompted to do so (Kaa & Lee 2025).
Kuan & Lee (2025) study whether SLMs can accurately perform three types of tasks without hallucination: (1) identify the presence of an object (e.g., "Ts there dog barking?"), (2) determine the temporal order of
26

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Latency comparison of speech-text models, measured milliseconds (ms). Official claims are denoted separately from benchmarkod results reported by URO-Bench Yan et 2025 and LLaMA- Omni2
[Fang al. 2025b).
Model
Size (# params)
Latency (ms)
Device
Moshi
7B
200
Official claim
GPT-4o
320
Official claim
Omeen--Omn-30B-A3BB
34B
234
Official claim
Latency reported from URO-Bench
Freeze- Ommi
7B
3675
A40
Mini-Omat
0.5B
399
A40
Mini- Omni2
0.5B
402
A40
GLM-4-Voice
9B
3244
A40
Latency reported from LLaMA-Omin
SpeechGPT
13B
5588
L40
GLM-4-Voice
9B
1563
I.40
LLaMA-Onen
8B
347
L40
LLaMA- -LaMA-Ommii200.5
0.5B
543
L40
LLMMA-Ommi2-14B
14B
663
I.40
sound events (e.g., "Is the dog barking before someone laughs?"), and (3) recognize the source of sound woman?"). The main finding is that all of the evaluated SLMs
event (e.g., "Who is laughing, hallucinate more than a simple cascade model that combines audio captioning with texi LLMs. This issue may arise because, while SLMs can generate accurate audio captions, they struggle to understand specific questions Kuan et al. (2024). The authors also proposed method that mitigates hallucination by prompting the model to produce output in multiple steps (similarly to chain-of- thouughht prompting (Wei et al. 2022b) ) by first describing the audio and then responding to the instruction based on that description.
7.4.2 Toxicity
Prevention of harmful, offensive, or inappropriate output a crucial issue for text LLMs, and the same challenge also applies to SLMs. Both Meta's SpiRit-LM (Nguyen et al. 2025) and OpenAl's GPT-4o voice mode (OpenAl 2024) evaluate the toxicity of their models' responses. These evaluations typically involve using specific prompts to elicit speech responses from the SLMs and assessing the toxicity of the text transcriptions of those responses. These evaluations therefore consider only verbal toxicity, i.e. toxicity within the word sequence. The evaluation of non-verbal loxicity (e.g., toxic sarcasi) has yet to be widely studied.
7.4.3 Bias
Similarly to toxicity, all bias measures used for text LLMs can also be applied to SLMSs For SLMs that produce spoken output, text-based methods can still be used to analyze their transcriptions (Lin et all 2024ch. In addition, because speech conveys information beyond the textual content, there may be factors outside the text that indicate bias. Lin et al. (2024b) investigate such biases by providing SLM with input utterances that share identical content but differ in speaker characteristics (such as gender and age) to assess whether these traits influence its responses. The results suggest that the lested SLMs exhibit minimal bias. However, this may be because current models lack sufficient sophistication to recognize differences in speaker characteristics, and therefore do not respond differently based on them. OpenAl reported in a blog post (OpenAl, 2024) on their efforts ensure consistency across user voices by post-training GPT-4o with a diverse set of voices and bias evaluation, which found significant variation in model behavior across
27

---

Published in Transactions on Machine Learning Research (10/2025)
While such findings are encouraging, regular and thorough bias benchmarking is needed to track the evolution of SLM bias as this field matures.
voices.
7.4.4 Deepfakes
Current SLMs can mimic a variety of voices to level indistinguishable from human speech (Chen et al, 2025b).
Unfortunately, malicious actors may exploit this technology, leading to misuse and security concerns, such as deepfake attacks (for example, creating fake news using the voice of a public figure). To address this issue, (Wu et al. 2024d: Du et al., 2025) introduced CodeeFake, a deepfake audio datasct, and found that detection models trained on CodecFake can counter fake audio generated by existing SLMs.
8 Challenges and future work
This survey has covered some of the key milestoncs that have been achieved in SLM research: the first pure speech language models that could generale convincing-sounding stretches of English speech (GSLM (Lakho- tia et al., 2021)); the first generation of models that could perform a variety of tasks with reasonable accuracy given natural language instructions (Gao ct al.
2022 Gong ct al. 2023b Wang et al. 2023b); models that can converse in "full-duplex" mode (Délossez et al.
2024); ); and benchmarks lailored for evaluating the modeling
and instruction-folle -onlllooioo-lloooinng capabilities of SLMSs
Dunbar et al. 2021 Huang al. 2025).
While rescarch in this area has produced many SLMs and exciting outcomes, it is safe to say that not yet close to the goal of truly universal speech processing systems. This section highlights several categories of challenges and open questions, which suggest directions for future research.
Model architecture The optimal representation of speech within SLMs remains unclear. Speech repre- sentations SLMss include both discrete and continuous varieties, derived from a wide range of encoders. This design choice can also influence other architectural choices in an SLM, for example depending on the information rate of the encoder and whether it encodes more phonetic or other types of information.
Another open question is determining the best method to combine speech and text, which applies to all as- pects of SLM modeling and training. We have described various choices of modality adapters and approaches for interleaving speech and text. These have not been thoroughly compared, so the effect of each modeling choice stiill unclear.
A final architectural challenge is that most current SLMs are large and slow, making them impractical for real-time and on-device settings. To some extent this is because various compression algorithms (e.g.. LLi et al. 2021 Peng et al. 2023a Ding et al. 2024)) and alternative architectures (e.g., Park et al. (2025) have not been widely applied to SLMs. However, there is also an inherent efficiency challenge that arises when combining multiple pre-trained components, sometimes with different. architect and frame rales.
Training There is a lack of public high-quality training data, particularly for instruction tuning and chat- based training. Zeng al. (2025) showed that scaling synthetie data generation enhances the performance of pure speech LMs; similar research efforts could be directed toward instruction tuning and chsed spoken data. Additionally, SLMs are trained on diverse datasets (including some proprictary datascts), making it difficult to analyze whether performance differences are caused by model design choices or training data. Thorough ablation studies focusing on the various design choices (Section 3) and training strategies (Section 4 are also essential and still lacking. Finally, scaling studies for SLMs are needed in order to better understand how SLMs scale with model and data size. Such studics would hopefully produce new scaling laws and help to speed up the development cycle. Cuervo & Marxer (2024) conducted the first such investigation for pure SLMs, and more recently Maimon et al. (2025a) empirically analyzed the optimization process for pure speech LMs, showing how one train high-quality pure SLM in 24 hours using single GPU. It remains uncertain whether scaling findings apply also to speech+text LMs or speech-aware text LMs.
Thus far, different SLMs have typically been evaluated on different datasets and tasks. Recent efforts to collect large numbers of tasks and standardize benchmarking (Huang et al.) 2024 Yang et. al.
Evaluation
28

---

Published in Transactions on Machine Learning Research (10/2025)
2024b; Huang et al. 2025) are promising, but as of this writing they not yet widely adopted for evaluating new models. Existing benchmarks also do not cover the full range of spoken language tasks. The largest current benchmarking effort, Dynamic-SUPERB Phase-2 (Huang et al., 2025), includes 180 tasks, which is similar to the size of the BIG-Bench suite of text LLM tasks (Srivastava et al., 2023). However, the range of spoken language tasks arguably much larger than that of text lasks, since speech tasks include the vast majority of text tasks (the ones that don't 't' relate explicitly to the textual form, such as transliteration) and in addition include variety of speech-specific tasks related speaker properties, accents, or prosody-specifie content. In addition, there is lack of standardized benchmarking for speech generation tasks. Finally, there is a need for more benchmarking of latency and turn-taking behavior in conversational SL.Ms (although, as mentioned in earlier sections, some benchmarks have begun to address these dimensions Lin el al. 2025b)).
Yan et al. 2025
Open research Very few SLMs are fully cww-sooorcc-incuudi code, model chockpoints, and training data which makes a comprehensive comparison between approaches virtually impossible. There has been progress in this direction, with some models having at least publicly available weights (sce Figure D and some support in open-source toolkits (Tian al. 20025).. Many of the items on the wish list above, such as controlled comparisons between multiple design decisions, will be infeasible to accomplish until more fully open models are released.
Improving inclusiveness and safety SLM research has, thus far, understandably focused on high- resource languages and settings. As SLMs become more performant and enter commercial products in daily use, it will be critical to make them accessible to as broad a range of users as possible, including a variety of languages, dialects, and speech-related medical conditions. Research in this area will likely follow the are of text LLM research, but again, there are speech-specific challenges that arise from speaker and style variation. Similarly, the area of safety and trustworthiness has only begun to be explored, and will require speech-specific solutions to speech-specific challenges like deepfakes and speaker type-related bias.
References
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. MusicLM: Generating music from text. arXiv preprint ar arXiv:2301.11995 2023.
Robin Algayres, Adel Nabli, Benoît Sagot, and Emmanuel Dupoux. Speech sequence embeddings using nearest neighbors contrastive learning. In Proc. Interspeech, 2022.
Robin Algayres, Yossi Adi, Tu Nguyen, Jade Copet, Gabriel Synnaeve, Benoît Sagot, and Emmanuel Dupoux. Generative spoken language model based continuous word-sized audio tokens. In Proc. EMNLP, 2023.
Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, Shengpeng Ji, Yabin Li, Zerui Li, Heng Lu, Haoneng Luo, Xiang Lv, Bin Ma, Ziyang Ma, Chongjia Ni, Changhe Song, Jiaqi Shi, Xian Shi, Hao Wang, Wen Wang, Yuxuan Wang, Zhangyu Xiao, Zhijie Yan, Yexin Yang, Bin Zhang, Qinglin Zhang, Shiliang Zhang, Nan Zhao, and Siqi Zheng, FunAudioLLM: Voice understanding and generation foundation models for natural interaction between humans and LLMs. arxiv preprint ar Xiv:22447 04051, 2024.
Junyi Ao, Yuancheng Wang, Xiaohai Tian, Dekum Chen, Jun Zhang, Lu Lu, Yuxuan Wang, Haizhon Li, and Zhizheng Wu. SD-Eval: A benchmark datasel for spoken dialogue understanding beyond words. In Proc. NewrIPS, 2024.
Siddhant Arora, Hayato Futami, Jee-weon Jung, Yifan Peng, Roshan Sharma, Yosuke Kashiwagi, Emiru Tsunoo, Karen Liveseu, and Shinji Watanabe. UniverSLU: Universal spoken language understanding for diverse tasks with natural language instructions. In Proc. NAACL, 2024.
Siddhant Arora, Haidar Khan, Kai Sun, Xin Luna Dong, Sajal Choudhary, Seungwhan Moon, Xinyuan Zhang, Adithya Sagar, Surya Teja Appini, Kaushik Patnaik, Sanat Sharma, Shinji Watanabe, Anuj Kumar,
29

---

Published in Transactions on Machine Learning Research (10/2025)
Ahmed Aly, Yue Liu, Florian Metze, and Zhaojiang Lin. Stream RAG: Instant and accurate spoken dialogue systems with streaming tool usage. or Xiv preprint or arXiv:2510.0204. 2025a.
Siddhant Arora, Zhiyun Lu, Chung-Chent Cheng Chiu, Ruoming Pang, and Shinji Watanabe. Talking turns: Benchmarking audio foundation models on turn-taking dynamics. In Proc. ICLR, 2025b.
Siddhant Arora, Jinchuan Tian, Hayato Futami, Jiatong Shi, Yosuke Kashiwagi, Emiru Tsunoo, and Shinji Watanabe. Chain-of-thought reasoning in streaming full-duplex end-to-end spoken dialogue systems. arXiv preprint Xiv:25000. 020066, 2025c.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Proc. NewrIPS, 2020.
Jinze Bai, Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chenggiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijic Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. ar preprint urx::22309. 160099 2023.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang, Semantic parsing on Freebase from question- answer pairs. In Proc. EMNLP, 2013.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pictquin, Matt Sharifi, Do- minik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. AudioLM: A language modeling approach to audio generation. IEEE/ACM Trans. Audio, Speech, Lang. Process., 31: 2523 2523-2533, 2023a.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. SoundStorm: Efficient parallel audio generation. arXiv preprint Xiv: Xtt:22355009666, 2023b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proc. NewrIPS, 2020.
William Chan, Navdeep Jaitly, Quoe V. Le, and Oriol Vinyals. Listen, attend and spell. In Proc. ICASSP, 2016.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. MaskGIT: Masked generative image transformer. In Proc. CVPR, 2022.
Kai-Wei Chang, Haibin Wu, Yu-Kai Wang, Yuan-Kuei Wu, Hua Shen, Wei-Cheng Tseng, Iu-Thing Kang, Shang- Wen Li, and Hung-yi Lee. SpeechPrompt: Prompting speech language models for speech processing tasks. IEEE/ACM. Trans. Audio, Speech, Long. Process., 32:3730 3744, 2024.
Kai- Wei Chang, En-Pei Hu, Chun-Yi Kuan, Wenze Ren, Wei-Chih Chen, Guan-Ting Lin, Yu Tsao, Shao- Hua Sun, Hung-yi Lee, and James Glass, Came-Time: Evaluating temporal dynamics in spoken language models. Xixiv preprint or arXi::2509.260 2025.
Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-LLM: Bootstrapping advanced large language models by treating multi-modalitics as foreign languages. arx preprint ar Xix:22300.00 04160, 2023.
30

---

Published in Transactions on Machine Learning Research (10/2025)
Liang Chen, Zekun Wang, Shuhuai Ren, Lei Li, Haozhe Zhao, Yunshui Li, Zefan Cai, Hongcheng Guo, Lei Zhang, Yizhe Xiong, Yichi Zhang, Ruoyu Wu, Qingxiu Dong, Ge Zhang, Jian Yang, Lingwei Meng, Shujie Hu, Yulong Chen, Junyang Lin, Shuai Bai, Andreas Vlachos, Xu Tan, Minjia Zhang, Wen Xiao, Aaron Yoe, Tianyu Liu, and Baobao Chang. Next token prediction towards multimodal intelligence: A comprehensive survey. airic preprint or arXiv:2412 12. 18619, 2024a.
Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, Yabin Li, Xiang Lv, Jiaqing Liu, Haoneng Luo, Bin Ma, Chongjia Ni, Xian Shi, Jialong Tang, Hui Wang, Hao Wang, Wen Wang, Yuxuan Wang, Yunlan Xu, Fan Yu, Zhijic Yan, Yexin Yang, Baosong Yang, Xian Yang, Guanrou Yang, Tianyu Zhao, Qinglin Zhang, Shiliang Zhang, Nan Zhao, Pei Zhang, Chong Zhang, and Jinren Zhou, MinMo: multimodal large language model for seamless voice interaction. arXiv preprint ar Xiv:2501.0628. 2025a.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujic Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yannin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. WavLM: Large-scale self-supervised pre-training for full stack speech processing. IEFE J. Sel. Top. Signal Process., 16(6):1505-1516, 2022.
Sanyuan Chen, Chengyi Wang. Yu Wu, Ziqiang Zhang, Long Zhou, Shujic Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers. IEFE Trans. Audio, Speech, Lang. Process., 33:705-715 2025b.
Wenxi Chen, Ziyang Ma, Ruiqi Yan, Yuzhe Liang, Xiquan Li, Ruiyang Xu, Zhikang Niu, Yanqiao Zhu, Yifan Yang, Zhanxun Liu, Kai Yu, Yuxuan Hu, Jinyu Li, Yan Lu, Shujie Liu, and Xie Chen. SLAM- Omni: Timbrl-controllable voice interaction system with single-stage training. In Findings of ACL, 2025c.
William Chen, Wangyou Zhang, Yifan Peng, Xinjian Li, Jinchuan Tian, Jiatong Shi, Xuankai Chang, Soumi Maiti, Karen Livescu, and Shinji Watanabe. Towards robust speech representation learning for thousands of languages. In Proc. EMNLP, 2024b.
Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. VoiceBench: Benchmarking LLM-based voice assistants, arXiv preprint arXiv:2410.11 17196, 2024c.
Yuan-Ping Chen, Ryan Price, and Srinivas Bangalore. Spoken language understanding without speech recognition. In Proc. ICASSP, 2018.
Zhehuai Chen, He Huang, Andrei Andrusenko, Oleksii Hrinchuk, Krishna C Puvvada, Jason Li, Subhankar Ghosh, Jagadeesh Balam, and Boris Ginsburg, SALM: Speech-augmenter language model with in-context learning for speech recognition and translation. In Proc. ICASSP, 2024d.
Zhehuai Chen, He Huang, Oleksii Hrinchuk, Krishna C. Puvvada, Nithin Rao Koluguri, Piotr Želasko, Jagadeesh Balam, and Boris Ginsburg. BESTOW: Efficient and streamable speech language model with the best of two worlds in GPT and T5. In Proc. SLT, 2024e.
Xize Cheng, Ruofan Hu, Xiaoda Yang, Jingyu Lu, Dongjie Fu, zehan wang, Shengpeng Ji, Rongjie Huang, Boyang Zhang, Tao Jin, and Zhou Zhao. VoxDialogue: Can spoken dialogue systems understand informa- tion beyond words? In Proc. ICLR, 2025.
Ju-Chich Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conncau, Alexci Baevski, and Michael Auli. Toward joint language modeling for speech units and text. In Findings of EMNLP, 2023.
Ju-Chieh Chou, Jiawei Zhou, and Karen Livescu. Flow-SIM: Joint learning of linguistic and acoustic information for spoken language modeling. In Proc. ASRU, 2025.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijic Yan, Chang Zhou, and Jingren Zhou. Qwen-Audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint ar (xXix:3311.07911 2023.
31

---

Published in Transactions on Machine Learning Research (10/2025)
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-Audio technical report. ar preprinl arXiv:2407.10759, 2024.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Proc. NewrIPS, 2023.
Santiago Cuervo and Ricard Marxer. Scaling properties of speech language models. In Proc. EMNLP, 2024.
Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Yiwen Guo, and Irwin King. Recent advances in speech language models: A survey, In Proc. ACL, 2025.
Nilaksh Das, Saket Dingliwal, Srikanth Ronanki, Rohit Paturi, David Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, Xilai Li, Karel Mundnich, Monica Sunkara, Sun- dararajan Srinivasan, Kyu J. Han, and Katrin Kirchhoff. SpeechVerse: A large-scale generalizable audio language model. Xrxiv preprint ar Xi0:2405.08295, 2024.
Maureen de Scyssel, Marvin Lavechin, Hadrien Titeux, Arthur Thomas, Gwendal Virlet, Andrea Santos Re- villa, Guillaume Wisniewski, Bogdan Ludusan, and Emmanuel Dupoux. ProsAudit, prosodic benchmark for self-super speech models. In Proc. Interspeech, 2023.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Trans. MLR, 2023.
Alexandre Défossc2, Laurent Mazaré, Manu Orsini, Amélic Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: A speech-text foundation model for real-time dialogue. arXiv preprint arXiv:24 10.00077 2024.
Keqi Deng, Guangzhi Sun, and Phil Woodland, Wav2Prompt: End-to-end speech prompt learning and task-based fine-tuning for text-based LLMs. In Proc. NAACL, 2025.
Jacob Devlin, Ming-Weert Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirec- tional transformers for language understanding. In Proc. NAACL, 2019.
Shaojin Ding, David Qiu, David Rim, Yanzhang He, Oleg Rybakov, Bo Li, Rohit Prabhavalkar, Weiran Wang, Tara N. Sainath, Zhonglin Han, Jian Li, Amir Yazdanbakhsh,, and Shivani Agrawal. USM-Lite: Quantization and sparsity aware fine-tuning for speech recognition with universal speech models. In Proc. ICASSP, 2024.
Jiawei Du, Xuanjum Chen, Haibin Wu, Lin Zhang, I-Ming Lin, I-Hsiang Chiu, Wenze Ren, Yuan Tseng, Yu Tsao, Jyh-Shing Roger Jang, and Hung-yi Lee. GadeeFaake-Ominn A large-scale codec-based deepfake speech dataset. arXiv preprint 25t::2501008255 2025.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijie Yan. Cosy Voice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXio preprint 2xXi:22407005000 2024.
Ewan Dunbar, Mathieu Bernard, Nicolas Hamilakis, Tu Anh Nguyen, Maureen De Seyssel, Patricia Rozé, Morgane Rivière, Eugene Kharitonov, and Emmanuel Dupoux. The sero resource speech challenge 2021: Spoken language modelling. In Proc. Interspeech, 2021.
Ewan Dunbar, Nicolas Hamilakis, and Emmanuel Dupoux. Self-supervised language learning from raw audio: Lessons from the zero resource speech challenge. IEEE J. Sel. Top. Signal Process., 16(6):1211 1226, 2022.
Erik Ekstedt and Gabriel Skantze. TurGGPT: A transformer-based language model for predicting turn-taking in spoken dialog. In Findings of ACL, 2020.
Ruchao Fan, Bo Ren, Yuxuan Hu, Rui Zhao, Shujie Liu, and Jinyu Li. AlignFormer: Modality matching achieve better zero-shot instruction--ollowing speech-LLM. IEFE J. Sel. Top. Signal Process., 2025.
32

---

Published in Transactions on Machine Learning Research (10/2025)
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. LLaMA-Omni.
Seamless speech interaction with large language models. In Proc. ICLR, 2025a.
Qingkai Fang, Yan Zhou, Shoutao Guo, Shaolei Zhang, and Yang Feng- L.aMA-Omni LLM-based real-time spoken chatbot with autoregressive streaming speech synthesis. In Proc. ACI, 2025b.
Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. Laamo Towards general-purpose speech abilities for LLMs. In Proc. NAACL, 2024.
Philip Gage. A new algorithm for data compression. C Users J., 12(2):23 38, 1994.
Marco Gaido, Mauro Cettolo, Matteo Negri, and Marco Turchi. CTC-based compression for direct speech translation. In Proc. EACL, 2021.
Heting Gao, Junrui Ni, Kaizhi Qian, Yang Zhang, Shiyu Chang, and Mark Hasegawa-Johnson. WavPrompt: Towards few-shot spoken language understanding with frozen language models. In Proc. Interspeech, 2022.
Gemini Team et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint 2rXiv:2403.05530 2024.
Gemma Team et al. Gemma: Open models basod Gemini rescarch and technology. ar preprint arXiv:2403.0822 2024.
Yuan Gong, Sameer Khurana, Leonid Karlinsky, and James Glass. Whisper-AT: Noisc-robust automatic speech recognizers are also strong general audio event taggers. In Proc. Interspeech, 2023a.
Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In Proc. ASRU, 2023b.
Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James R. Glass. Listen, think, and understand. In Proc. ICLR, 2024.
Aaron Grattafiori et al. The Llama 3 herd of models. arXiv preprint arXim:2407.2178, 2024.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal clas- sification: Labelling unsegmented sequence data with recurrent neural networks. In Proc. ICML, 2006.
R. Gray. Vector quantization. IEEE ASSP Magazine, 1(2):4 29, 1984.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In Proc. ICLR, 2022.
Yiwei Guo, Zhihan Li, Hankun Wang, Bohan Li, Chongtian Shao, Hanglei Zhang, Chenpeng Du, Xie Chen, Shujie Liu, and Kai Yu. Recent advances in discrete speech tokens: A review. arXiv preprint arXiv:2502.064 2025.
Parisa Haghani, Arun Narayanan, Michiel Bacchiani, Galen Chuang, Neeraj Gaur, Pedro Moreno, Rohit Prabhavalkar, Zhongdi Qu, and Austin Waters. From audio to semanties: Approaches to end-to-end spoken language understanding. In Proc. SLT, 2018.
Nadav Hur, Or Tal, and Yossi Adi arXiv:2505.1444 14470, 2025.
PAST: Phonetic-acoustic speech tokenizer. arXio preprint
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Défossez, Gabriel Synnacve, Emmanuel Dupoux, Roy Schwartz, and Yossi Adi. Textually pretrained speech language models. In Proc. NewrIPS, 2023.
William Held, Yanzhe Zhang, Minzhi Li, Weiyan Shi, Michael J Ryan, and Diyi Yang. Distilling an end-to- end voice assistant without instruction training data. In Proc. ACT, 2025.
33

---

Published in Transactions on Machine Learning Research (10/2025)
François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Esteve. TED- LIUM 3: Twice as much data and corpus repartition for experiments on speaker adaptation. In Speech and Computer, 2018.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degener- ation. In Proc. ICLR, 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly, trrameter-efficinn transfer learning for NLP. In Proc. 1CML, 2019.
Chi- Yuan Hsiao, Ke-Han Lu, Kei-Wei Chang, Chih-Kai Yang, Wei-Chih Chen, and Hung yi Lee. Analyzing mitigation strategies for catastrophic forgetting in end-to-end training of spoken language models. In Proc. Interspeech, 2025.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Ab- delrahman Mohamed. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio, Speech, Lang. Process., 29:3451 10:3451-3400, 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In Proc. ICLR, 2022.
Ke Hu, Zhehuai Chen, Chao-Han Huck Yang. Piotr Zelasko, Oleksii Hrinchuk, Vitaly Lavrukhin, Jagadeesh Balam, and Boris Ginsburg. Chain-of-thought prompting for speech translation. In Proc. ICASSP, 2025.
Shujic Hu, Long Zhou, Shujic Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, and Furu Wei. WavLLM: Towards robust and adaptive speech large language model. In Findings of EMNLP, 2024.
Chien-Ye Yu Huang, Ke-Han Lu, Shih-Heng Wang, Yhi-Yuan Hsiao, Chun- Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wet Chang, Jiatong Shi, Yifan Peng, Roshan Sharma, Shinji Watanabe, Bhiksha Ramakr- ishnan, Shady Shehata, and Hung-yi Lee. Dynamic-SUPERB: Towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech. In Proc. ICASSP, 2024.
Chien-Yu Huang et al. Dynamic-SUPERB Phaso-2: A collaboratively expanding benchmark for measuring the capabilitics of spoken language models with 180 tasks. In Proc. ICLR, 2025.
Xucdong Huang, Alex Accro, Hsiao Wuen Hon, and Raj Reddy. Spoken language processing: A guide to theory, algorithm, and system development. Prentice Hall PTR, 2001.
Shengpeng Ji, Yifu Chen, Minghui Fang, Jialong Zuo, Jingyu Lu, Hanting Wang, Ziyue Jiang, Long Zhou, Shujic Liu, Xize Cheng, Xiaoda Yang, Zehan Wang, Qian Yang, Jian Li, Yidi Jiang, Jingzhen He, Yunfei Chu, Jin Xu, and Zhou Zhao. WayChat: A survey of spoken dialogue models. arXiv preprint arXiv:2411 1.0.222111.1771 18577, 2024.
Feng Jiang, Zhiyu Lin, Fan Bu, Yuhao Du, Benyou Wang, and Haizhou Li. S2S-Arena, evaluat- ing speech2speech protocols on instruction following with paralinguistic information. arXiv preprint arXiv:2503.05000 2025.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In Proc. ICLR, 2024.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervisod challenge dataset for reading comprehension. In Proc. ACL, 2017.
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. In Proc. TCLR, 2023.
34

---

Published in Transactions on Machine Learning Research (10/2025)
Emmanuel Keuleers and Mare Brysbacrt. Wuggy: A multilingual pseudoword generator. Behavior research methods, 42:627-655 633, 2010.
Eugene KhariLonov, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Paden Tomasello, Ann Lee, Ali Elkahky, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. rextless-lib: A library for textless spoken language processing. In Proc. NAACL, 2022a.
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. Text-free prosody- aware generalive spoken language modeling. In Proc. ACL, 2022b.
Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaäl Marinier, Sertan Girgin, Olivier Pictquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Trans. ACI, 11:1703-1716 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proc. NAS, 114(13):3521 3526, 2017.
Yuma Koizumi, Kohei Yatabe, Heiga Zen, and Michiel Bacchiani. WaveFit: An iterative and non- autoregressive neural vocoder based on fixed-point ileration. In Proc. SLT, 2023.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. In Proc. NewrIPS, 2020.
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgan Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. Textless speech emotion conversion using discrete & decomposed representations. In Proc. EMNLP, 2022.
Chun- Yi Kuan and Hung-yi Lee. Can large audio-language models truly hear? Tackling hallucinations with multi-task assessment and stepwise audio reasoning. In Proc. ICASSP, 2025.
Chun-Yi Yi Kuan, Wei-Ping Huang, and Hung-yi Lee. Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models. Proc. Interspeech, 2024.
Cheng-I Jeff Lai, Yang Zhang, Alexander H. Liu, Shiyu Chang, Yi-Lun Liao, Yung-Sung Chuang, Kaizhi Qian, Sameer Khurana, David D. Cox, and Jim Glass. PARP: Prune, adjust and re-prune for self- supervised speech recognition. In Proc. NeurIPS, 2021.
Cheng-I Jeff Lai, Zhiyun Lu, Liangliang Cao, and Ruoming Pang. Instruction speech recognition. arXiv preprinl ar X30: 09843, 2023.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. On generative spoken language modeling from raw audio. Trans. ACL, 9:1336-1354, 2021.
Siddique Latif, Moazzam Shoukat, Fahad Shamshad, Muhammad Usama, Heriberto Cuayáhuitl, and Björn W. Schuller. Sparks of large audio models: survey and outlook. arXiv preprint orXiv:2308.122 12792, 2023.
LCM team, Loie Barrault, Amul-isbroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillanme Conairon, Marta R. Costa-jussà, David Dale, Hady Elsahar, Kevin Heffernan, João Maria Janeiro, Tuan Tran, Christophe Ropers, Eduardo Sánchez, Robin San Ro- man, Alexandre Mourachko, Safiyyah Saleem, and Holger Schwenk. Large concept models: Language modeling in sentence representation space. arXiv preprint ar Xrx::22412.0888 2024.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. Voicebox: Text-guided multilingual universal speech generation at scale. In Proc. NeurIPS, 2024.
35

---

Published in Transactions on Machine Learning Research (10/2025)
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. Direct speech-to-speech translation with discrete units. In Proc. ACL, 2022a.
Ann Lee, Hongvu Gong, Paul- Ambroisee Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. Textless speech-to-specch translation on real data. In Proc. NAACL, 2022b.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proc. CVPR, 2022c.
Junnan Lí, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proc. 1CML, 2023.
Guan-Ting Lin, Cheng- Henng-Hannoe Chiang, and Hung-yi Lee. Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. In Proc. ACI, 2024a.
Guan-Ting Lin, Shih-Yun Shan Kuan, Qirui Wang, Jiachen Lian, Tingle Li, Shinji Watanabe, and Hung yi Lee. Full-Duplex- Bench v1.5: Evaluating overlap handling for full-duplex speech models. arXiv preprint arXiv:2507.231 2025a.
Guan-Ting Lin, Jiachen Lian, Tingle Li, Qirui Wang, Gopala Anumanchppali,, Alexander H. Liu, and Hung yi Lee. Mull-Duple--ennde A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilitics. In Proc. ASRU, 2025b.
Guan-Ting Lin, Prashanth Gurunath Shivakumar, Aditya Gourav, Yile Gu, Ankur Gandhe, Hung-yi Lee, and Ivan Bulyko. Align-SLM: Textless spoken language models with reinforcement learning from Al feedback. In Proc. ACL, 2025c.
Yi-Cheng Lin, Wci-Chih Chen, and Hung-yi Loc. Spoken Stereosct: On evaluating social bias toward speaker in speech large language models. In Proc. SLT, 2024b.
Yi-Cheng Lin, Tzu-Quan Lin, Chih-Kai Yang, Ke-Han Lu, Wei-Chih Chen, Chun-Yi Kuan, and Hung-yi Lee. Listen and speak fairly: A study semantic gender bias in speech integrated large language models. In Proc. SLT, 2024c.
Heyang Liu, Yuhao Wang, Ziyang Cheng, Ronghua Wu, Qunshan Gu, Yanfeng Wang, and Yu Wang. Vo- calBench: Benchmarking the vocal conversational abilities for speech interaction models. arXiv preprint arXiv:2505.1572) 2025.
Penglei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Suro., 5599):195:1 195:35, 2023.
Chen-Chou Lo, Szu- Wei Fu, Wen-Chin Huang, Xin Wang, Junichi Yamagishi, Yu Tsao, and Hsin-Min Wang. MOSNet: Deep learning-bass objective asscssment for voice conversion. In Proc. Interspeech, 2019.
Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, He Huang, Boris Ginsburg, Yu-Chiang Frank Wang, and Hung-yi Lee. DeSTA: Enhancing speech language models through descriptive speech-text alignment. In Proc. Interspecch, 2024.
Ko-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang. Jagadeesh Balam, Boris Ginsburg, Yu- Chiang Frank Wang, and Hung-yi Lee. DeSTA2: Developing instruction-following speech language model without speech instruction-tuning data. In Proc. ICASSP, 2025a.
Ko-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Sung-Feng Huang, Chih-Kai Yang, Cheo-En Yu, Chun-Wei Chen, Wei-Chih Chen, Chien yu Huang, Yi-Cheng Lin, Yu-Xiang Lin, Chi-An Fu, Chun-Y' Yi Kuan, Wenze Ren, Xuanjun Chen, Wei-Ping Huang, En-Pei Hu, Tzu-Quan Lin, Yuan-Kuee Wu, Kuan-Po Huang, Hsiao- Ying Huang, Huang-Cheng Chou, Kei-Wei Chang, Cheng- Han Chiang, Boris Ginsburg, Yu- Chiang Frank Wang, and Hung yi Lee. DeSTA2.5-Andior Toward general-purpose large andio language model with self-generated modsl alignment. arXiv preprint Xrxi::2007. 02768 2025b.
36

---

Published in Transactions on Machine Learning Research (10/2025)
Ke-Han Lu, Chun-Yi Yi Kuan, and Hung yi Lee. Speech-IFEval: Evaluating instruction-folloving and quanti- fying catastrophic lorgetting in speech-aware language models. In Proc. Interspeech, 2025c.
Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, and Xie Chen, Language model can listen while speaking. In Proc. AAAT, 2025.
Gallil Maimon and Yossi Adi. Speaking style conversion the waveform domain using discrete self-supervised units. In Findings of EMNLP, 2023.
Gallil Maimon, Avishai Elmakies, and Yossi Adi. Slamming: Training a speech language model on one GPU in day. In Findings of ACL, 2025a.
Gallil Maimon, Amit Roth, and Yossi Adi. SALMon: A suite for acoustic language model evaluation. In Proc. ICASSP, 2025b.
Soumi Maiti, Yifan Peng, Shukjae Choi, Jec-weon Jung, Xuankai Chang, and Shinji Watanabe. VoxtLM: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks. In Proc. TCASSP, 2024.
Ryo Masumura, Tomohiro Tanaka, Atsushi Ando, Ryo Ishii, Ryuichiro Higashinaka, and Yushi Aono. Neural dialogue context. online end-of-turn detection. In Proc. STGdial Meeting Disc. Diol, 2018.
Raveesh Meena, Gabriel Skantze, and Joakim Gustalson. Data-driven models for liming leedback responses map task dialogue system. Computer Speech, & Language, 28(4):900 ()):933-922, 2014.
Ziqiao Meng, Qichao Wang, Wenqian Cui, Yifei Zhang, Bingzhe Wu, Irwin King, Liang Chen, and Peilin Zhao. Parrot: Autoregressive spoken dialogue language modeling with decoder-only transformers. In NewrIPS Workshop Al-Driven Speech, Music, and Sound Generation, 2024.
Microsoft et al. Phi-4-Mini technical report: Compact yet powerful multimodal language models via mixture- of-LoRAs. arXie preprinl arXiv:2503.01745 2025.
Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D. Havtorn, Joakim Edin, Christian Igel, Katrin Kirchholl, Shang- Wen Li, Karen Livescu, Lars Maaloe, Tara N. Sainath, and Shinji Watanabe. Self-supervised speech representation learning: A review. IEEE J. Sel. Top. Signal Process., 16(6):1179- 1210, 2022.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proc. NAACL, 2016.
Poonch Mousavi, Luca Della Libera, Jarod Duret, Artem Ploujnikov, Cem Subakan, and Mirco Ravanelli. audio and speech benchmark. ar preprind or arXin:2406. 1e200066.1224,, 14294, 2024a.
Pooneh Mousavi, Jarod Duret, Salah Zaiem, Luca Della Libera, Artem Ploujnikov, Cem Subakan, and Mirco Ravanelli. How should we extract discrete audio tokens from self-supervised models? In Proc. Interspeech, 2024b.
Pooneh Mousavi, Gallil Maimon, Adel Moumen, Darius Petermann, Jiatong Shi, Haibin Wu, Haici Yang, Anastasia Kuznetsova, Artem Ploujnikov, Ricard Marxer, Bhuvana Ramabhadran, Benjamin Elizalde, Loren Lugosch, Jinyu Li, Cem Subakan, Phil Woodland, Minje Kim, Hung yi Lee, Shinji Watanabe, Yossi Adi, and Mirco Ravanelli. Discrete audio tokens: More than survey! Trans. MLR, 2025.
Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mari- ooryad, Ehud Rivlin, RJ Skerry-Ryan and Michelle Tadmor Ramanovich. Spoken question answering and speech continuation using spectrogram-powered LLM. In Proc. ICLR, 2024.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomascllo, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, and Emmanuel Dupoux. Generative spoken dialogue language modeling. Trans. ACI, 11:250 266, 2023.
37

---

Published in Transactions on Machine Learning Research (10/2025)
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Ttai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, and Emmanuel Dupoux. SpiRit-LM: Interleaved spoken and written language model. Trans. ACL, 52, 2025.
OpenAl. Gpt-4o system card, 2024. URL
OpenAl al. GPT-4 technical report. arXiv preprinl orXiv:2303308774 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proc. NeurIPS, 2022.
Jing Pan, Jian Wu, Yashesh Gaur, Sunit Sivasankaran, Zhuo Chen, Shujie Liu, and Jinyu Li. COSMIC: Data efficient instruction-tuning for speech in-context learning. In Proc. Interspeech, 2024.
Se Jin Park, Julian Salazar, Aren Jansen, Keisuke Kinoshita, Yong Man Ro, and R.J Ryan.. Long-form speech generation with spoken language models. In Proc. ICML, 2025.
Ankita Pasad, Bowen Shi, and Karen Livescu. Comparative layer-wise analysis of self-supervised speech models. In Proc. ICASSP, 2023.
Jing Peng, Yucheng Wang, Yu Xi, Xu Li, Xizhuo Zhang, and Kai Yu. A survey on speech large language models. arXiv preprint arXiv:2410. 18908, 2024a.
Yifan Peng, Yui Sudo, Shakeel Muhammad, and Shinji Watanabe. DPHuBERT: Joint distillation and pruning of self-supervised speech models. In Proc. Interspeech, 2023a.
Yifan Peng, Jinchuan Tian, Brian Yan, Dan Berrebbi, Xuankai Chang, Xinjian Li, Jiatong Shi, Siddhant Arora, William Chen, Roshan Sharma, Wangyou Zhang, Yui Sudo, Muhammad Shakeel, Jee weon Jung, Soumi Maiti, and Shinji Watanabe. Reproducing Whisper-style training using an open-source toolkit and publicly available data. In Proc. ASRU, 2023b.
Yifan Peng, Yui Sudo, Muhammad Shakeel, and Shinji Watanabe. OWSM-CTC: An open encoder-only speech foundation model for speech recognition, translation, and language identification. In Proc. ACL, 2024b.
Yifan Peng, Shakeel Muhammad, Yui Sudo, William Chen, Jinchuan Tian, Chyi-Jiunn Lin, and Shinji Watanabe. OWSM v4: Improving open Whisper-style speech models via data scaling and cleaning. In Proc. Interspeech, 2025a.
Yifan Peng, Krishna C Puvvada, Zhehuai Chen, Piotr Želasko, He Huang, Kunal Dhawan, Ke Hu, Shinji Watanabe, Jagadeesh Balam, and Boris Ginsburg. Toxeeeext:le Augmenting large language models with speech capabilities via single-stage joint speech-text supervised fine-tuning. In Proc. NAACL, 2025b.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised repre- sentations. In Proc. Interspeech, 2021.
Krishna. C. Puvvada, Piotr Želasko, He Huang, Oleksii Hrinchuk, Nithin Rao Koluguri, Kunal Dhawan, Somshubra Majumdar, Elena Rastorgueva, Zhehuai Chen, Vitaly Lavrukhin, Jagadeesh Balan, and Boris Ginsburg. Less is more: Accurate speech recognition & translation without web-scale data. In Proc. Interspeech, 2024.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Πlya Sutskever. Language models are unsupervised multitask learners, 2019. URL https://cdn. opwnai. wowpaaaeooomettteeeeaaegeg madguage-modelss language_ caapuuageemdeel...mm are unsupervised, multitaa task learners.
38

---

Published in Transactions on Machine Learning Research (10/2025)
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proc. ICML, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Proc. NewrIPS, 2023.
Matthew Roddy, Gabriel Skantze, and Naomi Hartel. Investigating speech features for continuous turn- taking prediction using LSTMs. In Proc. Interspeech, 2018.
S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. MMAU: massive multi-task audio understanding and reasoning benchmark. In Proc. ICLR, 2025.
Thomas Schatz, Vijayaditya Peddinti, Francis Bach, Aren Jansen, Hynck Hermansky, and Emmanuel Dupoux. Evaluating speech features with the minimal-pair ABX task: Analysis of the classical MFC/PLP pipeline. In Proc. Interspeech, 2013.
Feiyu Shen, Yiwei Guo, Chenpeng Du, Xie Chen, and Kai Yu. Acoustic BPE for speech generation with discrete tokens. In Proc. ICASSP, 2024.
Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skervv-yyan, Rif A. Saurous, Yannis Agiomvrgiannakis, and Yonghui Wu. Natural TTS synthesis by conditioning Wavenet on MEL spectrogram predictions. In Proc. ICASSP. 2018.
Jiatong Shi, Jinchuan Tian, Yihan Wu, Jee-Weon Weon Jung, Jia Qi Yip, Yoshiki Masuyama, William Chen, Yuning Wu, Yuxun Tang, Massa Baali, Dareen Alharthi, Dong Zhang, Ruifan Deng, Tejes Srivastava, Haibin Wu, Alexander H. Liu, Bhiksha Raj, Qin Jin, Ruihua Song, and Shinji Watanabe. ESPnet-Codec: Comprehensive training and evaluation of neural codecs for audio, music, and speech. In Proc. SLT, 2024.
Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu J. Han. SLUE: New benchmark tasks for spoken language understanding evaluation on natural speech. In Proc. ICASSP, 2022.
Suwon Shon, Kwangyoun Kim, Yi-Te Hsu, Prashant Sridhar, Shinji Watanabe, and Karen Livescu. Dis- creteSLU: A large language model with self-supervised discrete speech units for spoken language under- standing. In Proc. Interspeech, 2024.
Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, and Yemin Shi. LLaSM: Large language and speech model. arXiv preprint arXiv:2308. 15930, 2023.
Amitay Sicherman and Yossi Adi. Analysing discrete self supervised speech representation for spoken lan- guage modeling. In Proc. ICASSP, 2023.
Rouard Simon, Orsini Manu, Roebel Axel, Zeghidour Neil, and Défossez Alexandre. Continuous audio language models. arXiv preprint arxi:22009 Xix:22099900666, 2025.
Hubert Siuzdak, Florian Grötschla, and Luca A Lanzendörfer. SNAC: Multi-scale neural audio codec. arXiv preprint 1xX1::2410.1411,, 2024.
Gabriel Skantze. Towards general, continuous model of turn-taking in spoken dialogue using LSTM recurrent neural networks. In Proc. SIGdial Meeting Disc. and Diol., 2017.
Aarohi Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Trans. MLR, 2023.
Tanya Stivers, Nick J. Enfield, Penelope Brown, Christina Englert, Makoto Hayashi, Trine Heinemann, Gertic Hoymann, Federico Rossano, Jan Peter De Ruiter, Kyung-Eun Yoon, Stephen C. Levinson, Paul Kay, and Krishna Y. Universals and cultural variation in turn-taking conversation. Proc. of the National Academy of Sciences, 106(26):00557-11555 2009.
39

---

Published in Transactions on Machine Learning Research (10/2025)
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In Proc. ICLR, 2024.
Jinchuan Tian, William Chen, Yifan Peng, Jiatong Shi, Siddhant Arora, Shikhar Bharadwaj, Takashi Mackaku, Yusuke Shinohara, Keita Goto, Xiang Yue, Huck Yang, and Shinji Watanabe. OpusLM: A family of open unified speech language models. In Proc. Interspeech, 2025a.
Jinchuan Tian, Jiatong Shi, William Chen, Siddhant Arora, Yoshiki Masuyama, Takashi Mackaku, Yi- han Wu, Junyi Peng, Shikhar Bharadwaj, Yiwen Zhao, Samuele Cornell, Yifan Peng, Xiang Yue, Chao- Han Huck Yang, Graham Neubig, and Shinji Watanabe. ESPnet-SpeechLM: An open speech language model toolkit. In Proc. NAACI, 2025b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint orXi::2307.09288 2023b.
Liang-Hsuan Tseng, Yi-Chang Chen, Kuan- Lee, Da-Shan Shiu, and Hung yi Lee. TASTE: Text-aligned speech tokenization and embedding for spoken language modeling. arXiv preprint orXiv22504.07055 2025.
Emiru Tsunoo, Hayato Futami, Yosuke Kashiwagi, Siddhant Arora, and Shinji Watanabe. Decoder-only architecture for streaming end-to-end speech recognition. In Proc. Interspeech, 2024.
Arnon Turetzky and Yossi Adi. LAST: Language model aware speech tokenization. arxiv preprinl arXiv:2409.0370 2024.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proc. NewrIPS, 2017.
Bandhav Vcluri, Benjamin N Peloquin, Bokai Yu, Hongyu Gong, and Shyamnath Gollakota. Beyond turn- based interfaces: Synchronous LLMs as full-duplex dialogue agents. In Proc. EMNLP, 2024.
Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy Chen. AudioBench: A universal benchmark for audio large language models. In Proc. NAACL, 2025a.
Chen Wang, Minpeng Liao, Zhonggiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong, and Jiajun Zhang. BLSP: Bootstrapping language-speech pre-training via behavior alignment of continuation writing. arXiv preprint arXiv:2309.00m 009166 2023a.
Chen Wang, Minpeng Liao, Zhongqiang Huang, Junhong Wu, Chengqing Zong, and Jiajun Zhang, BLSP- emo: Towards empathetic large speech-language models. In Proc. EMNLP, 2024a.
Mingqiu Wang, Wei Han, Izhak Shafran, Zelin Wu, Chung-Cheng Chiu, Yuan Cao, Yongqiang Wang, Nanxin Chen, Yu Zhang, Hagen Soltau, Paul Rubenstein, Lukas Zilka, Dian Yu, Zhong Meng, Golan Pundak, Nikhil Siddhartha, Johan Schalkwyk, and Yonghui Wu. SLM: Bridge the thin gap between speech and text foundation modols. In Proc. ASRU, 2023b.
Peng Wang, Songshuo Lu, Yaohua Tang, Sijie Yan, Wei Xia, and Yuanjun Xiong. A full-duplex speech dialogue scheme based on large language models. In Proc. NewrIPS, 2024b.
Xiong Wang, Yangze Li, Chaoyou Fu, Yike Zhang, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long MA. Peezze-Omni: A smart and low latency speech-to-speech dialogue model with frozen LLM. In Proc. ICLR, 2025b.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The benchmark of linguistic minimal pairs for English. Trans. ACL, 8:377-392, 2020.
40

---

Published in Transactions on Machine Learning Research (10/2025)
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In Proc. ICLR, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting clicits reasoning in large language modcls. In Proc. NeurIPS, 2022b.
WhisperSpeech. WhisperSpeech: An open source text-to-speech system built by inverting Whisper, 2024.
URL https:///gtthtte httppp///gttttb....//llaaaaaaaaa htt.th/...ccooololaaao////paepee
Anne Wu, Laurent Mazaré, Neil Zeghidour, and Alexandre Défossez. Aligning spoken dialogue models from user interactions. In Proc. ICML, 2025a.
Felix Wu, Kwangyoun Kim, Shinji Watanabe, Kyu J. Han, Ryan McDonald, Kilian Q. Weinberger, and Yoar Artzi. Wav2Seq: Pre-training speech-to-text encoder-decoder models using pseudo languages. In Proc. ICASSP, 2023a.
Haibin Wu, Xuanjun Chen, Yi-Cheng Lin, Kai-wei Chang, Ho-Lam Chung, Alexander H. Liu, and Hung-yi Lee. Towards audio language modeling-an overview. arXiv preprinl Xrxi::2402. 13236, 2024a.
Haibin Wu, Xuanjun Chen, Yi-Cheng Lin, Kaiwei Chang, Jiawei Du, Ke-Han Lu, Alexander H. Liu, Ho- Lam Chung, Yuan-Kuei Wu, Dongchao Yang. Songxiang Liu, Yi-Chiao Wu, Xu Tan, James Glass, Shinji Watanabe, and Hung-yi Lee. Codec-Superb @ SLT 2024 A lightweight benchmark for neural audio codec models. In Proc. SLT, 2024b.
Haibin Wu, Ho-Lam Chung, Yi-Cheng Lin, Yuan-Kuei Wu, Xuanjun Chen, Yu-Chi Pai, Hsiu-Hsuan Wang, Kai-Wei Chang, Alexander H. Liu, and Hung-yi Lee. Codec-SUPERB: An in-depth analysis of sound codec models. In Findings of ACL, 2024c.
Haibin Wu, Yuan Tseng, and Hung-yi Lee. CodecFake: Enhancing anti-spoofing models against deepfake audios from codec-based speech synthesis systems. In Proc. Interspeech, 2024d.
Haibin Wu, Naoyuki Kanda, Sefik Emre Eskimez, and Jinyu Li. TS3-Codec: Transformer-based simple streaming single codec. In Proc. Interspeech, 2025b.
Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrii Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, and Yu Wu. On decoder-only architecture for speech-to- to-txt and large language model integration. In Proc. ASRU, 2023b.
Shu Wu, Chenxing Li, Wenfu Wang, Hao Zhang, Hualei Wang, Meng Yu, and Dong Yu. Moddo-Thinker: Guiding audio language model when and how to think via reinforcement learning. arXiv preprint arXiv:2508. 00X000 2025c.
Weihao Wu, Liang Cao, Xinyu Wu, Zhiwei Lin, Rui Niu, Jingbei Li, and Zhiyong Wu. VoxRole: A com- prehensive benchmark for evaluating speech-bases role-playing agents. arXiv preprint or arXiv:2509.0300 03940, 2025d.
Zhifei Xie and Changqiao Wu. Mini-Omni: Language models can hear, talk while thinking in streaming. arXiv preprint fff::2400.1672555 2024a.
Zhifei Xie and Changgiao Wu. Mini-Omni2: Towards open-source GPT-4o model with vision, speech and duplex. arXiv preprint arXiv:2,10.11199, 2024b.
Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. Audio-Rcasoner Improving reasoning capability in large audio language models. arXiv preprint urxi::2003. 023000,,..3...... 2025.
Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. BigCodec: Pushing the limits of low-
bitrate neural speech codec. aixiv preprint wxXim2449.05397 2024.
41

---

Published in Transactions on Machine Learning Research (10/2025)
Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo Zheng, Rui Men, Fan Zhou, Bowen Yu, Jianxin Yang, Le Yu, Jingren Zhou, and Junyang Lin. Qwen3-omui technical report. arXiv preprint ar 250: 1 17765, 2025.
Wang Xu, Shuo Wang, Weilin Zhao, Xu Han, Yukun Yan, Yudi Zhang, Zhe Tao, Zhiyuan Liu, and Wanxiang Che. Enabling real-time conversations with minimal training costs. arXiv preprint arXiv:2409. 1 11727, 2024.
Hongfei Xue, Yuhao Liang, Bingshen Mu, Shiliang Zhang, Mengzhe Chen, Qian Chen, and Lei Xic. E-chat: smotion-sensitive spoken dialogue system with large language models. In Proc. ISCSLP, 2024.
Ruiqi Yan, Xiquan Li, Wenxi Chen, Zhikang Niu, Chen Yang, Ziyang Ma, Kai Yu, and Xie Chen. URO-Bench: A comprehensive benchmark for end-to-end spoken dialogue models. arXiv preprint arXiv:2502.170 17810, 2025.
Chih-Kai Yang, Neo Ho, Yen-Ting Piao, and Hung-yi Lee. Sakura: On the multi-hop reasoning of large audio-language models based speech and audio information. In Proc. Interspeech, 2025a.
Chih-Kai Yang, Neo S. Ho, and Hung yi Lee. Towards holistic evaluation of large audio-language models: A comprehensive survey. In Proc. EMNLP, 2025b.
Dongehao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Haohan Guo, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Zhou Zhao, Xixin Wu, and Helen M. Meng. UniAudio: Towards universal audio generation with large language models. In Proc. ICML, 2024a.
Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. AIR-Bench: Benchmarking large ladiolanguaaee models via generative comprehension. In Proc. ACL, 2024b.
Iddo Yosha, Gallil Maimon, and Yossi Adi StressTest: Can YOUR speech LM handle the stress? arXiv preprint (cXi::255.22766 2025.
Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. MEGABYTE: Predicting million-byte sequences with multiscale transformers. In Proc. NewrIPS, 2023.
Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Connecting speech encoder and large language model for ASR. In Proc. ICASSP, 2024.
Ze Yuan, Yanqing Liu, Shujie Liu, and Sheng Zhao. Continuous speech tokens makes LLMs robust multi- modality learners. arXiv preprint arXiv:2412 04917, 2024.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. SoundStream: An end-to-end neural audio codec. IEEE Trans. Audio, Speech, Lang. Process., 30:495 507, 2022.
Neil Zeghidour, Eugene Kharitonov, Manu Orsini, Václav Volhejn, Gabriel de Marmiesse, Edouard Grave, Patrick Pérez, Laurent Mazaré, and Alexandre Défossez. Streaming sequence-to-seqqueee learning with delayed streams modeling. arXiv preprind arXiv:2509. (00:2500000733, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proc. ACL, 2019.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Voice Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint or arXiv:2,12.02611 2024.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong, and Jie Tang. Scaling speech-text pre-training with synthetic interleaved data. In Proc. ICLR, 2025.
42

---

Published in Transactions on Machine Learning Research (10/2025)
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, and Xipeng Qiu. A Unified multimodal LLM with discrete sequence modeling. In Proc. ACL, 2024.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. SpeechGPT: Empowering large language models with intrinsic moosl conversational abilitics. In Findings of EMNLP. 2023a.
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. MM-LLMs: Recent advances in multimodal large language models. In Findings of ACL, 2024a.
Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, Chao- Hong Tan, Zhihao Du, and ShiLiang Zhang. OmniFlatten: An end-to-end GPT model for seamless voice conversation. In Proc. ACL, 2025.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Aujali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open pre-trained transformer language models. arXiv preprint 2rx::2200.001 01068, 2022.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. SpeechTokenizer Tokenizer: Unified speech tokenizer for speech language models. In Proc. ICLR, 2024b.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Xu Han, Zihang Xu, Yuanwei Xu, Weilin Zhao, Maosong Sun, and Zhiyuan Liu. Beyond the turn-based game: Enabling real-time conversations with duplex models. In Proc. EMNLP, 2024c.
Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhchuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, Andrew Rosenberg, Rohit Prabhavalkar, Daniel S. Park, Parisa Haghani, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Ramab- hadran, Tara Sainath, Pedro Moreno, Chung- Chiu, Johan Schalkwyk, Françoise Beaufays, and Yonghui Wu. Google USM: Scaling automatic speech recognition beyond 100 languages. arXiv preprint 3rXiu:2303.01097 2023b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a- judge with MT-bench and Chatbot Arena. In NewrIPS Datasets and Benchmarks Track, 2023.
A Appendix
43

---

Published in Transactions on Machine Learning Research (10/2025)
Table 4: Selected spoken language models. P.: Phonetic token. A.: Acoustic token. C.: Continuous fea- ture. For components not explicitly cited in the table, please see the following references: SSE (Algayres et al. 2022), USM (Zhang et al. 2023b), SNAC (Siuzdak et al.} 2024]. SenseVoice (An et al. 2024), and Canary (Puvvada et al. (0244) for speech representation models, and Tacotron (Shen et al., 2018), Wave-
Fit. (Koizuni al., 2023), and Cosye Voice (Du el al. 2024) for speech decoders. Modalities other than lext
speech (e.g., vision) are omitted. 1; Pre-trained ASR encoder.
Model
Input Representation
Speech
Modality Adapter
Description
- Output Representation
Decoder
Pure Speech LM
GSLM Lakhotia al. 20/1
HuBERT (P.)
Tucutron
X
The first pare speech LM, and the first tempt using phonetic tokens hasic units for language modeling.
PGSLM Kharitonov et al.
2022b
HuBERT (P.) + FO + dura-
HIFY-GAN
X
Augments GSLM with discrete prosodie
dGSLM Nguyen
2023
HuBERT (P.) duration
HiFi-GN
X
The first speech LM model buman versation end-to-end (but not full-duplex model, as It does not take euvironmental audio inpot).
tGSLM Algayres
2023
SSE (P.) HuBERT (P.)
Tacotron
X
Reduces sequence length and improves memorry efficiency of GSLM by using word- sized basic units.
AudioLM Borsos et
2023
w2y-BERT (P.) + Sound- SoundScream Stream (A.)
X
The first work to model audio codee units with language model, and also intro- duces the eoare-tto generation pattern In Figure gl).
TWIST Hassid et
2023
HaBERT (P.)
HiFi-GAN X
Demommtarats the beuefits initializing speech LM with pre-triieed LM. Improves the efficiency of coarse-to- fine generation in AudioLM by using non- antorgresive model to generate the fine- grained tokons.
SoundStorn Borses
2023h
w2v- BERT (P.)
Sound- SoundStrran
X
Streman (A.)
ALi Lin et al
2025c
HsBERT(P.)
HiF-GAN
X
First work to aligu pure speech LM genera- tions with reinforeement learning feedback.
Speech++ Tent LM
Moshi Défreses et al.
2024
Text + Mimi (A.)
Minn
Vocabulary Expansion
The first full-duplex speech langrage model, proposes the decoding pattern in Figure Ge).
GoMe--Volee Zeng al.
2021
Text Whisper' (P.)
Vosyeoice Voice
Vecabulary Expansion
Demoustrates the capabilities of fieed-ratio text-speech token interleaved prediction, without forced slignment.
SpdRit-LM Nguyen
2025
Text HOBERT (P.)
HIFI-GAN
Vecabulary
Demoustrats the decoding pattern in Fig-
Expansion
ure Edd).
MinMo Chen et al.
2025a
Text + SenseVoloef (C.)
Cosy Voice
Transformer
A fullduplex model trained with a multi-
Text + SeuseVoice! (P.)
+ CNN
stage aligument strategy, including newly proposed method using text model repre-
sentations context for the speweh de- coder.
VoxLLM Maiti et al. 2024
Text + HoBERT (P.)
HAPi-GAN
Vocabulary
Integrates text and speach modalities into one deroder-only model.
Expannsoon
AnyCPT Zhan 2024
Text Specch Tokenizer (A.) Speech Tokenizer
Vocabulary
An any-to-any multimodal langnage model, with various encoders and decodars
Expansion
(similar to SprechGP [Zhang et al.
2023a
MiniOmni
Xie & Wn
2024a
Text + Whisper! (C. Text SNAC
MLP
Demonstrates the decoding pattern in Fig- ure un)
+ SNAC (A.)
cabulary Ex-
pamsion
LLaMA-Oumi
Fang et
2025a
Whisper (C. + Text + Hu- BERT (P.)
HiFi-GAN
MLP
Demonstrates the decoding pattern in Fig-
ure bb).
SLAM-Ommi
C Chen et al.
2025c
Text Whisper" (P)
Voice
Linear
Proposes approach that decodes one
text token and several andio tokens simul- tanonsly.
Spectron Nachmani
2024
Text Mel-Spectrogram (C.)
WaveFit Text + XEUS (P.)+ ESPact- ESPnet-Codec
MLP
An LM that directly models spectrograms.
OpusLM
Tlan et al.
Vocabulary
Family of open-souree SLMs with compet- itive performance to closed-source APIs on speech and text processing lasks.
Codeo(A.)
Expansion
44

---

Published in Transactions on Machine Learning Research (10/2025)
Table 5: Selected spoken language models (continued).
Model
Input Representation
Speech
Modality Adapter
Description
Output Representation
Decoder
Speech-Awor Awore Teat LM
WrePrompt
Gao
al.
Text + Wav2vee 2.0 (C.)
ONN
The first speech-aware text LM. Requires few- shot demoustration.
2022
Text
X-I.TM Chen et al.
2023
Text + Conformert (C.)
CIF + Transfonnrr
One of the carliest multimodal LLMs that in- clude speech input. Combines a frozen LLM (ChatGLM) with frozen speech (and image) encoders, and trains modality adlpttess only.
Text
LTU-AS
Gong et al.
2023b
Text + Whisper! (C.) - Text
TLTR
Proposes an approsch for universal audio per- ception by training on QA-style data.
Speech- LLaMA
(Wu et
Mel-Speetrogram(( C.) Text
CTC compressor
Applies CTC compression to reduce the
2023b
Transformer
quence length of the ended speech.
LLaSM Shu et sl.
2023
Text Whisper! (C. + Text
X
Linear
An SLM that can follow instructions in both spokeu and written forms.
BLSP
Vang et al.
2023a
Text Whisspr! (C.)
CNN
Demonstrates that comlent-centric SLM can be trained using the respouse text of text- based LLM, generated by conditioning on sperch transcrptiinn..
Wang al.
Text + USMI (C.) Text
Transformer
Demonstrates the capability SLMs gen- eralize to unseen tasks while training, only the modality udapter. Focuses on tasks related to sperch content.
Chen es al.
2024d
Text + Conformer Encoder!
Conformer
Demonstrates the in-context learning capahil-
(C.) Text
ities of SLMs. Focuses tasks related speech content.
SALMONN Tang es al.
Text + Whisper! (C) +
Window-level
Demonstrates generalization to unsoen tasks. Reports the issue of task overittiing
2024
BEATS (C. + Text
Former
COSMIC Pan
2021
Text Whisper! (C.) Text
Window-level
Instruction tuning on speech comprebension tests demonstrates unseen task generalization and in-context learning ability.
Former
Qwen-Audio
Chu
et
al.
Text Whisper! (C.) + Text
Pooling layer
Scales up training diverse tasks and pro- pusce multitask training framework.
SpperhVVerse
Das et.
2024
Text Whisper! (C.) Toxt
CNN
Stake up SLM training to diverse tasks and studies inference strategies to improve gener- alisation to unsee lasks.
WavLLM
Hu et
2024
Text + Whisper! (C.) +
CNN + Bostleneck
Combines two sperch encoders capture dif. Ferent aspects of the input spoeth. Proposes the use of discrete tokeus for speech. aware toxt LMS.
WavLM (C.) Text
adapter Linear
DSscreteLLU
Shou
al.
Text WavLM (P.) + Text
X
Embbedingg + CNN +
2024
Transformrr + Lineur
SC.P.. Emo
Wang
Text Whisper! (C.)
CNN
Extends HLSP improved emotion standing. Trains LLM responses condi- tioned on speech transcripts conotion tags. Proposes a descriptive speech-text alignment approach based on speech captioning.
2024g
DeSTA
La et al. 2024
Text Whisper! (C) - Text
Q- Q-Former
AudiooAudio
Chu
Text Whisper! (C.) Text
Pooling layer
Extends Qwen-Audio. Scales up the training dataset to 500k hours. Includes multi-task pre-training, supervised fine-tuning, and di- rect preference optimization.
2024
DeSTA2 La
2025a
Text Whissee! (C.) Text
Q-Former
Demonstrates that versalile SLM trained using self-generated speech captions from text LLM, conditioned on speech meta- data, as its learning targes.
DIVA (Held 2025
Text Whisspe! (C.) Text
Q P.Forner
Trains a versatile SLM using sell- genwraardd speech captions from text LLM, conditioned on speech transcrpptioss, via cros-modla to- lon alignment los and distillation from out- put embodding distance.
Voice Toxt Blender
Peng
Text Canary Encoder! (C.)
Conformer
Proposes joint speech-text supervised fine- tuning strategy to preserve the text LM's orig- inal performance.
Iet al
2025h
Text
Multimaall
Microsoft
Text +
Conformer
En-
MLP
multimodal model that integrates text, vi- sion, and specch/audio inputs using modality- specitic routers to emable interforencs- free multimodal inference.
c al.
2025
coder(C C.) + Text
Audio-
Xie al.
Pine-tuned 2-dioo Instro-
large-seale audio language model deep reasoning tasks audio.
2025
Dest A-dioo
Lat al.
Text Whisper! (C. Text X
Q-Former
Extends DeSTA2 with more training data. Finds that using captioning data generated by text model aerves as the backbome the SLM is critical.
2025b
Audio-Tre Thinker
Wil al.
Fine-tuned from Aweon2-Audio-nnstruue Qwen2. 5-Ommi
A reinforeement learning approach to enlancce the reasoning capabilities of SLMs.
20125e
45

---

Published in Transactions on Machine Learning Research (10/2025)
Table 6: A representative set of training and evaluation strategies for speech-aware text language models, along with key tasks, details on the training data, and key findings. Triining
Model
Trnining
Training Tasks
Evaluation ASR, ST, Alpara tusles
Findings
Strategy
SLM Wang al.
IT
ASR, ST, contextual biasing, open-ended QA
multilinguu ASR, ST datasets, Alpaca with TTS (93k lis)
Fine tineittning only
20236
lightweight adapter
sufficient many
SALMONN
Pre-training
ASR, AAC, 15 andio and speech tasks
8 audio and speoch task
ASR, ST and SLU datasets (4k hrs)
Performs complex reasoning
al. 2024
+ IT
tasks like audio- storytelling in zero-shot setting.
unseen ST and SLU
tasks
LTU-AS .S Gong e5 al. 2023b
3-stage teaining
Classffcation, general QA
Audio and speech
Open-ASQA dataset (9.6M QA pairs)
Single instruction following model on both speech and fensible.
tasks, open-ended QA
COSMIC Pan
IT
ASR, QA
ASR, SQA, ST, contextual binsing,
TED-LIUM (152 hours)
Shows few-shot in-context
e5 al.
2024
learning ability.
ASR (out of domain)
LTU
Goug et
4-stage tralning
Clasification: + cesc..
Claseificctio,
AQA (5M QA pairs)
Performs open-ended andio tacks.
2024
cloee-ended QA, general
captioning, open-ended QA
QA
SALM
Chen
IT
ASR, ST
ASR, ST, keyword boosting
LibriSpeech (96th),
Biases the model to predict
IWSLT 2023 (4.8k hrs)
keywords instruction.
Cbu
Pre-training
speech, audio and music tasks, andio dialogue
Speech, audio music tasks,
No details his
Performs chat-based training to learn conversational ability.
05
2023
IT
ASR, > 123k hrs in total)
qualitative examples
audio
analysis/ mualysiseditinng
DeSTA2
La et al.
IT
Audio captioning
Dynamic-
Mixture of aeveral
Training only on audio captioning can guneralize other taoks.
2025ps
SUPERB
Huang
datasets (155 hours)
2021
AIR
Sench
Yang
m1
2024b
DiscreteST.U
Shon
ASR, SQA, sentimens annlysis, NER
WER, 52ST
(Hernandez 2018 &
Discrete speech token input be competitive with contimuuu representations in both seen and unseen speech domains, zero-shot setting.
al.
2024
al.
SLUE
Shon et al. 2022
46


Reviews:
[{"review_id": "nsaNykSXXr", "reviewer": "Reviewer_fvYi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Comprehensive coverage of models, training strategies, and applications in the SLM field.\n- Clear taxonomy that effectively organizes the diverse approaches.\n- Forward-looking insights into open challenges and future research directions.\n- Well-structured presentation\n\n\nWeaknesses\n- Lack of critical comparison/analysis among models and strategies.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comprehensive survey of Spoken Language Models (SLMs), categorizing them into pure speech LMs, speech+text LMs, and speech-aware text LMs. It offers a unified architectural framework, reviews model components, training strategies, decoding methods, and highlights representative models. The work serves as a timely and structured reference for developing universal speech processing systems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, this survey is logically structured and tells a coherent story, allowing readers to understand the latest developments and key challenges in the field of spoken language models (SLMs). To further improve the paper, the following changes are suggested:\n-  Incorporate more critical analysis and comparison across models and strategies, highlighting their trade-offs, limitations, and suitable application scenarios.\n- Simplify or summarise technically dense sections to make the content more accessible to readers less familiar with the field.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qStBL0y0g6", "reviewer": "Reviewer_2ajH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is comprehensive in its coverage of the field.\n\n- Informative and high quality figures help the flow of the paper and make digesting key concepts easy.\n\n- The timeline figure is interesting and informative, the author’s should strongly consider moving this to the main body.\n\n- Survey is timely and likely to be of interest to the audience.\n\nWeaknesses:\n\n- The author's make brief note of the closed-source nature of the field. I think expanding on this topic would be beneficial as it is somewhat atypical for machine learning and poses significant barriers to entry and progress compared to open-source domains.\n\n- Discussion of benchmarks and evaluation methods is present, but discussion of relative performance of different SLMs/components is rather light.\n\n- (Minor) Technical depth of the survey is fairly light when discussing existing works. \n\n- (Minor) The paper could likely be a bit more concise.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a survey of Spoken Language Models (SLMs). The manuscript provides an overview of the field's history, provides a general framework for SLMs and covers overarching topics such as tokenization and construction. The paper then also discusses past and contemporary contributions to the field under new categorizations. Limitations and future directions are also discussed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see my weaknesses section for details. In addition, I think moving the timeline figure to the main body would be beneficial.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NUWqBSJWM5", "reviewer": "Reviewer_VM9G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The survey covers a wide range of SLMs, including pure speech LMs, speech+text LMs, and speech-aware text LMs with clear categorizations. Meanwhile, it also captures recent advancements and benchmarks, making it highly relevant to current research.\n\n- The general formulation of SLM is invaluable for understanding and unifying diverse architectures of the involved works, which is useful for building a standard of the field.\n\n- The section of Challenges and future directions is well written, which figures out current critical problems in diverse aspects.\n\n- The milestone timeline in the appendix is rich and comprehensive, which provides a clear development trajectory of the field for the readers.\n\n**Weaknesses**\n- The current version of the survey is a bit dense in text and not very easy to understand the scope at start, especially SLM is not a standard and widely known concept. \n\n- While benchmarks are listed, there’s limited comparison of model performance. It's hard to learn the quantitative differences between the methods.\n\n- Typos, e.g., \"expemplified\" in Figure 4 caption.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey provides a thorough unification and overview of the field of Spoken Language Models (SLMs), categorizing them by architecture, training, evaluation, and challenges. It first gives a functional definition of SLM, identifying the discussion scope of this paper, and then makes in-depth discussions and analysis of the involved works. Finally, the achievements and limitations of existing methods are summarized, and then some research directions are provided for future works.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I consider it a high-quality survey, with a comprehensive overview and valuable insights. It could be better with the following minor changes:\n1. If format permitted, the milestone timeline of Figure 6 in the appendix could be put at an earlier position in the main paper. It would make the survey more accessible, especially for newcomers.\n2. Include comparison and analysis of quantitative results for key benchmarks to demonstrate the performance gaps.\n3. Check and fix the typos.", "broader_impact_concerns": "No additional impact statement is required.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5OHGwusF4A", "reviewer": "Reviewer_DySR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As mentioned in \"Related Surveys\" in page 3, there are a few contemporary survey papers on similar topics, in particular, [Cui et al, 2024](https://arxiv.org/pdf/2410.03751). Cui et al, 2024 and this paper are quite complementary in their strengths and weaknesses.\n\n### Strengths (of this paper)\n\n-   Comprehensiveness: While both papers cover the core of SLMs quite well, this paper covers the following aspects which are not sufficiently discussed in Cui et al, 2024:\n    -   Modality adaptors\n    -   Hierarchical token generation\n    -   Interleaved speech/token generation\n\n### Weaknesses (of this paper)\n\n-   Presentation: While the information is all there in this paper, Cui et al, 2024 arguably does a better job in presenting the same information in a more efficiently digestable manner.\n    -   The writing in this paper is often more verbose while Cui et al, 2024 is more succint.\n    -   Contents in the Appendix are actually highly valuable for offering a presentation of more structured information, and thus should be incorporated into the flow of the main paper similar to how it's done in Cui et al, 2024.\n\n-   Technical details: The descriptions of existing models and methods in this paper are often high level and abstract, and readers would find themselves knowing more technical details having read Cui et al, 2024. For example,\n    -   Cui et al, 2024 starts by a convincing and detailed argument of why SLMs are better than traditional cascades. While the same discussion can be found in the bottom of page 2, it does not get the point through as strongly and clearly.\n    -   Cui et al, 2024 is often better at clearly but succintly describing the individual contribution of important past work. While the same papers are also cited in this paper, they are usually only cited as part of the categorization effort.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Following the success of text LLMs, there has been a large body of work trying to apply the same core principles (e.g. large scale pre-training; instruction tuning; etc) to speech processing. This survey paper organizes and summarizes papers in recent years on the topic of building more universal speech processing models, which this paper calls \"spoken language models\" (SLM).\n\nRelevant models can be categorized based on a model's capability of ingesting and producing text/speech input (Table 1). This paper starts by outlining a functional definition of such SLMs (page 2):\n\n> 1. It has both spoken input and spoken output with optional text input and/or output. The spoken input may serve as either an instruction or a context.\n> 2. It is intended to be “universal”; that is, it should in principle be able to address arbitrary spoken language tasks, including both traditional tasks and more complex reasoning about spoken data.\n> 3. It takes instructions or prompts in the form of natural language (either speech or text), and not, for example, only task specifiers (Radford et al., 2023) or soft prompts (Chang et al., 2024).\n\nThe rest of the paper then surveys the various important topics around SLMs, including\n- Architecture components including speech encoders, modality adaptors, sequence models, and speech decoders (Sections 2 & 3)\n- Pre- and post-training strategies (Section 4)\n- Representative SLMs (Section 5)\n- SLMs that can simultaneously listen and speak (duplex models, Section 6)\n- Evaluation methods (Section 7)\n\nFinally, Section 8 outlines the challenges and discusses possible directions of future work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be greatly beneficial to readers of this paper if the weaknesses above can be better addressed.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EjNYHd4Ohk", "reviewer": "Reviewer_BhXd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper's main strength lies in its timeliness, clarity, and breadth of coverage. It introduces a coherent framework for understanding the landscape of SLMs, backed by clear terminology and structured analysis. The paper also succeeds in contextualizing recent developments within a broader trajectory from task-specific models to universal speech processing systems. Its writing is accessible, and the inclusion of architectural illustrations and classification tables greatly enhances comprehension.\n\nHowever, the paper offers limited comparative synthesis across models, making it difficult to assess performance trade-offs or practical design implications. Some model descriptions, especially for newer or less well-documented systems, are brief and could benefit from more elaboration. Additionally, deployment concerns such as latency, inference cost, or real-time viability are largely omitted. While the survey distinguishes itself conceptually from prior work, its novelty relative to concurrent surveys could be more explicitly articulated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comprehensive and well-structured survey of Spoken Language Models (SLMs), a rapidly growing area that seeks to extend the capabilities of large language models to speech. The authors propose a taxonomy of SLMs (pure speech LMs, speech+text LMs, and speech-aware text LMs) and systematically analyze their design choices, including tokenization strategies, modality adapters, sequence modeling architectures, and training procedures. The paper also reviews representative models in each category and introduces a functional definition of universal speech processing systems. Additionally, it identifies open challenges and outlines potential directions for future work. The survey synthesizes and unifies a fragmented body of literature, providing a clear reference point for ongoing and future research in this space.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While Table 1 already provides a helpful typology of SLMs, the paper might benefit from a supplementary table or visual summary that focuses on model-level comparisons. This could include input/output modalities, types of training supervision, instruction-following capabilities, or the scope of evaluation. A qualitative overview of such dimensions could help readers quickly grasp the practical differences between models.\n\nIt could also be valuable to include a brief discussion of real-world deployment considerations, such as latency, memory efficiency, and interactive usability, which are particularly relevant for conversational or duplex SLMs.\n\nTo more clearly establish the paper’s contribution in relation to other recent surveys, the authors might consider including a brief comparative discussion. Highlighting differences in scope, taxonomy, or recency of coverage could help clarify how this paper complements or improves upon concurrent work.\n\nFor some briefly mentioned models, especially recent ones, providing slightly more descriptive context or explicit pointers to relevant sections or appendix tables would make the paper easier to navigate without requiring major expansion.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)