TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling
Authors: Liang-Hsuan Tseng
Yi-Chang Chen
Kuan Yi Lee
Da-shan Shiu
Hung-yi Lee
Venue: ICLR 2026 Poster (2026)
Keywords: spoken language modeling
speech tokenization
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling

Abstract: Recent efforts target spoken language models (SLMs) that not only listen but also speak for more natural human-LLM interaction. Joint text-speech modeling is a promising direction to achieve this. However, the effectiveness of recent speech tokens for joint modeling remains under-explored. To address this, we introduce Text-Aligned Speech Tokenization and Embedding (TASTE), a method that directly addresses the modality gap by aligning speech token with the corresponding text transcription during the tokenization stage. We propose a method that can achieve this through a attention-based aggregation mechanism and with speech reconstruction as the training objective. We have conducted extensive experiments to demonstrate that TASTE can preserve essential paralinguistic information while dramatically reducing the token sequence length. Moreover, TASTE enables straightforward joint spoken language modeling by using Low-Rank Adaptation on the pre-trained text LLM. Our experimental results show that joint modeling with TASTE outperforms other pre-trained SLMs in tasks such as speech continuation and likelihood-based next-speech selection, showcasing its effectiveness. To our best knowledge, TASTE is the first end-to-end approach that utilizes a reconstruction objective to learn a joint tokenization and embedding tailored for text-speech spoken language modeling.

Published as a conference paper at ICLR 2026
TASTE: TEXT-ALIGNED SPEECH TOKENIZATION AND EMBEDDING FOR SPOKEN LANGUAGE MODELING
Liang-Hsure Tseng 23 Yi-Chang Chen Kuan- Lee Da-Shan Shiu'
Hung-yi Lee
Equal contribution 'MediaTek Research "Internship MediaTek Research
3 Graduate Institute of Communication Engineering, National Taiwan University
Artificial Intelligence Center of Research Excellence, National Taiwan University (yi-chang chen, ds. , shiu J@mtkresearch. com (f11921067, b10901091, hungyi lee @ntu.edu.tw
edu.tw
ABSTRACT
Recent efforts target spoken language models (SLMs) that not only listen but also speak for more natural numan-LLM interaction. Joint text-speech modeling is a promising direction to achieve this. However, the effectiveness of recent speech tokens for joint modeling remains underexplored. To address this, we introduce Text-Aligned Speech Tokenization and Embedding (TASTE), method that di- rectly addresses the modality gap by aligning speech token with the corresponding text transcription during the tokenization stage. We propose a method that can achieve this through attention-basee aggregation mechanism and with speech reconstruction as the training objective. We have conducted extensive experi- ments to demonstrate that TASTE can preserve essential paralinguistic information while dramatically reducing the token sequence length. Moreover, TASTE enables straig straightforward joint spoken language modeling by using Low-Rank Adaptation on the pre-trained text LLM. Our experimental results show that joint modeling with TASTE outperforms other pre-trained SLMs in tasks such as speech contin- uation and likelihood- based next-speech selection, showcasing its effectiveness. To our best knowledge, TASTE is the first end-to-end approach that utilizes a reconstruction objective to learn joint tokenization and embedding tailored for text-speech spoken language modeling. Our demo, code, and models are available at httpss//mtkkessaarchhh http://rrrerrrrrgggi.h.ooi////ie
M.github.io
INTRODUCTION
Spoken language modeling (SLM) is an intriguing direction nowadays that aims at creating models that can not only listen but also speak (Lakhotia et al. 2021 Yguyen et al. 2023 Défossez et al. 20244] Fang et al. 2024 Arora et al., 2025). Typically, building an SLM requires two stages: first, deriving speech to tokenizations; second, training language model based on the speech tokens. For the speech tokens, previous approaches either apply self-supervised learning (SSL) representations following by discretization techniques (Baevski al., 2020 Lakhotia al., [202 Nguyen al. [2023] Hassid et al.) 2023) or reuse units from neural codec models like EnCodec and SoundStream (Défossez et al.) 2023
Zeghidour al. 202 Kumar et al.
2023 Siuzdak al. 2024). Although autoregressive modeling
with these speech tokens shows great potential in text-to-speed (TTS) (Wang et al., 2023 Xin et al.
2024a Kim al. 2024] Chen et al.
2024), previous SLMs that model only speech tokens (Lakhotia
et al. 2021 Nguyen et al. 2023) have been shown to lack semantic fidelity (Lin et al. 2024).
To bridge this gap, one promising direction is to leverage which is rich in semantics semantics-duriinn during spoken language modeling. TWIST (Hassid et al. (202)) shows that SLMs can benefit from initializing with text LLMs. Building on this idea, recent work has shifted toward joint text-speech modeling to enhance semantic coherence in generated speech. Such approaches typically adopt hybrid decoding scheme that generates both text and speech tokens. However, combining the two modalities introduces a length mismatch, since speech token sequences usually much longer than their textual counterparts. Common remedies include interleaving text and speech tokens (Nguyen et al. 2025) inserting padding to synchronize sequence lengths (Défossez et al.] 2024 Xie & Wu 2024a

---

Published as a conference paper at ICLR 2026
how
non-trivial for joint modeling
straightforward joint modeling:
mismatch lengths
OOOOOOOOOO
TASTE
how
you
Speech Tokenizer
aligned
in
Conventional Speech Tokenizer
dual-modality input
lengths
Text-Aligned SLM
Hiant
how
are
you
<>>
how
arc
P
Figure 1: The concept overview. Conventional methods extract speech tokens solely from speech, inevitably carries overlapped information with text tokens and induces length-mismatch problem when conducting joint texc-speech modeling, By taking dual modalities as input, we generate speech tokenization that is aligned with text, facilitating straightforward and effective joint modeling.
Fang et al.
Xie & Wu 2024b), but these solutions rely on additional alignment procedures or heuristic rules, making joint modeling more complex.
2024
As hybrid text-speech decoding becomes the prevailing paradigm for joint SLM (Défossez et al. 2024 Xic & Wu 20224a] Fang et al. [2024] Li et al.) [025] [ie & Wu 2024b). the design of speech tokens should reconsidered in light of this setting. This motivates the development of more effective joint tokenization methods, which can be derived under the following two considerations: 1) a speech token should avoid redundantly encoding text content-already captured by the text tokens-ane instead focus conveying paralinguistic information; and 2) straightforward one-to- one correspondence between text and speech tokens preferable, as it allows SLMs to generate a text token and speech token simultaneously without any heuristics explicit alignments applied, mitigating the length mismatch issue during the tokenization stage.
In this work, introduce Text-Aligned Speech Tokenization and Embedding (TASTE), special type of joint tokenization tailored for text-speech joint spoken language modeling. By acknowledging that the length mismatch introduces additional complexity in joint modeling, develop our speech token to be aligned with its corresponding text transcription tokens. To achieve this, we first obtain the textual transcription of a speech with the ASR model; then derive the speech token based on the transcription through specialized cross-attentic that the full process can be accomplished in an end-to-end manner, with no explicit speech-text alignment required. Unlike previous speech tokens that are developed under a fixed stride with fixed down-sampling rate, our speech token has dynamic frequency as it text-aligned. Figure shoos an overall concept of TASTE, illustrating how our joint tokenization allows effective joint modeling.
mechanism for speech reconstruction. Note
To evaluate the effectiveness of TASTE, we first conduct extensive experiments on speech reconstruc- tion. Our results on LibriSpeech (Panayotov et al.) 2015) show that TASTE not only resynthesizes speech high quality, but also retains similarity to the original speech. TASTE achieves high-end reconstruction
extremely low bit rate (~150 bps); while the other comparable methods are often more than thousands of bps. We attribute the efficiency to the involvement of text tokens during encoding and decoding, and our speech tokens foeus carrying paralinguistic information, which is backed up by the demonstration that TASTE allows simple text-aligned speech editing. By exchanging the partial text-aligned speech tokens from different utterances with the same content, we demonstrate that the paralinguistic information such as duration and tone can be exchanged precisely following the words being exchanged, resulting in natural edited speech.
On the other hand, we demonstrate that TASTE successfully allows effective spoken language modeling. We perform straightforward joint modeling with TASTE under Low-Rank Adaptation (Hu) et al., 202T). We first perform speech continuation experiments with 3-second speech prompts given. The evaluation is three-fold. We use GPT-40 for evaluating the semantic aspect; UTMOS (Saeki et al., 2022) for the acoustic aspect; and the human listening test for the general evaluation. Results show that our SLMs not only generates natural, meaningful speech continuations, but also outperforms the other 7B pre-trained SLMs across all the continuation evaluation aspects with 1.3B parameters. We also evaluate our SLMs on two benchmarks, SALMON (Maimon et al. 2024) and StoryCloze (Hassid eet al. (2023) and our results show that our SLMs achieve comparable performance compared to the other text-speech joint modeling methods. Moreover, we show that our pretrained SLM can perform spoken question answering under the few-shot scenario.
2

---

Published as a conference paper at ICLR 2026
In summary, we derive TASTE, a specialized tokenization approach for text-speech spoken language modeling. By aligning speech tokens with their text counterparts, TASTE provides a simple yet effective form of joint tokenization. Our results highlight joint tokenization as key factor in joint modeling, offering a new perspective that may foster further research into more effective designs.
2 RELATED WORK
Recent SLMs often require speech tokenization to conduct language modeling with the next prediction objective as the text LLMs. Unlike text, the speech signal is continuous and lengthy, making it difficult
to derive proper speech tokenization for spoken language modeling. Common approaches may utilize
self-supervised learned (SSL) speech models followed by quantization techniques to extract speech
tokens (Baevski et al. 2020; Hsu et al. [2021] Lakhotia et al. 2021; Hassid et al. 2023 Nguyen al. 2025). In addition, audio speech codec models have also been used for tokenization recent SLMs (Zeghidour et al. 2021; Défossez et al. 2023 2024 Zhang et al. 2024). These models are
designed for resynthesis, where the speech decoders are jointly learned with the encoders.
With speech tokenization, GSLM (Lakhotia et al. 2021) first demonstrates the possibility of building an SLM that can generate speech. TWIST (Hassid et al. 2023) further shows that the SLM can benefit from initialization with the text-only LLM. With regard to the huge success of text-only LLMs. recent work shifts the focus towards joint speech modeling (Défossez et al., 2024 Xie & Wu 2024a). To facilitate joint modeling, Spirit LM (Nguyen et al. 2025) adopts interleaving strategy; moshi Défoossez et al. 2024) trains its own tokenizer with a reduced token frequency. Moreover, delayed sequential generation are introduced for joint modeling (Xie & Wu, (2024a).
Despite the increasing demand of joint speech-text modeling, we do not find any work discussing the effectiveness of current speech tokenization for it. However, we have observed that recent work is trying to mitigate the modality gap by reducing the token frequency (Défossez et al. 2024 Zeng ett al. 2024), conducting additional training stage for text-speech alignment (Xie & Wu 2024a),
leveraging hidden representations of the ASR encoder (Du et al., (0244)). This motivates us to design speech tokenization that is directly aligned with its text counterpart, tackling the mismatch issue during the tokenization stage.
In TASTE, we utilize a specialized mechanism based on attention to aggregate the encoder represen- tations. We clarify that the text-speech cross-attention mechanism has also been used for fine-grained control of TTS. More specifically, Chen Rudnicky (2022) propose content-style cross-attention to indicate their text-speech cross- aateenatttnionn mechanism that enables style transfer in TTS. Although both utilize a specialized text- rext-speech cross- coos-nattntioon mechanism, the design choices and problem formula- tions are completely different. We attribute of our main novelty to inventing text-aligned speech tokenization and embedding for joint spoken language modeling, and the text-speech cross-attention mechanism is considered and shown to be a clean and effective way of achieving it.
3 METHOD
We propose text-aligned speech tokenization and embedding (TASTE) to facilitate effective joint speech-text spoken language modeling. Here, we first introduce how we derive our joint tokenization in Section 3.1 and then discuss how we use TASTE for spoken language modeling ($ 3.2)
3.1 BUILDING TASTE
As depicted in Figure 2 TASTE is comprised of the two main components: the text-aligned speech tokenizer ($ 1 that produces the text-aligned speech tokenization; and the speech decoder ($ 3.222) to reconstruct speech based on the text token and the TASTE speech token aligned with it. The training objective of speech reconstruction is described in Section 3.1.3
3.1.1 TASTE SPEECH TOKENIZER
In TASTE, the speech tokenizer, denoted Tokenizer(-). is designed to generate the text-aligned speech tokenization and embedding with the speech-text pair X (u, v) taken as input, where represents the textual transcription of the speech utterance u, which can be easily obtained
3

---

Published as a conference paper at ICLR 2026
$ 2.1.1 TASTE Speech Tokenizer
$ 2.112 TASTE Speech Decoder
1
2/
33
key
RVQ Quantizer
Pre-trained Speech Encoder
TASTE
Attention-Based Aggregator
Speech Decoder
(Whisper)
value
query
(ASR)
how
you
how
11
22
you
23
2.1.3 Speech Reconstruction Training Objective
Figure 2: The overall framework of our text-aligned speech tokenization and embedding. The left side illustrate the process of obtaining the TASTE tokenization &, detailed in Section [31.I]] while the right side demonstrate how we reconstruct the speech with TASTE (Section 3.1.2). The training objective for our speech reconstruction is discussed in Section 3.1.3
through an automatic speech recognition (ASR) system. Recent developments in robust and efficient ASR (Radford et al. 2023; Gandhi al., 2023) allow to focus discussing how derive the text-aligned speech token effectively by assuming that v is of sufficient quality. The TASTE speech tokenizer is composed of three major components: an encoder, an aggregator, and a quantizer.
The encoder Encoder(-) contains L. layers of Transformer (Vaswani et al. 2017) encoder blocks and is used to extract high-dimensiioal speech representation. We employ the pre-trained Whisper ASR encoder (Radford et al., 2023) as our speech encoder, and it is frozen during training. For an input speech utterance u. the encoder produces a sequence of hidden states from each layer (h(), (2))...., ..... (e)).. In experiments, shallow representation (!) from the first half of the hidden representations of the encoder:
retain the last hidden representation (1.) and the
(5) h (1) = Encoder(u), where S [
Note that both of the hidden representations (L),, h) E TRxx have their length denoted T and the hidden dimension indicated by dn.
The hidden representations extracted from the encoder are then passed the aggregator. The aggregator is designed to obtain a more compressed speech representation z that is aligned in length with the text transcription e. Consider that e [V1, v2..... uN]. V2 V is a text token sequence with length N, the input and output of the aggregator can be denoted as:
Aggregator(e, (t), h ()), where z E Nxd. E VN, and (L), hD E Rxxd.
To make the speech representation text-aligned, mechanism based on the three inputs. Consider that the original multi-head attention in Vaswani et al. (2017) is denoted as MultiHead((,, K, V), our first layer attention the aggregator takes:
conduct simple yet effective attention
Q = text transcription v, K = encoder last hidden h (L),
V encoder shallow hidden (1)
By doing so, the length of our first multi-head attention output should follow the text transcription v. Note that the query of the following layers becomes the output from the previous layer. In addition, intuitions of using the encoder's 's last hidden representation as keys, and the shallow hidden representation as values can be described follows: In Transformer-based ASR models, the last hidden states often encode rich speech-text alignment cues; sometimes the cross-attention weight matrices can even be exploited as soft word- word-alignment maps (Radford et al. 2023 Gandhi et al. 2023). 2) The shallow representation has been shown to support high-quality speech reconstruction even when the quantization is applied (Du et al. 2024a b). Based on the above observations, we design aggregator that the text transcriptions, to aggregate the shallow encoder representations that have been shown to be beneficial for high-end speech reconstruction.
the soft attention maps obtained from the last encoder representations and
4

---

Published as a conference paper at ICLR 2026
After getting the text-aligned representation, the quantizer Quantizer(.) is adopted to discretize the text-aligned representation. We use the residual vector quantization (RVQ) to allow coarse-to-fine to-fine quantization. Given the text-aligned speech representation and the quantizer containing R residual vector quantization layers, we generate:
R Σ ((r)
q, 2 Quantizerzz);
q q= (=
2=
(I)
where each ((t) E CN denotes the r-th layer code sequence with code set C; and the quantized embedding & to be the summation over each layer of the codebook vectors. Note that both of the code sequence and the quantized speech embedding & are text-aligned, with the lengths to be N.
3.1.2 TASTE SPEECH DECODER
The speech decoder aims to perform speech reconstruction conditioned on the text token sequence and the text-aligned speech tokenization. shown in Figure the text and speech tokens are aligned in lengths and being fed into the speech decoder after weighted sum in autoregressive manner. The speech decoder is composed the two components: the unit decoder and the unit-to-speech vocoder.
The unit decoder Unit is Transformer-based decoder that takes the text token sequence v and the aligned speech embedding as condition and predicts the speech unit y for reconstruction:
y UnitDeccder(22 v)
(2)
Note that the additional speaker embedding is also taken as input to facilitate global speaker voice control in our spoken language models (Ju et al., (2024). After we generating the speech unit y. we unie-to-specch vocoder further transform the unit into the reconstructed speech.
3.1.3 TRAINING OBJECTIVE
Similar to other reconstruction--ase speech tokens (hhag et al. 2024, Liu et al. 2025 we derive TASTE by training it for speech resynthesis. To achieve this, we extract the speech unit unge with length 7' from the original speech u as the target unit for our speech tokenizer and speech decoder. Given the text transcription v, the TASTE speech embedding é, and the unit from the original speech argert as the target, the speech reconstruction through the tokenizer and the unit decoder parametrized by 0 under the next prediction schema can be considered as minimizing the cross-entropy loss below:
(0)
log pe (y:
farger target
2 Ui y<t
target
T
On the other hand, we employ the quantization loss as well tokenize the continuous representation extracted from the mooder-agggreatte Following prior works (Défossez et al. 22033 Zeghidour al., 2222),, given that (+)) is the r-th residual and ((r) indicates the r-th quantized residual, the the commitment loss defined as:
A Σ (1)
Lovg (0)
(4)
By summation over both losses, we formulate the overall loss for training TASTE as:
Luste Los + Lrvg.
(5)
3.2 TASTE FOR SPOKEN LANGUAGE MODELING
Next, we describe how we conduct effective spoken language modeling with TASTE, Following previous work (Hassid et al. 2023 Nguyen et al. 2025), we perform pre-training on speech data. The text transcription of the speech data is also used for joint speech-text pre-training of our text-aligned spoken language model (TASLM). Since TASTE tokenization already aligns with the text token sequence, we can conduct a straightforward joint modeling, as illustrated Figure To demonstrate perform two types of text-aligned spoken language modeling. First,
the robustness of TASTE, build TASLM over our text-aligned speech token q, discussed in Section 3.2.T Then, we show how build M with text-aligned speech embedding 2, detailed in Section 3.2.2
5

---

Published as a conference paper at ICLR 2026
3.2.1 MODELING TASTE TOKEN
As our speech tokens derived from the RVQ quantizer contain R layers of codes, we employ R linear heads for multi-head prediction in our TASLMoken Namely, the TASLMoken simultaneously predicts the next text token and the corresponding layers of speech tokens in each step. The overall training objective follows the original next token prediction scheme, but with multiple predictions across modalities at each step. Specifically, given the text transcription v and R layers of quantized RVQ codes q, the multi-head next-token prediction training objective can be formulated as:
N
R
Luken (Ø)
Σ (-log Pe
jext (Us Ues: q<s)
Σ
(t) (e) log Po ( pes.
(6)
IN
with & represents the parameter of the TASLM Moken and p(r) is the r-th probability prediction for the r-th RVQ code. As for inference, we directly sample the codes and the text simultancously, and transform the codes into the corresponding embedding for the speech decoder to generate speech.
3.2.2 MODELING TASTE EMBEDDING
2024 Meng et al. 2024 Besides the token code sets, recent progress on latent modeling (Kim et al.
Sun al. 2024| Fan et al. 2025) motivates us to conduct experiments on modeling our text-aligned speech embedding. Referencing MELLE (Meng et al. 2024), we employ a linear layer that predicts the mean vector μ4; and a -oggmnuie variance vector log of, where indicates the i-th frame of the sequence. And the final predicted latent of frame i is denoted as e; 14 + @ where c (O, I). Following MELLE, the straight- estimator is applied allow gradients to back-propagate properly during training.
To facilitate latent prediction, we apply the regularization loss and the Kullhack-Leibler (KI.) diver- gence loss druing training, which is described as follows:
d2
Creg(e) -lee - 112. CKL = ΣΣ Σ
+ ( &D))) - log ofll), (7)
where v indicates the parameter of TASLMemb, and d₂ is the dimension of our text-aligned embed- ding& The regularization loss Lreg is adopted to predict close latent towards the target embedding 2. The KL divergence loss calculates the KL divergence between the predicted latent distribution and the target distribution. Following MELLE, we select the target distribution to be ( I). This allows simplification of CKL. which can then be approximated with the predicted vectors (us, de, and the target embedding 2, Finally, the overall loss along with the text loss is described as:
N
1
Lemb (v))
Areg freg + AKI. LKI. +
log p (e))
N
1=1
where >reg AKL to be the weighted coefficients of the two losses, respectively.
4 EXPERIMENT SETUP
Model Configuration For our TASTE speech tokenizer, we initialize our encoder from Whis- per (Radford et al. 2023). By doing so, can reduce computational cost between obtaining the ASR transcription and extracting the TASTE tokenization with the TASTE encoder frozen during training. On the other hand, we use the S3 token from Du al. (2024a) the target unit for speech reconstruction. Since their speech tokenization facilitates additional speaker embedding, we follow the same procedure to obtain one. Adding speaker embedding allows global speaker voice control, which is a reasonable and useful scenario for spoken language models. The unit-to-speech vocoder comprised of a flow model (Lipman et al. 2022, Mehta al. 2022) and a HifiGAN. We use the published pre-trained ones from Du et al. (2024a). and they are not involved in our training. For the quantizer, we set the RVQ layer R -7= 4, the codebook size 512, and the codebook dimension to be 256. For the spoken language modeling, we follow previous work (Hassid et al. 2023) and initialize our spoken language model from a text LLM. However, this introduces the vocabulary mismatch problem between the ASR and LLM. We resolve this issue by using word-level TASTE tokenization and embedding, which is detailed in Appendix A.4 Moreover, we conduct LoRA tne-tuning of our TLSMMs, with hyperparameters rank = 64 and 128.

---

Published as a conference paper at ICLR 2026
Table 1: The speech tokenization evaluation results on the test-clean split of LibriSpeech. The evaluation is separated into the QUALITY and the SIMILARITY assessments, as introduced in Section [.1.1] We use gray text indicate the worst-performing methods in each metric. Freq. indicates the number of tokens per second. All reported results already account for the effect of ASR
errors whenever textual transcriptions are involved (Text-only and TASTE).
QUALITY
SIMILARITY
Method
Freq.
Bitrate
WER
UTMOS
DNSMOS VISQOL
Drtn. Con. Spkr. Sim. MUSHRA
Ground Truth
16k
256k
2.1%
4.09
3.84
76.6
75
1500
5.1%
1.58
3.26
3,46
0.94
0.63
Encodec"
75
3000
2.6%
2.35
3,48
3.81
0.96
0.78
25.6
50
500
5.2%
1.27
2.99
2.80
0.94
0.35
SpeechTokenizer"
50
2000
3.0%
3,56
3.60
3.65
0.97
0.80
53.9
50
4000
2.5%
3.90
3.76
4.03
0.98
0.92
Mimi?
12.5
1000
3.1 %
3.60
3,60
3.62
0.96
0.82
67.6
S3 token (topline)
25
600
3,0%
4.18
3,90
3,30
0.96
0.82
70.2
Text-only (baseline)
3
50
5.9%
4.31
4.11
2.44
0.57
0.78
42.6
TASTE (ours)
3
150
4.4%
4.29
4.10
3.05
0.91
0.80
68.3
Decossez et
2023
Zhang et
2024
Defossez et al.
2024
et al.
20240
Dataset We use two EitaasetsEEmille and LibriTTS as our training datasets. Emilia (He et al. 2024) is inlthe-wild dataset where the speech is web-scaled and the transcriptions pseudo-labeled. We use only the English subset of this multi-lingual corpus, which about 40,000 hours. LibriTTS (Zen al. 2019) is reading-style corpus based LibriSpeech (Panayotov al., 2015). We all the training splits in LibriTTS for training, which approximately 600 hours of speech. In addition, the test-clean split in LibriSpeech is used for evaluation purposes for our TASTE tokenizer and TASLMs.
RESULT
We separate our experimental results into two parts. Section [.1 discusses reconstruction quality while enables effective joint spoken language modeling: while Seciton 5.2 presents the additional results and ablation study of our joint tokenization and text-aligned SLM.
discuusss how TASTE strikes good
5.1 MAIN RESULTS
To demonstrate the benefits of our joint tokenization, we first evaluate the performance of TASTE on speech reconstruction; then introduce how it allows effective spoken language modeling. For simplicity, the evaluation metrics are introduced within each section.
5.1.1 TASTE FOR SPEECH RECONSTRUCTION
Evaluation
We evaluate joint tokenization two aspects: QUALITY and SIMILARITY. For QUALITY assessment, we use ASR- WER, UTMOS (Saeki et al. 2022). and DNS-MOS (Reddy (Hsu al. 2021)
et al.] 2021)) as metrics for evaluation. In ASR-WER, we use HuBERT-Land as the ASR model to transcribe the reconstructed speech, and then calculate the word-error rate (WER) the transcription. ! UTMOS and DNS-MOS are both neural-based MOS predictors. While both evaluate the speech quality, the design purpose of DNS-MOS makes it more suitable for evaluation regarding the noise levels. For SIMILARITY assessment, we measure ViSQOL, duration consistency (Drtn. Con.), speaker similarity (Spkr. Sim.), and the MUSHRA human listening test score. VISQOL (Chinen al. 2020 a production
tool that predicts speech quality via spectro- temporal image similarity comparisons. For the duration consistency, we first get the word-level alignment of the transcriptions
the original and the reconstructed speech using Montreal Forced Aligner (McAuliffe et al. 2017); then we calculate if the duration between each of the same words is matched under a preset tolerance window, which is set to 50 milliseconds. In the MUSHRA human listening test, follow the original protocal (Series) 2014) to instruct evaluators to rate similarity and quality on a scale to 100 with reference given.
18960

---

Published as a conference paper at ICLR 2026
Results Analysis Table reports the results of speech reconstruction LibriSpeech. To better understand the effectiveness of TASTE, we highlight three main observations. 1) Since our tokens are text-aligned. TASTE operates the lowest frequency and bitrate among all tokenization methods. We estimate these dynamic values by counting the total numher of tokens and accumulating the duration the testing 2) Despite this extremely low bitrate, TASTE achieves on-par even superior performance to higher-bitrate methods in the quality assessment. In particular, TASTE yields lower ASR-WER than the text-only baseline, which attribute to speech tokens carrying paralinguistic information that improves the naturalness of reconstructed speech. 3) In terms of similarity, TASTE performs comparably to high-bitrate, fixed wwnnssamling methods across multiple metrics. The inferior results on ViSQOL can be partly attributed to our use of flow-based vocoder, as both TASTE and the S3 token topline exhibit weaker ViSQOL performance- phenomenon also observed in Liu et al. (2025). This degradation on ViSQOL is not reflected in the MUSHRA listening test, where TASTE attains competitive perceptual quality and similarity from a human perspective. In general, TASTE significantly outperforms the text-only baseline, confirming that carries sufficient paralinguistic information to allow high-quality speech reconstruction.
5.1.2 TASTE FOR SPOKEN LANGUAGE MODELING
TASTE designed specifically to enable effective joint spoken language modeling (SLM). To examine its effectiveness, we train pretrained SLMs on top of TASTE following the methodology in Section 3.2 In line with prior work (Nguyen et al. 2025 Lin al. 2024), we evaluate these models from two perspectives: speech continuation evaluation and likelihood-base evaluation.
Speech Continuation Evaluation First, each pretrained SLM is conditioned on 3-second speech segments from LibriSpeech test-clean generate speech continuations under their own decoding schemes, following Hassid et al.] (2023); ;; Lin et al. (0024). The generated continuations are then evaluated along two main aspects: semantic coherence and speech naturalness. For the semantic aspect, we transcribe the continuations using ASR and ask GPT-4o to assign MOS scores based on their coherence, For the speech naturalness aspect, we compute UTMOS as an objective score of speech quality. In addition, human evaluators provide an overall MOS score that jointly considers both coherence and naturalness. The detailed instructions given to GPT-4o and human evaluators are provided in Appendix A.3.2
Likelihood- -Bssed Evaluation Following previous work (Hassid et al. 2023 Nguyen et al. 2025 Lin al.) 2024), also evaluate the SLMs through likelihood-based benchmarks, where the accuracy score is based on whether the pretrained SL. chooses the correct continuation from the two given speech utterances based its output likelihoods. We adopt two established benchmarks SALMON (Maimon el al. 2024) and spoken StoryCloze (Hassid et al. 2023 Mostalazadeh et al.
Table 2: Pretrained SLM speech continuation and Likelihood- next-speech selection results. The superscripts at the bottom of the table indicate the base models used by each SLM, indicated by superscripts. Cascade models refer to the pipeline with ASR (Radford et al.) 2023), text continuation by LMs (Touvron et al., 2023), and TTS (Du et al. 2024a). This allow us to evaluate SLMs with cascade models continuation perspective.
Finetuned base
CONTINUATION
Method
parameters
GPT-4o UTMOS Human SALMON StoryCloze Overall
Cascade
Cascade (LLaMA3.2-IB")
3.15
4.25
4,00
Cascade (LLaMA2-7B')
3.43
4.25
3.98
Spoken LMs
TWIST 1.3B
Massid al.
2023
1.3B 1.3B
1.48
3.25
1.95
62.5
61.5
62.0
TWIST Hassid
7B/ 7B"
1.44
3.27
2.04
63.4
64.7
64.1
Spirit LM
Nguyen et al.
7B
7B
2.79
3.41
2.38
59.1
72.0
65.6
Spirit LM Expr. (Nguyen
2025
7B
7B
1.90
3.40
2.41
69.0
66,2
67.6
Baseline 7S3 token)
45M
/1.3'"
1.37
4.04
2.84
50.2
58..7
34.5
TASLM (token)
"
3.08
4.07
3.93
60.8
76.5
TASLM (emhed.)
45M
L.3B"
3.16
4.22
4.16
57.7
76.7
67.2
Rase models 1LLaMA3.-11. PLLaMA2-7,, "LLAMA-7B, "OPT-L3B
8

---

Published as a conference paper at ICLR 2026
2016), which covers the acoustic aspect and the semantic aspect, respectively. Since both benchmarks contain multiple tasks, we report the average accuracy across these tasks within each benchmark for simplicity. The detailed results are in Appendix A.I.5 for the interested readers. We also report the mean of the SALMON and StoryCloze as an overall assessment for both aspects.
Results Analysis The results of TASLM compared other pre-trained SLMs are shown in Table 2 and three main advantages can be observed. 1) Compared to other pretrained SLMs. TASLM achieves substantially better performance on speech continuation across both human and machine evaluations, while also performing competitively the likelihood-based benchmarks. Notably, this is achieved with only LoRA finetuning on a relatively small 1.3B base language model, illustrating the effectiveness of TASTE for joint modeling. 2) Compared to cascade models with the same base LM, our TASLM achieves comparable scores on GPT-4o but higher human MOS. This indicates that its generated speech is more natural than cascade systems that rely solely TTS during continuation. TASLM is the only SLM that not only maintains but even surpasses the performance of its corresponding text-based model, highlighting the importance of speech token modeling. 3) Directly using the S3 token for joint modeling following Xie & Wu (2024a) yields poor performance across all aspects, even though it surpasses TASTE in speech reconstruction. This shows that while reconstruction quality is critical, it is not the sole consideration in tokenization for spoken language modeling. Taken together, these results highlight the central contribution of TASTE: building joint tokenization that facilitates more effective joint spoken language modeling.
5.2
ADDITIONAL RESULTS
5.2.1 TASTE FOR
SPEECH EDITING
Beyond the main results presented above, we report several intriguing observations that further showcase the versatility of TASTE. The first is that TASTE naturally enables text-aligned speech editing, as illustrated in Figure 3 Suppose we have two utterances with the same transcript but different paralinguistic characteristics. By exchanging their TASTE token sequences word by word, ask whether the associated paralinguistic traits are transferred clear, we select utterances that differ mainly in speaking rate and examine duration changes using MFA (MeAuliffe al. 2017). As illustrated in Figure B] swapping tokens at specific word positions causes the corresponding words to exhibit clear duration shifts, while untouched words preserve their original timing-evident
well. To make the effect
that TASTE enables precise, text-aligned manipulation. This observation also echoes our design principle introduced in Section I a speech token should avoid redundantly encoding text content and instead concentrate on conveying paralinguistic information. Additional examples targeting other paralinguistic dimensions are provided on our demo page.
5.2.2 TASLM FOR SPOKEN QUESTION ANSWERING
Next, we intriguingly find out that our TASLM exhibits spoken QA ability under few-shot sce- nario Brown et al. 2020). We are the only pretrained SLM in Table 2 that exhibits this capability. As
Original A
Original B modl today H6 Heaf
Result Visualization (an esample focuseed on duration) for text-aligned speech editing
niow meel you Indlyy
Original
Original
Extract the original TAS TE tokens
TASTE
TASTE
captamstoza boon...
CapTainta
gorle
now
manayyon hodkey
Swap text-aligned
speech editing gods
burine
Edited
Sparch
Decode edited TASTE tokans
Speech
(partally fast in slow)
Edited B
tael)
modyy
speach
laed bwwny
Deplawnsedece Jeck Bad
Edited
Edited
Figure 3: An illustration of TASTE for text-aligned speech editing. On the left shows the process of our text-aligned speech editing. We first extract the TASTE tokens; swap the tokens partially; and then decode the edited TASTE tokens into edited speech. On the right shows an example visualization. Only the durations of the words with exchanged TASTE tokens show significant difference.

---

Published as a conference paper at ICLR 2026
Table Evaluation of spoken question answering. Performance across modalities is compared row-wise, where T is text and denotes speech.
Table 4: Ablation study on the effects of each module in TASTE speech tokenizer. Enc. is encoder, Agg. is aggregator, and Quan. is quantizer. *: top-5 accuracy.
Method
Mode
Web Q.
LLaMA-Q.
Mini-Omni 0.5B(T-+1)
T
21.3
39.0
Modules
Freq. S3 token Acc. *
Mini-Omni 0.5B
T4$
4.5
11.6
Enc.
50Hz
0.98
Helium 7B (text)
T
32.3
75.0
Enc. + Agg
~3Hz
0.88
Moshi 7B
T4S
26.6
62.3
Enc. + Age. + Quan.
~3Hz
0.76
LLaMA3.1 118.3,1-88-IIniiui
T 60.4
71.7
Enc. (last)
50Hz
0.84
Manna-Omni-&B
T+S
35.5
67.3
Enc. + Agg. (last)
~3Hz
0.78
LLaMA3.2 2B
T
24.0
51.0
Text-only
~3Hz
0.65
T 1B (embed.)*
T+S
57.6
1 We apply few-shot leaming to facilitate question answering.
a result, we compare against other instruction finettned joint SLMs in Table 3 to hetter understand the performance. We use two spoken question answering benchmarks, Web Questions (Berant et al.) 2013) and LLaMA- LLaMA-Questions (Nachmani et al. 2024), following Défossez et al. (2024). We report the accuracy of answer containment. To more comprehensively assess the impact adding speech, we also report the performance of each system's 's underlying base text LLM. Notably. TASLM is the only approach that preserves its base text LLM's performance. We attribute this TASTE's joint tokenization strategy. Specifically, we employ a straightforward one-to-one mapping between text and speech tokens, which enables simple and effective joint modeling.
5.2.3 ABLATION STUDY ON TASTE SPEECH TOKENIZER
We run an ablation on TASTE speech tokenizer and use token top-5 reconstruction accuracy as proxy for information retention. Table first covers the module-wise ablations of our encoder, aggregator, and quantizer. The aggregator sharply reduces token rate with only a small drop in accuracy. Adding the quantizer lowers accuracy further, but performance is still well above the text-only baseline. Secondly, we show that using only the last hidden state (f) performs worse than using the shallow hidden states h()) (as values for the aggregator), confirming our design choice.
CONCLUSION
In this work, we propose Text-Aligned Speech Tokenization and Embedding (TASTE), to facilitate joint text-speech spoken language modeling. By aggregating proper encoder representation through the specialized cross-attention mechanism and taking the ASR model as initialization, we make the speech tokenization text-aligned in an end-to-end manner with no explicit word alignment required. With our text-aligned speech tokenization and embedding, joint text-speech modeling becomes straightforward and effective. We conduct extensive experiments demonstrating the benefits of developing joint tokenization tailored for spoken language modeling. We anticipate that these findings encourage further research on more effective joint tokenization for generative modeling.
Limitation Several limitations of our current work suggest promising avenues for future develop- ment. First, while our pretrained spoken language model generates high-quality audio continuations, lacks mechanisms for turn-taking and instruction following; developing a dialogue system is practical next step. Second, TASTE has so far been evaluated on English; confirming its general- izability across other languages remains future work. Third, our tokenization method is tailored for joint SLMM,, and its applicability to other generative tasks remains underexplored. Fourth, our pipeline currently focuses on single-speaker speech with lexical content and does not explicitly handle multi-speaker, overlapping, or non-lexical events (e.g., laughter, coughing). Future work could support these capabilities by incorporating target speech extraction (Zmolikova et al., 2023) and non-lexical event tags. Finally, system latency and streaming performance are yet be optimized for real-time applications. Overall, none of these limitations is a fundamental barrier; rather, they are natural extensions and research targets that will further enhance the versatility of TASTE framework.
10

---

Published as a conference paper at ICLR 2026
Ethics Statement TASTE enables the efficient development of spoken language models. It low- ers the barrier to building speech systems and improves the accessibility and convenience of hu- man compuerr interaction. At the same time, it raises security concerns: systems built with TASTE can more easily mimic person's 's voice and synthesize convincing personalized speech. Moreover, TASTE's text-aligned speech editing makes voice manipulation straightforward. Overall, TASTE offers clear utility for beneficial applications, but responsible geployment-pairee with consent, provenance, and anti-abuse safeguards-is essential mitigate misuse risks. On the other hand, this study includes human evaluations in the form of subjective listening tests. Annotators were recruited from Amazon Mechanical Turk and compensated fairly according to the platform's 's recommended rates. All participants provided informed consent, and no personal information has been collected. The audio material used in the evaluation does not contain sensitive content. The study adheres the ethical standards commonly adopted in speech perception research.
ACKNOWLEDGEMENT
This work is partially supported by the National Science and Technology Council (NSTC) in Taiwan under Grant NSTC 13-22634-F0002-0088, and Llang-Hsuan Tseng is supported by the NSTC Graduate Research Fellowship (NSTC-GRF). We also extend our gratitude to the anonymous reviewers for their constructive comments, which can be found at http:///ppmneeiiioeeeooooooo/ldd
REFERENCES
Md Mubtasim Ahasan, Md Fahim, Tasnim Mohiuddin, Akmmahbubur Rahman, Aman Chadha, Tariq Iqbal, M Ashraful Amin, Md Mofijul Islam, and Amin Absan Ali. DM-codec: Distilling multi- modal representations for speech tokenization. In Findings of the Association for Computational Linguistics: EMNLP 2025, 2025.
Siddhant Arora, Kai-Wei Chang, Chung-Ming Ming Chien, Yifan Peng. Haibin Wu, Yossi Adi, Emmanuel Dupoux, Hung- Yi Lee, Karen Livescu, and Shinji Watanabe. On the landscape of spoken language models: A comprehensive survey. arXiv preprint 2rXi::2504.08528 2025.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, 2020.
Jonathan Berant, Andrew Chou, Roy Frostig, and Perey Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, 2013.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
Li-Wei Chen and Alexander Rudnicky. Fine-grained style control in transformer- based cox-tto-speech synthesis. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers. arXiv preprint 24Xi:22006.03700 2024.
Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), 2023.
Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O' G'Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In 2020 twelfth international conference on quality of multimedia experience (QoMEX), 2020.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023.

---

Published as a conference paper at ICLR 2026
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real- time dialogue. arXiv preprint arXhv:2410.00037, 2024.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng. Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint 20002050770447 2024a.
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice Scalable streaming speech synthesis with large language models. CoRR, 2024b.
Lijie Fan, Luming Tang, Siyang Qin, Tianhong Li, Xuan Yang. Siyuan Qiao, Andreas Steiner, Chen Sun, Yuanzhen Li, Tao Zhu, et al. Unified autoregressive visual generation and understanding with continuous tokens. arXiv preprint arXiv:2503 13436, 2025.
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. CoRR, 2024.
Sanchit Gandhi, Patrick Platen, and Alexander M Rush. Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling. arXiv preprint wrXiv:2311.004300 2023.
Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv: 1211.371, 2012.
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, al. Textually pretrained speech language models. Advances in Neural Information Processing Systems, 2023.
Haorui He, Zengqiang Shang. Chaoren Wang. Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang. Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In 2024 IEEE Spoken Language Technology Workshop (SLT), 2024.
Harold Hotelling. Relations between two sets of variates. In Breakthroughs in statistics: methodology and distribution. Springer, 1992.
Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. Liger kernel: Efficient triton kernels for Ilm training, arXiv preprint 0rXiv:2410.10988) 2024.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions Audio, Speech, and Language Processing, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.09685, 2021.
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang. Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. International Conference on Machine Learning, 2024.
Jachyeon Kim, Keon Lee, Scungjun Chung, and Jaewoong Cho. Clam-tts: Improving neural codec
language model for zero-shot text-to-speech. ICLR, 2024.
Diederik Kingma. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 2023.
12

---

Published as a conference paper at ICLR 2026
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics. 2021.
Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Gu- osheng Dong, Da Pan, al. Baichuan-onea -omni--.n5 technical report. arXiv preprint arXiv:2501. 15368. 2025.
Guan- Minn-Ting&a Lin, Prashanth Gurumath Shivakumar, Aditya Gourav, Yile Gu, Ankur Gandhe, Hung-yi yi Lee, and Ivan Bulyko. Align-slm: Textless spoken language models with reinforcement learning from ai feedback. arXiv preprint arXiv:241 1.01834, 2024.
Yaron Lipman, Ricky TQ Chen, Heli for generative modeling. The Eleventh International Conference on Learning Representations, 2022.
Maximilian Nickel, and Matt Le. Flow matching
Alexander Liu, Sang-gil Lee, Chao-Han Huck Yang. Yuan Gong, Yu-Chiang Frank Wang. James R Glass, Rafael Valle, and Bryan Catanzaro. Uniwav: Towards unified pre- pre-training for speech representation learning and generation. The Thirteenth International Conference on Learning Representations, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Gallil Maimon, Amit Roth, and Yossi Adi. Salmon: A suite for acoustic language model evaluation. arXiv preprint arXiv:2409.07437, 2024.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech 2017, 2017.
Shivam Mchta, Ambika Kirkland, Harm Lameris, Jonas Beskow, Éva Székely, and Gustav Eje Henter. Overflow: Putting flows on top of neural transducers for better tts. Interspeech 2023, 2022.
Lingwei Meng, Long Zhou, Shujic Liu, Sanyuan Chen, Bing Han, Shujic Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, et al. Autoregressive speech synthesis without vector quantization. CoRR, 2024.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. corpus and evaluation framework for deeper understanding of commonsense stories. Proceedings of NAACL-HLT, 2016.
Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan and Michelle Tadmor Ramanovich. Spoken question answering and speech continuation using ceattoorrm-powweeer 1lm. In The Twelfth International Conference on Learning Representations, 2024.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet. Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres. Benoît Sagot, Abdelrahman Mohamed, et al. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 2023.
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R Costa-Jussa. Maha Elbayad, Sravya Popuri, Duquenne, Robin Algayres, Ruslan Mavlyutov, et al. Spirit-Im:
Christophe Ropers, Interleaved spoken and written language model. Transactions of the Association for Computational Linguistics, 2025.
Vassil Panayotov. Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ast corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015.
Ankita Pasad, Ju- Cu-Chich Chou, and Karen Liveseu. Layer-wise analysis of supe-supervised speech representation model. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021.
13

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, 2023.
Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery and data mining, 2020.
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2021.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. Interspeech 2022, 2022.
Monica Sekoyan, Nithin Rao Koluguri, Nune Tadevosyan, Piotr Zelasko, Travis Bartley Nick Karpov, Jagadeesh Balam, and Boris Ginsburg. Canary-1b-v2 & parakeet-tdt-O.6 b-v3: Efficient and high-performance models for multilingual asr and ast. arXiv preprint 2rXiv:2509.14128 2025.
Series. Method for the subjective assessment of intermediate quality level of audio systems. International Telecommunication Union Radiocommumication Assembly, 2014.
Hubert Siuzdak, Florian Grötschla, and Luca A Lanzendörfer. Snac: Multi-scale neural audio codec. Audio Imagination: NeurIPS 2024 Workshop Al-Driven Speech, Music, and Sound Generation, 2024.
Yutao Sun, Hangbo Bao, Wenhui Wang, Zhiliang Peng, Li Dong, Shaohan Huang, Jianyong Wang, and Furu Wei. Multimodal latent language modeling with next-token diffusion. arXiv preprint arXiv:2412.08663, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 2017.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111 2022.
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408. 16725, 2024a.
Zhifei Xie and Changqiao Wu. Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities. arXiv preprint arXiv:2410.11190 2024b.
Detai Xin, Xu Tan, Kai Shen, Zeqian Ju, Dongchao Yang. Yuancheng Wang, Shinnosuke Takamichi, Hiroshi Saruwatari, Shujie Liu, Jinyu Li, et al. Rall-e: Robust codec language modeling with chain-of-thoughe prompting for text-to-speech synthesis. arXiv preprint arXiv:2404.03204 2024a.
Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. Bigcodec: Pushing the limits of low-bitrate neural speech codec. arXiv preprint arXf::2409.05377, 2024b.
Hainan Xu, Fei Jia, Somshubra Majumdar, He Huang, Shinji Watanabe, and Boris Ginsburg. Efficient sequence transduction by jointly predicting tokens and durations. In International Conference Machine Learning, 2023.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.
14

---

Published as a conference paper at ICLR 2026
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. Interspeech 2019, 2019.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02212, 2024.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. ICLR, 2024.
Katerina Zmolikova, Marc Delcroix, Tsubasa Ochiai, Keisuke Kinoshita, Jan Černocký, and Dong Yu. Neural target speech extraction: An overview. IEEE Signal Processing Magazine, 2023,
15

---

Published as a conference paper at ICLR 2026
Table 5: The ablation study on how the ASR affects the perfor- mance of our TASTE tokenizer regarding speech reconstruction. GT: ground truth transcription.
Table 6: The ablation study on
how the ASR affects our SLM on spoken QA.
Method
WER UTMOS DNS-MOS ViSQOL Drtn. Con.
Spkr. Sim.
Methods
Web-Q LLaMA-Q
TASTE (W/ ASR)
4.4%
4.29
4.10
3.05
0.91
0.80
TASLM (w/ ASR)
27.1
57.6
TASTE (w/ GT)
4.6%
4.24
4.08
3.06
0.91
0.81
TASLM (w/ GT)
28.0
57.7
Table 7: The ablation study on using a different ASR model regarding the SLM continuation semantic evaluation. Overall, we do not observe significant relative performance difference.
Evaluation Models
TWIST 1,3B TWIST 7B Spirit LM Spirit LM Expr. S3 token TASLM (token) TASLM (embed.)
Whisper GPT-4o
1,48
1.44
2.79
1.90
1.37
3.08
3.16
nvidia-parakeet GPT-4o
1.38
1.49
2.76
2.06
1.42
3,20
3.37
A
APPENDIX
A.1
SUPPLEMENTARY RESULTS
A.1.1 ABLATION STUDY ON THE EFFECT OF ASR
Because our tokenization, SLM, as well as the evaluation using GPT-4o all rely on an ASR system to extract text transcriptions, we conduct several ablation studies to assess the impact of ASR on performance. 1) In Table and Table we study how the ASR affects the performance of our TASTE tokenizer on speech reconstruction and TASLM on spoken question answering. Our results indicate that on both the tokenization and the SLM stages, the performance drop introduced by the ASR errors are almost negligible, primarily attributed to the robustness of recent ASR systems. Note that we do not use any ground-truth transcriptions in the previous experiments in the main text. 2) We study how substituting the ASR used to produce transcripts before GPT-4o's semantic-coherence evaluation affects the reported scores. As shown in Table 7 we use another ASR model named nvidia- pardaeearaaket (Sekoyan et al., 2025). which employs an RNN-T (Graves) 2012 Xu et al., 2023) backbone. The results indicate that there is no significant relative performance difference between using Whisper and nvidia- nvidia-parakeet ASR systems. TASLMs achieve much better results in both evaluation setups compared to the other pretrained SLMs.
A.1.2 AGGREGATOR CROSS ATTENTION VISUALIZATION
To understand whether the aggregator has learned the text-speech aligned pattern, we visualize its cross attention map across all the layers and heads in Figure 4 and Figure 5
Us U0
Us 10
Loyer Feaed
Leger Menn
Lever Feael
Leyer Fear
Leper Meet
Figure 4: The cross attention map of the last layer in our aggregator. As illustrated, a lot of heads clearly demonstrate the text-speech aligned behavior.
16

---

Published as a conference paper at ICLR 2026
Lxyer For3)
hodd?
12 20
Add
Figure 5: The cross attention map of the first layer in our aggregator. As illustrated, the behavior is quite different from the last layer in Figure We observe some special heads: Head 8th showns clear alignment; while head 19th lights up at the silence part.
A.1.3 DISCUSSION ON THE SELECTION OF SHALLOW HIDDEN LAYER
In Section 3.1.1 we propose using the shallow hidden representation from Whisper encoder as the key in our specialized cross attention, it carries rich acoustic information. In practice, select I 1= 6. In Table [ this allows us to achieve a near-optimal reconstruction accuracy on the target S3 unit. To further justify this selection, we follow the analysis methodology Pasad al. (2021) and compute the Canonical Correlation Analysis (CCA: Hotelling (1922) between each Whis Whisper encoder layer and the target S3 unit embeddings. The resulting similarity curve shown in Figure The correlations peak at the 4-th to 8-th layers, indicating that these shallow layers encode representations most aligned with the S3 Importantly, Table 4 also shows that while the choice of the shallow layer affects the attainable upper bound, it does not impose strict limitation on our method: even using the final encoder layer, whose correlation to S3 is much lower, still yields reconstruction quality better than the text-only baseline.
our design choice of using a shallow hidden state.
CCA Between Whisper Layers and S3 Embeddings
CCA to 53 Embeddings
0
10
25
25
30
Whisper Layer
Figure 6: Canonical Correlation Analysis (CCA) between each Whisper encoder layer and the S3 target embeddings. Layers 4-8 exhibit the highest correlation with the S3 units, indicating that shallow representations contain the most target-relevant acoustic information.
17

---

Published as a conference paper at ICLR 2026
Table 8: The tokenizer robustness ablation study. We apply four different level of noise, with signal-to-noise ratio (SNR) ranging from 20dB (almost clean) to 5dB (very noisy). Our method substantially achieves good reconstruction quality across all noise levels.
SNR=20dB (clean)
SNR=15dB
SNR=10dB
SNR=5dB (noisy)
Method
Bitrate
WER L
Sim.
Rank
WER
Sim.
Rank
WER 4 Sim.
Rank
WER 4
Sim.
Rank
Ground Truth
256k
2.3%
2.3%
3.6%
9.2%
500
93.7%
0.193
97.7%
0.201
12
98. 6%
0.205
12
98.3%
0.202
13
DAC
1000
36.0%
0.292
55.0%
0.276
80.2%
0.278
94.5%
0.283
10
12000
2.5%
0.937
2.9%
0.924
4.7%
0.914
12.3%
0.907
500
98.7%
0.295
12
104.6%
0.264
12
107.1%
0.264
12
105.8%
0.283
12
DM-Codec
1000
33.9%
0.479
53.0%
0.447
77.4%
0.435
9
97.1%
0.415
9
4000
3.9%
0.737
6.4%
0.689
14.4%
0.647
38,5%
0.595
500
15.2%
0.334
33.9%
0.324
64.2%
0.301
91,7%
0.277
10
SpecchTokemizer'
2000
7.3%
0.773
16.3%
0.734
39,4%
0. 0.682
73.8%
0.600
4000
4.4%
0.864
8.1%
0.822
21.0%
0.765
49.2%
0.684
BigCodec
1040
10.1%
0.829
17.8%
0.785
33.5%
0.718
7
63.0%
0.625
Mimi
1000
5.1%
0,804
7.8%
0.772
14.7%
0.726
4
33.6%
0.673
4
token'
3.9%
0.860
5.2%
0.841
8.1%
0.815
16.7%
0.779
TASTE (ours)
1-15
4.8%
0.842
5.3%
0.830
6.9%
0.815
2
11.1%
0.792
Alason
Zheng
b024
Xix
2004:
Offowser
2034
Dueerll
2004
A.1.4 ROBUSTNESS ABLATION ON TASTE TOKENIZER
To evaluate our tokenizer robustness, we have conducted controlled noise level experiments on speech reconstruction. Specifically, we add 4 different levels of white noise to the original waveform and then evaluate the reconstruction performance. The noise levels are defined by signal-to-noise ratio (SNR) ranging from 20dB (almost clean) to 5dB (very noisy). We report two metrics: ASR-WER as an indicator of reconstruction quality, and speaker similarity (Sim.) as a measure of reconstruction similarity. For ease of comparison, we also provide the overall rank of each tokenizer considering both metrics. The results in Table that tokenizer remains stable across different noise levels, demonstrating strong robustness. Notably, TASTE achieves the best ASR-WER WER under the noisiest condition. This suggests that the underlying ASR system is not limiting factor for the general applicability of TASTE; rather, it serves as an effective and reliable source of semantic information for the reconstruction.
A.1.5 DETAILS ON SALMON AND STORYCLOZE
Our detailed results on SALMON and Story are reported in Table The introductions of the two benchmarks SALMON and StoryCloze are described below.
SALMON for Acoustic Evaluation SALMON offers a comprehensive set of metrics designed to evaluate SLMs in multiple dimensions. In summary, each test sample consists of positive sample and a negative sample. The negative sample differs from the positive sample by having some
Table 9: The evaluation results on SALMON and StoryCloze of different SLMs, and BG means background. We report likelihood-based accuracy on SALMON (acoustic aspect) and StoryCloze (semantic aspect). The baseline (S3 token) is conducted by joint speech-text modeling with the S3 token as speech tokenization.
SALMON (AcouSTIC CONSISTENCY)
STORYCLOZE
METHOD
LoRA
BG
Sentiment
Speaker
Gender
Room
(domain)
(rand.)
SSC/ISC
Previous Work
TWIST 1.3B
(Hassid et al.
2023
X
61,5+334
69.0=33
69.5+3.3
59.043.5
55.5+3.5
60.5+3.5
52.4408/ 224+08/706--00
70.6
TWIST 7B
lassd et al
61.51334
71.0132
70.0.322
62.013.4
55.513.5
60.51.55
553108/74.11
10.
Spirit Nguyen a 2115
54,5+33.
69.543.3
67.043.3
54.543.5
53.543.5
55.5+3.5
610-08/829-00.
Spirit LM Expe. ([Nguyen
2025
73,513.1
81.0188
85.022
54.513355
56.01.3
64.0344
569108/ 75.41..7
Ouri
Baseline (83 token)
49.5+3.5
48.443
48.843.5
49.543.5
55.343.5
49.5+35
544-08/630-08
TASLM (token)
59,0.3..5
70.5332
6101134
520135
$40135
6121008/899105
TASLM IB (emhedding)
57.5
67.0+33
75.513.
50.0135
47.03.5
49.0:3.5
640:108/8955-0
18

---

Published as a conference paper at ICLR 2026
segments altered. These alterations include changes in speaker, gender, environment (e.g., room acoustics), or sentiment in the middle of the utterance. The SLM serves as an anomoly detector that aims to distinguish between the pairs of positive and negative samples. The distinction is based on the likelihood score given by each SIM,, which is then evaluated with the overall precision between the ground truth and the prediction.
StoryCloze for Semantic Evaluation To evaluate the SLMs' ability to comprehend semantic coherence and logical reasoning, we employ the spoken version of StoryCloze test (sSC) and the Topic StoryCloze test (ISC) assembled by Hassid et al. (2023). Assessment of narrative understanding involves presenting a four-sentence story setup, followed by two possible endings. These tasks require the model to select the most appropriate conclusion, thereby testing its grasp of causal and temporal relationships within narrative. Similarly to SALMON, we measure the accuracy of the distinctions based on the likelihood scores.
Discussion on SALMON and StoryCloze On SALMON, we observe that our TASLM falls short for background-relater attributes (Room, Background) where the samples are added with environ- mental sounds (echoes, instruments, background noises from FSD50K). Since TASTE tokenization focuses natural speech and has not being trained on audio with environmental sound and noise, it may not be able to convey such information. However, on the speech related attributes, such as gender and speaker, our TASLM performs much better and is comparable to other SLMs. On StoryCloze, our TASLM successfully retains its semantic capability with effective joint modeling on TASTE, leading to the best results among all pretrained SLMS.
A.1.6 REPORT OF STANDARD DEVIATIONS
We report the standard deviations of our tables in the main text to allow further investigation.
Table 10: Results with standard deviations of Table
Method
Hituat
WERL
QUALITY
DNSMOS
VISQOL
SIMILARTTY
Con. Spkr. MUSIRA
Ground Trush
256k 2.07
4.0910.35
3.84:026
156
1300 5.11
3.26.0.4
3,6610.25
DALLO.IO
Facedas Difosser al. 3023)
3000 266%-000
2,3540.53
345=0025
3.020027
03660002
0.78=1007777
25
500 5.28-0.11
1,27+0.05
2.9-0.17
2,00-0.44
094-6.003
0.35+00:9
SpeechTokkmizer Zhung et 2024
2000
3.00::00.88
3,5610.43
3.601:0.88
3.65.0.22
0.97000022
0.80.0.06
53.9122.9
4000
2.5% 3,9020,36
3.766:0225
4,0320.17
0.96-000112
0.9220.444
Musii (Dlossee
1000 3:18:0009
3,0080.37
3.0000.0
3.62=00.6
D.S2=0006
67.6-19.8
53 wikan (lopline) Du al 2024a
600 3.0%-0.09
4.18+6027
3.00-0.24
3,30-6.26
0:96-0.002
0.82-0:09
20.2-1700
Text-only (baseline)
55
5.91
4,3110.16
4.11.1022
2.44.0.23
0,57.0.006
42.6177.1
TASTE (ours)
150
4.4%==0.11 4.2920.18
4.10.0.22 3,05:0.26 0:91:00000
0.5020:08
08.3217.1
Table 11: Results with standard deviations of Table 2
Finetuned base
CONTINUATION
LIKELIHOOD
Method
persmoiers
GPT-4o
UTMOS
Human
SALMON StoryClore
Ovzrall
Carcade
Ciscade (LLaMAS2-18")
1.15
42550222
400:1128
Ciscade (11.MA2-78")
1.3
4.55025
398+1229
Spaken
TWIST 1.38
at 2023
1.98/1.30 1.38""
1
3.25+0048
195+101
62.5+114
61.5=0.5
620+0.7
TWIST 78
Hisso d 2izes
7B/ 78"
144el1.70
5.27+0.55
20400.91
63.4+.4
647:0.5
6417
Spiri
Npnyon
78"
2.79::1066
3/4.10.11
2.38:081
19.1114
720:0.5
65610.7
Spiri 1M
Expr. Nguyon et 2025
7B/
78"
1:001103
5.40+-0.30
241+006
690+1.3
66:200.5
676+077
Binehiine (53 loken)
45M7
1.36"
133.00.77
40410.27
2.
302114
587166
"S4.1003"
TASLM (okea)
1.38
3.0811.37
4.0710.8
39511130
608114
76.520.5
TASLM 1B (embed)
45M/1.30
3.161133
4.221221
4.16.1120
57.7144
76.710.5
67210.7
"IL.M633 IR MI2 "LLaMAAR, 10P71.38
Table 12: Results with standard deviations of Table 3
Method
Mode
Q.b
Mini-Ouii 0.5B(T-))
T
31.3609
396628
Mini-Dmii 0.58 Na We 20046
T+A
4.5-05
116118
'A-hiin 76 Res
3599-1
Mushi Deloasr al 2024
T4A
LLaMA3. 30 Inttant
T
60.4211
717426
Liama Omni 33 [Fng at 2024
T1A 35.1111
673122
CLaMA32-10 110
T
510129
1B (cmbed))
T1A 27.110
57.6119
Twe spot loveba lomtn wnastaa saittos movvene.
19

---

Published as a conference paper at ICLR 2026
A.2 TRAINING DETAILS
A.2.1 HYPERPARAMETENN AND CONFIGURATIONS
We separate the training process into the two phases: deriving TASTE tokenization and conducting spoken language modeling with TASTE. In the tokenization phase, only the Aggregator, Quantizer, and the UnitDecoder is trainable. We use the Adam (Kingmal 2015) optimizer and the learning rate set to 0.0016. The batch size set to 160 seconds on each of the 8 NVIDIA A6000 GPUs we used. Note that in the first 2 epochs the quantization is not applied. From the beginning of the third epoch, quantization applied and the Quantizer starts to be updated. We train the TASTE tokenizer for epochs, which takes about 2 days for learning, with the learning rate gradually decayed.
As for the spoken language modeling training phase, we use the Adam (Loshchilov Hutter 2019) optimizer, the Consine scheduler with the learning rate set to le-5. We use 8 Nvidia A6000 GPUs for training. The total batch size summation over the GPUs is set to 768 samples with the gradient accumulation steps set to 2. To reduce the memory overhead and the computational cost, employ of olat mixed precision during training. Tools such as DeepSpeed (Rasley et al., 2020) and Liger Kernel (Hsu al. 2024) also applied speed up the fine-tuning process of the SLM..
A.3 EVALUATION DETAILS
A.3.1 HUMAN EVALUATION
We conduct human listening tests through Amazon Mechanical Turk. In each experiment, randomly select the same 20 samples from each method; and for each sample we collect more than 10 evaluation scores across different human evaluators.
MUSHRA
Table T] we have shown our result of the MUSRHA human listening test (Series)
2014). Following Zhang al. (2024), we conduct the evaluation with hidden reference but without a lowerpass-fittree anchor. instruct evaluators to rate the perceptual quality of the given samples with respect to the ground truth scale of to 100.
Speech Continuation MOS In Table 2 mention that we have conducted the human listening test to evaluate the overall performance the speech continuations. Here, we present the instruction for human speech continuation MOS evaluation follows:
Instruction for Human Speech Continuation MOS Evaluation
this test, each sample will contain a short audio clip called "prompt" (3 seconds) and longer audio clip called ppoompt+continaation" (~115 seconds).
You will be asked to rate the speech quality of the "prompt+continuttoon" audio clip, specifically focus the "continuation" part.
The rating should be based on how likely you think that the long audio a proper continuation of "prompt" audio.
Specifically, the rating should be based on the following seale:
Bad The "continuation" is not distinguishable
natural.
Poor The "continuation" 25% distinguishable.
3: Fair The "continuation" 50% distinguishable and natural.
4: Good - The "continuation" is 75% distinguishable and natural.
5: Excellent The "continuation" is distinguishable, meaningful, and natural.
Distinguishable means that the words
"continuation" " distinguishable.
Natural means that the "continuation" sounds like real human voice and natural continuation of the prompt without considering the content of the speech. Meaningful means that you only distinguish the words but also understand the meaning of the whole "prommt++continaatonn..
20

---

Published as a conference paper at ICLR 2026
A.3.2 GPT-40 FOR MOS EVALUATION
As introduced in Section 5.1 we use GPT-4o to assign MOS scores to the speech continuation
results (Chiang & Lee 2023 Lin al. 2024). Here, we describe the detailed procedure. First, whisper- -arre-re--- v3 is applied to transcribe the generated speech. Then, given the transcription, the text content from the prompt audio, and the instruction template, GPT-4o can produce score between and 5. The instruction template is provided below:
Instruction Prompt for GPT-4o MOS Evaluation
The task is evaluating the relevance and likel ihood of the predi cted text ination, gi the text prompt. You shculd also consicer whether the meaning of the text continuation is making sense. The text prompt is:
"[prompt) and the text continuation is "[content)"
You must give overall rating from to 5. The rating guideline is below:
1: The text continuation is very unlikely and irrelevant to the text prompt.
2: The text continuation is unl onlikely and marginally relevant to
the text prompt.
3: The text continuation mocerately likely and relevant the
text prompt.
4: The text continuation is likely and relevant to the text prompt.
5:
The text continuation very likely and highly relevant.
You should take the following steps to provide the score: First: briefly analyze the sample with the above definition.
Second: MUST follow the output format as: I would rate the score as
A.4 TACKLING THE VOCABULARY MISMATCH
The vocabulary mismatch problem lies in the fact that the vocabulary sets are different between the ASR and the LLM, and TASTE is aligned with the text transcription tokens from ASR. Consider that given text transcription v and the vocabulary sets of ASR and LLM denoted as Vasr and Vllm ASR tokenized sequence aar [ugar, 122 UN u E yaar and the LLM tokenized sequence Illm سللرر Ilm Jllm 1.Ilm E yllm can be different in terms of token ids and sequence lengths. Since the TASTE token and embedding are aligned with asr we need to derive method to align them with l for text-aligned speech-text modeling. Notice that u and u both represent v. propose to mitigate the issue through word-level grouping. averaging, and aligning, detailed in Algorithm]] I By crafting TASTE speech tokenization into the word level, we are able to align it with the text tokens of the LLM, denoted q, ž. In practice, also adopt the word-level averaging technique during the TASTE tokenization training phase, ensuring that the word-level TASTE tokenization facilitates high-quality reconstruction.
21

---

Published as a conference paper at ICLR 2026
Algorithm 1 Aligning TASTE with LLM Tokenization via Word-Level Techniques
1: Initialization:
Text transcription e = [word,, word2,.... wordw] ASR tokens of the transcription wast [ 22
N
TASTE embedding = [21, 22..... 2N)
LLM tokens of the transcription a 1u 22 Jm
mlm MM
2: procedure W WORDLEYEL
e 2, v )llm)
3: Since a a token sequence represents V, we can easily group by words:
4: U
"grouped
[uu u2",, ") (044) J2
(v (UN- 1. UN N
-Group wh by the words of
word; 5: With the word-level grouping from yasr
word2
wordw
gmaped we can group TASTE embedding as well: (24)2,.... (2N-1, 3
6: Egrouped < [(21, 82, 23)1,
7: Finally, we can group Im following the similar procedure of grouping utt:
8: willa "grouped (v 1/2 1,
(us [l J2...... (M -M Ms )w)
wonky 9: Due to the vocabulary mismatch, the grouping of groopped is different from w end procedure procedure NORDLEVELLVVRAGINNG grouped)
wordh
word2
grouped ggroppdd
11:
-
Initialize a new sequence
13:
for word group index i 1--10 to W do
word group (žj.... 2x) Egrouped
15: [[j.b] <- Average(($), ...,, )
Average the word group
16:
append Z(z:k] to z
17: end for
18:
Resulting in word-level TASTE embedding E RRW Wxx, where W is the word length of
19:
end procedure
20: procedure ALIGWWORRLLVEEEEEEDDDNGNWIIE
a lim grouped > Initialize a new sequence
21:
2 x-D
for word group index 1 to W do
23: word group (u)! ...., )l t- v lim
grouped
24:
M Length((u! ....., rxk {m))
Get the length of the word group.
25:
for 1 M do
>> add M æ|i] into the aligned sequence
26:
append z|1 to end for
27:
28:
end for
end procedure
30:
return The I.I.Maligned word-level TASTE embedding E and its codes form q
A.5 THE WEIGHTED SUM MECHANISM FOR MODALITY FUSION
The speech decoder takes the text-aligned speech embedding & and the text embedding input conditions. As depicted Figure in practice we perform weighted sum mechanism to obtain the final fused embedding Zjoint. The weights to fuse the two modalities are represented by two learnable parameters, denoted as W'sp and Wix- The whole process of modality fusion can be denoted follows:
' v' normalize( &) (nnomalizee Pru] softmax([ugp Wnr)), Zjoint Psp 2 + Pex V.
A.6 DISCUSSION ON THE USAGE LLM
We discuss our usage of LLM following the conference's policy. We use an Al assistant (ChatGPT specifically) to polish English prose, including grammar correction, wording refinements, consistent terminology and hyphenation, and minor restructuring for clarity and flow. The assistant suggests alternative phrasings, section bridges, and standard disclosure/impact wording based on author- provided content. It does not generate novel ideas, claims, analyses, figures, code, results, and it does not access proprictary data. All technical content and conclusions are our own, and we review and edit all Al-assisted text and take full responsibility for the final manuscript.
22


Reviews:
[{"review_id": "OSPOQMrXUk", "reviewer": "Reviewer_wxPH", "summary": "The paper proposes a novel tokenizer, called TASTE, specifically designed for text audio generative modeling. The motivation is that speech tokens generally have much longer length for the same content compared to their text counterpart. The proposed approach generates speech tokens of equal length as textual tokens. Technically, this is achieved by first passing the audio through a pre-trained speech encoder (Whisper). From this speech encoder, both last layer and shallow hidden states are extracted and fed into an attention based aggregator as key and value respectively. The queries for this aggregator are initialized by a text transcription of the speech. Since the queries are based on text, the final speech tokens match in length to the text tokens. The paper provides an intuition that these speech tokens focus on learning paralinguistic information as they are supposed to be used along with the corresponding text tokens. For decoding, a speech decoder uses both the text and speech tokens and decodes them back to speech using a vocoder. The tokenizer is trained on a speech resynthesis task wiht a cross entropy loss to predict the speech unit and commitment losses on the residual codebooks.\n\nApart from the tokenizer, the paper also proposes a speech language model which integrates the TASTE tokenizer, called TASLM. There are two variants - one which trains on the discrete codebook indices similar to a traditional language model and one which directly operates on the continuous latent embeddings.\n\nExperiments are done by training on Emilia and LibriTTS datasets and evaluating on LibriTTS test split. The paper evaluates the tokenizer for speech reconstruction, speech editing and  TASLM on speech continuation, likelihood-based next-speech selection and spoken question answering.", "questions": "- As the paper emphasizes that the goal for the speech tokens is to learn paralinguistic information, is there any sort of text encoding done to learn semantic information from the language? \n- It is unclear to me how the decoding is done from TASLM output. Since the decoder requires both text and speech, how is it achieved?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes an interesting way to formulate tokenizers by using multimodal speech text tokens to provide complimentary information. This can help in significantly reducing the number of tokens needed to model speech.\n- It is very interesting to see the behavior of this model in text-aligned speech editing. The model seems to be able to take paralingual information from one speech sequence and transfer it to another containing the same content. It would be interesting to pursue on this further and understand whether this type of editing can be done across very different voice types, like belonging to different genders etc.", "weaknesses": "- The proposed TASTE tokenizer tends to learn text aligned speech embeddings. One limitation of such an approach would be for sounds that occur commonly in speech but may not have a textual counterpart. For example, if a person laughs, it is unclear how the corresponding textual token would be retrieved. The approach may _collapse_ the laughing related information to the neighboring speech tokens if they have textual counterparts.\n- The paper does not evaluate on the sWUGGGY and sBLIMP ZeroSpeech 2021 benchmark which is standard in literature [1,2]. These benchmark test the lexical and grammatical knowledge of the model to understand how well model can identify real words vs phoenetically similar non-words. I believe this might be a limitation of this tokenization approach as it needs a text transcription of the speech input.\n- The paper does not explain its underperforming benchmarks. For example, In table 1, the proposed method has significantly worse WER compared to other tokenizers but the paper does not dive deeper into it. Instead, the text focuses on how WER is better than text-only baseline (343-344). Similarly, for table 2, it would be good to understand what factors contribute to the underperformance on SALMON benchmark. In table 8, this underperformance seems to happen for most of the experiments in SALMON and some discussion around it would be helpful.\n- While the paper cites [2], it is unclear why it is not compared against the experiments of the paper. \n- Related work section is missing from the main paper. While it is present in the appendix, it should be a part of the main paper. \n\nReferences:\n1. Spirit LM: Interleaved Spoken and Written Language Model, ACL 2025\n2. Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback, ACL 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dHYSBj5ACO", "reviewer": "Reviewer_8SAU", "summary": "The paper presents TASTE, a text-aligned speech tokenization approach that uses transcription-guided cross-attention and RVQ to better align speech and text representations. The goal is to address the speech-text alignment mismatch common in spoken language modeling. While the motivation is clear and the design is coherent, the method largely builds upon existing codec frameworks with a text-conditioned aggregation step. Experimental results are competitive but not stronger than prior models such as SpeechTokenizer, Mimi, and EnCodec, and the evaluation omits more recent baselines. Overall, the contribution is conceptually sound but represents a modest advancement without clear empirical evidence of improved alignment and performance.", "questions": "1. Please see the Weakness section for points where additional analysis, clarification, or experimentation would strengthen the paper.\n2. Can the authors provide clearer evidence that the observed improvements come from genuine text-speech alignment rather than architectural or bitrate differences? Specifically, can they (i) compare with stronger recent baselines under matched bitrates, (ii) justify the representational choice of K = h(L), V = h(l) with empirical analysis, and (iii) evaluate robustness under noisy, multilingual or imperfect ASR to confirm practical applicability?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Using dynamic token frequency for better alignment with text is an interesting and well-motivated approach. It directly targets the speech-text mismatch that often limits speech tokenizer alignment.\n\n\n2. Deriving speech tokens based on transcription-guided cross-attention is an interesting design choice that offers an interpretable mapping between continuous speech features and discrete text-aligned tokens.\n\n\n3. The text-aligned editing demonstration is interesting. Swapping word-level TASTE tokens to transfer duration and other attributes in a controlled way is well-motivated and could inspire future work on speech style transfer or voice cloning.", "weaknesses": "1. The core idea of using text to guide lower-frequency speech tokens is sensible, but similar strategies have appeared before. Prior works have reduced speech token rates or trained tokenizers with auxiliary text objectives/distillation. The paper should better position TASTE relative to SpeechTokenizer, TWIST, DM-Codec, TadiCodec, and other text included tokenization approaches. It should clearly delineate what is new beyond the addition of a cross-attention aggregator and RVQ. The claim that TASTE is the “first end-to-end reconstruction objective to learn joint tokenization” is not fully substantiated because the distinction from prior work remains unclear.\n\n2. The experimental evaluation is constrained. In Table 1, TASTE is compared only against SpeechTokenizer, Mimi, and EnCodec. While these are recognized baselines, they are relatively older and weaker compared to recent models such as WavTokenizer, DM-Codec, FACodec, and BigCodec, which have shown superior reconstruction performance. Similarly, Table 2 compares only to TWIST and SpiritLM, and should include newer SLMs approaches (Cosyvoice2, SparkTTS, and Llasa). Without such comparisons, the reported results lack convincing evidence of improvement. \n\n3. TASTE does not outperform existing baselines (e.g., SpeechTokenizer, and EnCodec) across key metrics (WER, ViSQOL, Duration Consistency, Speaker Similarity, and MUSHRA). This raises questions about the practical benefit of the more complex and resource-intensive TASTE training pipeline. The paper repeatedly attributes its lower performance to “lower bitrate,” but this argument is unconvincing without matched-bitrate experiments. The authors should directly compare TASTE with low-bitrate variants of established codecs (e.g., DAC, WavTokenizer, XCodec2, BigCodec). Without such controlled comparisons, the main claim of improved speech tokenization remains weak.\n\n4. The design choice of setting the aggregator's keys as last-layer encoder features (K = h(L)) and values as shallow-layer features (V = h(l)) is based on questionable representational assumption and not supported with experiments. The paper assumes h(L) captures alignment cues while h(l) preserves acoustic detail, but offers no empirical analysis (e.g., attention visualizations or layer ablations) to support this. It is unclear whether the observed effects are from meaningful alignment learning or coincidental correlations. \n\n5. The ASR dependence of TASTE limits general applicability. The model requires accurate transcriptions to derive aligned tokens, yet there is no analysis of robustness under realistic noisy ASR or low-resource conditions. Without such tests, the practical viability of TASTE is uncertain. Moreover, The evaluation is also restricted to English corpora (LibriSpeech, Emilia). This ASR dependence also limit TASTE's applicability to spontaneous or multilingual speech. Given that TASTE explicitly depends on text-speech alignment, it is unclear whether it generalizes to languages with more complex non-phonemic orthography.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iU9lB5FHoY", "reviewer": "Reviewer_tUyK", "summary": "This paper introduces TASTE (Text-Aligned Speech Tokenization and Embedding), a method for learning text-aligned speech representations that directly bridge the modality gap between text and speech. The approach aggregates speech features using an attention-based mechanism guided by text transcriptions and is trained end-to-end with a speech reconstruction objective. This alignment produces compact and semantically consistent speech embeddings that preserve paralinguistic information while reducing sequence length. By adapting a pre-trained text language model with low-rank adaptation, the resulting joint model can process and generate speech coherently without additional alignment heuristics. Experiments show that TASTE improves speech continuation and next-speech prediction performance compared to existing spoken language models, demonstrating its effectiveness for unified text-speech modeling.", "questions": "- How exactly is the attention map used in the alignment process, and how sensitive is performance to layer selection for keys and values?\n- What is the computational overhead of training the alignment module compared to standard speech tokenizers?\n- Could the model be extended to use unpaired speech in a semi-supervised manner to improve robustness?\n- How does TASTE handle inter-word silences or pauses during alignment and embedding generation? Are such regions ignored, or do they get represented implicitly in the embeddings?\n- What is the rationale for using the last hidden layer as key and shallow hidden layers as value? Have other layer combinations been tested, and how sensitive is performance to this choice?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper tackles an important problem in spoken language modeling: how to represent speech and text within a shared embedding space for joint generation. The proposed method is conceptually simple yet effective, combining attention-based feature aggregation with an end-to-end reconstruction objective to achieve text-aligned speech embeddings. The learned representations capture both linguistic and paralinguistic information in a compact form and integrate smoothly with pre-trained text LMs through lightweight adaptation. The experimental results are consistent across multiple tasks, and the qualitative analyses illustrate interpretable alignment behavior. The paper is clearly written and provides a strong empirical demonstration of how modality alignment can improve spoken language modeling.", "weaknesses": "While the paper is technically sound, several aspects of the design and analysis remain underexplored. The attention-based aggregation is central to the method, yet its behavior and learned alignment patterns are not clearly analyzed, making it difficult to understand how alignment emerges during training. The paper lacks ablations isolating the effects of key design factors such as the chosen encoder layers, the reconstruction objective, and the LoRA adaptation on the joint model’s performance. Moreover, the evaluation focuses mainly on speech continuation and next-speech prediction, without examining general spoken understanding tasks or perceptual quality metrics that would better validate the claim of improved paralinguistic preservation. These gaps limit the interpretability and completeness of the study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kE7MUBy0TL", "reviewer": "Reviewer_ajFq", "summary": "The paper introduces TASTE, a text-aligned speech tokenization and embedding method aimed at simplifying joint text–speech modeling. Given a speech–text pair, a frozen Whisper encoder provides shallow and last-layer features, an attention-based aggregator uses text tokens as queries over these features to produce a text-length speech representation, which is discretized with RVQ, and finally, a decoder reconstructs speech from the text and the aligned speech embedding for a reconstruction objective.", "questions": "- Please detail the decoder fusion: where are the gates/weights applied between $v$ and $\\hat{z}$, and how are they trained and stabilized?\n\n- Can you report out-of-domain continuation and SALMON/StoryCloze on non-LibriVox corpora, and include dedup checks between LibriTTS and LibriSpeech?\n\n- What are latency/memory costs for tokenization/decoding, and can TASTE stream in real time?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The key contribution is to align speech tokens to the text sequence during tokenization and show that this enables simpler, stronger joint modeling at low bitrate. The idea is clean and the empirical gains on continuation with a small LM are practically meaningful for SLM research.\n\n-  TASLM (1.3B + LoRA) outperforms prior 7B SLMs on continuation (GPT-4o, UTMOS, Human) and is competitive on SALMON/StoryCloze. The LoRA-only fine-tuning and word-level handling of ASR/LLM vocab mismatch increase practical relevance.\n\n- ASR robustness ablations (GT vs ASR transcripts; Whisper vs Parakeet) show limited sensitivity, and the word-swap editing demo nicely evidences alignment of paralinguistics to lexical units.", "weaknesses": "- While key idea of the proposed method is clean, novelty is rather moderate. Text-speech cross-attention has been used for fine-grained TTS control, and recent joint SLMs alleviate length mismatch via interleaving (e.g., Spirit-LM) or lower-frequency tokenizers (e.g., moshi).\n\n- Key \"semantic MOS\" relies on GPT-4o over ASR transcripts. Despite ablations, this double-black-box can amplify mistakes, such as prosody -> ASR error -> GPT score. Human semantic judgments or text-only LM plausibility on ASR outputs would be a stronger counterpoint, and code-switch/noisy-ASR conditions would probe robustness more realistically.\n\n- The autogressive \"weighted sum\" between text and aligned speech in the unit decoder is under-specified. I would suggest the authors include equations/ablation to clarify.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4DI4fQVaRE", "reviewer": "Reviewer_FzoP", "summary": "This work proposes TASTE, a speech tokenizer that enables the alignment with semantic text tokens. It can generate highly compressed audio tokens via minimization of the speech reconstruction loss. The experiments are performed to show that TASTE successfully reconstructs the audio and helps LLMs easily finetuned to be SLMs (Spoken Language Models).", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work proposes a novel highly compressible speech tokenizer named TASTE that can reduce the modality gap between text and audio pairs. This is practically useful to reduce the length of the token sequence for a given audio with little loss of reconstruction quality. \n\n2. TASTE helps finetuning LLMs to SLMs via Low-Rank Adaptation (LoRA). This is practically useful to speed up the development of SLMs or omni-modal LLMs.", "weaknesses": "1. One of the most practical uses of audio tokenizers may be training of SLMs, as this work experimented. \n- Thus, it may need to be studied whether the proposed tokenizers can lead to SOTA performance of SLMs. \n- For example, TASLM’s performance can be compared with that of SOTA SLM models such as LLaMA-Omni 2. \n\n2. The generalization of this tokenizer should be discussed. \n- What if the domains of text and audio corpora are changed? Do we need to train again? \n- Since the proposed tokenizers are forced to learn the joint representation of both “semantics” and “phonetics”, the learned tokenizer may be much more variable according to the training sets than the one learned only focusing on “phonetics”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)