Latent Speech-Text Transformer
Authors: Yen-Ju Lu
Yashesh Gaur
Wei Zhou
Benjamin Muller
Jesus Villalba
Najim Dehak
Luke Zettlemoyer
Gargi Ghosh
Mike Lewis
Srini Iyer
Duc Le
Venue: ICLR 2026 Oral (2026)
Keywords: Speech–Text Models
Latent Patching
Multimodal Alignment
Large Language Models
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce Latent Speech-Text Transformer, which patches long speech token sequences into latent units, improving text–speech transfer while cutting pre-training and inference compute, and significantly outperforming existing speech-text LLMs.

Abstract: Auto-regressive speech–text models pre-trained on interleaved text tokens and discretized speech tokens demonstrate strong speech understanding and generation, yet remain substantially less compute-efficient than text LLMs, partly due to the much longer sequences of speech tokens relative to text. This modality imbalance disproportionately allocates pre-training and inference compute to speech, potentially hindering effective cross-modal alignment and slowing performance scaling by orders of magnitude. We introduce the Latent Speech-Text Transformer (LST), which aggregates speech tokens into latent speech patches that serve as higher-level autoregressive units. This design aligns the sequence-modeling granularity between speech and text while improving computational efficiency. The resulting patches can align with textual units to facilitate cross-modal knowledge transfer and compactly capture recurring acoustic patterns such as silence. Across story-completion benchmarks under both compute-controlled and data-controlled settings, LST consistently improves speech accuracy while also improving text performance, achieving up to +6.5% absolute gain on speech HellaSwag in compute-controlled training (+5.3% in data-controlled training). Under compute-controlled scaling from 420M to 1.8B parameters in a near compute-optimal regime, gains grow with scale, and improvements persist up to 7B parameters under fixed-token budgets. These benefits extend to downstream tasks: LST stabilizes ASR adaptation and reduces the effective autoregressive sequence length during ASR and TTS inference, lowering computational cost without degrading reconstruction quality. The Code is available at https://github.com/facebookresearch/lst.

Published as a conference paper at ICLR 2026
LATENT SPEECH-TEXT TRANSFORMER
Jesus Villalba', Najim Dehak', Luke Zettlemoyer", Gargi Ghosh', Mike Lewis", Srinivasan Iyer Duc Le
Yen-Ju Luto, Yashesh Gaur?, Wei Zhou?o Benjamin Muller", Jesus
'Center for Language and Speech Processing, Johns Hopkins University Meta Superintelligence Labs
Joint last author
ylu 150 jhu edu, fsviyer, duchoangle, e) @meta. com
ABSTRACT
Auto-regressive speech-text models pre-trained on interleaved text tokens and discretized speech tokens demonstrate strong speech understanding and genera- tion, yet remain substantially less compute-efficient than text LLMs, partly due the much longer sequences of speech tokens relative to text. This modal- ity imbalance disproportionately allocates pre-training and inference compute to speech, potentially hindering effective cross-modal alignment and slowing per- formance scaling by orders of magnitude. We introduce the Latent Speech- Text Transformer (LST), which aggregates speech tokens into latent speech patches that serve as higher-level autoregressive units. This design aligns the sequence-mose modellig granularity between speech and text while improving compu- tational efficiency. The resulting patches can align with textual units to facilitate moosl knowledge transfer and compactly capture recurring acoustic pat- terns such as silence. Across story-completion benchmarks under both compute- controlled and data-controlled settings, LST consistently improves speech accu- racy while also improving text performance, achieving up +6.5% absolute gain on speech HellaSwag in compute-controlled training (+5.3% in data-controlled training). Under cumpute-controlled scaling from 420M to 1.8B parameters in a near compute-optimal regime, gains grow with scale, and improvements per- sist up 7B parameters under fixed-token budgets. These benefits extend downstream tasks: LST stabilizes ASR adaptation and reduces the effective au- toregressive sequence length during ASR and TTS inference, lowering compu- tational cost without degrading reconstruction quality. The code is available at /ttps:://iihub..co/ hwt.aauc..cooaaaaoooooeeeaaaaaaa
INTRODUCTION
Inspired by the strong zero- and few-shot understanding and generation capabilities of large auto- regressive textual language models with billions of parameters that are pre-trained on trillions of tokens, Lakhotia al. 202T) introduce the task of Generative Spoken Language Modeling (GSLM) a.k.a Textless NLP, where raw speech is encoded as sequence of discrete tokens based on dictio- nary of quantized speech features, and an auto-regressive language model (LM) is trained on these tokens with Next Token Prediction (NTP). While initially successful, Cuervo & Marxer (2024) esti- mate that this approach would require up to three orders magnitude more data obtain equivalent capabilities textual LLMs, largely owing to the same information requiring significantly larger number of speech tokens to represent compared to text. This increased sequence length also means that these models utilize considerably more compute during inference process the same amount of semantic content compared to text.
To improve scaling properties of large speech models by taking advantage of the comparatively larger corpus of web text compared to speech, recent efforts have leveraged transfer learning from textual modalities in the form of warm initialization from large pre-trained text models (Hassid et al.) 2023), pre-training with interleaved speech-text data (Nguyen et al. 2025). and modeling
Work done while Meta.

---

Published as a conference paper at ICLR 2026
52.5
52
52.2%
HalnSwan Accurary (N0)
HollaSwag Accuracy (N6)
50
47.5
47.0%
68
46
42.5
02
39.0%
40.2%
37.5
(peeech)
(Speect)
(Sowech)
(Spaech)
(Taat)
(iac))
250k
300k
350k
400kc
450k
500k
1608
2008
2308
2608
2908
330B
Total Training Iterations
Total Training Tokens
Compute-controlled: fixed training iterations
(b) Data-controlled: fixed data budget
Figure I: Comparison of LST and Baseline on HellaSwag story completion under two experimental setups, (a) compute-controlae same number of training iterations and (b) data-controlled: same amount of training data.
speech and text in multiple streams leverage the textual chain of thought "inner monologue" (Défossez et al., 2024). All these works attempted to some extent to achieve representational align- ment between and speech, where a perfect alignment means the model can treat the two modal- ities interchangeably without any performance difference. Despite this, there remains large gap between text-to-text and speech-to-speech performance on the same benchmarks, highlighting the incompleteness of the alignment. We hypothesize that the severe information-densic mismatch between speech and text tokens is one of the primary factors hindering speech-text alignment.
To overcome the aforementioned challenges, we introduce the Latent Speech- Text Transformer (LST) based on the byte-latent transformer (BLT) architecture (Pagnoni et al., 2024), comprising an encoder that dynamically groups sequences of speech tokens into higher-level speech patches, a global traecch-ransfrrmer that auto-regres regressively models interleaved sequences of textual tokens and speech patches, and light- weight transformer decoder (Vaswani et al. (017) that maps patches back into speech tokens of dynamic sizes. Working in terms of speech patches allows the model to encode more content given the same training cost, makes inference more efficient. These speech patches can represent higher-level speech concepts or prolonged silences, and serve to level the information density between speech and text, thus making them easier to align (see Figure I..
We first demonstrate that LST models with fixed-size speech patching schemes similar to what Yu al. (2023) did with text, are able significantly outperform their non-patching counterparts. Such models are aware of the internals of patches without expending much compute in the process, in contrast with methods that expand the speech token vocabulary by applying subword tokenization, which yield poor downstream performance (Cuervo & Marxer 2024). We further improve the per- formance by introducing speech-patchinn based on textual alignment the word/subword levels, which crucially also includes patching large sequences of silences. Since this approach requires text-speech alignment timestamps during training and inference, we also introduce curriculum- based method to eliminate the need for such alignments during inference.
To summarize, this paper makes the following contributions:
(1) We show that LST improves performance in both data- and compute-controlled settings com- pared to prior interleaved speech-text models on speech versions of popular text understanding benchmarks such HellaSwag (Zellers al. 2019) (see Figure I]. while substantially reducing training and inference compute and enabling efficient downstream ASR and TTS transfer.
(2) We introduce latent speech patching speech sequences and analyze static, alignment-based, and curriculum patching strategics.
unified mechanism for compressing autoregressive
(3) We demonstrate that the benefits of LST persist and grow with scale from 1B to 7B parameters, indicating improved sample efficiency and more favorable compute-optimal scaling behavior for spoken language modeling.
Collectively, these contributions demonstrate that aligning the autoregressive modeling granularity of speech and text helps mitigate key barrier efficient and scalable spoken language modeling.
2

---

Published as a conference paper at ICLR 2026
Better
text
lext
Toxt
to
to
Latent Speech- Text Transformer
to
Speech
speech
speech
Siiance
Sieence
Potch Decoder
Patch
spoken
spoken
83
Encoder
Latent Representation
s4
word
s5
s6
spoken
HuBERT:
86
57
Figure 2: Latent Speech-Text Transformer (LSST). The model encodes BPE text tokens and Hu- BERT speech tokens into shared latent space. A Patch Encoder compresses local speech segments into patch representations, which are jointly processed with text tokens. A Patch Decoder predicts future speech tokens from latent representations, enabling alignment and transfer across modalities.
2 BACKGROUND
Generalized spoken language models (Lakhotia et al., 2021
typically comprise three components:
(1) a speech tokenizer model that maps raw speech waverform to sequence of speech tokens (Su...., 8n ). (2) decoder-only transformer model (Vaswani et al. 2017 with parameters that models the distribution of the next speech token given the previous context i.c. (s<s) and (3) a vocoder model that maps speech token sequences back to speech waveform, such as HiFi-GAN (Kong al. 2020
Speech tokenization. Approaches for speech tokenization include semantic tokens represented by cluster-ids obtained by k-means clustering of frame representations in Hubert (Hsu et al., [202)) acoustic tokens obtained as discretized embeddings from residual-vector quantization bottlenecks from self-supervised neural codec models (Zeghidour al., 2021 Défossez et al., 2024). as well as additional tokens for expressivity and also, combinations different token categories. In this paper, follow Hassid et al. (2023); Nguyen et al. (2025) and use Hubert tokens using codebook of 501 speech tokens at 25Hz. Unlike Nguyen et al. (2025 we do not need to deduplicate Hubert tokens as this is organically handled by the LST architecture.
Sequence Modeling. Similar to LLMs for text, speech token modeling typically done using large transformer decoder model using causal self-attention, to maximize the likelihood of sequences from a large speech pre-training corpus (D) an auto-regressive fashion:
L(D;0) = ΣΣ log po (si(8ee))
SED
Interleaved Data. Since speech sequences are longer and less compact that their corresponding text sequences, such models can require several orders of magnitude more data in order achieve performance comparable to text models (Cuervo & Marxer 2024). In order to bridge the gap, Nguyen et al. (2255) find that training on interleaved sequences of text and speech data directly correlates with improved performance. For a subset of the pre-training dataset that contains the textual sequence (fo.... ,, fm J. where text tokens are obtained using a tokenizer (we use the Llama 2 tokenizer (Touvron al. 2023) in this paper) and each text token can correspond to span of speech tokens, the model is trained on an interleaved sequence obtained by replacing arbitrary spans of speech tokens in the sequence sequence with text tokens separated by special modality tokens. This allows the same model to be used for S S-S, S-TT T and T-+T tasks. We discuss the process of producing interleaved data from parallel text-speech data in A.I.1
3

---

hash Published as a conference paper at ICLR 2026
lec MaY T9P
hagy
101 MF
Aligned
Patching
.Wrgg
ssla
AWM .w0g
stt
Static
Patching
(a) Static patching segments speech into fixed-size patches, while aligned patching uses Way2Vec2+CTC boundaries. (sw spoken word and sil silence.)
(b) Example of alignment from Wav2 + CTC, where purple markers indicate word boundaries, aligning the audio signal with corresponding text.
Figure 3: Illustrations of alignment and patching methods.
LATENT SPEECH-TEXT TRANSFORMERS
The core idea of the LST architecture is rugo-regressively model latent patches of tokens (us- ing global transformer), rather than individual tokens, similar in spirit to BLT (Pagnoni et al.) 2024) which models dynamic-sized patches of bytes. The transformation of speech/text spans to patches and vice-versa takes place with the help a light-weight patch encoder and patch decoder, and the entire model is trained end-to-end using the same token-level likelihood as before. Figure 2 illustrates this architecture specialized to the task of speech-text modeling. The majority of the compute expended in terms of FLOPs in the global transformer. which yields savings by oper- ating on information-dense speech patches instead of granular speech tokens. Latent patching per- forms bounded-contex, temporal aggregation, preserving phonetic discriminability while reducing sequence-level redundancy.
Patch Encoder. Similar to BLT, the patch encoder uses a series of sliding window self-attention and cross-attention layers to aggregate token representations into patch representations. In LST, we only patch spans of speech tokens using strategies described in Section 3.1 Note that a simple alternative patching is to subword tokenization methods like Byte Pair Encoding (BPE)
the speech tokens. This was also explored by Cuervo & Marxer (2024) and similar to them, failed to improve performance in our experiments (ablations in Section 5). Unlike BLT, do
embeddings, as they did not provide improvements in our experiments.
Patch Decoder. light- transformer is used as decoder and trained with NTP loss, with cross-attention layers inserted between every transformer layer. Each token attends to both the previously generated speech patches and text tokens to incorporate patch-level information (using cross-attention) well as sliding window of the past 512 tokens (using self-attention). Further ar- chitectural details of the local patch encoder and decoder, including attention roles and initialization, provided in Appendix A.2.I
3.1 PATCHING
Let X = [2o, .T] E gd'xd be speech token embeddings obtained using a learned embedding matrix applied speech tokens {50, ..... . ST} The process of patching maps X to shorter quence of patch embeddings Z [z ... ΦT| E RT' xd by aggregating local frame segments. For trame-index set P: C {O.... T). T..a patch embedding is formed via the patch encoder:
24 PatchEEn(Xp.)).
integrating the frames indexed by P, into single patch embedding. Different patching strategies correspond to different segmentation (P.).
Static Patching. Speech sequence is split into non-overlapping segments of fixed length p (patch size). Each patch token is obtained by the patch encoder from the embeddings in the patch:
P. (ip,-. ... min((i +)p 1, 1,7))-
4

---

Published as a conference paper at ICLR 2026
55.3%
15.7%
45.0
34.0%
42.5
HellSwaag Accuracy
HellsSwan Accuracy
33.0%
$7.5
35.3%
35.0
32.5
(ppaccht
(Speact) (Tan
(Speeen) (llcc)
(Sprect)) (Toco)
(Tbc
420m
630m
810m
1.1b
2.40
1.80
10k
15k
20k
25k
Model Sizes
Total Training Iterations
(a) Compute-optimal scaling (420M-1.8B) LST performs baseline, with gains increasing scale.
(b) Sub-optimal token scaling 7B. Comparison
70B tokens, below scaling-law optimum (==140B).
Figure 4: Scaling behavior on HellaSwag (S (S-S and T TT)
For 3 and input embeddings X
[ro; 31; 322 33, F4, I5, ... J. the first patch is ( x1 x22) the second (Ig, F4, 255 ). and so on. Each segment encoded into a single patch embedding 2; by the patch encoder. This provides a uniform compression ratio independent alignment information.
Alignment Patching. To better synchronize speech and text at the semantic level, alignment patch- ing leverages forced aligmment timestamps between speech frames and textual units (e.g. words or BPE tokens). Let A = {(bx. ) k= denotes the aligned frame ranges, where [b,, ex. spans the k-th textual unit. The corresponding patch is
Ps [bk [bk.
Frames outside text spans (e.g., silence) are grouped into separate patches (Fig. 3a If consecutive words align to [2. 4] and [6, 7]. patches are {x2, æ3, 21} and {Io, ær), with silence forming { x1} and {.r). We obtain alignments with Wav2Vec2+CTC (Baevski al.) 2020), yiclding one patch text unit and silence segment (Fig. 36). While this enforces cross-modal correspondence, requires an auxiliary model inference, introducing possible errors. Curriculum patching (Sec.
mitigates this by gradually shifting from aligned to static patching during training.
Curriculum Patching. Curriculum patching gradually transitions from alignment- static patching. Let P(u) E [0, denote the probability of using alignment at training step
1, P(u)=
u < Th
1
T2
0.
u 2 72-
At step u, choose alignment patches with probability P(u) and static patches otherwise. This retains alignment benefits during early training while enabling simple static-only inference.
EXPERIMENTAL SETUP
4.1 TRAINING AND EVALUATION DATASETS
Our pre-training data comprises a mixture of text and interleaved speech datasets.
Text. Our text training data consists of web and academic corpora, sourced from a subset of the Llama 2 pre-training collection (Touvron et al. 2023), totaling 1.8T tokens. We follow the LLaMA setup and apply its SentencePicce (Kudo & Richardson 2018) BPE tokenizer with a 32K vocabulary.
Speech. Our speech training data includes speech which is discretized into HuBERT tokens (501- entry codebook at 25Hz) together with paired text transcriptions. We use LibriLight (60k hours), People's Speech (30k hours), Multilingual LibriSpeech (50k hours), and Spotify (60k hours), de- tailed Ta All corpora are aligned using the Wav2Vec2 + CTC framework to provide token- level correspondence between speech and text (Figure 3h

---

Published as a conference paper at ICLR 2026
Table 1: Speech training datasets with total speech hours and the amount of Hubert tokens.
Dataset
Hours
Hubert Tokens (B)
LihriL.gght (Kahn et al. 2020)
44,
3.7
People Speech (Galvez et al. 2021)
14,699
1.2
Multilingual LibriSpeech (Pratap et al. 2020)
50,601
4.2
Spotify (Clifton et al. 2020a)
55,309
4.6
We evaluate the model on three benchmarks, where each dataset provides a narrative context and candidate endings, and the model selects the most plausible continuation. Together, they test narra- tive understanding, commonsense reasoning, and topic coherence. We evaluate the model in both speech-to-speech (S=+S) and text-to-text (T-+T) modes. For the speech mode, we apply Kokoro TTS model (hexgrad) 2025) to generate the speech for evaluation.
sHellaSWAG (HS). We create speech version of HellaSwag (Zellers et al. 2019) with Kokoro TTS. This benchmark evaluates everyday commonsense reasoning with spoken inputs and outputs. To ensure fairness, we generate the speech for prompts and responses independently and concatenate them afterwards, that all responses evaluated against the same speech prompt.
StoryCloze and Topic Story Cloze (SC/TSC). SC (Mostafazadch et al., 2016) and its topic-based extension TSC (Hassid al. 2023) are widely used in prior multimodal work (e.g.. Nguyen et al. 2025) to test coherence and topic-sensitive reasoning. We resynthesize both datasets with Kokoro TTS for higher-quality speech inputs.
Table 2: Evaluation datasets for story completion (MC = Multiple Choice).
Dataset
Format
Focus
HellaSwag (Zellers et al. 2019
1-in-4 MC
Commonsense reasoning
StoryCloze (Mostalazadeh
2016)
1-in-2 MC
Narrative coherence
TopicStoryCloze (Hassid et al. 2023
1-in-2 MC
Topic consistency
4.2 LST MODELS AND BASELINES
LST Models. We explore four patching strategies for speech tokens:
Static Patching. Fixed-length patches (4 HuBERT tokens) as in Yu et al. (2023). indepen-
dent of alignment and consistent across training/inferenee.
Aligned Patching, Uses Wav2Ve22+CTC boundaries (Fig. 3b). For each text span (b. ex). we form patch set PA- (bk,...,es) ex), synchronizing speech and text tokens (Fig. 3ab.
Mixed Patching. Randomly applies static or aligned patching per sequence, combining the robustness static patching with the fine-grained sync of aligned.
Curriculum Patching. Training shifts from aligned (first third) to mixed (middle) to static (final), leveraging early alignment while ensuring robustness to static-only inference.
Baselines. We include two speechLLM systems as baselines:
Base Spe Processes speech tokens directly with text tokens, without patching. similar to SpiritLM (Nguyen et al. 2025).
BPE SpeechLLM. Maps speech tokens into BPE units using SentencePiece tokenizer (Kudo & Richardson 2018) trained on 100k random speech sequences, replacing speech tokens with BPE-derived units]]
4.3 TRAINING SETTINGS
To balance modalities, set speech tokens to account for about one third (33%) of the total training data, while the rest (67%) is text-only. This ensures that the model benefits from large-scale text pre-
We use the k configuration as BPE baseline, larger vocahularies (5k, 10k) showed henefit.

---

Published as a conference paper at ICLR 2026
Table 3: Main comparison of LST models and baselines under the same computation budget
scheme. Each dataset reports both S-xS and T-yT.
Model
Tokens (B)
HellaSwag
StoryCloze
TopicStoryClov
Int.
Text
S-+
T-+T
S-S
T-+T
S-*S
T-+T
Base SpeechLLM
87
175
39.0
47.0
59.1
67.8
87.5
95.7
BPE SpeechLLM
95
190
38.0
47.5
58.0
66.4
87.0
93.5
LST (Static)
108
217
44.3
51.1
60.5
70.3
87.7
96.2
LST (Aligned)
108
217
42.7
51.7
60.4
70.4
86.6
95.7
LST (Mixed)
108
217
44.3
51.9
61.4
70.8
88.0
95.9
LST (Curriculum)
108
217
45.5
52.2
61.2
71.6
87.9
96.1
Table Main comparison of LST models and baselines under the same speech/text tokens scheme. Each dataset reports both S-S and T-T.
Model
Compute Savings
HellaSwag S-S
StoryCloze
TopicStoryCloze
(%)
T-+
S-+
T-T
S-+
T-+T
Base SpeechLLLM
40.2
49.6
60.2
69.1
87.5
95.2
BPE SpeechLLM
8.2%
39.4
48.4
58.3
66.3
86.5
93.9
LST (Static)
19.3%
44.3
51.1
60.5
70.3
87.7
96.2
LST (Curriculum) 19.7%
45.5
52.2
61.2
71.6
87.9
96.1
training while still maintaining substantial exposure to speech for effective multimodal alignment. For comparison, SpiritLM (Nguyen et al., 2025) adopts different composition: 33% pure speech, 33% interleaved, and 33% text tokens. Since SpiritLM starts from a text-pretraings model, the rel- atively smaller text fraction is sufficient. In contrast, when training from seratch, find that using 33% interleaved and 66% text tokens yields better performance (see Appendix (A.6).
5 RESULTS
5.1 PERFORMANCE UNDER CONTROLLED BUDGETS
Conpute-controlled. We fix the number of training iterations and per-step sequence budget so that all methods process the same number of units (baseline tokens LST patches). Table shows three trends on HellaSwag. First, patching increases the effective token budget, benefit- ing both modalities: Curriculum Patching improves T-T by +5.2 (47.0-+5222) and S-S by +6.5 (39.0~>4..5). Second, Aligned Patching is less effective at evaluation, since variable word spans often yield longer patches, reducing the test-time compute. Finally, Mixed and Currieulum patch- ing combine the advantages of shorter evaluation patches with alignment information, consistently outperforming Static and Aligned across datasets.
Data-controlled. Here we fix the data budget with the same amounts of speech and text tokens. Since LST compresses sequences into patches, it processes fewer patch tokens than the baselines, leading to higher efficiency. Sable that the BPE baseline fails to surpass vanilla SpeechL.LMM L,,, whereas LST continues to achieve consistent gains. On HellaSwag, Curriculum Patching improves T-+T accuracy from 49.6 to 52.2 despite reduced computation, while boosting S-S from 40.2 to 45.5. Similar improvements are observed on StoryCloze and TopicStoryCloze. Overall, LST with Curriculum Patching reduces the speech-texx performance gap from 9.4 to 6.7, demonstrating that alignment through patching benefits both modalities while offering meaningful compute savings.
5.2 SCALING BEHAVIOR
Compute-Optimal Scaling, Figure 4 evaluates HellaSwag accuracy under compute-optime training from 420M to 1.8B parameters. Following Hoffmann et al.] (202)). text is trained with 20x model-size tokens, while speech uses half many tokens to preserve 2:1 text-speech ratio.

---

Published as a conference paper at ICLR 2026
Table 5: LibriSpeech ASR (WER) and TTS (CER) for the IB model, reporting context units for ASR and generation units for TTS.
Task
Model
Iters
clean.
other.
Ctx. Units
Gen. Units
lk
140
202
Baseline
2k
44.7
73.2
1.0x
ASR (WER %)
4k
20.7
42.4
1k
6.8
10.4
LST
0.25x
2k
6.0
13.3
Baseline
20k
14.1
15.1
1.0x 0.25
TTS (CER %)
LST
20k
14.1
16.2
At the smallest scale (420M), LST already outperforms the baseline, reaching 29.2% vs. 28.4% for speech and 31.6% vs. 31.1% for text. These improvements compound with scale: 1.8B, LST (Speech) attains 39.0% compared to 35.3%, while LST (Text) achieves 46.3% over 45.7%. Overall, LST provides consistent gains in both modalities, with advantages apparent from the earliest scale and amplified as model capacity increases.
Sub-Optimal Token Scaling at 7B. Figure 4b shows training dynamics for 7B model under fixed processed-token budget (70B tokens), which remains below the scaling-law optimal regime (=140B). Under this sub-optimal compute setting, LSTT exhibits consistently faster improvement and maintains higher accuracy throughout training. Additional comparisons 1B and 7B are sum- marized in Appendix A..4, where LST achieves larger gains at IB and persistent improvements at 7B. Notably, the speech data used our training has already been reused for multiple epochs (==6),, in- dicating that further token scaling becomes data-limited rather than compute-limited in this regime. The continued upward trend at 7B nevertheless suggests that training toward the scaling-las opti- mum with additional data would likely further amplify LST's advantage.
5.3 DOWNSTREAM TRANSFER AND ANALYSIS
ASR Adaptation and Efficientt TTS Generation. To assess ASR, we fine-tune both models on LibriSpeech clean for 4k-4k iterations (batch size 4, sequence length 4096; Table The baseline at 1k steps (140% 202% WER) frequently hallucinates transcripts and produces unreliable stopping behavior, consistent with observations in (Nguyen et al. (025). Although performance improves with iterations, it remains far worse even at 4k (>200% / 40% WER). In contrast, LST achieves 6.8% / 10.4% WER at Ik while reducing the context units during ASR inference. Under the same setup, we evaluate TTS reconstruction after 20k fine-tuning steps (Table 5) LST matches the baseline in CER and reduces the generation length by ~ 4x during TTS inference. CER is computed from Whisper- transcriptions (Radford et al., 2023). Together, LST enables faster ASR adaptation and preserves TTS quality while requiring fewer context units and decoding steps inference.
Visualization of Word-Level Speech Patch Embeddings We visualize word-level speech patch embeddings using t-SNE (yan der Maaten & Hinton 2008) (Fig. 5 from the aligned-patching LST model. Across different categories, embeddings of the same word consistently form tight clusters, while different words remain well separated. Each word forms its own cluster (c.g., he, she, they in pronouns; knife, scissors, sharpener in tools; boat, canoe, surfing in water-related terms). Related variants such as sail-sailing show stability under inflection, while semantically similar pairs like ccissors-shears also appear nearby despite being distinct words. These qualitative patterns match quantitative results: within-word similarity is high (~-0877), netween-word similarity is much lower (~~0.43), and silhouette scores (0.65-0.68) RRousseuu 1987) confirm well-separated clusters.
Ablation on Patching Strategies. compers static and aligned patching. Aligned patching uses word boundaries from alignment, producing semantically coherent patches. We consider two variants: Align (sil sep.), keeping silence spans as separate patches, and Align (sil merged), merging them with adjacent words. Both outperform static patching at similar patch sizes-for instance, Align (sil sep.) reaches 60.3 on StoryCloze S vs. 58.7 for static size 6, and Align (sil merged) scores 38.5 on HellaSwag S vs. 37.2 for static size 9. Curriculum starts with Align (sil sep.) and
8

---

Published as a conference paper at ICLR 2026
Pronouns
Sharp Tools
Water Movement
he they his her
bost
speedboet
Sail
sailing
knife
surfing
knives
paddless
scissors
paddling
pharpereer
sharpeners
sharpening
(a) Personal Pronouns
(b) Sharp Tool Terms
(c) Water-related Terms
Figure 5: Visualization of word-level speech patch embeddings from alignment patching models on HellaSwag speech, grouped by different linguistic categories. Clusters of the same word tight and well-separated from others.
gradually shifts to Static during training, retaining aligmment benefits while matching the shorter- patch evaluation regime; it yields the strongest and most consistent results (e.g., 41.3 on HellaSwag S-+S). Overall, aligned patching better preserves semantics than static, and curriculum combines alignment supervision with static-style evaluation for the best performance. For completeness, we also report BPE-aligned patching experiments in Appendix A.7
RELATED WORK
LLMs using speech tokens. Early neural audio generation methods included direct auto-regressive generation the speech waveform (van den Oord et al., 2016), or using adversarial approaches (Kong et al., 2020). Following this, textless NLP work (Lakhotia et al. (02T) showed that by us- ing discrete speech tokens obtained from self-supervised speech encoders (CPC, wav2vec 2.0, Hu- BERT) targets for language modeling, can enable fully spoken LLMs. AudioLM (Borsos al.) 2023) further uses hierarchical generation, first predicts semantic tokens, and subsequent stages pre- diet fine-grained acoustic tokens from SoundStream (Zeghidour al.) 202I). to achieve both high audio quality as well as long-term consistency. In addition to augmenting semantic speech tokens with pitch and style tokens explicitly model expressivity, SpiritLM (Nguyen al. (025) also introduced interleaving speech modeling with text-tokens. More recently, Moshi (Détossez et al.) 2024) propose a hierarchical inner monologue method, that jointly predicts time-aligned text and acoustic tokens (with distilled semantic information). together with modeling multiple-strean audio for handling full-duplex audio dialogues. Finally, similar to scaling laws for text LLMs (Hoffmann) al. 2022). Cuervo & Marxer (2024) fit scaling law curves predict the performance of spoken LLMs, and find that they scale upto three order of magnitude more slowly than text LLMs.
Transferring textual knowledge into speech LMs. Slower scaling trends, together with dis- proportionately lower amount data, lead to a knowledge and reasoning gap between speech and text L.MMs. To bridge this, AudioPaL.M and TWIST (Rubenstein al.) 2023 Hassid et al., 2023) initialize a spoken LLM from strong text model (PaL_M-2, ILaMM),, improving both speech under- standing/generation and cross-lingual transfer. SpiritLM demonstrates that interleaved speech-text training sigmificantly improves inter-modality knowledge transfer. Spectron (Nachmani et al.) uses "Chain-of-Modallty" pipeline to first produce text and then speech conditioned on the text, trading latency for stronger textual control, while Moshi (Défossez et al.) (024) uses similar approach but generates interleaved text and speech as an inner monologue. To improve latency, LLaMA-Omin (Fang et al. 2024) style systems decode text and speech simultaneously, by upsampling textual LLM hidden states to decode speech units, before proceeding to decode the next text token.
Speech model efficiency. Compared to text, speech yields much longer token sequences, owing to higher frequency audio codees, that consume many times additional compute to pre-train and generate. Efforts to mitigate this include methods to produce coarser speech units (Baade et al.) Tseng et al. 2025). hierarchical generation (Borsos et al., 2023), and producing residual tokens using parallel streams (Copet al. 2023). Attempts text-inspired approaches to compress token sequences such as BPE (Ren et al. 2022 Li et al., (024) achieved limited success. this paper, we take inspiration from recent dynamic patching approaches that have yielded improvements in other

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of patching strategies with approximately matched patch sizes. Static uses fixed patch lengths, Align (sil sep.) treats silence as separate patches, and Align (sil merged) merges silence into words, and Curriculum gradually shifts from Align to Static during training.
Patching Strategy
Ave Patch Size (tokens)
HellaSwagg T-T
StoryCloze S-*S
TopicStoryCloze T-xT
S-*
T-T
S->S
Static
4
40.5
48.8
58.2
69.4
86.2
95.1
Curriculum (sil sep.) Curriculum (sil merged)
5,8 + 4
41.3
49.2
58.6
67.8
86.6
95.4
9,4
+
40.3
48.9
58.7
68.9
86.8
95.4
Align (sil sep.)
5.8"
39.9
49.3
60.3
69.9
85.7
95.3
Static
39.4
49.2
58.7
69.6
84.9
94.9
Static Align, (sil merged)
37.2
49.4
57.5
69.7
84.7
95.9
9.4
38.5
49.0
58.8
69.7
86.9
96.0
The average patch length is 5.8 for words in Align (sil sep.), while silence has average of 3.7.
modalities such as text (Pagnoni et al. 2024 Yu et al. 2023 Videau et al. 2025) and vision (Png al. 2024 Beyer et al. 2023), and extend these methods to speech-text L.MMs.
Speech Understanding Benchmarks. Going beyond measuring only acoustic and phonetic capa- bilities of speech models using scores such as ABX (Kahn al., 2020), Nguyen al. (2020) estab- lished the Zero Resource Speech Benchmark 2021, comprising datasets/metrics to evaluate lexical (sWUGGY), syntactic (sBLIMP) and lexical-semantic (sSIMI) capabilities of spoken LLMs. Since these benchmarks contrast between very short speech segments, found that dynamic compute approaches such as ours, do not yield significant improvements (see Appendix [A.9). However, sub- sequently, [Hasidd et al. (2023) introduced the sStoryCloze and TopicStoryCloz datasets, which abili- ties of Spoken LLMs. We use these benchmarks in this paper, together with speech version of the popular HellaSWAG textual benchmark, also measuring commonsense reasoning capabilities.
story completion benchmarks in the speech modality measuring commonsense/understanee
LIMITATIONS
Our study has several limitations. First, focus half-duplex speech-text modeling, where speech and text alternate in turns, and do not yet address full-duplex interaction required for real- time dialogue such Moshi (Défossez et al. 2024). Second, our analysis is restricted to the pre- training stage, without exploring instruction fine-tuning or downstream adaptation, which we leave for future work. Third, some of our patching strategies, such as alignment and curriculum, rely on forced alignments during pre-training; although curriculum patching reduces this dependency at inference, fully alignment- approaches remain an open challenge. Finally, our experiments are limited to the speech- text modality, and we have not yet extended LST to additional modalities such as image or video, which represent a promising next direction.
8 CONCLUSION
We presented the Latent Speech-Ter Text Transformer (LST), autch-based framework that aggregates speech tokens into latent units to improve computational efficiency and balance across modalities in multimodal language modeling. Across controlled-budgee evaluations, scaling analyses, and down- stream transfer experiments, LST consistently outperforms SpeechLLM baselines while reducing autoregressive sequence length and inference cost. Importantly, the gains arise from shortening the effective autoregressive sequence through latent speech patching, enabling more compute-efficient efficient model scaling without sacrificing speech coverage reconstruction quality. These findings high- light token density imbalance as key bottleneck scaling spoken language models and suggest LST practical step toward compute-efficient unified seeech-text foundation models.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. We use only publicly available datasets (c.g.. Lib- riLight, People's Speech, Multillinuull LibriSpeech, Spotify) and did not collect any new human- subject data. No personally identifiable information is included. Our methods aim to improve efficiency and generalization in speech text text modeling, with potential positive impacts on accessibil- ity and multilingual applications. As with all large language models, there remain general risks of misuse, and we encourage responsible use of our work.
REFERENCES
Alan Baade, Puyuan Peng, and David Harwath. Syllablelm: Learning coarse semantic units for speech language models. In The Thirteenth International Conference on Learning Representa- tions.
Alexei Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli, wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 13:12449-1 122400, 2020.
Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Korbblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flex- ivit: One model for all patch sizes. In CVPR, 2023.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. 1EEE/ACM transactions on audio, speech, and language processing, 31:2523 25333333333,,3 2023.
Ann Clifton, Aasish Pappu, Sravana Reddy, Yongze Yu, Jussi Karlgren, Ben Carterette, and Rosie Jones. The spotify podcast dataset. arXiv preprint arXiv:2004.042777 042700, 2020a.
Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu, Rezvaneh Rezapour, Hamed Bonab, Maria Eskevich, Gareth Jones, Jussi Karlgren, Ben Carterette, et al. 100,000 podcasts: A spoken en- glish document corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 5903-5917. 2020b.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and controllable music generation. Advances in Neural Information Pro- cessing Systems, 36:47704 -477700-447720 2023.
Santiago Cuervo and Ricard Marxer. Scaling properties of speech language models. arXiv preprint arXiv:2404.00 00685, 2024.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dia- logue. arXiv e-prints, pp. arXiv-2411 2024.
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma. Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. arXiv preprint arxii:2409.06666 2024.
Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón, Keith Achorn, Anjali Gopi, David Kanter, Maximilian Lam, Mark Mazumder, and Vijay Janapa Reddi. The people's speech: A large-scale diverse english speech recognition dataset for commercial usage. arXiv preprint arXiv:2/ 1.0:://11.0044,, 2021.
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, et al. Textually pretrained speech language models. Advances in Neural Information Processing Systems, (6:63483-63501 2023.
hexgrad. Kokoro-822
when smaller means better in text-to-speech. https//wwgggnngfiee.ooooegggg//a
2025. Accessed: 2025-04-22.

---

Published as a conference paper at ICLR 2026
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. In Proceedings of the 36th International Conference on Newral Information Processing Systems, pp. 30016-30030, 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions audio, speech, and language processing, 29:3451-3460, 2021.
Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Fierre-Fmmande Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, al. Libri- light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673, IEEE, 2020.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022-17033, 2020.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, 2018.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336-1355 1354, 2021.
Bohan Li, Feiyu Shen, Yiwei Guo, Shuai Wang, Xie Chen, and Kai Yu. On the effectiveness of acoustic bpe in decoder-only its In Proc. Interspeech 2024, pp. 4134-4138, 2024.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint 1xXvv:660.00169 2016.
Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. Spoken question answering and speech continuation using spectrogram- poweered llm. In The Twelfth International Conference on Learning Representations.
Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling. In NeuRIPS Workshop Self-Superviser Learning for Speech and Audio Processing, 2020.
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R Costa-Jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Pabr-Ame Duquenne, Robin Algayres, Ruslan Mavlyutov, al. Spirit- Im: Interleaved spoken and written language model. Transactions of the Association for Compu- tational Linguistics, 13, 2025.
Artidoro Pagnoni, Ram Pasunuru. Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, al. Byte latent transformer: Patches scale better than tokens. arXiv preprint arXiv:24/2.0988 2024.
Yatian Pang, Peng Jin, Shuo Yang, Bin Lin, Bin Zhu, Zhenyu Tang, Liuhan Chen, Francis EH Tay, Ser-Nam Lim, Harry Yang, et al. Next patch prediction for autoregressive visual generation. CoRR, 2024.
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.033/1 2020.
12

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 18499-28851 PMLR, 2023.
Shuo Ren, Shujie Liu, Yu Wu, Long Zhou, and Furu Wei. Speech pre-training with acoustic piece.
In Proc. Interspeech 2022, pp. 2648-2646622 2652, 2022.
Peter Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20(1):53- 65, 1987.
Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that speak and listen. arXiv preprint arXiv:2306.1292. 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Liang-Hsuan Tseng, Yi-Chang Chen, Kuan-Yi Lee, Da-Shan Shiu, and Hung-yi Lee. Taste: Text- aligned speech tokenization and embedding for spoken language modeling. arXiv preprint arXiv:2504.07053 2025.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Semior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In Proc. SSW 2016, pp. 125- 125, 2016.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz. Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Mathurin Videau, Badr Youbi Idrissi, Alessandro Leite, Mare Schoenauer, Olivier Teytaud, and David Lopez-Paz. From bytes to ideas: Language modeling with autoregressive u-nets. arXiv preprint 2006:22506. 14761, 2025.
Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers. Advances in Neural Information Processing Systems, 36:78808-7885 -78003,, 2023.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 50:495-507, 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint arXiv:1905.07832. 2019.
A APPENDIX
A.1 DATA CONSTRUCTION AND LICENSING
A.1.1 INTERLEAVED DATA CONSTRUCTION
To generate interleaved sequences from parallel speech-text data, proceed follows:
1. Alignment. We obtain alignment information by using Wav2 Vec2 + CTC to determine the boundaries linking text tokens to their corresponding spans of speech tokens.
2. Span selection. For each training example, randomly select contiguous span of words. The selected span is replaced by text tokens, while the following span of approximately half that length is kept as speech tokens.
13

---

Published as a conference paper at ICLR 2026
3. Modality markers. We insert special tokens <t> and 
 indicate the start of text and speech segments, respectively. This ensures the model can disambiguate between modali- ties.
4. Dynamic sampling. Interleaved sequences are generated dynamically at training time, so each epoch exposes the model to different interleaving patterns for better robustness.
This process yields diverse interleaved training examples while preserving alignment between speech and text, allowing the same model to be applied uniformly to S-x,, S-T, T-+S, and T->T tasks.
A.1.2 DATASET LICENSING
We summarize the licenses associated with the speech corpora used for pre-training.
Libri-Light. Derived from LibriVox audiobooks that in the public domain.
The People's Speech. Includes audio released under CC BY and CC BY-SA licenses.
Multilingual LibriSpeech (MLS). Distributed under CC BY 4.0.
Spotify Podcast Dataset. The dataset is described by Clifton al. (Clifton al. 2020b).
available at https: arxiv. org/ abs 2004, 04200000044000 04270
A.2 ADDITIONAL. MODEL. DETAILS
A.2.1 LOCAL PATCH ENCODER AND DECODER DETAILS
We provide additional details on the local patch encoder and decoder that form the core representa- tion transformation in LST.
Patch Encoder. Following the BLT-style latent aggregation paradigm (Pagnoni et al. 2024), the patch encoder converts token-level speech representations into latent patch embeddings through alternating sliding-window self-attention and cross-attention layers. Within the encoder cross- attention, latent patch queries are initialized by pooling the hidden states of speech tokens within each local span. These pooled representations then act as queries, while the corresponding speech token features provide the keys and values. Self-attention operates over restricted temporal win- dow to preserve local coherence while controlling computational cost, after which cross-attention aggregates the token features into smaller set latent patch queries that summarize contiguous speech segments. Unlike BLT, LST: (i) applies patching only to speech-token spans determined by the strategy in Section 3.1 (ii) does not employ hash embeddings, as they showed no empirical benefit in our ablations. Additionally, LST does not rely subword tokenization (e.g., BPE applied to speech tokens), which we find provides no performance benefit in our experiments.
Patch Decoder. The patch decoder is implemented as lightweight Transformer trained with next- token prediction (NTP) loss. Each decoder layer contains: (i) causal self-attention over a sliding window of the previously generated 512 tokens to ensure autoregressive consistency, and (ii) cross- attention where current token hidden states act as queries, while previously generated speech patches together with text tokens provide the keys and values. This allows token-level prediction to condition simultaneously on higher-level latent patch structure and textual context. This dual conditioning enables efficient generation while preserving long-range semantic and acoustic information encoded in the latent patches.
Intuition. Conceptually, the local decoderr pair performs a token - latent patch -> to- ken information bottleneck. The encoder compresses dense speech-token sequences into structured latent summaries, while the decoder re-expands them for autoregressive prediction. This mecha- nism central LST's ability to reduce effective sequence length and computational cost without sacrificing linguistic acoustic fidelity.
A.2.2 MODEL ARCHITECTURE
Table 7 summarizes the hierarchical architecture used in our experiments. The model consists of a shallow patch encoder, deep global transformer, and moderately deep patch decoder. The local
14

---

Published as a conference paper at ICLR 2026
patch modules operate with restricted attention windows to capture fine-graines context, while the global transformer uses blusk- attention with rotary position embeddings (RoPE) to handle long-range dependencies efficiently. This design balances local detail preservation with scalable long-context modeling.
Table Model architecture configuration. Each module shown with its depth, hidden dimension, number of attention heads, and other relevant settings.
Module
Layers
Dim.
Heads
Notes
Patch Encoder
1024
16
Patch window = =512
Global Transformer
25
2048
16
Block-causal; RoPE (0 = 5x 10
Patch Decoder
1024
16
Patch window = 512
A.3 OPTIMIZATION AND TRAINING CONFIGURATION
We trained the model using the Adam W optimizer (3) = 0.9, B2 = 0.95, weight decay 0.1). The learning rate was initialized at 4 10 and scheduled with cosine decay, including warmup period of 2,000 steps and a minimum ratio of 0.01 at the final step. For the 1B model, training was performed 32 H100 GPUs with Per-GPU batch size of 4 sequences (sequence length 4,096 units), leading to a total batch size of 0.5M units. Mixed-precision training with 6 was used for efficiency. Gradient clipping was applied at 1.0, and gradient accumulation parallelism used a single partition, and Fully Sharded Data Parallel (FSDP) was enabled for memory efficiency. No dropout was applied. The 1B model was trained for 200k steps, corresponding to approximately trillion units, and required around 17 hours to complete on 32 H100 GPUs.
1. Model
A.4 FIXED-TORE TOKEN SCALING ACROSS MODEL SIZES
Table 8 compares the baseline SpeechLLM and LST at 1B and 7B under an identical processed- token budget, ensuring fair cross-scale evaluation. At IB, LST shows clear gains over the baseline (41.3 36.8 on S-S and 49.2 vs. 47.1 on T-*T), indicating improved sample efficiency at smaller scale. At 7B, the advantage remains consistent, with LST achieving 44.2/55.3 compared to the baseline's 42.0/54.8. These results demonstrate that the benefits of latent speech-text modeling persist across model scales and are not limited to low-capacity regimes.
Table 8: Scaling comparison between baseline SpeechLLM and LST at IB and 7B.
Model
Batch (M)
Iters (k)
HellaSwag S
T-+T
Baseline (IB)
0.5
200
36.8
47.1
LST (1B)
0.5
200
41.3
49.2
Baseline (7B)
4.0
25
42.0
54.8
LST (7B)
4.0
25
44.2
55.3
A.5 STABILITY ANALYSIS ACROSS TASKS
We further examine the robustness of patching strategies by repeating each experiment three times and reporting the average (Ave) and standard deviation (Std). Table summarizes results for Hel- laSwag (HS), StoryCloze (SC), and TopicStoryCloze (TSC) under both speech-to-speech (S SS) and text-to-text (T-T) directions. HellaSwag results are generally more stable than the other tasks: both Curriculum and the Baseline show near-zero std (0.13 and 0.22 for S-+S), while Static is rel- atively less stable with larger fluctuations (0.67). By contrast, StoryCloze and TopicStoryCloze exhibit considerably higher deviations. occasionally exceeding 1.5, which indicates greater insta- bility. Overall, Curriculum improves the average accuracy across all tasks while delivering highly consistent results on HellaSwag, underscoring its effectiveness in stabilizing training.
15

---

Published as a conference paper at ICLR 2026
Table 9: Average (Ave) and standard deviation (Std) across three runs. Each task is reported with both S-S and T->T directions.
Model
Evaluation
HellaSwag Ave
StoryCloze Ave
TopicStoryCloze
Std
Std
Ave
Std
Curriculum
S->S
41.4
0.13
59.2
0.68
87.1
0.45
T-+T
49.1
0.06
69.5
1.56
95.6
0.45
Static (4)
S-*S
40.9
0.67
58.5
0.50
86.6
0.52
T-T
48.5
0.37
69.4
0.11
95.1
0.19
Baseline
S-+S
36.5
0.22
58.3
0.21
86.3
0.52
T-+T
46.3
0.78
66.6
1.56
93.9
1.44
50
47.2%
Swaa Accuracy
45.4%
44
42
41.2%
38
37.0%
LST (Speach)
Baseline (Speach) (Text)
36
(Txxt)
14
1:3
1:1
Speech Text Takens Ratio
Figure 6: Effect of speech-to-text token ratio at 200k iterations. Results reported on HellaSwag under both S and T->T.
A.6 EFFECT OF SPEECH PROPORTION
Figure 6 illustrates the effect of varying the training speech-to-text token ratio on HellaSwag. Across all settings, LST consistently outperforms the baseline, and both methods exhibit the best speech-tex! trade-off at the 1:2 ratio. Moving from 1:3 to 1:2 improves LST (S+S) from 40.2 to 41.3 while keeping LST (T-T) high at 49.7; pushing further to 1:1 does not provide speech gain (41.2) but large text drop (47.2, 225)). The baseline shows the same pattern: at 1:2 it reaches 36.8 S) and 47.1 (T-T), whereas 1:1 gives only 37.0 on speech (+0.2) but lowers text 45.4 (-1.7). Averaging speech and text accuracies, the macro score peaks at 1:2 for both LST and the bascline. These results indicate that allocating about one-third tokens to speech (1:2) offers fair and robust operating point for both models avoid the substantial text-side degradation at 1:1 while securing clear gains over lower speech ratios.
A.7 BPE-ALIGNED PATCHING
In addition word- word-aligned patching, also explored BPE-aligned patching, where speech patches are constructed according to BPE segmentation of the text. To ensure comparability, we applied the same forced-alignment procedure at the character level and then mapped aligned spans their corresponding BPE units. While this provides finer granularity, the resulting boundaries are less pre- cise and the subword pieces do not always correspond to meaningful acoustic events. As shown in Table I0 word alignment generally outperforms BPE alignment in S (e.g., 59.4 vs. 55.6 on Sto- ryCloze and 84.8 vs. 79.6 on TopicStoryClozer reflecting the more reliable word-level boundaries. On the other hand. BPE achieves slightly better T-+T results, likely because its patching is directly aligned with the underlying text BPE tokens. Finally, curriculum training further boosts HellaS S performance, improving from 40.0/39.2 41 11.5/41.3 for Word and BPF, respectively.
16

---

Published as a conference paper at ICLR 2026
Table 10: Comparison of aligned patching strategies under a speech-to-texi token ratio of 1:4. Word Align uses word-level forced alignment, BPE Align uses BPE segmentation, and Curriculum gradually shifts from alignment- based static patching.
Model
Ave Patch Size (tokens)
HellaSwag S-xS
StoryCloze T-T
TopicStoryCloze S
T-T
S-+S
T->T
LST (Word Align)
5.8*
40.0
49,9
59.4
68.6
84.8
94.6
LST (BPE Align)
5.0"
39.2
50.1
55.6
69.1
79.6
95.6
LST (Word Curr.)
5.8++4
41.5
49.5
57.9
68.9
86.8
95.1
LST (BPE Curr.)
5.0->4
41.3
48.6
59.1
67.1
86.5
95.4
The average patch length is 5.8 for words, 5.0 for BPEs, and 3.7 for silence spans.
A.8 SC/TSC EVALUATION SETS
We additionally evaluate on the publicly available speech versions of StoryCloze (SC) and Top- ieStoryCloze (TSC) (Hassid al. 2023). The released TTS audio in these benchmarks contains noticeable synthesis artifacts and unnatural prosody, which may affect evaluation reliability, To pro- vide more realistic acoustic conditions, we generate improved TTS renderings using the same text prompts and release them for reproducibility. Table TI reports results both the original and im- proved versions. Across all four settings, LST consistently outperforms the baseline, indicating that the observed gains robust to TTS quality variations and do not depend a specific synthesis condition.
Model
SC (orig.)
SC (ours)
TSC (orig.)
TSC (ours)
Baseline
58.0
59.1
78.4
87.5
LST
60.8
61.2
79.5
87.9
Table 11: Evaluation on original and improved SC/TSC TTS sets.
A.9. FINE-GRAINED LEXICAL AND SYNTACTIC INFORMATION
Although main evaluation focuses narrative understanding and commonsense reasoning, LST does not discard fine-grained lexical or syntactic information. The patch decoder preserves token- level detail through localized reconstruction, while aggregation operates only over short spans with- out collapsing phonetic distinctions. To verify this, we evaluate on sWUGGY and sBLIMP, which probe subword discrimination and syntactic sensitivity in spoken language modeling. As shown in Table 12 LST performs on par with the baseline, suggesting that latent patching preserves the lin- guistic cues required for lowelevel discrimination without introducing trade-offs against higher-level reasoning performance.
Model
sWUGGY
sBLIMP
Baseline 72.5
58.9
LST
72.8
59.0
Table 12: Fine-grained lexical and syntactic evaluation.
A.10 SEQUENCE LENGTH, COMPUTE, AND INFERENCE SCALING
Under the compute-controlled setting, sequence composition is determined by token-level statistics of interleaved speech-text data. The data follow a fixed word ratio of 1:2 (speech: text), specified by the data construction. Because discrete speech units have higher temporal density than text tokens, this ratio corresponds in practice approximately 0.77 speech tokens and 0.23 text tokens within the interleaved portion of the baseline sequence. LST applies 4-+1 speech patching, reducing speech tokens from 0.77 to 0.19 while leaving text tokens unchanged. The total sequence length therefore
17

---

Published as a conference paper at ICLR 2026
Interleaved Speech Data (1:2)
Text Data
Total
Model
Speech units
Text units
Text units
Sequence length
Baseline
0.77
0.23
2.00
3.00
LST
0.19
0.23
2.00
2.42
FLOPs Reduction
0.19x
Table 13:
Token accounting under the compute-controlled setting. Interleaved speech-text data follow a 1:2 word ratio. Due to higher speech token density, the baseline allocates 0.77 speech tokens versus 0.23 text tokens within the interleaved portion. LST compresses speech tokens via 4-+1 patching, reducing the overall sequence length from 3.00 to 2.42 (~200% FLOPs reduction).
changes from 3.00 to 2.42 (0.81x). Given the quadratic dependence of transformer FLOPs on sequence length, this difference corresponds to an overall compute change of approximately 20%.
During TTS inference, speech patching reduces the number of autoregressive acoustic generation steps by approximately 4x. Since acoustic step count dominates generation cost, this leads to substantial reduction in overall generation computation. During ASR inference, the reduced number of acoustic sequence elements decreases the effective encoder context length by similar factor, yielding corresponding efficiency gains in acoustic encoding.
A.11 ALIGNER QUALITY ROBUSTNESS
Curriculum patching relies alignment boundaries primarily during the early training stage before transitioning to static patching. To assess robustness to alignment noise, we perturb Way2Vec2+CTC boundaries by =20-40 ms and re-evaluate alignment-based patching. This perturbation causes most~~1 % degradation in the setting most sensitive to boundary precision (pure alignment patching). Because curriculum patching depends less on precise boundaries and gradually removes alignment during training, these results indicate that its performance benefits are expected to persist even when using simpler or less accurate aligners.
A.12
LLM USAGE.
We used large language models solely for polishing some sentences in this paper.
18


Reviews:
[{"review_id": "s8I6MvEqCS", "reviewer": "Reviewer_fuu2", "summary": "This paper proposes _Latent Speech-Text Transformer_ (LST) for improving speech LLMs by using a pair of local encoder/decoder similar to the [Byte Latent Transformer](https://aclanthology.org/2025.acl-long.453.pdf):\n1.  The local encoder reduces the speech token rate via a cross attention layer turning speech tokens into _patches_.\n2.  The speech patches along with text tokens are processed by the global transformer. \n3.  The output from the global transformer can be used to directly predict the next text tokens. In order to predict the speech tokens, the corresponding global transformer output tokens are fed to the local decoder, which restores the tokens into the original speech token rate.\n\nThree patching schemes are explored:\n-   Static patching: Each patch consists of a fixed number of speech tokens without any overlap.\n-   Alignment patching: Each patch consists of a word / silence obtained from forced alignment using Wav2Vec2-CTC.\n-   Curriculumn patching: Training starts with alignment patching, then gradually transitions to static patching.\n\nModels trained from scratch using text and speech datasets are evaluated against a baseline transformer model that directly processes speech tokens in the same manner as text tokens. Evaluations are conducted on HellaSwag, StoryCloze, and Topic StoryCloze. For evaluating the speech processing capability of the proposed model, these test sets are also TTS'd by the authors using Koroko TTS. Evaluation results show LST leads to a noticeable improvement in both text and speech versions of these test sets.", "questions": "Could you clarify what \"compute-controlled\" means in line 321? I'd expect the baseline model to need much more compute to process the same speech input due to the higher number of HuBERT tokens and the quadratic time complexity of transformer. But Table 3 seems to suggest that the number of interleaved tokens is only slightly lower different in the baseline compared against LST. In a setting where the baseline uses roughly the same flops, shouldn't it see far fewer speech tokens?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Originality: This paper applies Byte Latent Transformer, originally designed for better byte based language modelling, to a new problem of speech language modelling. This is a reasonably novel approach compared to other approaches such as low bitrate speech codecs (e.g. SpeechTokenizer or encodec). The curriculum learning scheme that transitions from alignment patching to static patching is also novel.\n- Quality: The use of latent transformer is a simple and well motivated method for reducing the speech token rate, eventually resulting in a better language model for both speech and text.\n- Clarity: Most part of the paper is well organized and clearly written.\n- Significance: The proposed method compares favorably against the baseline model in both the text and TTS versions of HellaSwag, StoryCloze, and TopicStoryCloze evaluations.", "weaknesses": "-   Clarity: To help readers not already familiar with Byte Latent Transformer, the description of the local encoder/decoder (161-185) could use some expansion. This is central to the main idea of this paper, and thus it would be great if the readers do not have to refer to another paper to understand the core ideas of this paper.\n-   Quality: Several changes in evaluation & experiments would be necessary to better support the claim of this paper.\n    -   In terms of speech modelling, all the evaluation test sets in this paper are produced by TTS. There is no evaluation on actual human speech, which often differ greatly from clean synthesized speech. For comparison, [the Spirit LM paper](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00728/127457), which is this paper's baseline model, included results on ASR (Table 5) and a comparison against a cascade system.\n    -   It is not clear why the authors chose to evaluate on their version of synthesized StoryCloze/TopicStoryCloze while there exists [a widely used version from the original authors](https://github.com/slp-rl/SpokenStoryCloze). This makes comparison against results from other papers very difficult.\n    -   The model is supposedly capable of both understanding and generating speech (line 269), however this is only evaluation of speech understanding in the current draft. In constrast, there is TTS evaluation in the Spirit LM paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ylAPrGM0S8", "reviewer": "Reviewer_SwjZ", "summary": "This paper presents the latent speech-text transformer, a pre-training method for speech-text language models, with the goal of reducing the computation required for long speech sequences and the sequence length mismatch between text and speech. Specifically, the model works by aggregating discrete speech tokens into latent patches.The authors train LST models on interleaved speech-text data and ablate the different patching strategies used. Overall, LST models perform better than simple speech-text LMs on textual and spoken versions of commonsense and narrative coherence benchmarks.", "questions": "- How are the output hubert tokens converted to the actual multiple choice answer? Its unclear if ASR or some other method is used to map the model output to the actual answer choice.\n- Do they spoken Hellaswag / SC / TC datasets have multiple speakers or only use a single speaker?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to understand.\n- The latent speech-text transformer (LST) is a novel architecture that attempts to address the long sequence length of speech tokens and the length mismatch with text. This is a step towards solving a significant problem that makes scaling speech models difficult\n- The authors propose and benchmark a variety of patching techniques that aggregate speech tokens together, and show how they can be combined into a stronger and faster model through curriculum learning\n- They analyze several factors related to the proposed method, such as scalability and compute equivalence.", "weaknesses": "- Unclear evaluation procedure: the chosen evaluation sets rely on multiple choice QA. This means that in the speech setting, the model outputs HuBERT tokens. How are the output tokens converted to the actual multiple choice answer? Its unclear if ASR or some other method is used to map the model output to the actual answer choice. Without such information, reproduction and fair comparisons against this work are challenging.\n- Small test coverage: the evaluation only covers the aforementioned mQA tests. While this is a good evaluation of the model's \"intelligence\"-related capabilities, I am surprised there was no evaluation on more traditional speech tasks like ASR or TTS that measure the model's phonetic and cross-modal capabilities. I believe that it would be vital to test such abilities, since they may be affected by the compressed representations, which would lower the impact of the proposed method. Such experiments are done in the SpiritLM paper, which can be compared to.\n- While fine for understanding tasks, the proposed technique would be non-trivial to extend the multi-stream speech LMs that use neural codecs, which are the SOTA for generation tasks, limiting its impact.\n- While the scaling figure is nice, I think the data points appear too close together to be very meaningful (only from 10K to 25K iterations) for models that are trained for 200K iterations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jMX42JeWQy", "reviewer": "Reviewer_cWAg", "summary": "This paper addresses a significant and well-known challenge in auto-regressive speech-text models: the information density mismatch between modalities. Speech, when tokenized (e.g., using HuBERT), results in disproportionately long sequences compared to the equivalent text tokens. The authors hypothesize this mismatch hinders speech-text alignment and leads to poor computational efficiency and scaling laws. To solve this, the paper introduces the Latent Speech-Text Transformer (LST), an architecture inspired by the Byte Latent Transformer (BLT). The core idea is to aggregate sequences of speech tokens into \"latent speech patches\" using a patch encoder. A global transformer then processes a shorter, more balanced sequence of interleaved text tokens and these latent speech patches. A patch decoder maps the latent representations back to speech tokens for generation. Moreover, the proposed LST architecture, and especially the curriculum patching method, is an innovative, practical, and effective solution. It demonstrates state-of-the-art performance gains over strong baselines in a rigorously controlled and very convincing experimental setup. Moreover, the work provides a practical and scalable architectural solution that improves the efficiency and alignment of multimodal speech models, which is of broad interest to the ICLR community.", "questions": "I would appreciate clarification on the following points:\n- The authors mention omitting sWUGGY and sBLIMP. While I understand the focus on narrative reasoning, could you speculate on how LST's aggregation mechanism might perform on these tasks? Would the patch decoder be sufficient to reconstruct the necessary fine-grained information, or do you expect a trade-off?\n- How sensitive is the curriculum patching method to the quality of the Wav2Vec2+CTC aligner? Would the benefits be held if a simpler, less accurate, or different-style aligner were used during training?\n- The t-SNE plots in Figure 4 are a nice qualitative illustration of LST's alignment. This claim would be significantly strengthened by including a parallel t-SNE plot showing the speech token embeddings from the baseline model for the same words. A direct visual comparison of cluster separation and tightness would make the \"improved alignment\" point much more immediate and convincing.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper's primary strength is its direct and effective approach to a clear and significant problem in speech-text modeling. \n- The motivation is well-articulated, and the proposed LST architecture is a logical adaptation of patching techniques from other domains. The experimental design is rigorous, particularly the use of both compute-controlled and data-controlled settings, which provides a robust validation of the method's efficiency gains. \n- The most substantial contribution is the curriculum patching strategy. This is an innovative and highly pragmatic solution that achieves the \"best of both worlds\": it leverages the rich semantic guidance of an external aligner during training to produce robust representations, but transitions to a simple, dependency-free static patching method for inference. \n- The strong and consistent performance gains of this curriculum-based model over all baselines, combined with promising scaling results up to 7B parameters, makes it very effective.", "weaknesses": "- The evaluation is focused on high-level narrative and commonsense reasoning tasks, with fine-grained lexical and syntactic benchmarks like sWUGGY and sBLIMP explicitly omitted. \n- This leaves open the question of how LST's token aggregation might affect performance on tasks requiring very fine-grained acoustic-phonetic or syntactic judgments. \n- Furthermore, the provided ablation on LST (Static), which is the \"pure\" architecture without aligner supervision, shows mixed results: it demonstrates a clear and strong advantage over both baselines on HellaSwag but underperforms the base model on TopicStoryCloze. This inconsistency, however, serves to strengthen the paper's main claim by justifying why the novel curriculum patching approach is necessary and superior, as it successfully resolves this instability and delivers robust performance across all tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SX8VH3Cxq5", "reviewer": "Reviewer_Xr9g", "summary": "The paper proposes the Latent Speech‑Text Transformer (LST), which models patches of speech tokens instead of individual tokens to reduce the speech‑vs‑text compute/sequence‑length imbalance in interleaved speech‑text LMs. A lightweight local encoder/decoder forms and reconstructs speech patches, while a global transformer models interleaved text tokens and speech patches; alignment‑based patching (Wav2Vec2+CTC word boundaries) and curriculum patching (aligned→static) are introduced to better synchronize content while enabling simple, static‑only inference. Across compute‑controlled and data‑controlled protocols, LST improves both S→S and T→T on HellaSwag, StoryCloze, TopicStoryCloze, with clearer gains from curriculum patching and competitive compute savings; the method also scales from 1B→7B parameters with consistent improvements.", "questions": "- Can you confirm that Patch Encoder/Decoder in Fig. 2 are exactly the Local Encoder/Decoder of §3 and make that explicit in the text/caption? Also, in the local cross‑attention, what is the query and what are the keys/values?\n- Did you compare Wav2Vec2+CTC to Whisper alignment or MFA for word/BPE boundaries in terms of accuracy, speed, and downstream impact on LST vs. baseline? If not, any evidence that W2V2+CTC is near‑optimal for your training/inference regimen?\n- For BPE SpeechLLM, how did you determine the SentencePiece tokenizer is sufficiently trained (beyond vocab size sweeps)? Could a better BPE (more data, longer training, different seed, or unigram LM) close the gap to LST? Please provide tokenizer diagnostics or an expanded ablation.\n- You start curriculum from Align (sil sep.). Did you try starting from Align (sil merged) (arguably closer to static evaluation) and, if so, how do results compare across patch sizes?.\n- Given Spotify (55k hrs) is a substantial portion of training, can you clarify license, current accessibility, and whether research‑only use is still permissible? A short line in §4.1/Ethics would help replicability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "LST is a practical, well‑motivated way to mitigate speech‑text length and compute imbalance: it pairs a local encoder/decoder (restricted windows) with a global transformer to operate on information‑dense speech patches; alignment‑based patching improves semantic synchrony while curriculum avoids reliance on aligners at test time. The paper executes careful compute‑controlled and data‑controlled comparisons showing consistent S→S and T→T gains, compute savings under a fixed speech/text token budget, and robust scaling from 1B→7B. The patching‑strategy ablation clarifies why curriculum starting from Align (sil sep.) works well.", "weaknesses": "- Originality\n    - The novelty is largely an application of BLT‑style patching to the speech‑text setting; stronger baselines (e.g., late cross‑attn fusion, gating/MoE variants) are not contrasted head‑to‑head under the same compute.\n    - Alignment‑aware and curriculum schedules are natural extensions; their conceptual leap is moderate relative to prior patching ideas in other modalities.\n- Quality\n    - Aligner choice is fixed to Wav2Vec2+CTC; there is no comparison to Whisper forced alignment or MFA‑style alternatives, so the robustness of the alignment dependency is unclear.\n    - The BPE SpeechLLM baseline uses 1k SentencePiece trained on 100k speech sequences, with a note that 5k/10k didn’t help; however, the paper doesn’t show tokenizer sufficiency checks (e.g., stability/perplexity/segmentation quality) that would rule out an under‑trained BPE baseline.\n    - Curriculum vs. Align (sil merged): Table 6 shows Align (sil sep.) generally stronger than sil merged on S→S, but it’s not shown whether a curriculum initialized from “sil merged” could match or exceed “sil sep.” for specific patch sizes.\n    - Compute savings are reported, but a concise methodological paragraph clarifying how savings are measured versus baseline token→patch conversion would aid interpretation.\n- Clarity\n    - Figure 2 terminology: “Patch Encoder/Decoder” appear to correspond to §3’s Local Encoder/Decoder; making this mapping explicit in the caption/text would reduce confusion (Fig. 2; §3).\n    - Local Encoder cross‑attention: clarify what queries what. Do latent patch embeddings query a local window of speech token embeddings (keys/values)?\n    - Dataset licensing/availability: Spotify Podcast (55k hrs) is listed (Table 1); please state license and present‑day access status for reproducibility/ethics.\n- Significance\n    - The gains are solid on story completion tasks synthesized with Kokoro TTS, but the generalization to other speech understanding tasks is not explored; a short discussion of when patching helps most would strengthen the takeaways.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)