Long-Form Speech Generation with Spoken Language Models
Authors: Se Jin Park
Julian Salazar
Aren Jansen
Keisuke Kinoshita
Yong Man Ro
RJ Skerry-Ryan
Venue: ICML 2025 oral (2025)
Keywords: spoken language models
long-form generation
state-space models
evaluation
Primary area: applications->language_speech_and_dialog
TL;DR: We introduce the first long-form spoken language model (16 min. of audio at once), discuss key design choices (e.g. state-space modeling), and propose new benchmarks.

Abstract: We consider the generative modeling of speech over multiple minutes, a requirement for long-form multimedia generation and audio-native voice assistants. However, textless spoken language models struggle to generate plausible speech past tens of seconds, due to high temporal resolution of speech tokens causing loss of coherence, architectural issues with long-sequence training or extrapolation, and memory costs at inference time. From these considerations we derive **SpeechSSM**, the first 
speech language model family to learn from and sample long-form spoken audio (e.g., 16 minutes of read or extemporaneous speech) in a single decoding session without text intermediates. SpeechSSMs leverage recent advances in linear-time sequence modeling to greatly surpass current Transformer spoken LMs in coherence and efficiency on multi-minute generations while still matching them at the utterance level.
As we found current spoken language evaluations uninformative, especially in this new long-form setting, we also introduce: **LibriSpeech-Long**, a benchmark for long-form speech evaluation; new embedding-based and LLM-judged metrics; and  quality measurements over length and time. Speech samples, the LibriSpeech-Long dataset, and any future code or model releases can be found at https://google.github.io/tacotron/publications/speechssm/.

Long-Form Speech Generation with Spoken Language Models
Se Jin Park 12
Julian Salazar Aren Jansen
Keisuke Kinoshita"
Yong Man Ro RJ Skerry-Ryan
Abstract
1000
We consider the generative modeling of speech multiple minutes, requirement for long- form multimedia generation and audio-native voice assistants. However, textless spoken lan- guage models struggle to generate plausible speech past tens of seconds, due to high temporal resolution of speech tokens causing loss of co- herence, architectural issues with long-sequence training or extrapolation, and memory costs inference time. From these considerations de- rive SpeechSSM, the first speech language model family to learn from and sample long-form spoken audio (e.g., 16 minutes of read or extemporane- ous speech) in a single decoding session with- out text intermediates. SpeechSSMs leverage re- cent advances in linear-time sequence modeling greatly surpass current Transformer spoken LMs in coherence and efficiency multi-minute gen- erations while still matching them at the utterance level. As we found current spoken language evalu- ations uninformative, especially in this new long- form setting, we also introduce: LibriSpeech- Long, a benchmark for long-form form speech eval- uation; embedding-basee and M.LM-uudeed metries; and quality measurements over length and time. Speech samples, the LibriSpeech-Lonm Long dataset, and any future code or model releases can be found at https://goolle.githbbb githuu.ii/taccooron// publications/sppechsssm/
SpeechSSM-X (ours)
800
Max Training Length (secs)
600
Moshi
SpeechSSM (ours)
Spirit LM
200
dGSLM
TWIST
AudioLM
Spectron
0
GSLM
200
400
600
800
1000
Max Evaluated Generation (secs)
Figure Maximum sequence lengths considered by various spo- ken LMs. Italicized models used text intermediates at generation time. Our models can generate indefinitely due to their constant memory footprint, but we cap our evaluations to 16 minutes.
its paralinguistic aspects, such as prosody (Kharitonov et al., 2022) and turn-taking (Nguyen al., 2023b). These capabil- ities make speech-native language models (LMs) promising for applications like media understanding and co-creation, audio-native voice assistants, and textless NLP. However, real-world use-cases of spoken LMs require the ability to both understand and generate long-form speech. For exam- ple, voice interactions can last many minutes, requiring a model to maintain a growing conversational history in real time, and expressive media like audiobooks and podcasts can require semantic, paralinguistic, and speaker coherence over a chapter or episode.
1. Introduction
Generative spoken language models (Lakhotia al., 2021; Dieleman et al., 2021; van den Oord et al., 2017) are autore- gressive models of invertible audio representations, enabling the direct learning and generation of intelligible speech and
This presents significant challenges for existing spoken lan- guage models, as spoken audio's s textual content entangled with paralinguistic and acoustic properties that detract from learning invariant representations of meaning. Additionally, current audio representations have high temporal rates, re- quiring 10+ speech tokens to cover the duration of 1-2 text tokens (Hassid et al., 2023). Hence, models must disentan- gle, aggregate, and generate content coherently over longer time horizons. 2017) LM is impractical, as its initial cost grows quadrati-
Priiaayy contributors. Se Jin's work was done as student researcher at Google DeepMind. Google DeepMind. *Integrated Vision and Language Lab, KAIST. Correspondence to: Se Jin Park <jinny9608 12 kaist.ac ac kr>, Julian Salazar <julsal @ google.com>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
A single-stage Transformer (Vaswani et al.,

---

Long-Form Speech Generation with Spoken Language Models
Prompt (ASR): "Oh, it's casy enough to make a fortune, Henry said. "It seems to be easier than it I begin think," " replied Philipp
SpeechSSM Continuation (ASR):
Spiritt Continuation (ASR):
"Well, If you' ve for fortune in your head, It's not that yet, Phillp sald. "But needo' 't attaid of Desert Mountain, a Rosalind Coral Mountain,
Well, you II see. sald his brother, getting in his hat and coat.
Now here's the second cup.
the book, you know, When you mention that forture, your head will get disturbed, and you'll find you all sorts pictures which give you a dinner time."
with your
Here's 's the second cup,
you 'i bear, take it. Huh.
Okay, pay five helper.
down now, and now, this time, don't say is here. Well. Depend on yourself. Airight. Here for quick off, two perice.
There was no telling what they had disturbed the day before. Phillip, who had a tendency to see the fallures of trave) with a good deal of ambition, and a perfectlly happy way, too, moved over to the beeee with the little river fronts
And with the thrust of his waist, to hasten his much quick, he went off. to get back, Phillp. He filled the cup he was thought, he felt little cup, with much made thick thought, from hra. Your universe, for your face because of experieence, for Virginia secretaty. tells how they'd have responded to the request. Well, the director it coming from fifty thousand students, and half lllon dollars of fundlngg for Confederate meets, lost urgent ascendancy, possum hospital, over fourteen days, preparing assemble, and questions University of Momgerson, Massachusetts, Medford Health and Medical Research and vullnerable training, And we' 've been trying have mock epic of any violent programs since 2017. Their for university and transfer program. For the simple part, freshly from droppin'.
But then he'll come back, top into agalin well
The great buildings resembled the pile of
banging down in length before them. The weather temperature also, was if Philip had laid and make money. There devil. And such
transsersallly clear. it would
pair of scissors, and
used the one
little hox, and tried poll the other giving songs
river
where people were abusing
body of music
been given Horace Barrows, who had come in
world Gross) and Tortel, to sing wonderful songs. The apple groves were in a small cataract where the people could pick out the little gardens full of ugly
metal, and stng their strange low melody, no better then their practice.
don't it think there''s is the nearest acrial to the traveling song to do, said Philipp
Great, Possible photograph from
"I didn't hear the music long ago, forl learned it in such tones," " replied
[Laughter]
Boraaee Barrows.
This PTSD. This likee me, This is great. This great. This great. This is
(more speech...)
what. This is great. This great. This great. This great. This is great. This
four c'clock, when Plerre called Prince Albert and explained what four o' o'clock had
great. This is great. This great.
great. This great. This is great.
had happened. The school was lost in completing
great. great. This great. This is great. great. great. This meant: This is great, This is great. This is great. This great. This is great. This is it. This great, This is great. This is great. This is it. This is probably the best when people are interrupting each other right now: You know, I know, I know what it's, you know I know what I fell, and I can't you right away. There's a rate, It's cutting out in this song, because It's quests, like Donald's, s. where the spirits listen though. Although, other stuff, I hate doesn't just focus artists. Perhaps putteers bait. It's balt, it's balt, says Actually, sald that
great. great.
been driven
Westehester in very few minutes.
is great.
great.
At last Philli was exhausted. He didn't 't bear a word, but entered the coach have some news to discuss.
and was driven by Prince Albert into the street.
Captalm Kidd will come for presently, was the cordial response of the
Princess Maria Theresa, who was still in her green satin. They all tallied gallantly, and Philip believed that it was genuine disaster. "They would have
take him for supger, and Mr. Phineus, suppose wansed to the handsome fellow,
sport. told you boy.
Philip smiling fran kly at the
(more sperch...)
(more speech..)
So Philln and Princess Marla went to look for him the Dragon Springs garden, and there most of the guests were gone. Philip was tertibly excited. "They must get some of the guests in kitchen, exc laimed the guests, selecting a bright fire in the bright, sunny spot. Phillp had never felt so happy in his life before. "Did
It's going to be. It's going be, It's goling to be. It's going to be. You can see it
tonight. It's going to be. It's going to be, So freaking so fun. So fricking fun. So fum. so fricking fun. So fricking. So fricking So fricking. So fricking. So fricking. So fricklig So fricking. So fticking So fricking So fricking So fricking. So fricking So fricking. So fricking. Su fricking. So fricking. So fricking so fricking, So fricking So fricking. So fricking. So fricking, So fricking. fricking So fricking, fricking.
fo
hear about the Dragon Spring?" he asked, after pause. "Yes, they say that when they were strong enough, the famous Albanians, British and Germans, were worthy
Figure 2. Automated transcripts of 4min speech continuations generated by SpeechSSM-2B (ours) and a Spirit LM Expressive (7B) model (Nguyen et al., 2025) under slide-and- anddproomp generation (Section 7), extending 10-second audio-only prompt from our proposed LibriSpeech-Lone Long (test-clean). Aspects like recurring proper nouns show SpeechSSM's relative semantic consistency over time.
cally with prompt length, and its per-step cost grows linearly when decoding. Furthermore, it may also be ineffective, as suggested by Transformer's s degraded performance on long- range tasks (Tay et al., 2021). Though a few works have improved speech coherence via joint modeling with text (Section 2), the challenge of directly modeling long-form speech, particularly generation, remains unstudied by exist- ing spoken LMs (Figure 1) and the field overall. Our work proposes and makes initial progress generative long-form speech:
eration evaluation are noisy and poorly discriminative, and propose the use of reference- based semantic metrics, side- by-side LLM-as-judge, and time-stratified evaluations for speech generation. To scale these to long-form evaluation, we introduce the LibriSpeech-Lone processes LibriSpeech's 's (Panayotov et al., 2015) dev and test sets' original chapter-level audio into utterance-aligned spans of up to 4 minutes. This enables much longer prompts and ground truths for reference-based evaluations in tasks like long-form speech continuation, speech recognition, and text to speech.
benchmark, which re-
Modeling, We discuss the design choices required to en- able the practical training, generation, and extrapolation to tens of minutes of audio, from tokenization to speaker conditioning complexity with respect to sequence length. The result is SpeechSSM, new (textless) spoken language model family (2B, 9B) designed for long-form generation, being the first to model and generate unbounded long-form speech in bounded memory and the first state-space spoken LM. As baselines, we also train spoken Transformer LMs to perform multi-minute generations. Finally, we demonstrate SpeechSSM-X, an extemporaneous variant for naturalistic spontaneous speech.
We find that SpeechSSM matches existing spoken LMs on short generations, while outperforming their sliding win- dow extensions on long generations (e.g., Figure 2), and that our proposed metrics and benchmark distinctly quantify the quality gaps between past work, our work, and human- level speech release examples of read- and extemporaneous--ttye gen- erations of up to 16 minutes in length,' 1 and we release the LibriSpeech-LLng evaluation dataser² under a CC-BY 4.0 license.
-enattiinnnnnnnnnnggggegn future development. We
hhtps://goolle.githbb http://wwgg..tuuuu..oooooooooooo wwt.a/iwwww..cccooooooolloosspoo http://hhhhhuooooo/o/ooddded//id
Evaluation. We observe that existing metrics in speech gen-
2

---

Long-Form Speech Generation with Spoken Language Models
2. Related Works
(auto-BLEU), along with human evaluations of intelligibil- ity and meaningfulness with mean opinion scores (MOS and MMOS respectively). For their spoken LMs, zero-shot (non- (oon-generative) metrics based on logprobs of contrastive pairs (sWUGGY and sBLIMM; Nguyen et al., 2020) were predictive of generation performance, though scores varied with token vocabulary size. However, these initial metries seem to lack robustness or are saturating with respect to newer spoken LMs. Hassid et al. (2023) found transcript PPL and auto-BLEU to be noisy, favoring MMOS and ex- panding zero-shot metrics (sStoryCloze and tStoryCloze). In turn, Défossez et al. (2024) found that SWUGGY and sBLIMP scores degraded despite experiential improvement from noise augmentation and instruction-tuma -unnin,, instead fa- voring spoken question- answering (Nachmani et al., 2024) evaluated via ASR.
Generating with Spoken LMs. The family of GSLM mod- els (Lakhotia et al., 2021; Kharitonov et al., 2022) are Transformer decoder LMs trained on discrete units obtained from k-means clustering of HuBERT (Hsu et al., 2021) features and synthesized via speetto--peetrogram or unit-to- waveform models. This approach gave promising temporal coherence but poor audio quality, and so AudioLM (Bor- sos et al., 2023a) proposed separate LMs, one for semantic tokens as before, and two for modeling coarse-to-fine acous- tic tokens that are residual codes of neural audio codec (Zeghidour et al., 2022); this was simplified and made non- autoregressive by Borsos et al. (2023b). TWIST (Hassid et al., 2023) found that text LM initialization improved content-level coherence, atop which VoxtLM (Maiti et al., 2024) and Spirit LM (Nguyen et al., 2025) found that joint or interleaved training with text gave further improvements.
Closest to our work was the use of LLMSs to assign absolute, reference-free scores to assess the instruction-oolloinng of turn-based text-and-speech LMs in Zhang et al. (2023a; 2024). As for observing saturation, Borsos et al. (2023a) found that humans could not distinguish between syn- thetic 7s continuation versus the real 7s continuation of a 3s prompt on holistic side-by-side evaluation, suggesting the need for more targeted and longer-form evaluations.
Beyond the scope of this work, there are audio-text models like SpeechGPT (Zhang et al., 2023a) trained for sequence- to-sequence and not generative continuation; there are also dual-channel and -ext-intermediate models like dGSLM (Nguyen et al., 2023b) whose semantic evaluations are <20s, Spectron (Nachmani et al., 2024) which passes through text, and Moshi (Défossez et al., 2024) which had few-minute dialogues via time-aligned text.
3. Unbounded Speech Generation
State-Space Models for Long-Form Audio. State-space models (SSMs; Gu et al., 2021) have become popular among efficient (sub-quadratic) replacements for Transformer- based architectures, giving the first model (S4; Gu et al., 2022) to perform all tasks in the Long-Range Arena (Tay et al., 2021), outperforming the vanilla Transformer. They utilize constant computation and memory requirements to generate tokens during inference and can be efficiently trained. Recent focus has shifted to hybrid models (Glo- rioso et al., 2024: Lenz et al.. 2025; De et al., 2024) which integrate state-space layers and variants like linear recurrent units (LRU; Orvieto et al., 2023) with finite-context self- attention layers. Recent works have considered SSMs in audio, primarily support long speech imputs for text-out tasks like automatic speech recognition (ASR) and summa- rization. None are spoken LMs for speech continuation, with only one considering (acoustic-level) tokens (Gao & Chen, 2024); most works involve spectrogram encoders or outputs (Shams et al., 2024; Erol et al., 2024; Lin & Hu, 2024; Miyazaki et al., 2024). Closest in spirit is SaShiMi (Goel et al., 2022), a multi-scale S4 operating directly on waveform samples; though they generated only 1s of speech, this corresponds sequence of 16k discretized scalars.
We begin by proposing a set of requirements for a general, unbounded, speech generation system:
Constant memory during decoding, to enable indef- inite AR sampling without running out of memory.
Infinite context, so that arbitrarily distant dependen- cies can be expressed, least in theory. With the above, this means relevant context must fit in a fixed-size state.
Generative length extrapolation, so that speech qual- ity remains consistent over time, in particular beyond audio durations seen during training.
The first leads to linear-complexity sequence modeling with a fixed-size state. The second leads us to models with aggregation mechanisms such as recurrences or compressive memories. We show that with some care, one can also achieve the third requirement of generative extrapolation.
Finally, there is also a soft requirement for efficient training. e.g., train-time dependence on sequence length that sub- quadratic, to enable longer sequences and reduce reliance on extrapolation. This favors a parallelizable weight learn- ing scheme, which naturally leads to state-space models (broadly defined, i.e., including linear recurrence models and certain hybrid variants; Patro & Agneeswaran, 2024; Dao & Gu, 2024) and thus SpeechSSM, a family of hybrid
Evaluating Spoken LM Generations. Lakhotia et al. (2021) was first to evaluate the generations of spoken LMs, proposing ASR as a path to automated text metries like text perplexity (PPL) and proportion of repeated k-grams
3

---

Long-Form Speech Generation with Spoken Language Models
SpeechSSM
USM-v2 Tokens
USM-v2
SoundStream Tokens
Tokens
Windowing
USM v/ Encoder
Embedder
Recurrent
Residual Block Block
Recurrent Residual Blockk Block
Local Attention
Residual Block Block
Embedder
Windowing
SoundStorm
SoundStream Decoder
10
X N
Speoker Reference
Figure 3. Overview of SpeechSSM. Left: A causally-masker hybrid state-space model (Griffin) is trained with an LM objective on semantic tokens (USM-v2) encoded via overlapping fixed-size windows. Right: autorrrrrreeeevvvee synthesizer (SoundStorm) converts overlapping windows of semantic tokens to the acoustic tokens a neural codee (SoundStream) in a cenditr-cond manner.
state-space spoken language models for efficient long-form speech generation that fulfills all these desiderata:
HH
Architecture. For our decoder-only hybrid SSM we choose Griffin (De et al., 2024), which interleaves gated variant of LRUs (Orvieto et al., 2023) and local (sliding-window) multi-query attention (MQA) blocks in fixed pattern (two recurrent, one local-MQA; see Figure 3, left). Local at- tention efficiently captures recent context, while the states of the gated recurrences transmit information across arbi- trary distances. Griffin's performance matched comparable Transformers while greatly improving inference speed and enabling context-side extrapolation least 4x longer than seen in training. As RoPE (Su et al., 2024) in the local-MQA blocks still encodes absolute position, we follow recent work on position embeddings (PEs) under causal self-attention (NoPE; Kazemnejad et al., 2023) and remove all explicit PEs from SpeechSSM to promote extrapolation.
Repasted Padding
With
(a) Tokenization
(b) Decoding
Figure 4. Depiction of how input and output windowing work, shown here with 5-token window widths and 2-token overlaps.
use a SoundStorm model (Borsos et al., 2023b) to non- autoregressively generate SoundStream tokens (Zeghidour et al., 2022), a standard neural audio codec that efficiently reconstructs to high-quality audio. Notably, one can train SoundStorm support 38 voice prompts frozen prefix of semantic and acoustic tokens), such that output acoustic tokens reflect its speaker characteristics.
Initialization. Inspired by Hassid et (2023)'s success with text-initialized spoken language models (TWIST), we initialize models with RecurrentGemma- /2B. )2 IT (Botev et al., 2024), which are open-weight LMs with the Griffin architecture, trained on 2 trillion text tokens. We discard the pretrained text token embeddings and initialize new ones for our audio token vocabulary.
By choosing a token and model decomposition that isolates speaker characteristics to the acoustic stage, SpeechSSM focuses capacity on modeling semantic coherence along the temporal axis.
Semantic Tokenizer. We use the pretrained USM-v2 speech tokenizer (Vashishth et al., 2024; Rubenstein al., 2023). Its encoder (Zhang et al., 2023b) is trained with masked language modeling on untranscribed audio and an auxiliary ASR loss on transcribed audio. Inner represen- tations are vector-quantized into 32k units that serve as fixed-rate (25Hz) pseudo-text for our speech LM. Vashishth al. (2024) found that USM-v2 was by far the most speaker- invariant token among common speech tokenizers.
Windowed Tokenization and Decoding. To process long- form speech while bounding the memory of the (non-SSM) semantic tokenizer and acoustic decoder, we divide audio into fixed segments, with overlaps between neighbors. Each window is tokenized independently, then merged into a sin- gle stream each overlap by taking the first half's tokens from one window and the second half's tokens from its successor (Figure 4a). For synthesis, fixed token windows, each conditioned on a short speaker prompt (3s), are syn- thesized into waveform independently then merged with the same boundary overlap adjustment (Figure 4b). We find these strategies minimize boundary artifacts while enabling continuous tokenization and decoding over time.
Speaker-Prone Prompted Audio Synthesis. Following Borsos et al. (2023a), we have second stage that generates low- level acoustic tokens conditioned on semantic tokens. We

---

Long-Form Speech Generation with Spoken Language Models
Avoiding Implicit EOSes. Despite having no end-of- sequence (EOS) tokens, our early models did not gener- atively extrapolate (e.g., a 4min model reaching 4.5min before degrading to noise/silence). In non-causal semantic tokenizers like USM-v2, we found the remaining length-in- window may be implicitly encoded in tokens, making tokens in final windows look "different." As evidence, padding the last window to 30s using silence, tokenizing, then dropping those tokens led to silence, as "future" silence was now in the kept tokens. What worked was (1) to pad the last win- dow to 30s using speech from the beginning of the example (depicted in Figure 4a) so that final tokens were tokenized as if there was further speech, and (2) in the case of Libri- Light, to still drop the last 10s of examples- as the padded beginnings were disproportionately "Chapter <number>"! "!
Table I. Statistics of our proposed LibriSpeech- Long benchmark, which was generated with maximum target duration of 4 minutes.
Subset
Hours # Examples
Avg. Dur. (s)
# Chapters # Spkrs
dev-clean
16.0
295
194.8
97
40
dev-other
9.5
188
182.4
91
33
test-clean
14.8
270
197.6
87
40
>3.5min
12.6
193
234.2
82
40
test-other
10.7
207
185.9
90
33
>3.5min
8.2
126
234.4
77
32
of public-domain audiobooks which have been excluded from the standard LibriLight training set (Kahn et al., 2020) used by many spoken LMs, we reprocessed their source uncut audio files, similar to the convenience script in Libri- Light which agglomerates utterances up to target length of 4 minutes (240s) along utterance boundaries. The results enable longer prompts and references for side-by-side and similarity metrics. Statisties are shown in Table I; 64%- 76% of each split's examples are >3min long.
4. Improved Evaluations for Spoken LMs
Updated NLG Evaluations. The shortcomings found in recent work (Section 2) align with recent developments in natural language generation (NLG) evaluation, which have moved beyond intrinsic and/or surface word metrics like PPL, auto-BLEU, self-BLEU (Zhu al., 2018), especially for open-ended generation. One major shift has been the metries, where distances are computed between embeddings of generated versus ref- erence text (Sai et al., 2022). A more recent trend uses instruction-tuned LLMs to perform automated Likert-scaled evaluations (Li et al., 2024b), as applied by Zhang et al. (2023a; 2024) to text-instructed speech generation and could be extended to speech as well.
Generation Quality over Time. While ASR can capture degenerate cases like repeated words (Figure 2), we found that it can fail on cases exacerbated by long-form generation, such as extended silences and voiced non-speech, suggest- ing the continued importance of audio-native evaluations. Furthermore, we find that generation failures generally in- crease as decoding progresses over time, which we must quantify to determine if our model has the desired prop- city of generative length extrapolation (Section 3). Towards this, we propose computing semantic and acoustic met- ries that are stratified over the decoding process. This progression can be expressed in terms of semantic content (number of words into the ASR transcript), or acoustic con- tent (time offset into the generated speech). We describe our implementations of both in Section 7.2.
adoption of embedding-basse
However, to tackle the saturation of acoustic evaluations and to leverage text references, we in particular propose auto- mated side-by-sides (LL.Maas---Judge) Zheng et al., 2023) of total transcripts to scalably compare systems against the ground truth and each other. This has particular advantages for spoken LMs: (I) It mitigates the noise from ASR issues highlighted by Hassid et al. (2023), as the compared gener- ations will be both afflicted (the ground truth should also be re-transcribed by ASR, for fairness). (2) It works around the subtle issue of fixed-duration slices occurring mid-word, degrading PPLs; instead, one can always transcribe prompt and continuation together, leaving the L.M to focus on the contrast. We implement both proposed evaluation types in Sections 6.2 and 7.1.
5. Experimental Setup
Training and Generation. Following Borsos et al. (2023a), Nachmani et al. (2024), and others, we train on the unlab- 60k split from LibriLight (Kahn et al., 2020). Unlike prior work, we study the effect of sequence length on long-form generation, segmenting the audiobooks into train- ing sequences of up to a target duration. The default for SpeechSSM is 4min (240s) during training, though we com- pare with target durations of 30s and 16min (960s) as well ("with 30s/16mih, segments"). We train -2B and -9B variants, corresponding to RecurrentGemma. Each model is trained with 16 TPUs (v5p) and data parallelism for 100k steps with 768k tokens per batch, and a checkpoint is chosen via transcript perplexity on LibriSpeech-
To extend reference- baseed metrics to long-form speech, one needs long-form reference speech and transcripts. Although 38 prompts from LibriSpeech (Panayotov et al., 2015) have been the standard benchmark for spoken LMs since Lakhotia et al. (2021), their provided references have an average length of 10s, making them un- suitable beyond 7-10s continuations. Observing that the LibriSpeech dev and test sets are derived from full chapters
LibriSpeech-LLong
dev-clean; more details in Appendix B.1. We sample semantic (USM-v2) tokens with temperature 1. Then the SoundStorm model
hinni-lghhddaaappppaaaainnnn vad on GitHub.
5

---

Long-Form Speech Generation with Spoken Language Models
Table 2. Results on short-form generation LibriSpeech test-clean. Generations are 7s continuations of 38 prompts. Bolded ours. Text-PT, F, FT denote pretraining (via LM init.) and finetuning with text. Win denotes the win rate of the model over the ground truth. |Vaniol denotes speech token vocabulary size. For naturalness mean opinion score (N-MOS) report 99% confidence intervals.
Text-Based (ASR)
Speech-Basee
Method
Text-PT
Text-FT
PPL.
SBERTT
Win
Vandoo
SpkrSimt
N-MOS1
SWUGGY1
sBLiMP
GSLM (0.2B)
X
6.28
0.17
1.4
100
0.36
2.23 =.13
64.8
54.2
AudioLM (0,9B)
X
1k
71.5
64.7
TWIST-L.3B
V
7.25
0.18
3.6
500
0.41
3.09 = 0.12
72.7
57.0
TWIST-7B
6,54
0.20
15.5
500
0.41
3.24 0.13
73.9
59.0
(1.3LM(
57.1
Spirit LM Expressive (7B)
6.17
0.19
7.7
665
0.45
3.00 0.08
65.0
54.2
SpeechSSM-2B
5.76
0.23
7.9
32k
0.79
3.87 = 0,07
55.8
60.9
without LM init.
X
6.15
0.23
7.7
32k
0.79
3.80 0.07
with Transformer instead
6.16
0.22
8.4
32k
0.79
3.74 + 0.08
with 30s segs. instead
5.73
0.22
10.5
32k
0.79
3.84 = 0.08
57.3
61.1
with 16min segs. instead
5.84
0.20
4.0
32k
0.79
3.86 + 0.07
54.3
60.4
SpeechSSM-9B
5.60
0.23
13.5
32k
0.79
3.94 0.07
Ground Truth
5.63
1.00
50.0
0.84
4,02 # 0.07
prikkkee with the first seconds of the prompt) and windowing (30s with 4s overlaps) give acoustic (Sound- Stream) tokens, which a SoundStream codec turns into waveform; model details for these are in Appendix B.2.
continuation under Gemma-2B. as an initial proxy for con- tent fluency.
Speaker Similarity (SpkrSim): To analyze voice preser- vation, we speaker-embed both the prompt and its generated continuation and compute their cosine similarity. We use a speaker classifier used in AudioLM (Borsos et al., 2023a) as the speaker embedder.
Baselines. We compare SpeechSSM with decoder-only speech LMs. We use GSLM (Lakhotia et al., 2021)'s best model (HuBERT-Ló tokens with vocab 200), trained on the clean 6k hours of LibriLight. For TWIST (Hassid et al., 2023), use both the OPT-1.3B and LLaMA-7B versions, trained on 150k speech hours. For the 7B Spirit LM (Nguyen et al., 2025), we use the Expressive version model which adds expressiveness via pitch and style tokens in addition to HuBERT tokens. We also cite numbers from AudioLM (Borsos al., 2023a) and VoxtLM (Maiti et al., 2024), which are both 2B models trained on unlab-60k. VoxtLM and Spirit LM see text data during training. Due to variations in token, initialization, and training data choices, we also define SpeechTransformer ("with Transformer", a spoken LM initialized with Gemma-2B (Gemma Team et al., 2024) but otherwise matched with SpeechSSM-2B.
Naturalness Mean Opinion Score (N-MOS): We evaluate how natural the speech sounds, ignoring the grammar and content of the utterance; this focuses attention on issues not visible on transcripts, ranging from synthesis issues and unintelligible speech through to coherent but unnatural prosody; rater details in Appendix C.
SUUGGY and SBLiMP: These probe the spoken lan- guage model can implicitly perform lexical and syntactic contrasts (Nguyen et al., 2020).² One reports the & of time the model's log-likelihood ranks the semantic tokens of a correct utterance versus its incorrect counterpart.
Our models' continuations (Table 2) are most speaker- similar to the prompt, which we attribute to our speaker- promptable acoustic stage; in contrast, GSLM and TWIST can only propagate speaker identity via their semantic to- kens (plus coarse style and pitch tokens in Spirit LM Expres- sive). Our N-MOS scores suggest high semantic-to -o---ttoouuuoo synthesis quality, which we attribute to USM-v2's large vo- cabulary (32k) and the staged approach via existing codec (SoundStream). SpeechSSM's naturalness on par or better than comparable Transformer and close real speech; see supplementary. Meanwhile, sWUGGY scores much worse, while our sBLiMP scores are neutral to above-
6. Short-Form Continuation Experiments
Before considering long-fone form. compare SpeechSSM to its Transformer-be basdd counterparts as in past work. This takes 3s prefixes from LibriSpeech's 's test-clean set and gen- erates 7s continuations, which one then transcrihes with an ASR model (our details in Appendix B.3). We used test examples with ground-truth continuations 25s.
6.1. Existing Metrics and Results
Transcript Perplexity (PPL): As in prior work, we com- pute the log-perplyxity of the transeript of the generated
WUUGYY audio pairs of real word and fake but similar- sounding word. sBLIMP: audio pairs of syntactically correct sentence and an incorrect one.
Note that Gemma saw 50% more text than RecurrentGemma.
6

---

Long-Form Speech Generation with Spoken Language Models
average. These do not positively correlate with text-based scores or subjective quality; instead, they match Lakhotia et al. (2021)'s observations and Borsos et al. (2023a)'s Fig- ure 2. where sWUGGY scores hit relatively sharp maxima at vocabularies of a few hundred tokens. We argue increased noise is expected with larger vocabularies, as then tokens of an audio utterance represents less of the probability mass of all possible renditions of its text. Hence, separately from Défossez et al. (2024), we move towards transcript-based evaluations, as these marginalize over spoken renditions to get less noisy evaluations. Finally, even the 2B SpeechSSMs had the lowest transcript perplexities, which is surprising given the other models are larger (7B) and/or have jointly trained with text, and one would not expect SSMs to confer semantic edge in the <10s speech horizon. We note that Lakhotia et al. (2021) already caveated-and Hassid al. (2023) actively discouraged the use of ASR PPL given its sensitivity to e.g. audio sampling temperature; such metrics may simply indicate model repetitiveness at default settings.
inative than a holistic human side-by-side audio task in selecting the synthetic sample, where humans performed random in Borsos et al. (2023a), as it focuses on the content of the speech rather than superficial naturalness. Though our model is not the most fluent in this regime, SBERT fore- shadows benefits in faithfulness (our models' continuations are semantically closest to the true ones).
7. Long-Form Generation Experiments
We conduct the first evaluation of long-form speech gener- ation, taking extended prompts of 10s from our proposed LibriSpeech-Long (test-clean) and having each model con- tinue them through to 4 and 16 minutes. As other off-the- shelf models trained on sequences well below 4min and were not designed generate beyond their training length (e.g,, use of position encodings), we found them unable to generate beyond a minute without being stuck in noise or silence. To give functional baselines, we propose applying slide-and-prompl generation (Borsos et al., 2023b) to the semantic LM itself; that is, we generate to each model's maximum completion length (Figure 1) first, and then re- peat generation using the last 3s of the previous window as context. Example generations are in Appendix E.
6.2. Proposed Metrics and Results
The above points-noise from contrastive audio probes, saturation of N-MOS and SpkrSim, suspicious results from transcript perplexity-aa all motivate our proposed shift to newer, reference-based NLG metrics (Section 4):
7.1. Semantic and Acoustic Results
Semantic Similarity (SBERT). We measure the distance between the semantic embedding of the transcriptions of the generated speech and the reference, using Sentence- BERT M-niL---L6-v2 (Reimers & Gurevych, 2019) as the semantic embedder. This expresses contextual alignment between the generated text to the ground truth, focusing on semantic meaning over surface-form patterns.
We again measure existing and our proposed metrics, with two key changes: first, we replace Centence-BERY with Gecko (Lee et al., 2024) as embedder for semantic similarity as Sentence- Centence-BERTTs 51 512-token context cannot handle the transcripts of 4min+ generations; more long-form evaluation details in Appendix B.3. Second, extrapolation failures cause e.g.. GSLM and TWIST to generate far less than our models; for faireess in side-by-sides, we truncate to the shorter transcript's 's length.
Side-by-Side Win Rates (Win vo.. model). We ask the model to analyze then rate (Chiang & Lee, 2023), fork- ing the format of Arena-Hard-Auto LLM-Judge System Instruction (Li et al., 2024a). Given the book domain and the relatively unconstrained nature of speech continuation, we base our criteria on questionnaires from story generation evaluation (Xie et al., 2023) around fluency, coherence, logi- cality. and interesttinnnss;; see Appendix D for the template. We evaluate each prompt twice with order of presentation flipped. Gemini 2.0 Flash (Gemini Team et al., 2025) re- transcribes both model and true audio (without windowing and jointly with prefix) and performs judgments.
Our results are in Table 3. The SpkrSims of Speech former, SpeechSSM, and ground truth increased from 0.79 in short-form to 0.85, likely from increased confidence from longer prompt and continuations; all other decreased. This shows the advantage of our more speaker-invariant USM- v2 tokens and a modulated by the semantic-to-acouse model, instead of consuming capacity, and being imperfectly carried by, the semantic LM and windowing. SpeechSSM has the best ASR PPL and wins a majority of time vs. all models; our 2B Transformer variant is close at 30sec, but not at 16min (Table 4) recall that 16min is 24,000 USM-v2 tokens!
audio stage; identity is
Benefits are evident in side-by-sides versus a transcript of the ground truth, with results (Table 2) now matching qualitative experience and expectations (larger models per- forming better, with Spirit LM Expressive underperform- ing TWIST perhaps due to capacity spent pitch/style). However, even SpeechSSM-9B and TWIST-7B win <20% versus transcribed ground truth, suggesting that (automated) bide comparison on transcripts is more discrim-
However, models achieve zero wins over the ground truth. As generation length increases, faults in fluency, coher-
'Note this means the relative order of models single Win model column should not he read into too much as chal- lenger models induce truncation to very different lengths.
7

---

Long-Form Speech Generation with Spoken Language Models
Table 3. Evaluations on LibriSpeech-Long test-clean. I denotes model extension via windowed generation. Generations are 4m completions of the 10s prompt. Win zu and WinSter are model wins versus SpeechSSM-28 and ground truths (>3.5min) respectively. For naturalness MOS over time (N-MOS-7), the same 5s time span is sampled over all models from each minute of completion.
Text-Based (ASR)
Speech- Based
Method
N-MOS-TT 1 (99% CI)
PPL.
Geckot
Wi
WinSarr
SpkrSimt
< <miinn
1-2min
2-3min
23min
GSLM (0.28) O
4.74
0.67
17.6
0.0
0.33
2.00 0.22
2.01 0.23
1,93 0.21
2.09 i 0.22
TWIST-1.3B
5.70
0.60
16.2
0.0
0.38
1966 =
1.59 0.22
1.40 0.09
0.09
TWIST-78*
4.93
0.65
24.0
0.0
0.37
1.83 + 0.20
1.84 0.18
Spirit LM Expressive (7B)
5.71
0.63
24.2
0.0
0.41
2.77 2
2.52
2.56 + 0.11
2.60 + 0.10
SpeechSSM-2B
3.75
0.70
50.0
0.0
0.85
4.12 0.08
4.13 0.06
4,13 = 0.07
4,16 0.08
withou init.
4.57
0.71
42.4
0.85
3.76
3.76
3.71 0.10
3.80 0.08
with Transformer instead
4.71
0.71
33.9
0.0
0.85
4.03
3.88 0.08
0.09
3.89 0.12
with 16min segments instead
3.83
0.70
46.1
0.0
0.85
3.83 0.08
3.89
3.87 = 0.09
3.88 i 0.09
SpeechSSM-99
3.57
0.71
75.0
0.0
0.85
3.86 0.08
+
3.88 + 0.07
3.95 + 0.07
Ground Truth
3.67
1.00
100.0
50.0
0.92
4.12 0.08
4,11 + 0.08
4.10 0.08
4.09 + 0.08
Table 4. 16min completions of 10s prefixes of our LibriSpeech- Long test-clean. As there are no 16min ground truths, we only take reference-fiee free metrics. B denotes model extension via slide- and-prone prompt. Win denotes win rate over SpeechSSM-2B.
Ground Truth
SPeechSSM-2B
0.56
GSLM
Tramsformer
TWIST-L.3B
w/ 16min segs.
0.54
TWIST-78*
SpeechS
Semantic similarity
Spirit LM Expr.
0.52
Method
PPI.,
Win 1
GSLM (0.2B)
4.48
18.4
0.50
TWIST-L.3B
4.76
17.5
T
4.45
32.8
0.48
Spirit LM Expressive (7B)
5.66
37.7
SpeechSSM-2B
3.59
50.0
with Tronsformer instead
5.91
24.9
0.44
with 16min segments instead
3.55
48.4
SpeechSSM-99
3.39
68.3
0.42
6
12
14
18
20
Distance from the prompt (x100 words)
ence, logicality and interestingness become increasingly apparent, showing that side-by-side comparison versus LibriSpeech-Long ground-truth, is a new and difficult benchmark for (long-form) spoken language generation.
Figure 5. Semantic similarity between a 10s prompt and the 100- word segment starting word 100L in the 16min completion, as measured by cosine similarity of Gecko embeddings (SC-L).
7.2. Proposed Extrapolation Metrics and Results
that of each 100-word segment 01006:10((+-1)-
Naturalness Mean Opinion Score over Time (N-MOS- T). To balance cost and informativeness, for each exam- ple select 5 sec. span from each minute; specifically [fprompt 60), [60, 120), [120, 180), and [180, ) where (promps is prompt duration and max is ground truth duration, and extract each span's 's audio from every model's 's generated continuation.
These metrics, motivated in Section 4 and shown in Ta- bles 3 and 4, show that windowed model extension is poor length extrapolator. GSLM and TWIST are al- ready low in our proposed N-MOS-T by minute one, having trained on even shorter sequences. Spirit LM, which has seen sequences up to 200s (though rarely, due to text inter- leaves), degrades acoustically over time, though slower.
Semantic Coherence over Length (SC-L). To evaluate se- mantic faithfulness the original prompt over time while mitigating the effect of speech rate, we take each continua- tion's 's transcripts and divide them into spans of 100 words (as determined by whitespace). Each 100-word segment represents 30 seconds of speech, with the advantage of normalizing out differences in generated speech rate as well as degenerate silences. Our SC scores are cosine similarities between the embedding of the original prompt Opramps with
Our proposed SC-L is plotted for 16 minutes in Figure 5 (table in Appendix A). As generation progresses, we see a decline in SC-L scores, aligning with the natural flow of speech starting on topic and evolving over time. However, existing models except GSLM sharply drop in semantic coherence (SC) scores around 200 words (~ 1min; around their training lengths). GSLM fares better as its failures are untranscribed noise which do not show in SC-L (but do
8

---

Long-Form Speech Generation with Spoken Language Models
in N-MOS-T), but still worse than models which are closest to the 3-4min ground truths' performance. At 16min (Table 4, Figure 5), we do not see obvious degradation from SpeechSSM-2B generating beyond its training length. In contrast, TrneeccTraanoommerr-.. win-rate, and SC scores up to its training length, but these metrics quickly degrade at 16min. This suggests an edge for SSMs in long-form speech generation. complement- ing past work (Gu al., 2022). In all, SpeechSSM's is design and architecture lead to generative length extrapolation.
Max tokens/sec decoded
35k
SpeechSSM-2B
30k
w/ Transformer
25k
had comparable PPI.,
20k
15k
10k
5k
128
256
512
1024
2048 4096 8192 16.4k
Sample length
7.3. Qualitative Discussion
Figure 6. Max throughput under batch decoding per model and sampling length on one TPU v5e (unconditional generation).
We invite the reader to listen to the samples our website. For convenience, Figure 2 shows that even SpeechSSM-2B generates intelligible and more coherent speech over the 4min duration, with ongoing references to the "Philip" in the prompt, along with new recurring entities like "Prince Maria", "Prince Albert", and "Horace Barrows" also consis- tently appear, maintaining contextual relevance. We provide further transcripts for ours well other models in Ap- pendix E, showing how their decline in SC-L is qualitatively visible as e.g., degeneration into repetitive outputs.
500
SpeechSSM-2B
Time taken (sec)
400
w/ Transformer
300
200
100
0
Given our use of windows during synthesis (Section 3), one may wonder if audio quality differs at transitions (every 25 + 23n seconds; the multiple comes from 30s, minus 3s for the prompt and 2s truncation per side). We ran hu- man evaluation comparing 5-second windows centered at chunk boundaries vs. chunk midpoints, and found no clear effect on MOSes or six categories of aater-flaggable issues (Appendix A.2), which matches our informal listening."
128
256
512
1024 2048 4096 8192 16.4k Sample length
Figure 7. Time taken to decode a single sample (batch size 1) to a target length on one TPU v5c (unconditional generation).
7.5. Extemporaneous Speech Generation
With few exceptions (dGSLM, Spirit LM Expressive), most spoken LMS are trained on read speech such as Libri- Light. However, long-form multimedia and assistant ap- plications likely require modeling of spontaneous speech, which has its own long-term discourse structures (e.g., podcasts; Nishimura et al., 2025). Hence, we also devel- oped SpeechSSM-X, a 2B SpeechSSM model trained on 216k-hour corpus of eXtemporancous monologues (see Appendix B.4 for details). We find that it is able gener- ate natural monologue speech in more informal, extem- poraneous style, while showing similar coherence at the multi-sentence level; see website for examples.
7.4. Inference Efficiency
Throughput measures the maximum number of tokens that can be successfully decoded per second given fixed memory, e.g., by increasing the batch size. In Figure 6, SpeechSSM attains higher throughput due to its recurrent layers that maintain a constant-size state; furthermore, its self-attention has maximum context of size 2048, giving a lower bound. In all, this confirms SpeechSSM's 's capped memory use and thus capacity for unbounded generation, with >120x the throughput of SpeechTransSonee Transformer once de- coding 16.4k-token sequences in batch (also depicted in Ap- pendix A.3). Decoding time measures the speed at which a sample can be decoded to a desired length Figure We see that SpeechSSM also increases speed per step versus SpeechTranstone
8. Conclusion
We considered the task of generative modeling of long-form speech. For modeling. this led us to SpeechSSM, the first spoken state-space LM, allowing generation than can go in- definitely without running out of memory. For evaluation created the LibriSpeech-Long benchmark and proposed new evaluations for long-form speech continuation. We hope our work will simplify and enable new audio generation ap- plications involving long-for form media, such as audiobooks, podcasts, voice agent sessions, and video-related content.
additional factor in SpeechSSM's improved throughput. On the TPU v5e, the 2B SpeechSSM decodes 16.4k tokens (roughly 10.9 minutes) in just over 100 seconds, a real-time factor well under 0.2x.
Thha said, targeted listening does reveal e.g., subtle loudness shifts at times; still a massive reduction versus synthesizing without overlaps without fixed prompt.

---

Botev, A., De. S., Smith, S. L., Fernando, A., Muraru, G., Haroun, R., Berrada, L.., Pascanu, R., Sessa, G., Dadashi, R., Hussenot, L., Ferret, J., Girgin, S., Bachem, O., Andreev, A., Kenealy, K., Mesnard, T., Hardin, C., Bhupatiraju, S., Pathak, S., Sifree, L... Rivière, M., Kale, M. S., Love, J., Tafti, P., Joulin, A., Fiedel, N., Senter, E., Chen, Y., Srinivasan, S., Desjardins, G., Budden, D., Doucet, A., Vikram, S., Paszke, A., Gale, T., Borgeaud, S., Chen, C., Brock, A., Paterson, A., Brenman, J., Risdal, M., Gundluru, R., Devamathan, N., Mooney, P., Chauhan, N., Culliton, P., Martins, I.. G., Bandy, H., Huntsperger, D., Cameron, G., Zucker, A., Warkentin, T., Peran, L., Giang, M., Ghahramani, Z., Farabet, C., Kavukcuoglu, K., Hassabis, D., Hadsell, R., Teh, Y. W., and de Frietas, Long-Form Speech Generation with Spoken Language Models
Acknowledgements
language modeling approach audio generation. IEEE ACM Trans. Audio Speech Lang. Process., 31:2523-2533, 2023a.
We are grateful to Zalán Borsos for their technical feedback; to Tongzhou Chen, Roshan Sharma, and members of our Speech and Griffin teams for infrastructure support; and to Soroosh Mariooryad for general assistance. We also thank the anonymous reviewers and area chair for their support and helpful feedback.
Borsos, Z., Sharifi, M., Vincent, D., Kharitonov, E., Zeghi- dour, N., and Tagliasacchi, M. SoundStorm: Efficient parallel audio generation. CoRR, abs/2305.09636, 2023b.
Impact Statement
Progress on coherent, efficient, and unbounded audio-native speech language models will improve the availability and ac- cessibillity of mudio-based human- comanec interfaces, par- ticularly towards the generation of speech with paralinguis- tic nuances often inexpressible in a standard text-to-speech prompt, as well as enabling models in primarily oral lan- guages (textless NLP). should also stimulate the use of synthetic speech in long- long-form creative multimedia applica- tions, and could conceivably generalize to non-speech audio such music.
We recognize that increased efficiencies in speech language modeling may increase the proliferation of audio deepfakes or low-quality synthetic media. However, for these uses we do not believe our work exacerbates what is already achievable (at higher content quality) by cascading text large language models (LLMs) with modern promeppro text-to-speech systems. We hope our work increases public awareness that direct methods in deep learning now enable large-scale generation of expressive speech, analogous to the situation in text media due to LLMs.
N. RecurrentGemma: Moving past Transformers for ef- ficient open language models. CoRR, abs/2404.07839, 2024.
Chiang, D. C. and Lee, H. A closer look into using large language models for automatic evaluation. In EMNLP (Findings), pp. 8928-8942. Association for Computa- tional Linguistics, 2023.
Dao, T. and Gu, A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In ICML. OpenReview.net, 2024.
As in prior work for generative speech language modeling, speaker prompts used in our read-speech demos come from public domain LibriVox audiobooks and are only used to generate spoken continuations in the same setting, albeit for longer durations. For our spontaneous-speed demos, the voices have been explicitly licensed for speech synthesis. We do not currently release model weights.
De, S., Smith, S. L., Fernando, A., Botev, A., Muraru, G., Gu, A., Haroun, R., Berrada, L.. Chen, Y., Srinivasan, S., Desjardins, G., Doucet, A., Budden, D., Teh, Y. W., Pascanu, R., de Freitas, N., and Gulcehre, C. Griffin: Mixing gated linear recurrences with local attention for efficient language models. CoRR, abs/2402. 19427, 2024.
References
Ardila, R., Branson, M., Davis, K., Kohler, M., Meyer, J., Henretty, M., Morais, R., Saunders, I.., Tyers, F. M., and Weber, G. Common Voice: A speech corpus. In LREC, pp. 4218 4218-4222. European Lan- guage Resources Association, 2020.
Défossez, A., Mazaré, L.,, Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., and Zeghidour, N. Moshi: speech- text foundation model for real-time dialogue. CORR, abs/2410.00037,. 2024.
Dieleman, S., Nash, C., Engel, J. H., and Simonyan, K. Variable-rate discrete representation learning. CoRR, abs/2103.06089, 2021.
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self supervied learning of speech representations. In NeurlPS, 2020.
Erol, M. H., Senocak, A., Feng. J., and Chung, J. S. Au- dio Mamba: Bidirectional state space model for audio representation learning. IEEE Signal Process. Lett., 31: 2975-2979, 2024.
Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D.. Teboul, O., Grang- ier, D., Tagliasacchi, M., and Zeghidour, N. AudioL A
10

---

Long-Form Speech Generation with Spoken Language Models
Gao, X. and Chen, N. F. Speech-Mamba: Long-context speech recognition with selective state spaces models. In SLT, pp. 1-8. IEEE, 2024.
Kharitonov, E., Lee, A., Polyak, A., Adi, Y., Copet, J., Lakhotia, K., Nguyen, T. A., Rivière, M., Mohamed, A., Dupoux, E., and Hsu, W. Text-free prosody-aware generative spoken language modeling. In ACL (1), PP. 8666-8681. Association for Computational Linguistics, 2022.
Gemini Team, Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., Marris, L.. Petulla, S., Gaffney, C.. Aharoni, A., Lintz, N., Pais, T. C., Jacobsson, H., Szpektor, I., Jiang, N.-J., Haridasan, K., Omran, A., Saun- shi, N., Bahri, D., Mishra, G., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. CoRR, abs/2507.06261, 2025.
Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J., Baevski, A., Mo- hamed, A., et al. On generative spoken language model- ing from raw audio. Transactions of the Association for Computational Linguistics, 9: 9:1336-13555 2021.
Lee, J., Dai, Z., Ren, X., Chen, B., Cer, D., Cole, J. R., Hui, K., Boratko, M., Kapadia, R., Ding, W., et al. Gecko: Versatile text embeddings distilled from large language models. CoRR, abs/2403 203277, 2024.
Gemma Team, Mesnard, T., Hardin, C., Dadashi, R., Bhu- patiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. CoRR, abs/2403.08295. 2024.
Lenz, B., Lieber, O., Arazi, A., Bergman, A., Manevich, A., Peleg, B., Aviram, B., Almagor, C., Fridman, C., Padnos, D., Gissin, D., Jannai, D., Muhlgay, D., Zimberg, D., Ger- ber, E. M., Dolev, E., Krakovsky, E., Safahi, E., Schwartz, E., Cohen, G., and al. Jamba: Hybrid Transformer- Mamba language models. In ICLR. OpenReview.net, 2025.
Glorioso, P., Anthony, Q., Tokpanov, Y., Whittington, J.,
Pilault, J., Ibrahim, A., and Millidge, B. Zamba: A compact 7B SSM hybrid model. CoRR, abs/2405.16712. 2024.
Goel, K., Gu, A., Donahue, C., and Ré, It's raw! Audio generation with state-space models. In ICML,, volume 162 of Proceedings of Machine Learning Research, pp. 7616-7636 PMLR, 2022.
Li, T., Chiang, W., Frick, E., Dunlap, L.,, Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena- Hard and BenchBuilder pipeline. CORR, abs/2406. 1939, 2024a.
Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., and Ré, C. Combining recurrent, convolutional, and continuous-time time models with linear state space layers. In NeurlPS, pp. 572-585 2021.
Li, Z., Xu, X., Shen, T., Xu, C., Gu, J., Lai, Y., Tao, C., and Ma, S. Leveraging large language models for NLG evaluation: Advances and challenges. In EMNLP, pp. 16028-16045. Association for Computational Linguistics, 2024b.
Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. In ICLR. Open- Review.net, 2022.
Hassid, M., Remez, T., Nguyen, T. A., Gat, Conneau, A., Kreuk, F., Copet. J., Défossez, A., Synnaeve, G., Dupoux, E., Schwartz, R., and Adi, Y. Textually pretrained speech language models. In NeurlPS, 2023.
Lin, J. and Hu, H. Audio Mamba: Pretrained audio state space model for audio tagging. CoRR, abs/2405. 13636, 2024.
Maiti, S., Peng, Y., Choi, S., Jung, J., Chang, X., and Watan- abe, S. VoxtLM: Unified decoder-only models for con- solidating speech recognition, synthesis and speech, text continuation tasks. In ICASSP, pp. 13326 13330. IEEE, 2024.
Hsu, W., Bolte, B., Tsai, Y. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE ACM Trans. Audio Speech Lang, Process., 29:3451-3444 3460, 2021.
Kahn, J., Riviere, M., Zheng. W., Kharitonov, E., Xu, Q., Mazaré, P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. Libri-Light: A benchmark for ASR with limited or no supervision. In ICASSP, pp. 7669-7673. IEEE, 2020.
Miyazaki, K., Masuyama, Y., and Murata, M. Exploring the capability of Mamba in speech applications. In INTER- SPEECH. ISCA, 2024.
Nachmani, E., Levkovitch, A., Hirsch, R., Salazar, J., Asawaroengchai, C., Mariooryad, S., Rivlin, E., Skerry- Ryan, R. J., and Ramanovich, M. T. Spoken question answering and speech continuation using spectrogram- powered L.M., In ICLR. OpenReview.net, 2024.
Kazemnejad, A., Padhi, I., Ramamurthy, K. N., Das, P., and Reddy, S. The impact of positional encoding on length generalization in Transformers. In NeurlPS, 2023.

---

Long-Form Speeeh Generation with Spoken Language Models
Nguyen, T. A., de Seyssel, M., Rozé, P., Rivière, M., Kharitonov, F.,, Baevski, A., Dunhar, E.,, and Dupoux, E. The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language model- ing. CoRR, abs/201 111888, 2020.
H., Padfield, D., Qin, J., Rozenberg, D., Sainath, T. N., Schalkwyk, J., Sharifi, M., Ramanovich, M. T., Tagliasac- chi, M., Tudor, A., Velimirovic, M., Vincent, D., Yu, J., Wang, Y., Zayats, V., Zeghidour, N., Zhang, Y., Zhang, Z., Zilka, I.,, and Frank, C. H. AudioPal.M- A large language model that can speak and listen. CoRR, ftbss/300..19225. 2023.
Nguyen, T. A., Hsu, W., D' Avirro A., Shi, B., Gat, L., Fazel- Zarandi, M., Remez, T., Copet, J., Synnaeve, G., Hassid, M., Kreuk, F., Adi, Y., and Dupoux, E. Expresso: A benchmark and analysis discrete expressive speech resynthesis. INTERSPEECH, pp. 4823-4822 ISCA, 2023a.
Sai, A. B., Mohankumar, A. K., and Khapra, M. M. A survey of evaluation metrics used for NLG systems. ACM Comput. Surv., (5(2):26:1-2::333 2022.
Shams, S., Dindar, S. S., Jiang, X., and Mesgarani, N. SSAMBA: Self-supervised audio representation learning with Mamba state space model. In SLT, pp. 1053-1055 IEEE, 2024.
Nguyen, T. A., Kharitonov, E., Copet, J., Adi, Y., Hsu, W., Elkahky, A., Tomasello, P., Algayres, R., Sagot, B., Mo- hamed, A., and Dupoux, Generative spoken dialogue language modeling. Trans. Assoc. Comput. Linguistics, 11:250-266 266, 2023b.
Su, J., Ahmed, M. H. M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:1 127063, 2024.
Nguyen, T. A., Muller, B., Yu, B., Costa-Jussa, M. R., EI- bayad, M., Popuri, S., Ropers, C., Duquenne, P.-A., Al-
gayres, R., Mavlyutov, R., al. SpiRit-LM: Interleaved spoken and written language model. Transactions of the Association for Computational Linguistics, 13:30-52, 2025.
Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L.., Ruder, S., and Metzler, D. Long Range Arena: A benchmark for efficient Transformers. ICLR. OpenReview.net, 2021.
Nishimura, Y., Hirose, T., Ohi, M., Nakayama, H.,, and Inoue, N. HALL-E: Hierarchical neural codec language model for minute-long zero-shot text-to-speech synthesis. In ICLR. OpenReview.net, 2025.
van den Oord, A., Vinyals, O., and Kavukcuoglu, K. Neural discrete representation learning. In NIPS, pp. 6306-6315, 2017.
Orvieto, A., Smith, S. L., Gu, A., Fernando, A., Gülgehre, Ç., Pascanu, R., and De, S. Resurrecting recurrent neural networks for long sequences. In ICML,, volume 202 of Proceedings of Machine Learning Research, pp. 26670- 26698. PMLR, 2023.
Vashishth, S., Singh, H., Bharadwaj. S., Ganapathy, S., Asawaroengchai, C., Audhkhasi, K., Rosenberg, A., Bapna, A., and Ramabhadran, B. STAB: Speech tok- enizer assessment benchmark. CoRR, abs/2409.02384, 2024.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. LibriSpeech: An ASR corpus based on public domain audio books. ICASSP, pp. 5206-5210. IEEE, 2015.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS, pp. 5998 0008,600,001 2017.
Patro, B. N. and Agneeswaran, V. S. Mamba-360: Survey of state space models as Transformer alternative for long se- quence modelling: Methods, applications, and challenges. CoRR, abs/2404. 16111111122 12, 2024.
Wang, C., Rivière, M., Lee, A., Wu, A., Talnikar, C., Haz- iza, D., Williamson, M., Pino, M., and Dupoux, E. VoxPopuli: large-scale multilingual speech corpus for representation learning, semi-supervised learning and in- terpretation. In ACL/IJNNLP (!), pp. 993-1003 Associa- tion for Computational Linguisties, 2021.
Pratap, V., Xu, Q. Sriram, A., Synnaeve, G., and Collobert, R. MLS: A large-scale multilingual dataset for speech re- search. In INTERSPEECH, pp. 2757-2761. ISCA, 2020.
Xie, 7., Cohn, T., and Lau, J. H. The next chapter: A study of large language models in storytelling. In INLG, pp. 323-351 Association for Computational Linguistics, 2023.
Reimers, N. and Gurevych, I. Sentence- Sentence em- beddings using Siamese BERTTnetworks. In EMNLP/IJC- NLP (1), pp. 3980-3990. Association for Computational Linguisties, 2019.
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. SoundStream: An end-to-end neural audio codec. IEEE ACM Trans. Audio Speech Lang. Process., 30:495-507, 2022.
Rubenstein, K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., de Chaumont Quitry, F., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., Muckenhirn,
12

---

Long-Form Speeeh Generation with Spoken Language Models
Zhang, D., Li, S., Zhang. X., Zhan, J., Wang, P., Zhou, Y., and Qiu, X. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities. In EMNLP (Findings), pp. 15757-15773. Association for Computational Linguistics, 2023a.
A. Additional Results
A.1. SC-L (16 min.)
Table 5. Semantic coherence scores are over the indicated spans of words for the transeripts of 16-minute completions. B denotes model extension via windowed generation. GSLM's blank is due many words Truth's blanks are due to its audio being <miin.
Zhang, D., Li, Z.,, Wang, P., Zhang, X., Zhou, Y., and Qiu, X. SpeechAgents: Human-communicatte simulation with multi-modal multi-agent systems. CoRR, 02//2001.03945, 2024.
to not generating
time period. Ground
SC-L1
Method
Zhang, Y., Han, W., Qin, Wang, Y., Bapna, A., Chen,
0-100
600-700
1200-1300
1800-1900
Z., Chen, N., Li, B., Axelrod. V., Wang, G., Meng, Z., Hu, K., Rosenberg, A., Prabhavalkar, R., Park, D. S., Haghani, P.. Riesa, J., Perng, G., Soltau, H., Strohman, T., Ramabbaddan, B., Sainath, T. N., Moreno, P. J., Chiu, C.. Schalkwyk, J., Beaufays, F., and Wu, Google USM: Scaling automatic speech recognition beyond 100 languages. CoRR, abs/2303.01033 2023b.
GSLM (0.2B)
0,531
0.464
0.451
TWIST-1.3B
0.499
0.420
0,429
0,423
TWIST-7B
0.520
0.429
0.420
0.423
Spirit LM Expressive (7B)
0.540
0.422
0.417
0.416
SpeechSSM-2B
0.549
0.482
0.470
Transformer instead
0.533
0.464
0.445
with 16min segments instead
0.537
0.483
0.473
0.469
SpeechSSM-9B
0.561
0.487
0.472
0.471
Ground Truth
0.556
0.498
Zheng, L., Chiang, W., Sheng, Y., Zhuang. S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang. H., Gonzalez, E., and Stoica, I. Judging LLM-as-a-judde with MT-bench and Chatbot Arena. In NeurIPS, 2023.
A.2. Human Ratings around Synthesis Boundaries
From SpeechSSM-2B long- form continuations 4 min- utes, take subset of 50 prompts (Appendix B.3). For each, take five-second windows centered at synthesis chunk boundaries versus synthesis chunk centers, giving 10 windows of each type. Each gets 2 ratings, to give 1,000 ratings per condition (Table 6). The mean opinion scores (MOSes) are nearly identical (4.05=0.077 vs. 4.07+0.07) and there no clear trend in synthesis issues, suggesting con- catenation boundaries do not lead to net loss in naturalness to attentive but non-specialist listener.
Zhu, Y., Lu, S., Zheng, L.,, Guo, J., Zhang, W., Wang, J., and Yu, Y. Texygen: A benchmarking platform for text generation models. In SIGIR, pp. 1097-1100. ACM, 2018.
Table 6. Human evaluation of generation quality of SpeechSSM- at and away from chunk boundaries. report Mean Opinion Scores (MOS) with 99% confidence intervals along with rater- flagged error categories.
Speed Prosody Sentiment
Chunk boundayy 4.05
115
16
41
38
Chunk center
4.07 + 0.07
122
16
53
78
32
20
A.3. Relative Throughputs
In Figure 8 we compare SpeechSSM-2B's throughput versus SpeechTransformer.
B. Additional Implementation Details
B.I. Model Training and Selection
SpeechSSM-2B w/ 30s, 4min, and 16min segments get 750, 5760, and 24k tokens per segment respectively, under USM-v2's frame rate of 25Hz. For the 4min's 's length extrap- olation (Section 3), we drop the last 10s of each segment.
13

---

Long-Form Speeeh Generation with Spoken Language Models
For long-form evaluations: Gecko is a text embedding model trained variety of tasks including document re- trieval, semantic similarity, and classification using long passages, and is more suited for extracting semantic embed- ding of long texts in open-domain generation. The choice of task is given via prompting; we use 'search results' as the prompt which was used to train on clustering tasks. For win-rates, mitigate length bias we only consider the 193 examples 23.5min (71 % of test-clean). For MOS computa- tions, we randomly selected 50 from these.
SpeachSSM ve. SpecckTrancfemer
100
Throughuu Speedop
30-
922 Figure Ratio of SpeechSSM to Transforrass 's through- put single TPU v5e performing unconditional generation at diffcrent sampling lengths, based on Figure
1004 2048
8392
Sample
B.4. SpeechSSM-X
We speaker-based detection categorize audio files into long-form monologue content, with activity detection. We take contiguous turns of up to 16min in length as training sequences.
turns marked by voice
Each version is trained groups of 512, 128, and 32 se- quences respectively, amounting to 768k tokens per batch. We train with the Adam optimizer with a learning rate of 5e-4 and weight decay 0.6 for 100k steps (with a warmup of Ik steps and a cosine decay schedule to 1/20th the learning rate). We select the best model on a fixed random subset of LibriSpeech-Long dev-clean, by generating continuations with each model to its target length over 5 checkpoints, then choosing the one which minimizes transcript perplexity (Section 1).
C. Additional MOS Evaluation Details
For short-form N-MOS, we took our subset of 200 prompts (Appendix B.3) and collected 6 ratings for each model con- tinuation. For N-MOS-T, we took our subset of 50 prompts and collected 24 ratings for each model continuation and slice. Hence, each score represents 1200 ratings unless stated otherwise.
The following prompt was used in all cases:
B.2. SoundStorm and SoundStream
Our SoundStorm follows the original hyperparameters for voice-promptable model described in Borsos et al. (2023b) (30s windows), except we double the number of layers to 24 to give 600M parameters, and our model was pretrained on a mixture of Common Voice v10 (Ardila et al., 2020), Multilingual LibriSpeech (Pratap et al., 2020), and VoxPopuli (Wang al., 2021). This comparable to other works which do not restrict their synthesizer data to LibriLight: TWIST (Hassid et al., 2023) does not explicitly indicate its vocoder data, but for HuBERT and the speech LM itself the English splits of these three datasets were used in training among many others; Spirit LM (Nguyen et al., 2025) used Expresso (Nguyen et al., 2023a) to train its vocoder.
This task zequires you to listen to audio clips using headphones in quiet environment.
In this task, you will be given audio clips. For each clip, please listen to the speech very carefully and then select rating on a scale of (very unnatural) to (very natural) using 0.5 point increments. The rating should be based on how natural unnatural the speech sounded. Please do not judge the grammar or the content of the sentence. Instead, just focus on how natural the speech sounds.
Possible Ratings
B.3. Model Evaluation
1: Bad
1.5
For text-based evaluations, we transcribe the generated speech enable the application of natural language gener- ation (NLG) metrics (Lakhotia al., 2021). Unless stated otherwise, use 1e122ee2-bbse--900
2: Poor
2.5
3: Fair
3.5
(Baevski et al., 2020) for ASR, applied on 180s windows. To reduce cost and to mitigate length/duration as an indicator for ground truth, for more intensive evaluations (N-MOS, side-by side-by-side by-sides) we randomly selected 200 test-clean utterances with ground- truth continuations 2 7s.
4: Good
4.5
5: Excellent
14

---

Long-Form Speeeh Generation with Spoken Language Models
D. LLM-as-a-Judge Example
This is our model prompt, with the example of comparing a transcript of the ground truth versus a transcript of GSLM's (Lakhotia al., 2021) generation. This example was chosen highlight the rater model's acknowledgement the prompt request to not penalize incomplete sentences:
Instructions
Please act an impartial judge and evaluate the quality of two texts which occur in the context of book. These texts are transcribed from audic recordings that were truncated to a fixed duration, Your job is to consider the following criteria to evaluate which text is better:
Fluency: How granmatically correct is the text?
Coherence: How well do the sentences of the text fit together?
logicality: How much does the text obey common sense?
Interestingress: How enjoyable is the text to read?
First, read text A and consider its fluency, coherence, logicality, and interestingness. Do not penalize the text for ending mid-sentence mid-paragraph.
Then, read text and consider its fluency, coherence, logicality, and interestingness. Do not peralize the text for ending mid-sentence or mid-paragraph.
Afterwards, compare the Eluency, coherence, logicality, and interestingness of the texts. Do not penalize either text for ending mid-sentence or mid-paragraph.
Finally, after providing your explanations, you must output only one of the following choices your final verdict with label:
is significante ficantly better: [[A>>]]]
2. Text is slightly better: [[A>B]]
3. Tie, relatively the same: [[A=B]]
4. Text is slightly better: [[B>A]]
Text is significantly better: [[B>>A]]
Example output: "My final verdict is tie: [[A-B]]".
# Comparison task
Text
Pearl accordingly ran to the bow window at the further end of the hall and looked along the vista of garden walk carpetea with closely shaven grass and bordered with some
H
Text B
Pearl accordingly ran the bow window at the further end of the table. "Palmer Austin and two boys," said the next fore keeper.
#
Detailed Comparison of Continuations
Below is the rater model's response:
**Pluency Both texts demonstrate good grammatical structure. Text is complete sentence, while Text ends mid-sentence, but this isn't penalized as per the instructions. Thus, they are roughly equal fluency.
**Coherence:** Text presents seemingly disconnected statement. Who is Palmer? Why Austin and two boys being mentioned? Without further context, it lacks coherence. Text deseribes Fearl's action and the scene she observes, creating more coherent picture.
15

---

Long-Form Speech Generation with Spoken Language Models
# Text A lacks logicality due to the lack of context. It a granmatically correct sencence, but the information presented doesn't make much sense on its own. Text B describes a plausible scenario \u2013 somecne running to a window and looking at a garden.
** **Intereatingness:. Text B is more interesting. The imagery of the garden walk with closely shaven grass is more evocative than the seemingly random information presented in Text A.
Overall, Text B is superior due to its better coherence, logicality, and interestingnest despite both having good flueney.
My final verdict is: Text B is slightly better: [[B>A]]
E. Generated Samples
Figure 9 presents transeripts of 4min and 16min speech continuations generated by SpeechSSM-2B, and Figure 10 compares 4min speech continuations generated by SpeechSSM-2B with those from other models. The recurring proper nouns highlighted in consistent colors, demonstrating SpeechSSM's superior performance in semantic coherence and context maintenance over long generations. '4min' denotes the default SpeechSSM-2B and '16min' the variant trained on 16min segments.
SpeechSSM 4min On Sunday afternoon Alevander remembered Miss Rouryoin's invitation and called her apartment. He found a delightful little place and met charming people
there,
For five minutes when Alexander came downstairs, talking as long he had time to know, he said that Miss Bourgoin had been shown some pictures. He soemed to be quite satisfied. He was afraid to talk to Alexander, not much of him. He was afraid of one but Alexander and this afternoon's recreation he heard her chattering away in his dining room.
He was very ansious get to picture for short but the picture pleased him and helped him much. At the first glance he found decidedly refined and ornamental. He was wearing a handsome, dark mustache autumn. And the impression he made on him was especially great at the beginning, a large strip of pink and blue. (more speech...)
There one little picture which Mr. Bourgoin tried to put in his way when be had one, he really carried with him. that wasu't the entertsionent they'd have to wait, he mentioned, and began patting in into his pockethook, which was written on huge black needles, and which was surrounded by a confussin of rounded tiles and someone had large scattered notebook it. It was written psir of rather large crinoline boots that appeared every minute and made a pleasant dream. Mademoiselle not make more than a picture. This was great experience with some pictures containing Alexander's pocketbook.
(more speech...)
"What time it" I asked, lonkinn up at the handsome disembudied sky. "You know, Mr. Bourgoin calls for it."
Mr. Bourgoin came downstairs and everyone came to see us. I liked the picture, and I found Mr. Bourgoin in the same attitude with family card in his hands and having private trouble face before him.
"I hope so, Mademoiselle, said. "I suppose it is a shame to bang with a horse when he has not got any food."
SpeechSSM 16min
from the evil seed of lust, all other deadle sins had sprung forth, Pride in himself and contempt of othens, covelousness using money for their subsequent life, memorre of poverty kindling the auger of the judge. Hairy, staining clothes of brothers, rum slaving sisterly and dear children in a pitiless passion. The white spots and sober face and rough, hard-mannered features, but wealth of evidence to decide. This distress would not he terrible.
(more speech...)
"And so you give me a reason for it.' 'Sid Tobioos "Why then should you choose a resson for it? For why could I not accept what believed to be good?" yeast answered with effort, 'Lord, what this that I have not said? You are skilltul hatchet. I am going sit with you here. labor have my yeast among you better place. will make you ask."
little one to you." 'He added. "Ah, how fine is your friend! You were
not all walk alone a (more speech...)
"I wantyou understand." Said Tobinos. "You want to know whether you deserve what you have given to believe. Would you still like to believe it? Lard, I want know whether can believe and love all and such a man. I was there thinking. Lord, what will not I do? Lord, what will not do?" Thereupon, Gorab lifted himself lightly from the winard's chest, coughed, took his foot out the chest, and said, "Lord, I labor to give you life and strength. (more speech...)
"Go straight up. He thou one the men in the world and bless your pride.' Came from Gombh as he opened his eyes. The rod king was really blinded in body. And his glances contracted with fixed look. He felt the coffer coming and going fast. Gorab stepped towards him and shouted some well- koll-know word him, and ordered the wizard bound him in a bowl bread into the cave. "Mumgulul, Samko, wake up. Said Tobinos. (more speeeh...)
And he approached the worker at the same instant. "What the consequence? He said. short time the young man, his body bent, showing one leg, above the knee, His hands dropped his cheek, and he sat up in the depths of his berth. 'I must follow," said Tobinos. "and send the strongest of the soldiera." IIt was not possible to like a flash, as it is generally understood for's stranger to be so lying the hammer. "Yes, who cometh forward to disgrace us? The Bescilian? Yes, but thou lass already bought the Brazilian."
sold thy goods the Brazilian? No. but his enemy by calling out the old man. (more speech...)
Tobince. He went, however, not in good humor,
desperate desice delude
Her right foot, with its long brown fingers, her fair bair, the truth sprang from her eyes. "Arthur can't 't afford to do this." She said. "And it's no use scolding me or me alone. And besides, be might well feel the wrong way with his gun." She turned her face away, but still, he went on, 'Let me look at the man who did those things." "
Figure 9. Automated transcripts of 4min and 16min speech continuations generated by SpeechSSM-2B, atop 10-second audio-only prompt from our proposed LibriSpeech-Le (test-clean). The prompt is underlined, and parts abbreviated with (more speech...) for emphasis. Recurring proper nouns are highlighted in consistent colors show SpeechSSM's relative semantic consistency over time.
16

---

Long-Form Speech Generation with Spoken Language Models
Prompt (ASR): Yes, it must be confessed, Polly was homesick, All her imaginations of her mother's hard work, increased by her absence, loomed up before her till
GSLM Continuation (ASR):
felt the wright of their last fraction. then she pansed, pausing upon poor child, while George was waiting for her.
"What? What?" [Noise] What kind a man could he be in his house it he's going to...? Hie tumned his head and stepped across the lawn, looking down at the window, then turned to her with grove smile- [Noise] which all the rest of that piece sound came, and Ronnie's face pale. His hair grow full and stately. His head began brag and faint, and he breathed heavilly feeling quite certain that Polly could be his house by digging bread for money for his pour child, and having bread for money, and having bread in his hand for Bonnle's.
"Oh, mother," cried Polly indignantly, "Yon ought wait on Polly." " [Noise]] "That's me, too. "There lay as if she did not laugh and entreat. How long, no one could tell. The The magistrate sucdenly assumed surprise, followed by a burst of delight, and the more today, Now, the milk and supper with those words of selfishness, she had written home better than she was before, [Noise]
(more speech...)
Thoughtful that the man [Noise]] who is wonderful, would not be the slave, but the master. He opposed his limhs to be upon the top with the worthy company. Accordingly,
TWISTL3B Continuation (ASR):
teatime. The little boy was suddenly cold and told his trouble. Two of the poor old black women whom she had visited for the parpose of looking at the soome, and for toic with epicures, [Noise]
contemplating the wretchodness of the crime. The purchase of prison, the pardoning Babylon and apology. The incorruptible
(more speeeh...)
However, the burger. Barger, hurges, burger, barger, hurger, barger, burger. Burger, burger, burges, burger, burger, hurger, barger, burger, burger, burger, burger, burger. Burger, burger, burger, burger, burger, burger, burger, burger, burger, burger, burger, burger, burger, burger, burger.
TWIST7B Continuation (ASR):
she was ashamed to continue her studies, and he could not keep ber thoughts from occupying heraelf with Abby, who was so devoted to ber. He had put her in the way of enjoying herself when she young. and now she said the housekeeper in halting tones, giving Hannah to stand as be brought her to the door. "Wherre's ' Hannah Winnyard?" Mrs. Manstay went again to her window, and heard and saw, or felt she did in what the schoolmaster was doing. The boy was dressed in the dark blue gown and broad white collar that she might keep her sight of him. [Noise] Mrs. Manstay said nothing, putting her temp him. [Noise] She had families to attend She got out to catch the night train, but the meaning which the moment of felt gave her a slight shock, and she looked down at the piece of paper without a word. [Noise]
the way giving herself her willingly others. He
hand bell the door. "Mrs. Manstay, if you please,"
(more speech...) Sn, one thing
want do, want know when you think of Star Trek, what do you think of? [Nolae] Star Trek. think about this. Olay. What you think Star Trek? You Star Trek. What do you think of? What do people [Noise] That would be Stat Trek. I mean, that would be Star That would be Star That would be Star StarTrek and then you're like "Ian't Stor Trek trend?' [Noise] like medieval of Star Trek. No. it's trend.[ [Noise]
Spirit LM Continuation (ASR):
they seemed a reality, She was a trifle prettier than usual, but she is not a child of one any longer, And there was no optimism here for the paid to hope that she would be able to keep Juva completely supplemented by the family happiness. At the time Lucy the math global music time in public education a little while perhups and then go sit suceessfull students. But it would be better not to try to carn the money to pay some students by compromising discriminate traders of under the provisions the United States Constitution. Thank you. Coming up. Thank you, your honor. Think you satisfy that roquest as relates compilers promation damages, that crossed the line between constitutional and state forums. (more speeeh...)
child of She might taken make that worried to the village public school to
Caesar, rotations the bullion been completed, she could kind down
of money
There also different impression on the record which are entizely distinct from case. There was no suppression, but distraction in the telling X plus the member suspleion him committing taking down turning him back to the carriage and looking for him in orderto to get rid of him which was wanting to take him something closer to the Larson but she wouldn't going back that he's close death and he couldn't wait for him so he went back get the arm chair.
(more speech...)
That's why I feel like a high school student. I feel like I'm in a 1960 student. That's why I can't go back to secure me. That's why I can't be put in, they can be stealing coftins, can't put that in the end comparisous. That's why can't be put a Gield leadership and comparison. That's why can be a teacher so cares. That's why dad said I can't be, that's why I can't be practical. That's why I can't be see.
Speech Trunsformer Continuation (ASR):
she hardly remembered them anyone clse. She for boarding girl, 18 old, and any who had some influence over her that to be would find out. Your kennel men, you see, she was for a choral business. do believe you would call more work on a professional matter, a day of two, young, ladies, with you at nights to catch sight of, She's that grown up, isn't she?""Oh, "Oh, it's a pity, unele," Annie exploded with feeling. There's six boys, and the lads and one, I don't know which. The widow here has little girl of her own, Geoffrey and Chuffy, came. Her mother raised her up springless, and Billy marshaled his friends in thick garments. She had to go down to his office next and take her place at his hotel. The boarder of Grace's offices came in, smiles and greetings, and Dolly made a call. (more speech...)
whom you meet Shendent Bullen platforms. "Allow me select for you girl." said Polly, and Dolly
He brooded over her until he contemplated his new dress. "Don't you look as homely as all that, Bobby dear?" murmured Pisoechio salemnay. "Ob, don't wear?' asserted Annic. And amiled brightly after the little Gustave, who was half menacing in his gost. His face grew very pale, and his rumpled brown hair fell drags in front of him. Held up in benignant readiness as a long tunic of just that shape she had left. "There's as something telling, Mr. Cart," began Dolly, looking silently down. "Nothing, I'd rather the overseer do in this name." (more speech...)
How many more come out? Three chains and old stuff. "Where?" demanded Polly. Three for Miss Gould, who had silver bends. I'm sure she didn't wear any of her old beads. Annie gathered herself together, and patting her adoring head back at the doorstep. strutted away the clearing. Polly lowered her old, gentiemanly head. I didn't have time to go over to lady's Place. Then Miss Gibson is up. And I'm sure she wants.
SpeechSSM Continuation (ASR):
she had fallen into an ugly sleep. When she awoke, the children were sleeping, but not frightened at all, for fear they should stir step, and one had to wake them before she bedroom window. Mrs. Hopkins sprang from her tray and dressed herself and her mother, her face, but she was still busy, and brought up tea. "Who is Polly?" asked Mr. Hampton as she laid down her tea. "Polly? She hasn't 't caten anything already!" The children, overcome with grief, turned leave the room. And for some moments the shivered and wept because, when she awakened, the room smelled of bad, cold milk That night, they were more worried than they had expected. They found the handeuffs driven in kitchen sills, and the pot hammer barning in kitchen fireplace. They'd d said when and how they were to come. The only thing that startled Polly and her mother was the notion of the handcutis being driven in Mr. Hampton's s (more speech...)
awoke, And then heard a gust snap and
ran wash
I'm different from Polly and I want her to a hero too. She's a very nasty child and doesa't know exactly how to speak to me." "That's the kind of life the old woman, the poor old woman, goes into now. must not give up my mother. It's right to have a continuation in still things" "Don't you think she's good enough for me?" asked Elizabeth spoken slowly. "Do you?" "Not any us. This little hand of yours after you've ve been sleeping, and get away all the same."" "one."WWy "Why are you going away so quietly? can't remain, Polly." (more speecht..)
hontible for delay. I couldn't draw around you and tell you lie down well as can. when perhaps, can you that
You must not abandon the idea, Elizsheth, the flesh is too weak, It's too stealthy and hurts. is horrible to say so. "That don't know, isn't worth living, If you have no sympathy with my dislike, you shall have more too." He stood by the door with beave, small voice as he wiped his dry face and patted ber on the arm.
Figure 10. Automated transcripts of 4min specch continuations generated by SpeechSSM and baselines. Parts abbreviated with (more speech...) J for emphasis. Recurring proper nouns are highlighted in consistent colors to show SpecchSSM's 's relative semantic consistency over time. Non-sense sentences and proper noun errors are highlighted in grey.
17


Reviews:
[{"review_id": "yeeTQeuk1J", "reviewer": "Reviewer_k44A", "summary": "The paper proposes SpeechSSM, a spoken language model based on state-space models that can generate long-form audio in a single pass. The paper also proposes reference-based semantic similarity and LLM-based pairwise judgment to evaluate the generated long-form audio. They also released a new dataset LibriSpeech-Long for evaluating speech continuation. The SpeechSSM outperforms the baselines in both short-form and long-form generation. The new metrics and dataset enable better evaluation for long-form speech generation.\n\n## update after rebuttal: I decided to keep my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive reading of our work!\n\n> How important is the text-based initialization (using RecurrentGemma-2B) for speechSSM's performance?\n\nFollowing the insights from Hassid et al. (2023)'s TWIST model, which demonstrated that initializing spoken language models with pretrained textual knowledge enhances semantic coherence and fluency, we initialized SpeechSSM with RecurrentGemma 2B IT’s weights. As both SpeechTransformer and SpeechSSM are initialized with first-generation Gemma 2B models, we believe **the text-based initialization is not important to SpeechSSM’s gains over SpeechTransformer**. Furthermore, Spirit LM is initialized with Llama 2 7B, which is comparable to Gemma 2B (Table 6 of [the Gemma paper](https://arxiv.org/pdf/2403.08295) gives scores of 46.9 vs. 45.0 averaged over 18 benchmarks), so **we believe it is not important to our gains over Spirit LM either.**\n\nThough we did not prioritize this ablation (as LM initialization is “free” and benefits were shown by TWIST and Spirit LM), it is an interesting question and new for speech SSMs. **We have trained a SpeechSSM 4m 2B with random initialization.**\n\n[Here is the training plot](https://raw.githubusercontent.com/demo474/ID4403_Demo/refs/heads/master/figures/rebuttal-2b-randominit.png). Despite starting from a higher training loss (expected), it converges to a lower training loss (unexpected)! We suspect that our removal of RoPE may have harmed transfer, as keeping RoPE gives a similar loss curve (see orange). Note **these are audio token losses**; the generated _text_ may be worse. There are also considerations beyond loss; early experiments suggested that RoPE prevented unbounded length generation, a key requirement for our system. **We will validate this ablation and share results during the discussion period if possible.**", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem at hand.", "theoretical_claims": "The paper does not have any proofs for theoretical claims.", "experimental_designs_or_analyses": "The baselines are extensive and comparable.\nThe analysis on semantic coherence and error modes is comprehensive.\nThe Librispeech-Long benchmark is well-designed for the task.", "relation_to_broader_scientific_literature": "The core contribution of this work is built on the developments in many different areas, most importantly language models and speech representation learning. It deals with the problem of spoken language modeling, with the core challenge being the choice of the right model and speech representation for modeling.\nPrevious approaches, like GSLM, used transformers to model speech tokens from HuBERT. Later, speech representation modeling improved. SoundStorm, for example, incorporated hierarchical modeling with RVQ which inspired AudioLM to do better modeling. However, the previous approaches were not good at modeling long speech sequences. That was until state space models like S4 arrived and substantially improved time-series modeling for long sequences.\nFor the evaluation methods introduced in this paper, it is also backed by the recent advancements in large language models (LLMs). Many papers are now starting to make use of LLMs to build evaluation metrics for their methods.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "How important is the text-based initialization (using RecurrentGemma-2B) for speechSSM's performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4CR4DoZXhF", "reviewer": "Reviewer_9NCy", "summary": "The paper proposes SpeechSSM, a spoken language model designed for long-form speech generation. It is based on state-space models enabling efficient generation with constant memory consumption. To evaluate the model on long generations the authors propose LibriSpeech-Long benchmark and new evaluation metrics including embedding-based similarity, LLM-judged scoring, and time-stratified quality assessments. Results show that SpeechSSM performs on par with existing spoken language models for short-form generations and outperforms them on long-form speech generation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and thoughtful feedback about our work!\n\n> **Major Weaknesses:**\n>\n> No human evaluation beyond N-MOS\n\nNote **we have strengthened our MOS results**; see response to Reviewer QbKA (#4).\n\n> LLM-as-judge vs. human correlation analysis\n\nLLM judges align closely with human preferences (over 80% agreement) in pairwise comparison (Zheng et al, 2023), with SxSes more stable than single-answer grading. When we started, our LLM as judge (Gemini 1.5 Pro) was [the top generative LLM-as-judge](https://x.com/aseveryn/status/1793605627232833887) on [RewardBench](https://huggingface.co/spaces/allenai/reward-bench), with a score of **88% (roughly viewable as human agreement %)**. Our rubric is based on story generation evaluations (Section 5.2 + Appendix D), a reading/style comprehension task we believe is similar to tasks evaluated by these works.\n\nThat said, a correlation analysis specific to our transcript SxS would be informative; we'll investigate this for the camera ready.\n\n> Limited dataset diversity... proposed model is trained and evaluated on the same audio/speaker conditions ... other models have been trained on different data. Experiments on different datasets...\n>\n> **Other Weaknesses:**\n>\n> Only audiobook generations... other scenarios like dialogues?\n\nWe restricted our main model to LibriLight for fair comparison with non-Spirit LM works. However, to demonstrate diversity, **in 6.4 we also trained a version (SpeechSSM-X) on extemporaneous podcast-style monologue.** Our model **replicated this data's expressive and varied nature**, as can be heard in the [SpeechSSM-X samples on our demo page](https://demo474.github.io/ID4403_Demo/). That said, our models generate a single voice as they disentangle speaker identity (a design contribution of our work). **We see no blockers to training a model with entangled tokens to model many voices in dialogue.**\n\n> Ablation... architectural choices (e.g. hyperparameters) impact model performance?\n\nWe ablated train/test lengths and Transformer vs. SSM. **We also now ablate text LM initialization and model size**; see Reviewer k44A (#1) and Reviewer QbKA (#4) rebuttals for details. We could explicitly discuss more things we tried e.g., RoPE length issues or overlap parameters in the camera ready.\n\n> **Q:** EXPRESSO, EmoV?\n\nTo quantify SpeechSSM-X's performance, we are trying Expresso and could share results during this period if completed in time. We are considering context-based continuation (e.g. [\"Subjective metrics\" by Sesame](https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice)) but welcome suggestions.\n \n> **Q:** window parameters, [tokenizer] overlap, and performance\n\nFor our top concern of unbounded generation, we found that avoiding \"implicit EOSes\" was key (Section 3), so we used the largest window supported by USMv2; we only overlap to avoid edge tokens. It's possible that intentionally smaller windows may improve performance, which we leave to future work.\n\n> token widths and performance\n\nLonger audio token widths are an active research area which we believe is driven by poor long-form performance of current models. **We intentionally focus on architecture/training improvements _instead_ of larger tokens**, which we'll clarify in the camera ready. Future work can combine both approaches.\n\n> [synthesizer] overlap and performance\n\nWe tested 0s, 2s, 4s, and 8s overlaps, and found that **no overlap produces artifacts at boundaries**. The smallest overlap we tried already significantly reduced these artifacts, so we chose it. See final part of our response to Reviewer QbKA (#4) for metrics.\n\n> **Q:** different prompt durations? 10 seconds for long-form generation instead of 3 or 5 seconds?\n\n**Shorter prompts are insufficient initial context to act as anchor** to compare against for long-term coherence, e.g., our SC-L metric of semantic similarity between the prompt and segments of the continuation. A prompt of at least 10 seconds (1-2 sentences) differentiates models (Figure 5) and constrains the space of valid continuations for LLM judging.\n\n> **Q:** >16min speech generation; semantic coherence and speaker identity?\n\n**We have generated with SpeechSSM up to 30 minutes** and see no issues going further. The model maintains speaker identity and speech naturalness indefinitely (as expected from moving speaker modeling to the acoustic layer), with a gradual semantic drift as one may expect from SC-L scores in Figure 5. However, paragraph+ coherence issues [seen in our 16min samples](https://demo474.github.io/ID4403_Demo/) should be improved first before such generations are worthwhile.\n\n> **Q:** non-linguistic vocalizations such as laughter or sighs?\n\nAs long as the model is trained on data with such vocalizations, it can learn to reproduce them. **In the third [demo page](https://demo474.github.io/ID4403_Demo/) sample, from 45s - 1:15s, there are sighs, exhalation, and fillers (“uh”, “um”, “uh-huh”).**", "claims_and_evidence": "The paper makes the following main claims:\n  1. Model-wise: SpeechSSM is the first state-space language model for speech and provides long-form speech generation\n  2. Evaluation-wise: The paper introduces new evaluation metrics for long-form speech generation, including side-by-side LLM-as-judge scoring and time-stratified quality assessments.\n\nThe paper provides clear and convincing evidence for both claims through experimental comparisons.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in the paper are well-suited for the problem of long-form speech generation. To evaluate the model's performance the authors propose four evaluation methods:\n1. LibriSpeech-Long benchmark: To assess the model’s performance on extended speech continuations the authors reformatted the original LibriSpeech dataset into longer 3-4 minute segments creating the LibriSpeech-Long benchmark.\n2. Embedding-Based Metrics: Semantic embedding-based metrics are used to evaluate the content preservation of the generated speech.\n3. LLM-as-Judge: Large language models are utilized as judges to provide side-by-side evaluations of generated speech samples.\n4. Time-Stratified Evaluations: The paper introduces time-stratified quality assessments to analyze the consistency of speech over different time intervals within the long-form generation.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Two main experiments have been performed:\n\n  1. Short-Form Continuation Experiments: The model is evaluated on 7s continuations given a 3s prompt assessing short-term coherence and speaker similarity.\n  2. Long-Form Generation Experiments: The model generates up to 16 minutes of speech from a 10s prompt, evaluating semantic coherence, fluency, and speaker consistency across extended durations.\n\nMain weaknesses:\n\n1. No human evaluation is performed beyond N-MOS. While the authors employ LLM-as-a-judge, a human LLM correlation analysis would be very interesting.\n2. Limited dataset diversity in training and evaluation. The proposed model is trained and evaluated on the same audio/speaker conditions (e.g. background/noise, neutral speech) whereas the other models have been trained on different data. If experiments on different datasets are provided it would demonstrate the robustness of your approach.", "relation_to_broader_scientific_literature": "The paper's key contributions align closely with advancements in the broader scientific literature. In particular, while state-space models have been used in speech generation, none have been applied as a spoken language model operating directly on acoustic tokens. Unlike prior transformer-based approaches such as TWIST and AudioLM, the proposed method leverages state-space models for more efficient long-form generation with constant memory complexity.\n\nAdditionally, the paper builds upon standard evaluation metrics for short-form generation and extends them for long-form evaluation. First, it introduces the LibriSpeech-Long dataset, created by reformatting the LibriSpeech dev and test sets to enable longer speech evaluations. To evaluate long-form generated samples it proposes new evaluation metrics for assessing the quality and coherence of extended speech generation (LLM-as-Judge, embedding-based metrics and time-stratified evaluation).", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths\n\n1. Efficient and scalable approach. SpeechSSM has constant memory complexity and linear-time inference.\n2. New evaluation metrics and the LibriSpeech-Long benchmark are important long-form speech research.\n\nWeakness\n\n1. Ablation analysis is missing. How different architectural choices (e.g. hyperparameters) impact model performance?\n2. Only audiobook generations are considered. How would the model perform on other scenarios like dialogues?", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. How does the choice of windowing parameters impact performance? How do different token widths and overlap sizes influence the model’s performance?\n2. How does your model perform with different prompt durations? For example, why did you choose 10 seconds for long-form generation instead of 3 or 5 seconds?\n3. Have you evaluated SpeechSSM on more expressive speech datasets, such as EXPRESSO or EmoV?\n4. Have you tested the model for longer than 16min speech generation (e.g. 60 min)? Does it maintain semantic coherence and speaker identity?\n5. How does your model handle non-linguistic vocalizations such as laughter or sighs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6RNH174UJe", "reviewer": "Reviewer_qRRp", "summary": "This paper introduces the first speech-language model ***SpeechSSM***. Two new metrics and a new benchmark are proposed. Experiments and analysis are comprehensive.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and support!\n\n> (Minor) The architecture of SpeechSSM appears somewhat straightforward and could benefit from more innovative design considerations.\n\nAs our primary contribution is advancing long-form speech generation over tens of minutes, and being the first work on the topic, we outlined the task's minimum requirements (Section 3, first paragraph) and **intentionally meet them in a straightforward way** so that we could also focus on dataset and metric design. Nonetheless, this already involved several key design choices, such as: using hybrid SSMs from text LMs; disentangling speaker identity, acoustic, and semantic aspects; windowed tokenizing and decoding; repeated padding and avoiding implicit EOSes; using NoPE.\n\nBy establishing a strong baseline and benchmark, we hope future work can focus on new and innovative architectures for long-form speech.\n\n> (Minor) The windowed tokenization and decoding approach represents a compromise solution rather than an optimal one.\n\nThe overlap windowing approach was a practical solution to effectively utilize a bounded tokenizer and decoder to achieve unbounded speech generation. Future work could explore streaming tokenizers to handle extended speech generation, but these have to train from scratch.\n\nIn contrast, our method works on default non-causal tokenizers like HuBERT and USMv2. **It is close to optimal from the viewpoint of \"working out of the box\", and also from mitigating boundary artifacts**, as we quantify in the end of reply to Reviewer QbKA (#4). We consider our design and comprehensive details (e.g. avoiding implicit EOSes, where to speaker prompt) a contribution in its own right.\n\n> The reviewer acknowledges that the weights are currently not released. Will the model and weights be released in the future? This is important for further research.\n\n**SpeechSSM is largely RecurrentGemma finetuned on a public dataset** and so we believe it is easily reproduced. As USMv2 is not widely available, another speech tokenizer would be required, of which there are now many viable candidates (e.g., SpeechTokenizer, Mimi). Due to safety considerations and a proprietary dataset we are unlikely to release SpeechSSM-X. \n\nWhile for institutional reasons we can't promise a full release of the (LibriLight) SpeechSSM, **we could release finetuning code, e.g., atop [the Gemma finetuning library](https://gemma-llm.readthedocs.io/en/latest/colab_finetuning.html) and a public speech tokenizer**, to support further research.", "claims_and_evidence": "**Yes**", "methods_and_evaluation_criteria": "**Yes**", "theoretical_claims": "**Yes**\n\nThere is no theoretical claim in this paper.", "experimental_designs_or_analyses": "**Yes**\n\nThe authors conduct experiments for short- and long-form speech generation separately.\n\nThe experiments are well-designed. Comparisons and evaluations are comprehensive.", "relation_to_broader_scientific_literature": "The key contributions of this paper include:\n1. **SpeechSSM** -- the first speech-language model for long-form speech.\n2. Two new metrics and a new benchmark for long-form speech evaluation.\n\nThe related works seem to be discussed in the Section 2.", "essential_references_not_discussed": "**Maybe not**\n\nI thought the related works were discussed.", "other_strengths_and_weaknesses": "**Strengths:**\n1. This paper is well-written. The figures and tables are clear and easy to understand.\n2. The results generated by the proposed method are very competitive.\n3. As the authors claimed, the proposed methods significantly increase the audio length during inference and training, which are meaningful for real-world applications.\n4. The two new metrics and the new benchmark designed for long-form speech are useful.\n\n\n**Weaknesses:**\n1. (Minor) The architecture of SpeechSSM appears somewhat straightforward and could benefit from more innovative design considerations.\n2. (Minor) The windowed tokenization and decoding approach represents a compromise solution rather than an optimal one.", "other_comments_or_suggestions": "There is no more comments from the reviewer.", "questions_for_authors": "The reviewer acknowledges that the weights are currently not released. Will the model and weights be released in the future? This is important for further research.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7PWaAMheaV", "reviewer": "Reviewer_QbKA", "summary": "This paper aims to develop a speech language model (LM) capable of modeling long-form speech. The key distinctions of this paper compared to previous studies are as follows: (1) Introducing State Space Models (SSMs) into the speech LM at the speech-token level, resulting in improved perplexity (PPL) and other evaluation metrics compared to existing baselines. (2) Highlighting shortcomings in previous evaluation methods for speech LMs, proposing a more effective evaluation approach, and introducing a new benchmark, \"LibriSpeech-Long,\" specifically designed to evaluate these capabilities.\n\n## update after rebuttal\nAs I mentioned in my official comment below, I have read the authors' rebuttal and slightly increased my score accordingly. Once again, thank you for your prompt and thoughtful response.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough reading of our work!\n\n> W1: SpkrSim and N-MOS reflects speech tokens; decoupled tokenizer/vocoder baselines?\n\n**We agree N-MOS and SpkrSim are primarily due to tokenizer/vocoder**, hence why it felt unnecessary to isolate them from the backbone architecture change as these metrics would focus on our \"acoustic\" improvements. Then, to isolate backbone improvements from tokenizer/vocoder, we introduced SpeechTransformer as a baseline, sharing the same speech tokenizer (USM-v2) and vocoder (SoundStorm) with SpeechSSM. \n\nAlso, despite using existing tokens, **we view our _deliberate integration_ as the contribution**. Other schemes entangle acoustic and semantic properties, and so their N-MOS is tied to long-form ability (-T); ours is not due to our fixed context and semantic-to-acoustic stage. Other models have speaker drift, while our speaker-invariant tokens (USMv2) *and* speaker-prompted acoustic stage ensure even and high SpkrSim.\n\n> W2: N-MOS sufficient for SSM over Transformer? \n\nDue to shared tokenizer/vocoder/etc., **the N-MOS gap being narrow between SpeechTransformer and SpeechSSM is _expected_**. This is also why our proposed metrics are important, such as Win%, PPLs at 4x train (Tab. 4) and SC-L (Fig 5), isolating the semantic delta of SSM vs. Transformer.\n\nRegardless, **we’ve greatly expanded our human evals. We report(ed) 99% CIs**, where each model had 200 items, 3 ratings each (incorrectly ‘5’ in Appendix C). We 2x this to get **[tighter intervals for Table 2](https://raw.githubusercontent.com/demo474/ID4403_Demo/refs/heads/master/figures/rebuttal-updated-table2.png)**. \n\nOur N-MOS-T was a different, smaller rater pool due to technical/time issues. We've now aligned it with short-form N-MOS to give **[far better 99% intervals for Table 3](https://raw.githubusercontent.com/demo474/ID4403_Demo/refs/heads/master/figures/rebuttal-updated-table3.PNG)**: 1200 ratings per (model, slice). \n\nWhile magnitudes in the top range decreased, trends remain. In particular, **SpeechSSM now clearly improves over SpeechTransformer in N-MOS-T**; their 99% intervals do not even overlap in two of the columns. This also **shows N-MOS-T is affected by the backbone model also.**\n\n>  W3: Simply applied SSM; modifications/adaptations specifically designed for speech data?\n\nAs our primary contribution is advancing long-form speech generation over tens of minutes, and being the first work on the topic, we outlined the task's minimum requirements (Section 3, first paragraph) and **intentionally meet them in a straightforward way**. Our architecture, benchmark, and evals _stem from the task_ -- issues with the obvious SpeechTransformer baseline, issues with metrics that capture acoustic issues rather than semantic quality, etc.\n\nThis did involve key modifications: using hybrid SSMs from text LMs; disentangling speaker identity, acoustic, and semantic aspects; windowed tokenizing and decoding; repeated padding and avoiding implicit EOSes; using NoPE. Though these are not architectural changes in the narrow sense, they contribute to length generalization, enabling the model to effectively generate speech of unbounded length.\n\n>  Q1: Would performance significantly improve with increased scale? \n\nWe expect performance to continue improving beyond 2B. This expectation is supported by the Griffin paper, the underlying hybrid recurrent architecture used by RecurrentGemma and thus SpeechSSM, which showed consistent gains across 1B, 3B, 7B, and 14B models on downstream text tasks.\nTowards this, **we have started training a SpeechSSM 9B**, initialized from RecurrentGemma 9B which was unavailable at the start of our work. **[Its training curve versus 2B is very promising and suggests headroom](https://raw.githubusercontent.com/demo474/ID4403_Demo/refs/heads/master/figures/rebuttal-9b-inprogress.png).**\n\n> Q2: Would concatenation process introduce artifacts at the boundaries? \n\nYes, **there are artifacts due to waveform splicing, but they are very subtle due to overlapping,** as the mismatch in the contexts used to generate audio before vs. after the splice only starts 50 tokens (2s) away.\n\nSince SoundStorm uses 3s for speaker prompting, it synthesizes in 27s chunks, whose last 4s are overlapped with the next chunk. Hence, **when boundary artifacts occur, they occur at `25 + 23*N` seconds.** In the third file on the [demo page](https://demo474.github.io/ID4403_Demo/), at :25, the male voice gets briefly louder.\n\nWe take 5s windows centered at the boundaries (“at concat”) and 5s windows at the chunk centers (“not at concat”). There are 10 of each window type in a 240s generation, giving 50 x 10 x 2 ratings/item = 1000 ratings per type. At 99% CI we get 4.05 ± 0.07 vs. 4.07 ± 0.07, showing **artifacts do not appear in MOS, [nor do they appear systematically in the categories raters can flag](https://raw.githubusercontent.com/demo474/ID4403_Demo/refs/heads/master/figures/rebuttal-concat-artifacts.PNG).**", "claims_and_evidence": "The authors have established several criteria that a speech LM designed to model long-form speech should satisfy. They also described how State Space Models (SSM), along with their proposed model and dataset, were employed to meet these criteria.", "methods_and_evaluation_criteria": "I have a few questions below regarding the evaluation scheme, specifically parts that were slightly unclear or raised my curiosity. Aside from these points, everything else seems appropriate.", "theoretical_claims": "Since this paper focuses primarily on applying SSM, there do not appear to be specific issues regarding proofs or theoretical claims.", "experimental_designs_or_analyses": "I have some questions specifically about the evaluation rather than the experimental design itself. I'll address these points separately below.", "relation_to_broader_scientific_literature": "I agree with the authors' contribution in integrating SSM into a token-level speech LM, which has not been addressed by previous studies. Additionally, highlighting the limitations of existing evaluation methods and proposing an improved evaluation metric also constitute contributions of this paper.", "essential_references_not_discussed": "I found no issues with references.", "other_strengths_and_weaknesses": "### Strengths\n1. Introducing SSM at the speech-token level to enable effective long-form modeling.\n2. Proposing both a benchmark and novel metrics designed explicitly for evaluating long-form speech generation.\n3. Overall, the approach is reasonable, and the authors have leveraged the advantages of SSM for long-form generation.\n\n### Weaknesses\n1. Regarding the N-MOS (ignoring grammar and content) and SpkrSim metrics, I believe these primarily reflect the qualities inherited from speech tokens proposed by previous works such as USM-v2 or SoundStream, and the speech decoder (SoundStorm), rather than being novel contributions of this paper. \n- If the goal was to compare these aspects, the baselines should also have been trained as personalized speech synthesis models, or similarly, the USM-v2 tokens used by baselines should have been decoded with a separately trained vocoder such as HiFi-GAN, similar to the previous works. (The controlled variables should have been introduced only in the proposed SSM-based model.)\n\n2. Regarding the perplexity (PPL) results, the authors show that SSM is effective compared to Transformers. However, other metric, particularly for the N-MOS results in Tables 2 and especially 3, I notice the confidence intervals are notably large. I am curious why these confidence intervals are so wide, what the corresponding p-values are, and whether it is indeed possible to conclusively determine the advantage of SSM over Transformer-based models based on these numbers.\n\n3. In terms of novelty, I personally feel the contributions might be somewhat limited. Aside from introducing a new benchmark and proposing new evaluation metrics, the main impression is that the paper simply applied SSM to speech tokens. If I have missed any additional structural modifications or specialized adaptations specifically designed for speech data within the proposed model, I would appreciate further clarification from the authors.", "other_comments_or_suggestions": "The paper is well-written, and I did not notice obvious typos.", "questions_for_authors": "1. In recent language model trends, scalability has emerged as an important factor. I am curious about how the performance of the proposed SSM-based approach changes as the model scales up in Speech LLMs. Specifically, would performance significantly improve with increased scale, or is there a clear performance upper bound?\n\n2. Additionally, during speech generation, the authors mention generating audio chunks with overlapping segments for memory efficiency, then concatenating them. I would like to know whether this concatenation process introduces any artifacts at the boundaries. If artifacts exist, it would be helpful to understand their nature.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary**

The paper designed a speech language model (spoken language model) based on the existing techniques but placed them in a right combination and first time demonstrating the spoken language model which is able to generate consistent long form speech outperforming all prior works. The key components authors placed together are 1) introducing and using Space State Models for the first time for the spoken language models 2) switching back to no positional embedding to have better long form extrapolation 3) initialization from the pretrained LLM 4) semantic tokens to model language and further reuse of acoustic tokens generation via SoundStorm 5) windowed version of tokenization to reduce boundary effects in the long form generation. Authors reuse the prior metrics and benchmarks to compare with prior works but also introduce the long-form variant of the LibriSpeech (and release it) as well as adopt/propose the reference-based semantic metrics, side-by-side LLM-as-judge, and time-stratified evaluation specifically to evaluate the long-form speech.

**Strengths and weaknesses**

All reviewers are supportive of the paper, pointing out the contributions of designing long-form generative speech language model, new benchmark and evaluation protocol, comprehensive evaluation and reasonable approach and evaluation. Some concerns were raised about details of human evaluation, scaling properties, novelty (as SSM is not novel), absence of ablations on the hyperparameters. During rebuttal authors strengthen the paper providing more insights on these all aspects showcasing scalability to the 9B model, extended human evaluation and some insights on the ablations. In the end all reviewers were satisfied with the authors extra details, ablations and clarifications which was reflected in keeping scores on acceptance or increasing the final score for the paper.

**Recommendation**

Given well written paper, strong contributions and comprehensive study, rebuttal from the authors with clarifications and extensive additional details on the results, supportive position from all reviewers, and also importance of the results showing for first time that the spoken language model can be trained for long form speech generation and giving more or less simple recipe to do so, **I strongly recommend acceptance of the paper.**
Decision: Accept (oral)