SyllableLM: Learning Coarse Semantic Units for Speech Language Models
Authors: Alan Baade
Puyuan Peng
David Harwath
Venue: ICLR 2025 Poster (2025)
Keywords: Generative Spoken Language Modeling
Audio
Textless NLP
Representation Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce a new self-supervised method to learn semantic speech units (pseudo-syllables) that dramatically lowers bitrate and improves spoken language modeling compared to prior work.

Abstract: Language models require tokenized inputs. However, tokenization strategies for continuous data like audio and vision are often based on simple heuristics such as fixed sized convolutions or discrete clustering, which do not necessarily align with the semantic structure of the data. For speech in particular, the high resolution of waveforms (16,000 samples/second or more) presents a significant challenge as speech-based language models have had to use several times more tokens per word than text-based language models. In this work, we introduce a controllable self-supervised technique to merge speech representations into coarser syllable-like units while still preserving semantic information. We do this by 1) extracting noisy boundaries through analyzing correlations in pretrained encoder losses and 2) iteratively improving model representations with a novel distillation technique. Our method produces controllable-rate semantic units at as low as 5Hz and 60bps and achieves SotA in syllabic segmentation and clustering. Using these coarse tokens, we successfully train SyllableLM, a Speech Language Model (SpeechLM) that matches or outperforms current SotA SpeechLMs on a range of spoken language modeling tasks. SyllableLM also achieves significant improvements in efficiency with a 30x reduction in training compute and a 4x wall-clock inference speedup. Our code and checkpoints are available at https://www.github.com/alanbaade/SyllableLM

Published as a conference paper at ICLR 2025
SYLLABLELM: LEARNING COARSE SEMANTIC UNITS FOR SPEECH LANGUAGE MODELS
Alan Baade, Puyuan Peng, David Harwath
Department of Computer Science
The University of Texas Austin
ABSTRACT
Language models require tokenized inputs. However, tokenization strategies for continuous data like audio and vision are often based on simple heuristics such as fixed sized convolutions or discrete clustering, which do not necessarily align with the semantic structure of the data. For speech in particular, the high res- olution of waveforms (16,000 samples/second or more) presents significant challenge as speech-based language models have had to use several times more tokens per word than text-based language models. In this work, introduce controllable self-supervised technique to merge speech representations into coarser syllable-lik like units while still preserving semantic information. We do this by 1) extracting noisy boundaries through analyzing correlations in pretrained encoder losses and 2) iteratively improving model representations with novel distillation technique. Our method produces controllable-rate semantic units at as low as 5Hz and 60bps and achieves SotA in syllabic segmentation and clustering. Us- ing these coarse tokens, we successfully train SyllableLM, Speech Language Model (SpeechL.M) that matches or outperforms current SotA SpeechLMs on range of spoken language modeling tasks. SyllableLM also achieves signifi- cant improvements efficiency with 30x reduction in training compute and 4x wall-clock inference speedup. Our code and checkpoints available at https: / www github. com/ http//aaaaaaadSyyyaalll
INTRODUCTION
Learning to generate speech solely from listening to spoken language is fundamental task in speech processing. It requires abstracting beyond the underlying acoustics of speech into phones, syllables, words, and sentences to process correlations across long ranges of time. But while current textual language models (Touvron et al., 2023: Zhang et al. [022 Brown al.) (020)) can compose highly realistic text, language models on spoken language still struggle to output semantically meaningful speech. An increasing focus has coalesced around Generative Spoken Language Modeling (GSLM)
(Lakhotia et al. 2021), which sets out to achieve this goal.
The most successful approaches to GSLM are Transformer Decoder Language Models (Vaswani ct al. 2017) such as AudioLM (Borsos et al. 2023) and TWIST (Hassid et al. 2023). These Speech Language Models (SpeechLMs) operate on discrete tokens output by self-supervised learning (SSL)
based encoder models (Hsu et al. 2021 Chung et al. 202Tb. However, existing SSL tokenizations predominantly capture phonetic information (Choi et al. (2224) at high temporal resolution of 25-50 tokens per second, much greater than the typical human speaking rate of 2-5 words per second. This large number of tokens substantially impairs both training and inference speed (Hassid et al. 2023). and it is unclear whether modeling speech with such high granularity harms semantic understanding.
Very recently, there has been significant progress extracting coarser speech unit representations from raw audio. In particular, SD-HuBERT (Cho et al. 2024) finetunes HuBERT (Hsu et al. 2021
with DINO-like distillation objective, and VG-HuBERT (Peng & Harwath 2022 Peng et al. 2023)
contrastive loss against cross moosl visual inputs. We continue and significantly improve
upon this line of research, resulting in high quality speech tokens (units) suitable for SpeechLMs that exhibit temporal resolution roughly corresponding to syllables. Specifically, demonstrate

---

Published as a conference paper at ICLR 2025
[1-] Imli
Teacher HuBERT
215
cluatering
aaglome
108
73B
HuBERT
Student
527
527
tagio
LossPred
SylBoost
Syllabic Units
Figure Left-Top: The loss prediction matrix C, where brighter is higher likelihood placed the teacher label. A time-aligned transcript is on the bottom, and predicted cluster unit boundaries span vertically as dashed-lines. Left- Seft-Bottom: A Mel-Spectrogram of the input waveform with example masked timespan in gray. The losses on tokens at timesteps covered by the solid blue and dotted red spans are mapped their corresponding rows and columns in C' as described Section 3.1 Right: Visual of SylBoost. We train a student to match intermediate teacher features pooled over regions generated by pseudo-syllabie cetllo-yylaabe-buddddddee boundaries.... We use a min-cut algorithm on the feature self-similarity matrix to extract boundaries, and then apply K-Means and Agglomerative clustering to obtain discrete units.
significant improvements in textual reconstruction from low-bitrate units of SSL models, reducing the word-error-rate (WER) from 37% using SD-HuBERT units to 7%, and more than halving realized bitrate of previous SpeechLM units from 175bps low as 60bps. We additionally find that our units correlate strongly with syllables both in boundary detection and in cluster quality.
Furthermore, we evaluate the effects of training SpeechLMs on these new units and obtain state-of- the-art results across wide-variety of metrics, competitive with or outperforming AudioLM (350M parameters), all TWIST model sizes (125M-13B parameters), and the newly released Moshi-7B (Défossez et al. 2024) with significantly fewer parameters, lower training time, and faster inference speed. Our contributions are as follows:
1. We propose an unsupervised algorithm named LossPred that reveals noisy syllable-like segmentation of unannotated speech signals by analyzing the loss a pretrained self- supervised model (e.g. HuBERT) across different masking spans.
2. We propose a method named SylBoost that sharpens the feature space of self-supervised speech models by bootstrapping from LossPred syllable-like segmentations. SylBoost achieves SotA unsupervised syllabic segmentation, categorization, and low-bitrate unit-to- audio resynthesis.
3. Using quantized SylBoost units as a basis for tokenization, we train SyllableLM, a SpeechLM that outperforms or matches prior approaches on range of tasks while be- ing 30x faster to train, 4x faster for inference, and having a 2.5x reduction in unit bitrate.
2 RELATED WORK
Self-Supervised Encoder Models There has been a great amount of work in learning livellevel
representations from data by reconstructing corrupted inputs across speech (Baevski et al. 2020 Hsu
al. 2021 Baevski et al. 2023), audio (Gong et al. 202T). text (Devlin et al.] 2019 Clark et al.
2020). and vision (Caron et al. 2021 He al. 2021). To navigate the lack of simple discrete targets in speech, much work has been placed finding high-quality targets, such as iterative clustering (Hsu al.] 2 and by predicting the representations of a teacher network based on a running average of student model weights (Baevski et al. 2022 2023). An alternate but similar line work has been placed into learning low-bitrate units for the task of resynthesis (Défossez al.] 2023] Zeghidour et al.) 2021 Yang et al.) 2023a Kumar et al. 2023 Zhang et al., 2023 Du et al.) (203),, which include losses focused reconstruction and information bottleneck to enforce compression.
2

---

Published as a conference paper at ICLR 2025
Applications of Neural Codees
The discrete units generated by these self- encoders
versatile and fundamental to much of the recent progress in speech research such as Text-To- Tort-T--Speer To-Speech
(Wang et al., [2023 Ju et al. 2024| [ong et al. 2024 Peng et al. (0024), joint audio-text foundation
models ( Yang et al. 2023b Chou et al. 2023 Nguyen et al. 2024). unsupervised speech recognition
(Baevski al. 2021 discrete unit resynthesis (Polyak al. 2021 Défossez al., 2023 Zeghidour
ct (021), lext-to-audio (Kreuk et al. 2022, Agostinelli et al. 2023 Copet et al. (023), and
generative spoken language modeling (Borsos et al. 2023 Hassid et al.] 2023 Lakholia et al. 202T).
Each of these methods operates on audio units exc exclusively greater than or equal to 25Hz, which
has been frequently cited area for future work to improve on (Hassid et al.
(023). Recent work (Elkahky et al. 2023) has also explored training speech encoder models with coarser units as targets.
Extracting Semantic Units from Raw Data Additionally relevant to our work are several ap- proaches, particularly in vision and audio, that generate emergent semantic clusterings from self- supervised transformer (Vaswani et al. 2017) models. In particular, the DINO approach in Caron al. (02T)) observes object representations in attention maps through student-teacher distillation. Similar ar techniques have been also applied to audio to discover emergent syllable boundaries (Cho al. 2024] Peng et al. (023). These behaviors can vary heavily with small changes in pretraining
strategy explored in Darcet et al. (2024). Merging similar features has also been shown to produce significant vision model speedups such as in Bolya et al. (2023). Most similar to our work, Algayres
ct al. (2023) extracted coarse continuous representations for GSLM, however these results trail behind
language modeling approaches.
3. LEARNING SELE-SUPERVISED, SYLLABLE-LIKE REPRESENTATIONS FROM RAW SPEECH
In this section, we describe the bootstrapping process by which we extract low-bitrate speech units. We first describe LossPred, our algorithm to analyze outputs of self-supervised speech model loss functions to generate initial syllable-like unit boundaries. Following this, we define SylBoost, procedure to iteratively refine these boundaries with student-teacher distillation. We also propose a new algorithm for the efficient extraction of boundaries from feature self-similarity matrices to fix the bottleneck slowing down VG-HuBERT and DD-HuBERT extraction.
3.1 LOSSPRED: EXTRACTING SYLLABLE-LIKS SEGMENTATION FROM RELATIONS IN HUBERT'S LOss
HuBERT has previously been shown learn phone-like units with its K-Means clusterings (Hsu et al. 2)2T) which have formed the basis of subsequent works on GSLM and unsupervised ASR (Baevski ct al. 202T] Lakhotia et al. 2022 Hassid et al., 2023). However, other work (Pasad et al. 2023 2024) has shown that the representations learned by these models also correlate with higher level structure such as words, despite these structures not immediately appearing during clustering. Our goal in this section is propose method that can be applied to a pretrained HuBERT model in order to automatically extract unit boundaries at the level of syllables or words rather than phones. Although we apply our method to HuBERT, we expect that it could also be applied other SSL speech models that utilize a similar loss function such as WavLM (Chen et al., [022) or wav2vec2.0 (Baevski et al., 2020). The crucial commonality between these models that they all utilize a masked language modeling (MLM) training objective, whereby input speech tokens are randomly masked and the model is trained to predict the masked inputs conditioned on the unmasked inputs.
We ground our intuition for LossPred with the following thought experiment: if the input tokens corresponding to an entire word were replaced with mask tokens, we would expect the HuBERT model loss at these timesteps to be relatively high, as HuBERT would have to jointly predict word identity and the underlying acoustics to predict the missing span. On the other hand, if only the latter portion of a word were masked out, infilling this masked region given the word prefix may be easier by comparison. With this, we iteratively shift a contiguous mask over span of tokens and look at the loss, would suspect to see strong decrease in the loss throughout the timesteps corresponding to a masked semantic unit (word, syllable, or otherwise) as the beginning or end of the unit was partially revealed to the model.
3

---

Published as a conference paper at ICLR 2025
LossPred operates using two frozen off-the-shelf HuBERT models, student model and teacher model, where the student model has been optimized to predict the quantized representations of the teacher model at layer L, and the teacher model comes from the prior iteration of pretraining as described in Hsu et al. (202T). Formally, given an input waveform W, we extract the teacher labels by passing W into the teacher model and then quantizing the representations layer I. with K-Means. We denote these teacher labels as YI...r3. where I is the number of timesteps. As done during HuBERT pretraining, we give the student model a corrupted version of W where outputs from student CNN feature extraction at select timesteps are replaced with the HuBERT 'mask' embedding. We denote these student CNN features as X.M (1...T) where M {fI.... f' } is a set of masked out timesteps. Following the pretext task, the student then predicts the teacher labels at masked timesteps Y,, E M conditioned on X M and is evaluated using cross-entropy loss:
LHUBERT == - -logp(Y; X²
We define a loss prediction matrix C, shown in Figure ! which looks at the losses of the student model as move contiguous sliding window of timesteps mask. Specifically, C models the raw probabilities of the losses where the column is the timestep being predicted and the row represents the closest unmasked token.
PYY X M fr+ 1,r 2, 23..,1+)30
< c -
Cr. ERT
P(Y. X M -rr (r- 1,r -... if > c, c
otherwise.
We separately calculate the upper and lower triangles of C, relating to the closest observed waveform being before the mask and after the mask respectively. For simplicity. assume here that the losses in the first half of the mask span are uncorrelated with information after the mask and vice versa. In column e is equal to p(Y: X²M given that the mask span
the upper triangle, each entry Cr.c in XM starts just after time r. Inversely, in the lower triangle, Cr.e is equal to p(Y: X xM) given that the mask span ends just before time r. We use span size 50 corresponding to I second this duration is long enough to mask the majority of spoken words. We calculate the upper triangle using the first 25 timesteps of the mask span, and the lower triangle using the last 25.
To extract k semantic regions with boundaries B = (b < b2 < b T + 1) from C. we use the min-cut algorithm from (Shi & Malik 2000) discussed further in Peng et al. (002)). treating C' as the input naature-similarig matrix:
bes--
Σ i,j=b
Cis
B
arg max
b T
b
(b <ha by
Σ (C,s + Cz,a) b=b+ 1=1
Σ
Cis
By choosing i to be proportional to the length of the utterance, our boundaries. We explore modifying this parameter in-depth throughout our experiments. We find that semantic units extracted by LossPred tend be syllable-like (both via inspection, and also confirmed experimentally in our segmentation and clustering experiments) and so we focus on syllable-resolution units for the rest of the paper.
control the sample rate of
LossPred is expensive to run due to using one forward pass for each sliding window location. To make this efficient, we extract multiple masked spans simultaneously with a gap between spans of three seconds. This results in roughly 200 forward passes of the student model to calculate C' on an arbitrarily-sized audio. Fortunately, we only ever need to run LossPred once to initialize SylBoost training (described in 3.2). so this does not affect later semantic unit extraction speed.
3.2 SYLBoOST: BOOTSTRAPPING
-YLLIO---YLLBICCCC UNITS WITH ITERATIVE DISTILLATION
Given the initial boundaries predicted by LossPred, follow the paradigm of shiyy-studentteechhe learning (Xie et al. 2020) to extract semantic representations. Our goal is to "sharpen" the syllabic organization in the feature space of an input student model, seen in Figure We choose
4

---

Published as a conference paper at ICLR 2025
dockans station attrain
dockeda attrain
station
"Mrors "ror tor3lo ca.
brain at station
docked
HuBERT
D2V2
SylBoost 8.33Hz
SylBoost 6.25Hz
SylBoost 5.0Hz
Figure 2: Qualitative results on SylBoost controllability for boundary detection. We plot the feature similarity matrix A, described in 3.2 for HuBERT, Data2Vec2, and SylBoost Data2Vec2 when trained at different unit rates. The number of cuts k is selected dynamically as described in 5.2
pretrained HuBERT (Hsu et al., 2021) or Data2 (Baevski et al. 2023) to initialize our SylBoost student and teacher models, with the teacher model parameters held constant.
The SylBoost procedure is depicted in Figure [ For set of hypothesized speech segment boundaries group together all temporal tokens between two boundaries into disjoint groups. G: = {t b, S b.+1}. For notation, we let H, be the index of the group that contains layer I., which we select based on syllabic correlation as explored in detail in Peng et al. (003); Pasad et al. (2023 2024). This results in student features ( and teacher features Vi....) The loss is then the mean squared error between the student features X(² at each timestep and the mean of the teacher features in the timestep's 's corresponding group:
B (b, = b2 < ... bb+11 + 1,
E G.. We apply the SylBoost loss (Equation 4) model features at
T
LyylBoott = T Σ
X
(T)
[HH
sEGN;
After training. SylBoost results in a student model with sharpened latent features X( from which we calculate matrix A of pairwise frame distances where element Ai,j equals the L2 distance between the features ( and X(4) as depicted in Figure 2 We then extract boundaries from A using
cut algorithm described in the next section (Sec. 3.3 With this, we can generate new pseudolabel
boundaries and run SylBoost again to extract even better boundaries, which we perform twice. Finally, we extract discrete clusters by mean pooling the features at layer L over each SylBoost boundary group, applying K-Means to produce large number of clusters, and then using Agglomerative Clustering to merge similar clusters into a final desired number of units.
3.3 EFFICIENT EXTRACTION OF UNIT BOUNDARIES FOR SYLBoOST
To extract boundary indices from learned feature representations Peng et al. (2023) proposed adapting the mincut approach in Malioutov & Barzilay (006)). However, for speech this approach is slow in practice and difficult to parallelize, bottlenecking our ability to extract boundaries in bulk across the large corpora necessary for downstream language modeling. Inspired by the SylBoost objective, propose a more efficient approach for extraction: given + potential boundaries, we seek to choose groups that minimize the sum of the distances from each unit to the mean of its assigned group:
b+11-
b
B
arg min
Σ
Σ
x
X (E)
(5)
(br
T+1)
1=1
j=b;
bi+1
by
1=
We further restrict the setting by choosing a maximum group length G tokens, where we choose G' 50 to correspond to second of tokens, as syllables or words longer than this are fairly rare. With this, we can then split our algorithm into 1) calculating a distance array D E RTXG where Dz.g the cost of the group of length g ending token t and then 2) solving the minimal interval cover from this distance array with dynamic programming. An efficient implementation using Py PoTch (aaszk et al. 2019] CUDA (NVIDIA al. 2020 runs in ((k) data-aware sequential steps.
5

---

Published as a conference paper at ICLR 2025
4 SYLLABLELM: SPEECH LANGUAGE MODELING ON COARSE UNITS
GSLM (Lakhotia et al. 202T defines a pipeline for modeling raw audio as three stages: 1) Audio-to- unit Tokenization, 2) Running language model on these units, and 3) Decoding the tokens back into a waveform. For Audio-to-unit Tokenization, we use the second iteration of SylBoost as described in In this section, we lay out our language modeling and resynthesis.
Section 3
4.1 LANGUAGE MODEL
Like AudioLM and TWIST, we use an autoregressive transformer decoder language model to approximate p(æt 2t-1,-..5 F1) given an input token sequence ø1, .1....::2 Zr. We refer to this language model trained on the discrete clusters output by SylBoost as the main SpeechLM. Like TWIST, we prepend a <BO5> token and make no other special changes.
4.2 TOKEN TO SPEECH DECODING
To convert SylBoost tokens back into audio, adopt the interleaved decoding strategy from Song et al. (2024) to output the units from TWIST (Hassid et al.) 2023). obtaining a waveform by cascading this output into their provided vocoder. This interleaving strategy demonstrates superior performance
in high-difficulty settings compared to other text-to-speech models like VALL-E (Wang et al. 2023).
and
it for all resynthesis experiments. To interleave our units during training, we sort the start-timester of every SylBoost unit and TWIST-unit in ascending order. We subtract 0.08s (the length of two TWIST units) from each SylBoost unit start time before sorting account for noisy SylBoost boundaries. For the rest of the pipeline, we follow Song et al. (2024) without global advance using our syllables as drop-in replacement for phones and use greedy decoding. We call this model the Interleaved Vocoder-L.M
Although incorporating this additional resynthesis model slows down generation compared TWIST, most model scaling happens in the SpeechL... For example, the TWIST paper still observes scaling improvements in semantic understanding with a Speechl.M of 13B parameters while current SotA speech synthesis models such as Ju et al. (2024) operate with fewer than IB parameters. Taking the same approach as TWIST, our work focuses purely on improving the semantics of generated audio and is entirely orthogonal work audio synthesis quality and voice cloning.
We then generate continuations for a sample by 1) Extracting syllable-ume units and TWIST units from the sample, 2) Sampling syllable-unit continuations from the SpeechLM, 3) Continuing TWIST units with our interleaved model conditioned on sample TWIST units, sample syllable-units, and continued syllable-units, and 4) Resynthesizing these into speech using the vocoder.
EXPERIMENTS
5.1 TRAINING DATASETS
We train our tokenizer using LibriSpeech (Panayotov et al. 2015), which contains 960 hours of audio books. We noticed that SylBoost converges before all data is used, and we randomly subsample LibriSpeech to 100 hour train set and train for five epochs and two iterations for all experiments. We train our SpeechLMs using all of LibriLight (Kahn et al. 2020), which provides roughly 55k hours of speech. As a note on fair comparison, although AudioLM uses exactly this split of LibriLight, TWIST collects an additional 100k hours of data, totaling 155k hours.
5.2 SYLBoOST UNIT CONFIGURATIONS
By varying the number of boundaries input our cut algorithm at each stage in the SylBoost pipeline. we can arbitrarily control our rate temporal tokenization. We evaluate three main unit rates at 8.33Hz, 6.25Hz, and 5.00Hz, the latter which matches the empirical rate of SD-HuBERT units on LibriSpeech dev-clean. To account for differences between slow and fast speakers, we dynamically choose & during SylBoost to best match the number of distinct feature groups in the embedding space (seen as blocks in the feature self-similarity matrix A, Figure 2 We do this by setting a threshold and taking the fewest boundaries k such that the extraction objective (Equation 5) per timestep is less

---

Published as a conference paper at ICLR 2025
Table 1: Unsupervised syllable boundary detection and clustering accuracy on LibriSpeech (Panayotov et al. 2015) test. For Fl scores, the superscript is tolerance threshold in ms. All other metrics use 50ms. Higher is better.
Approach
Backbone
Training
F1 50
F1
Pr.
Re.
R
CPur
SPur
Feat-Sim(Peng et al. 2023
HuBERT
no
47.3
24.7
46.6
48.0
54.5
28.0
30.0
LossPred 5.0HZ (Ours)
HuBERT
no
59.6
31.4
54.9
66.7
56.3
SD-HuBERT Cho et al. 2024
HuBERT HuBERT
yes
66.1
32.2
64.9
67.4
70.7
43.2
45.0
SylBoost 5.0H7 (Ours)
yes
70,9
40.1
70.8
71.4
75.1
28.9
47.8
SylBoost 5.0Hz (Ours)
Datu2
yes
73.2
44.6
72.1
74.4
76.9
33.6
54.9
Table 2: Boundary detection
Table 3: Controllability of unit rate measured on LibriSpeech dev-clean boundary detection.
Table 4: Inference speed measured in Real- Time-Factor (RTF), the seconds per pro- cessed second. 32 batches with 25s of audio
with different initialization using
HuBERT on LS dev-clean
D2V2, 50ms threshold. P.Phone, S:Syllable, Woword
each, I GPU, 16 Cores. Hardware in A.4
Model
FI
Pr.
Re.
Tokenizer Encoder
RTF
Feat-Sim
46.7
48
45
SD-HuBERT (Cho al.
2024
2.72F-3
-Iter 1
51.1
50
52
F1-P
F1-S
F1-W
HuBERT+(Pcng et at. 2023
1.14E-2
-Iter 2
50.4
51
50
8.33
72.0
63.5
56,8
HuBERT+Our Extraction
2.05E-3
LossPred
60.1
53
68
6,25
65.2
71.8
66.0
SpeechLM, 90M, Cached Units
-Iter 1
67.1
67
68
5.0
58.7
73.0
71.8
-Iter 2
70.2
70
70
4.3
54.3
73.2
74.0
TWIST
1.28E-4
Ours 6.25Hz 8k
2.88E-5
than for 75% of timesteps. We choose ô so that average unit rates over LibriSpeech dev-clean match a desired frequency. We also explore modifying the number of clusters generated by K-Means and Agglomerative Clustering, which combined with changing the unit rate results in fine-grand fineggrined bitrate control. We note that although prior approaches like SD-HuBERT and VG-HuBERT apply cut algorithm with cuts, there no way to control the actual number of distinct feature groups in the self-similarity matrix during training. As result, we cannot increase the frequency of SD-HUBERT units by changing k; Additional cuts result in close-to-identical representations that map to the same quantized clusters.
5.3 RESULTS: EVALUATING UNIT QUALITY
We evaluate the quality of our semantic units with two approaches 1) measuring correspondence with syllables and 2) running speech resynthesis followed by ASR. To measure correspondence with syllables, we use the development and test sets of LibriSpeech (Panayotov et al. 2015) and follow the approach from Peng et al. (2023, extracting timesteps for phones using the Montreal Forced Aligner (McAuliffe et al. 2017) and then converting these phones into syllables with rule-based method [Gorman 2014). We evaluate the quality of syllable boundary detection with a ground truth boundary marked as hit if proposed boundary is present within tolerance threshold. We report F1, Precision, Recall, and R score (Räsänen et al. 2009). We ablate Fl scores with tolerance windows of 20ms and 50ms. Given boundaries, we also evaluate the purity of our clusters at 4096 units. Syllable Purity measures the probability that a syllable is mapped to its most corresponding cluster unit, and Cluster Purity measures the probability that a cluster is mapped to its most corresponding syllable unit.
Even if units do not correspond with syllables, they can still be useful to SpeechLMs if they can resynthesize back into speech that matches the original text. Additionally, training resynthesis model provides stronger description of the semantic information contained in units than purity metrics, which may be unreliable because SD-HuBERT does not provide a unit at every timestep while methods do. To evaluate resynthesized speech, we follow AudioLM and measure Word Error Rate (WER) and Character Error Rate (CER) on the set of 4-10 second segments from LibriSpeech test-clean. For ASR, follow VALL-E (Wang et al. 2023) and use the public HuBERT-basee CTC ASR model provided by Hsu et al. (2021).
Table I shows our syllabic correspondence results against the prior-state-oftther SD- HuBERT (Cho ct al. 2024) and the HuBERT-basse feature similarity strategy from Peng et al. (202)). Applying SylBoost to either HuBERT or Data2Vec2 improves correspondence ciross-the-boarde except for
7

---

Published as a conference paper at ICLR 2025
Table Unit Resynthesis. WER/CER results on 4-10 second examples on LibriSpeech (Panayotov et al. 2015 test-clean. Hz and Bitrate are measured post deduplication on LibriSpeech dev-clean.
Model
Changes
Hz
#Units
BPS
WER
CER
SD-HuBERT (Cho et
2024
4096
60
37.3
22.7
SylBoost (HuBERT)
+Our Clustering
5.0
4096
60
18.5
10.2
SylBoost (Data2 Vcc2)
+Use Data2Vec2
5.0
4096
60
12.8
6.4
SylBoost (Data2Vec2)
+Increase #Units
5.0
16384
70
9.1
SylBoost (Data2Vec2)
+ +Tune unit-rate, #Units
8.33
2048
91
8.0
3.7
SylBoost Vec2)
+Tune unit-rate, #Units
6,25
8192
81
7.0
3,2
TWIST Tokenizer (upper bound) (Hassid ct al. 2023)
19.5
500
175
6.3
2.5
in cluster purity. Although LossPred trails SD-HuBERT in performance, it pushes the boundary for syllable recognition using HuBERT without additional training. Improvement across iterations and with different loss initialization can be found in Table 2 which shows that using LossPred as bootstrapping source instead of HuBERT similarity (Peng et al. 2023) makes SylBoost converge to much higher segmentation accuracy. We discuss initialization more in Appendix [A.2 In Table we demonstrate the controllability of SylBoost to extract boundaries at different granularities: higher unit rates successfully correspond more with phones and lower unit rates correspond more with syllables and words. In Table 4 we show the speedup of our efficient unit extraction.
We demonstrate the step- step-by-step changes used to improve unit cluster re- sen-hytthess quality as compared to SD-HuBERT in Table S with the same clustering parameters as SD-HuBERT. We further decrease WER by a third by using Vec22 and from there by modifying the unit sample rate and number of clusters can reach as low as 2048 clusters and a WER of 7%. These results demonstrate by far the lowest bitrate we are seff-supervised-unii resynthesis. Resynthesis for all models we train is done back into TWIST-Tokknizer units, bounding potential quality at WER of 6.3%.
We observe over a 50% decrease in WER and CER by using SylBoost
aware of for "reasonable-quaii
5.4 SPEECH LAUGUAGE MODEL CONFIGURATION
All of the SpeechLMs we implement, as well as our Interleaved- AnterlaveddVVccode---e Vocoder-LM, follow the OPT (Zhang et al. 2022) architecture and default using 12 Transformer layers, an embedding dimension of 768, and learned positional embeddings. This totals to 90M non-embedding parameters, and represents an identical model architecture to TWIST- 125M. We also experiment with a larger 24 layer 1024 dimension model totaling to 300M non-embedding parameters, the same as AudioLM and TWIST-350M, For all language model pretraining experiments, we randomly crop files to 25 seconds, use batch size of 80000 tokens, and train for 200k steps, which amounts the same compute as in TWIST. To make our approach entirely textless, we do not use TWIST initialization. We tokenize using SylBoost on Vec2 (Baevski et al. 2023), which provided the best results in low-bitrate WER resynthesis (Table We discuss the base encoder further in Appendix A.3 Additional hyperparameters and hardware details Appendix A.4
5.5 SPEECH LANGUAGE MODEL BASELINES
We compare against a range of SotA SpeechLMs: AudioLM (Borsos et al. 2023). TWIST Hassid et al. 2023), GSLM (Lakhotia et al. 2021), and the audio-only version of Moshi (Défossez et al. 2024). As a tokenizer, M uses on tokens from HuBERT model that the TWIST authors pretrain for additional iteration on multilingual data, Lakhotia et al. (202T) uses HuBERRT-base and K-Means, and Moshi operates on residual codec with the first code distilled to match WavLM (Chen et al.) 2021). AudioLM and TWIST are the most directly comparable models to SyllableLM with respect to dataset and transformer size and output units at 25Hz followed by consecutive deduplication.
tokens (Chung et al., 202T). TWIST operates
We reimplement a 90M parameter model using the TWIST tokenizer units without textually partaly-ee initialization (Cold- Init in the TWIST paper) our data split for all-cise-hed hheld equal comparison on unit type. We additionally reimplement Byte Pair Encoding (BPE) to train a SpeechLM as done in Shen al. (2024), resulting in the lowest bitrate encoding of speech outside of our model. We
8

---

Published as conference paper at ICLR 2025
Table 6: Main SyllableLM results. We evaluate on SWUGGY Lnca,,,,, All, Out-of-Vocabe sBLIMP from Nguyen et al. 2020), and IStoryCloze from Hassid et al. (2023). Higher is better. *Estimated
GPU Hours, see Appx. A.4 for details. Bold is best, underlined is second-best bestt large models in gray.
#Dara
#Dmt
SWUGGY
Model
Params
#Units
Hz
BPS
Hours
Toks
Hours
All
IV
OOV
sBLIMP
ISC
Phone Topline
90M
70
12.5
76
55K
2.5B
70
81.4
95.2
67.7
68.8
80.6
Syllablc Toplinc
90M
28k
5.0
74
55K
1B
70
79.5
95.1
65.9
69.3
76.6
Lakhotia et al. 2021
150M
100
332
6K
1.1B
64.8
54.2
66.6
Audiol Biossss etal 2023
300M
1K
250
55K
5B
2,9K*
71.5
83.7
59.3
64.7
TWIST HHassed of 2025
300M
19.5
175
155K
9B
295
70.6
80.3
61.0
56.2
69.9
TWIST
1.38
500
19.5
175
15SK
9B
1.
71.8
81.1
62.3
57.0
70.6
TWIST
7B
72.7
TWIST
131
500
19.5
175
155K
9B
100.
73.9
84.1
63.7
59.2
76.4
Defossez 2024
7B
2K
1.1K
2.5T
74.8
TWIST-CI
90M
500
19.5
175
55K
3.9B
84
69.7
79.8
59.7
55.5
69.0
(Sben 2024
2B
SyllableLM
90M
2k
8.3
91
55K
1.6B
70
72.2
81.7
62.6
62.4
71.4
SyllableLM
8k
6.25
55K
1.28
75
72.1
62.9
SyllableLM
90M
16k
5.0
70
55K
1B
82
67.6
76.9
58.3
63.2
69.0
SyllableLMM
300M
8k
6.25
55K
1.2B
290
72.2
82.2
62.0
63.7
75.4
Table 7: Holding number SylBoost units and unit rate constant. ZeroSpeech development set.
Table Continuation Metrics. measure PPX @ Oracle- VERT and VERT@Oracle-PPP (Descrihed in 5.6
Hz
#Units
sWUGGY
sBLIMP
Model
P-X
VERT @O-P
8.33
4k
72.9
61.8
TWIST 300M
205 + 24
24.0 1.0
6.25
4k
69.3
63.3
TWIST 1,3B
175 + 14
22.6 - 1.2
5.00
4k
65.7
62.8
BPE 90M
148 + 12
17.3 + 0.9
8.33
2k
72.1
62.0
8.33Hz 2k 90M
159 =
15.1 0.9
8.33
4k
72.9
61.8
6.25Hz 8k 90M
139 + 12
20.1 + 0.7
8.33
8k
72.9
61.2
5.00Hz 16k 90M
131 + 11
15.2 + 1.0
6.25Hz 8k 300M
116 = 7
15.8 1 0.9
run BPE on deduplicated TWIST units and we grid search find that the minimum bitrate for BPE occurs at 4k units, which we use for all experiments (Shen et al. (2024) originally operated on 50Hz units, meaning that the 118bss rate obtained here also new). For textual toplines, we train on corresponding LibriLight text transcripts from Kang et al. (2023) and convert text to phones and syllables using the same methods as in Section 5.3
5.6 RESULTS: SPOKEN LANGUAGE MODELING
The end-to-end GSLM pipeline consists of several components, and it is essential to have metrics to independently evaluate different stages. To evaluate our SpeechLM stage, we follow Lakhotial al. (202T) and the ZeroSpeech (Nguyen al. 2020 sWUGGY and sBLIMP evaluation. The SWUGGY dataset tasks the model with outputting higher perplexity on similar but fake spoken words (e.g. brick blick). Similarly, the sBLIMP dataset checks syntactic correctness (c.g. the dog sleeps vs the dogs sleeps). We also evaluate the SpeechLM on the tSC set from Hassid et al. (2023). which operates like the ZeroSpeech metries on spoken version of the StoryCloze dataset MMostaazaadeh et al. 2016) with the last sentence in negative samples randomly chosen. For all metries we follow prior work and output the mean perplexity per token.
The results for SpeechLM metrics are depicted in Table We find that training with our syllable units improves performance across-the-board on speech understanding tasks. With under 90 hours of training, SyllableLM outperforms the 13B parameter TWIST and 7B Moshi on sBLIMP, which best evaluates semantic understanding (Lakhotia et al. 202I). Measured on all of WUUGGY we also beat AudioLM with 30x less GPU compute and TWIST model sizes up to 1.3B parameters in lexical understanding. On tSC, we observe that SyllableLM benefits from scaling with our large model approaching the performance of our textual topline. Because units much lower frequency, we see a 4.5x speedup at inference time with the same parameter count in Table Due to compute requirements, unable to scale further.

---

Published as a conference paper at ICLR 2025
We notice decrease in SWUGGY quality with our 5.0Hz units, which we suspect is in part caused by the short length of the dataset audios making input tokenization excessively short. We further ablate changing the unit rate and unit frequency in Table 7 We also find that BPE, despite having the lowest bitrate outside of our approach, does not approach the quality gains created by our syllable-like units.
To measure the quality of end-to-end continuations, use the VERT@ MIRTTO-PPI @O-PXX and PPX O-VRERT metrics proposed in Lakhotia et al.] (202I), which are shown to be the automatic metrics correlating best with human meaningfulness judgements. VERT@O-PP2 measures the diversity of output at the sampling temperature where the perplexity of generated audio transcriptions matches that of the ground truth text, and PPX@O VERT performs the inverse. Like Lakhotia al. (202I). we generate 10-second continuations from 1000 randomly sampled 3-second crops from LibriSpeech test-clean, and measure results using their provided environment and parameters. To prevent errors from randomly cutting off audio in the middle of syllable-like unit, we discard the last encoded unit for all models when generating continuations. We report these in Table with two sigma error bars and find that SyllableLM outperforms TWIST 300M and 1.3B.
6 DISCUSSION AND LIMITATIONS
Though speech is very general medium, there are number of challenges in adapting our methods to generate low-bitrate units angled towards other audio tasks or other domains such as vision. Our LossPred technique assumes that the semantic units to learn are separable across time, one- dimensional, and contiguous. In audio tasks or settings with multiple speakers, sounds or words can occur simultaneously. Images and video have partially occluded or overlapping objects.
Despite this, LossPred is unique in that it finds boundaries using the values (losses) that the pretrained model is directly trained to optimize. Meanwhile, other approaches such as DINO (Caron al., 2021) and SD-HuBERT rely on difficult-to-contro) intermediate features for semantic discovery. SylBoost then demonstrates that we can use LossPred to extract emergent and interpretable features even when they are not clear in the original model. Because of this, we believe that LossPred and SylBoost provide a very promising direction for controllable and scalable semantic feature clustering and extraction in domains like speech, audio, music, and video.
Low-frequency units such as ours also provide a significantly more computationally tractable path toward the scaling that has seen such success in textual language models. On the other hand, SylBoost units may be losing out on useful paralinguistic features like tone whose impact is only salient on non-audiobooks or scale. We suspect that hybrid models operating low-bitrate and high- bitrate units could be constructed to balance these considerations and demonstrate significant quality improvements in multimodal speech-text language modeling.
CONCLUSION
We introduce a new method to tokenize speech for use in SpeechLMs. We do this by proposing a method to elicit syllabic organization in pretrained speech encoder models, bootstrapping feature- space clustering algorithm from a static analysis of correlations in off-the-shelf SSL encoder model losses across time. We demonstrate the success of our technique both in having strong associations with syllables and as an extremlely low-bitrate codec of speech for textual resynthesis. Using this tokenization strategy, successfully train SyllableI.M, a Speechl.N .M that out-performs comparable state-of-the-ard approaches across a diverse range of metrics with a significant training and inference speedup. We further ablate several design decisions such as quantization strategy, loss initialization, and the effects of controllability for downstream usecases. Compression is a crucial aspect of learning, and we hope that these significant improvements in the unsupervised learning of low-bitrate speech units can serve as foundation for approaches towards understanding spoken language and general representation learning.
8 REPRODUCIBILITY
We focused heavily on reproducibility throughout our work, and we have released all parameters and code used upon deanonymization. We provide the exact equations used to calculate LossPred,
10

---

Published as a conference paper at ICLR 2025
SylBoost, and our Efficient Extraction Pipeline, allowing for reimplementation solely from the main text. The SyllableLM architecture and Interleaved- Vocoder-LM are well-defined and follow publicly sourced prior work. All datasets used for training and evaluation are listed explicitly in the main text, and all evaluation procedures are described as would be needed to calculate them.
ETHICS
It is important to note that large textual language models can have harmful effects, such as enabling the generation of misinformation in mass, Although generative spoken language models have not yet caught up to their textual counterparts, it is still necessary to be aware of potential misuses that could arise the future.
REFERENCES
Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. Musiclm: Generating music from text, 2023.
Robin Jonathan Algayres, Yossi Adi, Tu Anh Nguyen, Jade Copet, Gabriel Synnaeve, Benoît Sagot, and Emmanuel Dupoux. Generative spoken language model based on continuous word-sized audio tokens, 2023. URL /ttttt:://ppeermt
net/forunm 2ee
Alexei Baevski, Henry Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: framework for self-supervised learning of speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020.
Curran Associates Inc. ISBN 9781713829546.
Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Unsupervised speech recognition. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:://ppenss htt.:eee..eeeeeeeeeeeeeeeee forum?
QmxF so FRvW9)
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language, 2022.
Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In Proceedings of the 40th International Conference on Machine Learning, ICML'23 23. JMLR.org 2023.
Loic Barrault, Yu-An Chung. Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li, Daniel Licht, Jean Mail- lard, Ruslan Mavlyutov, Alice Rakotoarison, Kaushik Ram Sadagopan, Abinesh Ramakrishnan, Tuan Tran, Guillaume Wenzek, Yilin Yang, Ethan Ye, Ivan Evtimov, Pierre Fernandez, Cynthia Gao, Prangthip Hansanti, Elahe Kalbassi, Amanda Kallet, Artyom Kozhevnikov, Gabriel Mcjia Gonzalez, Robin San Roman, Christophe Touret, Corinne Wong, Carleigh Wood, Bokai Yu, Pierre Andrews, Can Balioglu, Peng-Jen Chen, Marta R. Costa-jussà, Maha Elbayad, Hongyu Gong, Francisco Guzmán, Kevin Heffernan, Somya Jain, Justine Kao, Ann Lee, Xutai Ma, Alex Mourachko, Benjamin Peloquin, Juan Pino, Sravya Popuri, Christophe Ropers, Saflyyah Saleem, Holger Schwenk, Anna Sun, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang, and Mary Williamson. Seamless: Multilingual expressive and streaming speech translation, 2023.
Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: httpp:://opereviiiiie http:///ppeeviii.eeem//e net taattfoorune Forum?id-
JrozRaRw ( Eu
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: language modeling approach audio generation, 2023.
11

---

Published as a conference paper at ICLR 2025
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- neurips cc cappaper...p..e..eeeeeeeeeeee..e files/paper)
ciates, Inc., 2020. URL
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Micheal Zeng, and Furu Wei. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16: 1505-11518, 2021. URL https::/ /ap.. semanticschon
wwt. ID; :t::://opppui::::::::::: 239885872
Cheol Jun Cho, Abdelrahman Mohamed, Shang- Wen Li, Alan W Black, and Gopala K Anu- manchipalli. Sd-hubert: Sentence-level self-distillation induces syllabie organization in hubert. 2024.
Kwanghee Choi, Ankita Pasad, Tomohiko Nakamura, Satoru Fukayama, Karen Livescu, and Shinji Watanabe. Self-super speech representations are more phonetic than semantic. In Interspeech, 2024.
Ju Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau, Alexei Baevski, and Michael Auli. Toward joint language modeling for speech units and text. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https
Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. w2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 244-250, 2021. URL https: /atss//aapaa /api. semant icscholar. DrrgCooppssID
237048255
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning, Electra: Pre-training text encoders as discriminators rather than generators. International Conference on Learning Representations, 2020. URL pttctpps://ppnnrvieee net fett:/oooumidddllllela
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre D' efossez. Simple and controllable music generation. ArXiv, wwtp//236.002284 2023. URL htt.::ttps://ppp semant ascho ar wogg/Corpuseeeo :
2591 259108357
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL
stttps://opee httpss////ppeeeeew...o net http:/oommmmmmmmmmmmmmm
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi High fidelity neural audio compression. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. httss Featured Certification, Reproducibility
Soopenreview.net/EEee wtt.::/eeoooommmmssiiaceeae..ee. Certification.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation fommlattotttommditiioo model for real-time dialogue. Technical report, Kyutai, September 2024. URL http:::/://// .org/Moshi. ..o.
12

---

Published as a conference paper at ICLR 2025
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics, doi: 10.18653/v1/N199
1423
Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng. Funcodec: A fundamental, reproducible and integrable open-s toolkit for neural speech codec. ArXiv, http//2299007000 2023. URL
https://api. www.seemaattttee seantoll wrt//orrpsssirne 2 61330655
Ali Elkahky, Wei-Ning Hsu, Paden Tomasello, Tu-Anh Nguyen, Robin Algayres, Yossi Adi, Jade
Copet, Emmanuel Dupoux, and Abdelrahman Mohamed. Do coarser units benefit cluster prediction- based speech pre-training? In ICASSP 2023 2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023. doi: 10. 10.10AA4SS449935..22223 10096788.
Tzeviya Sylvia Fuchs and Yedid Hoshen. Unsupervised word segmentation using temporal gradient pseudo-labels. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023. doi: 10.1 1A110/CASPP493372000000 10095363.
Yuan Gong, C Lai, Yu-An Chung, and James Glass. Ssast: Self-superviied audio spectro- gram transformer. ArXiv, abs/2110.097844, 2021. UBL_ittppsr/sall..eentttcsssos
org/CorpusID: :230002:::39992333
Kyle Gorman. master 2014.
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Défossez, Gabriel Synnaeve, Emmanuel Dupoux, Roy Schwartz, and Yossi Adi. Tex- tually pretrained speech language models. In Thirty-seventh Conference Neural Information Processing Systems, 2023. htLL/tttss///penrreee htwpeeerrreeeeeeeeeeeeeeeeeeeeee http://ereeeeeeeeeeeeeeeeeeoeeea Hue
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked
autoencoders are scalable vision learners. arXiv:2111.0633 2021.
Wei-Ning Hsu, Benjamin Bolte. Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 29:3451 1093351-3460, oct 2021. ISSN 2329-9290. doi: 1001109///LLP22213332222
URL httpt::t///att org/10
1109/TASIA 2021 3122291
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang. Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang. Zhizheng Wu, Tao Qin, Xiang- Yang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei Hc, Jinyu Li. and Sheng Zhao. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. ArXiv, 0bs/24433003000 2024. URL https://aag www.sseemanncnooommm wwwant wrtp/oororroooooo/ooo//ooaa I
J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko. G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673, 2020. https:// //ithbu com/ http://accooooeearreeeeaaeeaaaaa
Wei Kang, Xiaoyu Yang. Zengwei Yao, Fangjun Kuang, Yifan Yang. Liyong Guo, Long Lin, and Daniel Povey. Libriheavy: 50,000 hours asr corpus with punctuation casing and context, 2023.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D' D'efossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. ArXiv, abs/2209.15352, 2022. 2526687611
URL https: /tppi.ppp.a//// www.ntiisccheeeooeo org/CorpusID;
13

---

Published as a conference paper at ICLR 2025
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvqgan. ArXiv, abs/2300.005466 2023. URL https
org/
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte,
Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. On generative spoken language modeling from raw audio. Transactions of the Association for
Computational Linguistics, 9:1336-1354, 2021. doi: 10.1162/tac_ Bacpuul__0003330 00430. URL https
//aclanthology.o ort.:org///2// 2ttp://221......1.77.
Igor Malioutov and Regina Barzilay, Minimum cut model for spoken lecture segmentation. In Nicoletta Calzolari, Claire Cardie, and Pierre Isabelle (eds.), Proceedings of the 2 International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Compu- tational Linguistics, pp. 25-32, Sydney, Australia, July 2006. Association for Computational Lin- guistics. doi: 10.3115//220175. 11 1220179. URL httpssts:: / ac wcl.ahoooontoooo.ooooooooooo org/p06-1004
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi, In Proc. Interspeech 2017, pp. 498-502 502, 2017. doi: 10.21457/mneexppecch.211/ 2017//rspppeech2211888666
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper under- standing of commonsense stories. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 839-849, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/ 1098. URL https://aclanthonn http////aaatooloooor/////i 1098
Tu Nguyen, Benjamin Muller, Bokai Yu, Marta Ruiz Costa-jussà, Maha Elbayad, Sravya Popuri, Paul- Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Gabriel Synnaeve, Juan Pino, Benoit Sagot, and Emmanuel Dupoux. Spirit-Im: Interleaved spoken and written language model. ArXiv, abs/2402.05755, 2024. URL https: h //api. semanteeeiiiom wcwcho.ttccccco.oooo lar art.rr/Coopus.DD.. 267547793
Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. The zero resource speech benchmark 2021: Metries and baselines for unsupervised spoken language modeling, 2020.
NVIDIA, Péter Vingelmann, and Frank H.P. Fitzek. Cuda, release: 10.2.89, 2020. URL https 7/developer wwwiiii.ii......... com/ halmm//uar cuda-toolkis
Vassil Panayotov. Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ast corpus based on public domain audio books. In 2015 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015. doi: 10. 10.11000 119//CCASSP2221557788644
Ankita Pasad, Bowen Shi, and Karen Liveseu. Comparative layer-wise analysis of self-supervised speech models. pp. 1-5, 06 2023, doi: 10.1 D0.10000CASSPP445522002... 10096
Ankita Pasad. Chung-Ming Chien, Shane Settle, and Karen Livescu. What Do Self- Self-Supervised Speech Models Know About Words? Transactions of the Association for Computational Linguistics, 12: 372-391, 04 2024. ISSN 2307-387X. doi: 16.1162/tacl a_ 00656. URL 10. /2//t2/accc
6
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, pigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL
files/paper/
14

---

Published as a conference paper at ICLR 2025
Puyuan Peng and David Harwath. Word discovery in visually grounded, self-supervised speech models. In Interspeech, pp. 2823-2827, 09 2022. doi: 10.2145//mnexspecch22222 10652.
Puyuan Peng. Shang- Wen Li, Okko Räsänen, AbdeIrahman Mohamed, and David Harwath. Syllable segmentation and cross-lingual generalization in visually grounded, self-supervised speech model. In Interspeech, 2023.
Puyuan Peng, Yao (Bernie) Huang, Daniel Li, Abdelrahman Mohamed, and David F. Harwath. Voicecraft: Zero-shot speech editing and text-to-speech the wild. ArXiv, abs/2403. 16977733..6,,, 2024. URL SRRLhhtp::
Semant http:/pppp.smmmaiiccccccoco.oooe ar wttp:a/ro//oo/ooooooaooo 26868186633
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. Speech Resynthesis from Discrete Disentangled Self-Supervised Representations. In Proc. Interppecch 2021, 2021.
Okko Johannes Räsänen, Unto Kalervo Laine, and Toomas Altosaar. An improved speech seg- mentation quality measure: the r-value. In Interspeech 2009, pp. 1851-1854, 2009. doi:
10.21447/mterrpecch200-955
Feiyu Shen, Yiwei Guo, Chenpeng Du Xie Chen, and Kai Yu. Acoustic bpe for speech generation with discrete tokens. In ICASSP 2024 2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp- 11746-11750, 2024. 10446063.
doi: 10.1 N0.109//AASP444455.200
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888-905, 2000.
Yakun Song, Zhuo Chen, Xiaofei Wang, Ziyang Ma, and Xie Chen. Ella-v: Stable neural codec language modeling with alignment- guided sequence reordering, 2024.
segment-based Zheng-Hou Hua Tan, Achintya kr. Sarkar, and Najim Dehak. rvad: An unsupervised segment- based robust voice activity detection method. Computer Speech & Language, 59:1-21, 2020. ISSN 0885-2308. com/
201.:...00.00000666... 9300920 sci httpc///rtiiceep./////e//
doi: https:/doi.org//
URL fttt:tttt////w/ c scw.ecediiiiicccooco encedi rect rectt
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302. 13971, 2023. URL httpps://ap) semant. deemanticscho oscho ar org/ CorpusID:257219400
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz. Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.. 2017. URL ht htetts:://proceedosss.eeeeee www.neuuuuu..pppppeeepee files/ F11es//aperr/2177 2ile/3f5ee243547deee hww.:444c6.eeeeec0cccceccaaaaaa.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codee language models are zero-shot text to speech synthesizers, 2023.
Qizhe Xie, Minh-Thang I Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In 2020 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, 2020. doi: 10. 1001099CCVPP22200220000000
Donghhao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi- codec: Group-residual vector quantization for high fidelity audio codec. ArXiv, abs/2305 02765, 2023a. htt.h:tp/////
semant cschola ar Ltrg/Corpus. ID: 2584772::999
Dongchao Yang. Jinchuan Tian, Xu Tan, Rongjie Huang. Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, Zhou Zhao, Shinji Watanabe, and Helen Meng. Uniaudio: An audio foundation model toward universal audio generation, 2023b.
15

---

Published as a conference paper at ICLR 2025
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec, 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.
Opt: Open pre-trained transformer language models. ArXiv, abs/2205.0100 2022. URL https //api. semant icschnlarrcccc otg/CorrpssDD:24449223
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified
speech tokenizer for speech large language models, ArXiv, abs/2308. 16692, 2023. URL https
//api. www.mmnttiicoooioooo. wttp.///o/ooop.SSD2223333999999
16

---

Published as a conference paper at ICLR 2025
A APPENDIX / SUPPLEMENTAL MATERIAL
A.1 RANDOMLY SAMPLED EXAMPLE SEGMENTATIONS
We provide randomly sampled example segmentations from the LibriSpeech (Panayotov al. 2015 dev-clean set. All models are the second iteration of Data2Vec2, which we use for our SyllableLM experiments in Section [.6] Top: Feature self-Similarity matrix, darker green closer. Segmented cuts span vertically in blue from the top, ground truth boundaries span vertically in red at the bottom. Bottom: time-aligned Mel-Spectrogram. We call attention to the interesting behavior of global correspondences appearing when words or syllables are repeated. Best viewed zoomed in.
8.33Hz
6.25Hz
5.0Hz
17

---

Published as a conference paper at ICLR 2025
18

---

Published as a conference paper at ICLR 2025
A.2 DISCUSSION: OTHER BOOTSTRAPPING STRATEGIES
Of course, there already exist several strategies for unsupervised syllable and word segmentation such as in Fuchs Hoshen (2023), Pasad et al. (2024) and Peng et al. (2023) that could be used to bootstrap first pscudolabels. We have not conducted an exhaustive search over initializing with these strategies, however it is intriguing that the approach from |eng et al. (023) achieves significantly lower quality, relatively and absolutely, from SylBoost bootstrapping than LossPred. We suspect that this may be caused by the fact that although the representations of these models correlate with boundaries, there is no modeling in the pretraining loss pushing the representations linearly separate across semantic differences. Meanwhile, the loss is forced to change across semantic boundaries due to the difficulty of language modeling. albeit noisily.
A.3 DISCUSSION: BASE ENCODER
50Hz base encoder to match SD-HuBERT and have fine- fineigrained boundary control during syllable segmentation, we cannot use the 25hz encoder from TWIST. Unfortunately, this means that the quality of the base encoder may be confounding factor in our SpeechLM evaluation. We choose Vec22 SpeechLMs on syllable-like units because we find its quality enables lower bitrates than HuBERT, but it is older and trains on less-data than the TWIST tokenizer, and it has 6x fewer parameters than w2v-BERT, used by AM. We suspect that applying newer encoders like w2v-BERT' 2 from Barrault et al. (2023) could enable even better performance, which we leave to future work. Because the Data2Vec2 loss function doesn't 't natively work with LossPred, we initialize Data2Vec2 SylBoost from the same HuBERT loss boundaries discussed in 3.]
Because
(Baevski et al.) 2023) as middleground for training
A.4 HARDWARE AND HYPERPARAMETERS
We implement all experiments using NVIDIA A40 46GB GPUS with Intel Xeon Gold 6226R CPU @ 2.90GHz. Estimated speeds are made using these results as well as scaling from ZZhng et al. (2022).
Hyperparameters for pretraining our models are below. We note that the Batch Size is in terms of tokens, which means that higher unit rates will have fewer seconds of raw audio per batch to keep GPU compute roughly equal per model.
For LossPred, we use a HuBERT-large student and HuBERT-base teacher they are the only public checkpoints for a corresponding student and teacher model. We preprocess the audio input to LossPred and Feature Similarity (Peng et al. 2023) using unsupervised voice activity dection model (Tan et al. 2020).
Table 9: Speech pre-training hyper-parameters.
SyllableLM Base
SyllableLM Large
Layers Embed Dim
24 1024
768
MLP Dim
3072
4096
GPUs
2
4
Learning rate
2 x 10-4
2 x 10
Adam 31 / /3
0.9 0.98
Weight decay
0.01
0.01
Learning rate schedule
Linear Decay 0.1
Linear Decay
Dropout
0.1
LayerDrop
0.0
0.0
Warmup updates
8,000
16,000
Batch size (tokens)
80,000
80,000
Updates
200,000
200,000
Position Embeddings
Learned
Learned
19

---

Published as a conference paper at ICLR 2025
Table 10: Sylboost Prarameters.
HuBERT Base
Data2 Base
(One-Indexed)
11
Learning rate
5e-5
5e-5
Epochs
LibriSpeech Data
100 hours
100 hours
Batch Size
32
32
Iterations
2
2
K-Means clusters (before Agglom.)
16384
24576
A.5 ADDITIONAL EXPERIMENTS
Table 11: Relative change in the number of tokens under different audio speed modifications.
New Length Original Length
0.5x
0.6x
0.7x
0.8x
0.9x
1.11x
1.25x
1.43x
1.66x
2x
Tokenizer
Speedups
Slowdowns J
TWIST
0.60x
0.70x
0.79x
0.86x
0.93x
1.07x
1.13x
1.22x
1.36x
1.52x
SD-HuBERT 5.0H 4096 Units
0.70x
0.78x
0.84x
0.90x
0.95x
1.6xx
Lilx
121x
1.32x
1.46x
SylBoost 5.0Hz 4096 Units
0.34x
0.52x
0.75x
0.87x
0.94x
1.03x
1.0%x
1.19x
1.25x
1.33x
We evaluate the robustness of different tokenizations to different audio speeds in Table [1 Changing the speed audio does not change the content of the words spoken, so we would expect that tokenizer that captures information content at abstract level, like syllables words, would output roughly the same number of units when run on a sped-up or slowed-down version of an audio file. To measure this, evaluate the relative change in the number of tokens under speedups and slowdowns across all audio files in LibriSpeech dev-clean. We evaluate this for SylBoost, SD-HuBERT, and TWIST-Units. For fair comparison, with an average unit rate of 5.0Hz and 4096 clusters, as sequential deduplication is affected by the number of clusters.
evaluate SylBoost and SD-HuBERT under the same settings,
For speedups, we find that SylBoost performs comparably to SD-HuBERT at light speedups; however, the unit rate collapses under more extreme speedups. Anecdotally, we find that model resynthesis for SD- HuBERT and SylBoost under speedups of 0.7x or higher is largely gibberish, suggesting that these numbers should considered carefully. For slowdowns, we find that SylBoost is significantly more robust to changes in length.
We additionally run AB human opinion tests to evaluate the grammar, meaningfulness, and diversity of generated continuations, following the Human Meaningfulness-MO9 from Hassid al. 2023). We sample 50 random 10-second continuations from TWIST-1.3D and SyllableLM 6. 6.25Hz 300M, following the approach in Section 5.6 and have five raters evaluate each sample. We find that raters prefer SyllableLM responses 42.2% of the time, TWIST responses 26.6% of the time, and rate results equally meaningful 31.2% of the time, with 95% confidence interval of +5.9%.
A.6 SAMPLE CONTINUATIONS
Below are sample continuations generated with temperature sampling parameter chosen to best match Oracle VERT diversity scores. We provide continuations of roughly seconds of audio, sampled randomly from LibriSpeech test-clean. This text is given output by our HuBERT ASR Model from HHsu al. (202T), with transcription errors present and with no additional modifications. The source text bolded, and sometimes cuts off mid-word, which can behave differently per sample based on unit rate and quantization artifacts.
20

---

Published as a conference paper at ICLR 2025
Model
Continuation
TWIST 300M
HE FILLED FOR AND FOUND THE WIZARD'S BLACK CLOTH THE SQUIRE WAS QUITE DEVOTED TO CARRYING IT IN THE PARKS I COULD TELL YOU THAT HE RATHER LOVED ALL THE WIZARDS IN THE MATTER OF HONEST FIGUROUS TEMPERED MEN THEN IMMEDIATELY AND BECAUSE HE DID
TWIST 1.3B
HE FILLET FOR AND FOUND THE WIZARD'S BLACK CLOTH
THE SQUIRE WAS QUICK IN MOVEMENT A WHEEL OF HIS
JOURNEY MADE HIM THE IMPRESSION OF HARE'S BLOOD-
SHED THERE WAS NO HELP FOR IT HE SURMISED AND HE
DID SO
SyllablcLM 2k 8.33Hz 90M
HE FILLED FOR AND FOUND THE WIZARD'S BLACK
CLOTH THE SQUIRE WAS GLAD THAT HIS LAD HAD NOT
RETURNED TO KING LENO FOR PROBABLE THAT HE
WOULD HAVE BECOME SO UNWELL THAT HE COULD NOT HAVE SPOKEN
SyllabieLM 8k 6.25Hz 300M
HE FILT FOR AND FOUND THE WIZARD'S BLACK CLOTH
THE SQUIRE WAS HALF WAY DOWN THE PATH WITH THE WIZ-
ARD BUT HE STOPPED SHORT BEFORE THE GATE CLUTCHED HIS FISTS SEIZED HIS GOWN AND
Model
Continuation
TWIST 300M
WHEN THEY WERE OUTSIDE UM SIMPLY LATCHED THE DOOR AND STARTED UP THE STAIRS AND EVEN WENT INTO THE LITTLE HOUSE AROSE IN THE SETTING SUN SET FORTH THE PLACES WHERE PEOPLE LIVED FROM MON HONEY- SUCKLE HANNEY
TWIST L.3B
WHEN THEY WERE OUTSIDE UM SIMPLY LATCHED THE DOOR AND STARTED UP THE TURNPIKE OL HAT DON KILL ME THE SLING DE IN YORN ME WEAVING OUT CHARLEYS SENSE EXAMINED WHAT HE MADE EXAMINES
SyllableLM 2k 8.33Hz 90M
WHEN THEY WERE OUTSIDE UM SIMPLY LATCHED THE DOOR AND STARTED WALKING IN THEY WERE TOO OLD TO CARE MUCH ABOUT GOING HOME THEIR RELATIVES LEFT
SyllableLM 6.25Hz 300M
WHEN THEY WERE OUTSIDE UM SIMPLY LATCHED THE DOOR AND STARTED SLOWLY DOWN THE CORRIDOR AND MISSUS BAKER WALKED BESIDE THEODORA THEY WERE
NEAR THE OUTER DOOR WHERE
Model
Continuation
TWIST 300M
DO BE OR NOT TO BE THAT IS THE QUESTION WHETHER TIS NOBODY SIBL LINE IN OTHER SHIRTS OR CHOCOLATE NOS MICOTTON BUTTER WHAT WE WERE DO WE SEE THESE HITS WE'VE GOT THE GHOST HERE THEY'RE LOOKING
TWIST L.3B
DO BE OR NOT TO BE THAT IS THE QUESTION WHETHER TIS NO GOOD EITHER THAN TO GO THROUGH THE JUDG- MENT OF GAUL AND YOUR DOCTRINES THE LORD YOUR GOD AND YOUR GOSPEL IN RESPECT OF THE POWER OF THIS
SyllableLM 2k 8.33Hz 90M
DO BE OR NOT TO BE THAT IS THE QUESTION WHETHER
TIS NO OTHER THAN ESO'S OWN DESTINY YOU SEE IT LA MISTER PRIOR THAT THIS IS THE CASE
SyllableLM 8k 6.25Hz 300M
DO BE OR NOT TO BE THAT IS THE QUESTION WHETHER
TIS NO REGRET OR NO PLEASURE THAT MAY BE RUSHED
INTO ACTION AT ONCE WITH THE GREATEST EAGERNESS OF
IMPULSE AND ELASTICITY OF HEART
21

---

Published as a conference paper at ICLR 2025
Model TWIST 300M
Continuation
HE IS CALLED AS YOU KNOW THE APOSTLE OF THE INDIAN KING WHO IS SO GLORIOUS AND ACTING WHY IS THE OTHER PRINCE NOT BELIEVED BY HIM IN EVERY FAITH THAT IS FREE WILL EXCEPT WHEN HE
TWIST 1.3B
HE IS CALLED AS YOU KNOW THE APOSTLE OF THE INDIES SAW WHAT HAD PASSED THROUGH HIM LATER IN ANOTHER BOOK AMONG THOSE WHO HAD ENGRAVED IT THIS VOLUME MISTER PICKWICK THOUGHT RIGHT NOT TO INSULT YOU
SyllableLM 2k 8.33Hz 90M
HE IS CALLED YOU KNOW THE APOSTLE OF THE IN- VIDISIBLE BEFORE THEY RECEIVED THE GRACE OF GODAD CHRIST THEN HAD IN THE FAITH OF HIS SON
SyllableLM 8k 6.25Hz 300M
HE IS CALLED AS YOU KNOW THE APOSTLE OF THE INDIES HE IS THE FORERUNNER OF TEACHING AND FAR BEYOND IT HE THE EXACT SCIENTIST WHO MEASURES
THE MOVE
22


Reviews:
[{"review_id": "fuCdAHOkM5", "reviewer": "Reviewer_HE4J", "summary": "This paper proposes a method to extract speech units from existing pre-trained models such as HuBERT or Data2vec2 at syllable boundaries. They show that they are able to significantly lower the number of units used to encode a particular utterance while maintaining and improving on the quality of re-synthesized speech, or continuation of the speech if used with a language model trained on these new speech units. The method is relatively straight forward and, if it works, allows significant savings in the number of tokens needed to accurately model speech data.", "questions": "see weaknesses section for suggestions on what to add to the paper", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the paper describes the method in sufficient detail to reproduce it\n- this appears to be a novel way (although based on much prior work) to discover unit boundaries in speech data, which turn out to be syllables.\n- the results are compelling in light of much concurrent work on speech-unit language models, especially as they allow to significantly reduce the cost of training such models due to higher compression ratio", "weaknesses": "- while using speech units from HuBERT and other models has been widely explored, one downside of such units is that they tend to mostly capture only the semantic content. Presumably, TWIST units may capture some of the prosodic information, but it is not clear how the authors measure the prosody and expressiveness of this method. An alternative for generative tasks is to simply model speech as text (e.g. via an ASR model) and add additional style vectors. Speech understanding tasks may simply use the pre-trained models which this work bootstraps on directly. It would be very interesting to see some quantitative and qualitative comparisons on this topic.\n- \"Language models require tokenized inputs\" - this is not strictly true depending on how one defines \"tokens\". While discrete tokens are a popular choice, one can certainly build a language model that uses continuous inputs (\"tokenized\" by splitting it into distinct representations on the time axis or through methods like what is described in this paper) with discrete or continuous targets on the output.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hjBcaVtfRf", "reviewer": "Reviewer_nVDU", "summary": "The authors propose a novel method to extract rough syllabic segment boundaries utilizing loss patterns in HuBERT induced by different input mask spans. With those rough segments, a learning method, SylBoost, is proposed to learn a segmented representation by fine-tuning pretrained SSL models. By controlling the granularity of the initial segments, the authors train multiple models with different frequency of speech segments, ranging from phoneme to word level. Syllables are more accurately and efficiently detected than the previous model, SD-HuBERT. By training LM on those syllable-like units from clustering, the authors suggest SyllableLM and demonstrate that spoken LMs at syllabic granularity can outperform or on par with previous SOTA spoken LMs using smaller model and training data.", "questions": "In the intro, what is evidence for “language models on spoken language still struggle to output semantically meaningful speech.”? \n\nIn method, “We apply the SylBoost loss (Equation 4) to model features at layer L, which we select based on syllabic correlation as explored in detail in Pasad et al. (2024).” The cited paper is about words not syllables, where I couldn’t locate a particular mention of syllabic correlation. Please let me know in which part of the reference paper demonstrates it.\n\n1. Provide motivations/reasons for some design choices: Why 75% of timesteps in Section 5.2? Why 8.33, 6.25, and 5.00Hz?\n\n2. On the statement \"training a resynthesis model provides a stronger description of the semantic information contained in units than purity metrics”, please provide the rationale for this. It is counter intuitive since resynthesis will be better if the input embedding is closer to low-level, like acoustics. For example, VQ-VAE based models like EnCodec show great performance in resynthesis but are claimed to lack higher-level information like semantics.\n\n3. Typo in Table 2: Boundary detaction —> detection.\n\n4. In Table 6, sBLIMP is designed for evaluating syntax (or grammar), which should not be categorized under “semantics”. Also, sWUGGY-all and tSC are the best in Moshi but bold annotations are erroneously given. Please fix them if the numbers are correct.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "LossPred and Sylboost are well-motivated, novel ways to learn speech representation beyond phonemes, allowing different model training with different frequency units. The method is meticulously explained. Authors depict real speed up in using these units compared to TWIST. Furthermore, results shown achieve near SOTA results on syllable boundary detection and discovery. Table 5 well ablates their design choices and results shown in Table 6 show clear improvements over existing baselines. In particular, the gain in length of tokens and bitrate is outstanding compared to previous speech unit LMs, which is an important factor in LM.", "weaknesses": "There are several concerns regarding some claims/analysis, as detailed below.\n\n1. Lack of verification of syllabic correspondence of higher frequency models: The higher frequency models (6.25 HZ or 8.33 Hz) seem sub-syllabic or representing di- or tri-phones based on Figure 2 and Table 3. For example, there is a huge gain in phoneme boundaries correspondence (F1-P) by changing from 5Hz to 6.25Hz. Though, F1-S is still higher than F1-P, but this would be due to the usage of monophones in evaluation. This evokes a question whether the 6.25Hz GSLM – the best one beating SOTA, is a “syllable” LM. The 5Hz version seems the most syllable-like (and the only evaluated frequency in Table 1), but it is the worst among different frequency versions according to Table 5 and 6, showing high WER and no outstanding performance in LM.  Detailed analysis of the linguistic characteristics of the units at different frequencies, perhaps including examples and a discussion of how they relate to syllables, phonemes, or other linguistic units will make this clear.\n\n2. Overclaiming controllability: The controllability of the segment frequency is given during training not at inference. However, the statement in the abstract, ”Our method produces controllable-rate semantic units” is misleading as it strongly implies that the controllability is given at the inference time. It requires rephrasing or a significant improvement in clarity. Besides, the authors claim a lack of controllability of previous studies, but the mincut approach cited by the authors has knobs that can control the frequency of resulting segments. The previous method may have a controllability though the prior works didn’t leverage the functionality. If true, the claimed novelty on controllability should be reassessed, may be provide additional experiments as evidence; and the manuscript must be accordingly updated.\n\n3. Adaptability on different speaking rate at inference: The audiobook datasets have relatively consistent speaking rate as the speech is very carefully delivered. However, in the real world, people may use very different speaking rates based on contexts or based on accents or dialects. Can a model work on speech with a different speaking rate? For example, if we put speech elongated by two times, would the model output the same number of syllables or doubled? The authors explain that the k is determined by optimizing timestep losses but not clear how this loss is calculated at the inference time. \n\n4. Lack of validation of semantic unit discovery: As the authors are citing Choi et al. (2024), I assume semantics here means concepts, e.g., “dog” is similar to “puppy” though they are phonetically distant. However, syllables by definition are not conveying such conceptual meaning, rather structured phonetic information. While the authors are frequently claiming semantic unit discovery, the paper has no supporting evidence that the discovered units or embeddings are semantics. The low error rates in resynthesis cannot verify whether the feature encodes semantic information. The approach suggested by Choi et al. (2024) may be useful to verify the claims.\n\n5. Lack of valid comparison/evaluation of LossPred: The proposed method for initial boundary extraction, LossPred, is novel and makes a lot of sense. Table 2 shows comparisons using Feat-Sim, but why not VG-HuBERT or SD-HuBERT? Those prior works already showed that the Feat-Sim on the original HuBERT is not accurate in terms of syllabic segmentation. As Feat-Sim works best with VG-HuBERT or SD-HuBERT, a fair comparison should involve Feat-Sim on them, not on HuBERT.\n\n6. Absence of audio quality evaluation of generated speech: The paper severely lacks evaluation of the quality of generated audio which is a very crucial aspect of generative model or codec language model of speech. Though it is not claimed to be the main focus, some quality metrics like Mean Opinion Score (MOS) should be reported.  The WER/CER metrics are not sufficient to show the audio quality.\n\n7. Erroneous calculation of RTF: As defined by sources like https://www.openvoice-tech.net/index.php/Real-time-factor and https://link.springer.com/article/10.1007/s11042-020-10073-7, real time factor is defined as the ratio between the time to process an input over the duration of the input. However, it seems like here, the authors chose to report the reciprocal, which is inadequate. The scores should be corrected to comply with the RTF definition. Also, the RTF comparison over TWIST should be calculated end-to-end, not from cached speech units. This is because RTF is aimed to measure efficiency in “real-world” scenarios, so the calculation should start from raw waveform, simulating real-world inference as much as possible. Moreover, this correction is critical as SyllableLM has an additional layer of segmentation in comparison to TWIST, which is not reflected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OlIVivb8lj", "reviewer": "Reviewer_7QjU", "summary": "This paper suggests a discrete unit extraction method for speech, where each unit has a variable length, unlike most discrete audio units with a fixed rate (often 25/50 Hz).\nTo segment variable length units, they use a pre-trained self-supervised audio model trained on masked language modeling.\nThey first run the SSL model on the raw sequence to obtain the GT discrete units (teacher predictions).\nThey then mask a fixed-sized segment and use the SSL model again to measure the likelihood of every teacher token. They repeat this over sliding windows throughout the input, resulting in a square asymmetric “similarity” matrix.\nAs in previous work, they apply a min-cut algorithm to segment the audio into variable-length units.\nThey then train a neural model for feature matching, which leverages the produced boundaries in its loss function (average feature teacher feature across the segment). \nThis is done to make the unit extraction more efficient (resolve the computational-heavy sliding windows) and hopefully generalize better.\nTo finally derive discrete units, they extract features from the boosted neural model and use a more efficient cut algorithm to segment the sequence. \nThey then calculate the mean feature over each segment, and cluster using K-means and agglomerative clustering to discretize the units. \nUsing their novel variable-length discrete units, they train speechLMs and see benefits in several metrics (e.g. sWUGGY, sBLIMP, and tSC).\n\nEdit: I increased my score from 5 to 6 (weak-accept) and the contribution from 2 to 3 (good).", "questions": "Comments:\n- It would be interesting to develop a joint algorithm for segmenting and clustering together. \n- Table 5: When training your tokenizer on smaller datasets with a single language and low input variability, it is easier to compress them and reduce the bitrate, as you have less information to distill into the discrete units.  \n\nWriting/Typos:\nLines 14-17 (abstract) can be simpler and more concise\n\nLine 172: Can you clarify: “C models the raw probabilities of the losses”\n\nLine 158: “Two frozen Hubert models” - you used the same single model, right? (otherwise, their token space wouldn’t match..) \n\nLine 280: TWIST citet->citep\n\nTable 2: Detaction->Detection\n\nline 462: Regarding “the E2E GSLM pipeline is deep” — how about rephrasing it to “consists of several independently trained components”?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem of extracting variable-length representations for audio is important, as the amount of information within different audio segments changes. \n- The approach reduces the token rate by x2-3 and SpeechLMs trained on these units obtain better results compared to the fixed-rate VQ/K-means discrete unit extraction.\n- The idea of segmenting the audio before clustering is interesting, and seems to be quite efficient.", "weaknesses": "Weaknesses:\n- The approach for variable-length unit extraction is quite complicated, with a wide range of hyperparameters to control (span size, number of segments, different ways to combine two clustering algorithms, etc). \nI’m a bit concerned that this method is overfitted to boost the specific metrics evaluated.  I find the core idea to be grouping before clustering - would you agree? \nA simpler algorithm can be: (1) compute a similarity matrix with SSL embeddings (maybe K-band matrix as you did to reduce complexity) (2) segment the audio using the similarity matrix (3) run K-means on the average representation per segment. (no training, no sliding windows)  .\n\n- The metrics calculated to evaluate speechLMs mostly include likelihood ranking and perplexity calculations. However, generative models are mostly used to generate content. \nA sample page with audio continuations would help qualitatively check the differences in the generative capabilities of the models. ([1] may be useful to quantify those generative capabilities).  \nI admit that likelihood ranking is the common practice for evaluating speechLMs, but IMO they don’t measure the thing we’re interested in (I’ll use the model mainly to generate, not to measure likelihood), so evaluating generated outputs is also important and should be added.  \n\n- One reason to have variable-rate units is to dynamically distinguish between high-info segments and low-info segments. \nIn your approach, the number of semantic regions is proportional to the length. In that case, 10 seconds of silence and 10 seconds of high-speed speech would result in the same number of tokens. \nThe units have a variable length, but each sample has a fixed sample rate. With BPE, in contrast, we might get fewer tokens for 10 seconds of silence compared to 10 seconds of fast speech.  \n\n- Table 6: What is the reason for the gray shade on several TWIST? In several metrics, the bolded result is not the best (in contrast to what is stated in the caption). I assume you want to state it is unfair to compare 7/13B models with 300M models (and I agree). In that case, it should be clear from the caption. \n\n- The tokenizer is trained on 960 hours of audio, which got me concerned about scalability in the unit extraction process. Can your algorithm scale so we can compute the units based on (e.g.) 55K hours of data? Can you elaborate on the sample complexity of the process (w.r.t. the length of each sample and the number of samples) \n\n- Table 6: The results suggest BPE (an alternative approach to producing a variable-length representation) degrades the results. This contradicts the findings of other several papers reporting benefits from BPE (additional references attached): \n[1] Exploring the Benefits of Tokenization of Discrete Acoustic Units (Interspeech 2024) \n[2] On the Effectiveness of Acoustic BPE in Decoder-Only TTS (Interspeech 2024) \n\n- Section 5.2: The explanations on the final unit extraction using clustering are vague. How did you combine K-means and Agglomerative clustering? \n\n- I find the task of unsupervised syllable segmentation is more suitable for speech-oriented venues such as ICASSP and Interspeech.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NTFQqvcBxU", "reviewer": "Reviewer_p1N2", "summary": "This work introduces a controllable self-supervised technique to merge speech representations into coarser syllable-like units while still preserving semantic information. The propsed method produces controllable-rate semantic units at as low as 5Hz and 60bps and\nachieves SotA in syllabic segmentation and clustering. Using these coarse tokens, authors successfully train SyllableLM, a Speech Language Model that matches or outperforms current SotA SpeechLMs on a range of spoken language modeling tasks.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "Overall, I believe this is a valuable work, featuring a clear motivation, good technical contributions, and impressive experimental results. The use of semantic units as low as 5Hz undoubtedly holds significant value for training SpeechLLMs. Additionally, with under 90 hours of training, SyllableLM outperforms both the 13B parameter TWIST and the 7B Moshi on sBLIMP, indicating that this paper contributes meaningfully to both the speech and multimodal research communities.", "weaknesses": "The reason I gave a \"boadline rej\" is that I still have several concerns due to the somewhat unclear presentation of this paper. If my main questions can be adequately addressed during the rebuttal phase, I would be willing to raise my score.\n\n1) If I understand correctly, SylBoost tokens are extracted from the waveform and include not only a sequence of tokens but also the start and end timestamps for each token. However, it’s unclear how SpeechLLM decodes these tokens when performing speech continuation, particularly in terms of predicting timestamps. Specifically, how exactly are timestamps handled during decoding? Is there a step-by-step process the authors could provide to clarify this? Additionally, the frequent mention of TWIST in Section 4.2 is somewhat confusing. My personal suggestion would be to focus more on the proposed method itself in the methods section, and simply add appropriate citations for references to other works where relevant.\n\n2) I noticed that the authors used a cold-init approach for the 90M model. What are the potential advantages and disadvantages of using a larger text pre-trained model compared to the cold-init approach? How might this affect performance on specific downstream tasks?\n\n3) In addition to semantic information, can SylBoost tokens also model acoustic information? Furthermore, regarding SyllableLM, what specific modifications to SyllableLM might be needed to improve its zero-shot TTS voice cloning capabilities?\n\n\nQuestion (3) is intended solely for discussion and does not require corresponding experiments to demonstrate these capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)