SyllableLM: Learning Coarse Semantic Units for Speech Language Models

## Abstract

Abstract Language models require tokenized inputs. However, tokenization strategies for continuous data like audio and vision are often based on simple heuristics such as fixed sized convolutions or discrete clustering, which do not necessarily align with the semantic structure of the data. For speech in particular, the high resolution of waveforms (16,000 samples/second or more) presents a significant challenge as speech-based language models have had to use several times more tokens per word than text-based language models. In this work, we introduce a controllable self-supervised technique to merge speech representations into coarser syllable-like units while still preserving semantic information. We do this by 1) extracting noisy boundaries through analyzing correlations in pretrained encoder losses and 2) iteratively improving model representations with a novel distillation technique. Our method produces controllable-rate semantic units at as low as 5Hz and 60bps and achieves SotA in syllabic segmentation and clustering. Using these coarse tokens, we successfully train SyllableLM, a Speech Language Model (SpeechLM) that matches or outperforms current SotA SpeechLMs on a range of spoken language modeling tasks. SyllableLM also achieves significant improvements in efficiency with a 30x reduction in training compute and a 4x wall-clock inference speedup.

## 1 Introduction

Learning to generate speech solely from listening to spoken language is a fundamental task in speech processing. It requires abstracting beyond the underlying acoustics of speech into phones, syllables, words, and sentences to process correlations across long ranges of time. But while current textual language models (Touvron et al., ; Zhang et al., ; Brown et al., ) can compose highly realistic text, language models on spoken language still struggle to output semantically meaningful speech. An increasing focus has coalesced around Generative Spoken Language Modeling (GSLM) (Lakhotia et al., ), which sets out to achieve this goal.

The most successful approaches to GSLM are Transformer Decoder Language Models (Vaswani et al., ) such as AudioLM (Borsos et al., ) and TWIST (Hassid et al., ). These Speech Language Models (SpeechLMs) operate on discrete tokens output by self-supervised learning (SSL) based encoder models (Hsu et al., ; Chung et al., ). However, existing SSL tokenizations predominantly capture phonetic information (Choi et al., ) at a high temporal resolution of 25-50 tokens per second, much greater than the typical human speaking rate of 2-5 words per second. This large number of tokens substantially impairs both training and inference speed (Hassid et al., ), and it is unclear whether modeling speech with such a high granularity harms semantic understanding.

Very recently, there has been significant progress in extracting coarser speech unit representations from raw audio. In particular, SD-HuBERT (Cho et al., ) finetunes HuBERT (Hsu et al., ) with a DINO-like distillation objective, and VG-HuBERT (Peng & Harwath, ; Peng et al., ) uses a contrastive loss against cross-modal visual inputs. We continue and significantly improve upon this line of research, resulting in high quality speech tokens (units) suitable for SpeechLMs that exhibit a temporal resolution roughly corresponding to syllables. Specifically, we demonstrate significant improvements in textual reconstruction from low-bitrate units of SSL models, reducing the word-error-rate (WER) from 37% using SD-HuBERT units to 7%, and more than halving realized bitrate of previous SpeechLM units from 175bps to as low as 60bps. We additionally find that our units correlate strongly with syllables both in boundary detection and in cluster quality.

Furthermore, we evaluate the effects of training SpeechLMs on these new units and obtain state-of-the-art results across a wide-variety of metrics, competitive with or outperforming AudioLM (350M parameters), all TWIST model sizes (125M-13B parameters), and the newly released Moshi-7B (Défossez et al., ) with significantly fewer parameters, lower training time, and faster inference speed. Our code and checkpoints are available at [https://www.github.com/alanbaade/SyllableLM](https://www.github.com/alanbaade/SyllableLM). Our contributions are as follows:

- 1.
We propose an unsupervised algorithm named LossPred that reveals noisy syllable-like segmentation of unannotated speech signals by analyzing the loss of a pretrained self-supervised model (e.g. HuBERT) across different masking spans.
- 2.
We propose a method named SylBoost that sharpens the feature space of self-supervised speech models by bootstrapping from LossPred syllable-like segmentations. SylBoost achieves SotA unsupervised syllabic segmentation, categorization, and low-bitrate unit-to-audio resynthesis.
- 3.
Using quantized SylBoost units as a basis for tokenization, we train SyllableLM, a SpeechLM that outperforms or matches prior approaches on a range of tasks while being 30x faster to train, 4x faster for inference, and having a 2.5x reduction in unit bitrate.

## 2 Related Work

Figure: Figure 1: Left-Top: The loss prediction matrix $C$, where brighter is higher likelihood placed on the teacher label. A time-aligned transcript is on the bottom, and predicted cluster unit boundaries span vertically as dashed-lines. Left-Bottom: A Mel-Spectrogram of the input waveform with an example masked timespan in gray. The losses on tokens at timesteps covered by the solid blue and dotted red spans are mapped to their corresponding rows and columns in $C$ as described in Section 3.1. Right: Visual of SylBoost. We train a student to match intermediate teacher features pooled over regions generated by pseudo-syllable-boundaries. We use a min-cut algorithm on the feature self-similarity matrix to extract boundaries, and then apply K-Means and Agglomerative clustering to obtain discrete units.
Refer to caption: x1.png

#### Self-Supervised Encoder Models

There has been a great amount of work in learning high-level representations from data by reconstructing corrupted inputs across speech (Baevski et al., ; Hsu et al., ; Baevski et al., ), audio (Gong et al., ), text (Devlin et al., ; Clark et al., ), and vision (Caron et al., ; He et al., ). To navigate the lack of simple discrete targets in speech, much work has been placed in finding high-quality targets, such as iterative clustering (Hsu et al., ) and by predicting the representations of a teacher network based on a running average of student model weights (Baevski et al., ; ). An alternate but similar line of work has been placed into learning low-bitrate units for the task of resynthesis (Défossez et al., ; Zeghidour et al., ; Yang et al., ; Kumar et al., ; Zhang et al., ; Du et al., ), which include losses focused on reconstruction and use an information bottleneck to enforce compression.

#### Applications of Neural Codecs

The discrete units generated by these self-supervised encoders are versatile and fundamental to much of the recent progress in speech research such as Text-To-Speech (Wang et al., ; Ju et al., ; Song et al., ; Peng et al., ), joint audio-text foundation models (Yang et al., ; Chou et al., ; Nguyen et al., ), unsupervised speech recognition (Baevski et al., ), discrete unit resynthesis (Polyak et al., ; Défossez et al., ; Zeghidour et al., ), text-to-audio (Kreuk et al., ; Agostinelli et al., ; Copet et al., ), and generative spoken language modeling (Borsos et al., ; Hassid et al., ; Lakhotia et al., ). Each of these methods operates on audio units exclusively greater than or equal to 25Hz, which has been a frequently cited area for future work to improve on (Hassid et al., ). Recent work (Elkahky et al., ) has also explored training speech encoder models with coarser units as targets.

#### Extracting Semantic Units from Raw Data

Additionally relevant to our work are several approaches, particularly in vision and audio, that generate emergent semantic clusterings from self-supervised transformer (Vaswani et al., ) models. In particular, the DINO approach in Caron et al. () observes object representations in attention maps through student-teacher distillation. Similar techniques have been also applied to audio to discover emergent syllable boundaries (Cho et al., ; Peng et al., ). These behaviors can vary heavily with small changes in pretraining strategy as explored in Darcet et al. (). Merging similar features has also been shown to produce significant vision model speedups such as in Bolya et al. (). Most similar to our work, Algayres et al. () extracted coarse continuous representations for GSLM, however these results trail behind language modeling approaches.

## 3 Learning Self-Supervised, Syllable-Like Representations from Raw Speech

In this section, we describe the bootstrapping process by which we extract low-bitrate speech units. We first describe LossPred, our algorithm to analyze outputs of self-supervised speech model loss functions to generate initial syllable-like unit boundaries. Following this, we define SylBoost, a procedure to iteratively refine these boundaries with student-teacher distillation. We also propose a new algorithm for the efficient extraction of boundaries from feature self-similarity matrices to fix the bottleneck slowing down VG-HuBERT and SD-HuBERT extraction.

### 3.1 LossPred: Extracting Syllable-like Segmentation from Relations in HuBERT’s Loss

HuBERT has previously been shown to learn phone-like units with its K-Means clusterings (Hsu et al., ) which have formed the basis of subsequent works on GSLM and unsupervised ASR (Baevski et al., ; Lakhotia et al., ; Hassid et al., ). However, other work (Pasad et al., ; ) has shown that the representations learned by these models also correlate with higher level structure such as words, despite these structures not immediately appearing during clustering. Our goal in this section is to propose a method that can be applied to a pretrained HuBERT model in order to automatically extract unit boundaries at the level of syllables or words rather than phones. Although we apply our method to HuBERT, we expect that it could also be applied to other SSL speech models that utilize a similar loss function such as WavLM (Chen et al., ) or wav2vec2.0 (Baevski et al., ). The crucial commonality between these models is that they all utilize a masked language modeling (MLM) training objective, whereby input speech tokens are randomly masked and the model is trained to predict the masked inputs conditioned on the unmasked inputs.

We ground our intuition for LossPred with the following thought experiment: if the input tokens corresponding to an entire word were replaced with mask tokens, we would expect the HuBERT model loss at these timesteps to be relatively high, as HuBERT would have to jointly predict word identity and the underlying acoustics to predict the missing span. On the other hand, if only the latter portion of a word were masked out, infilling this masked region given the word prefix may be easier by comparison. With this, if we iteratively shift a contiguous mask over a span of tokens and look at the loss, we would suspect to see a strong decrease in the loss throughout the timesteps corresponding to a masked semantic unit (word, syllable, or otherwise) as the beginning or end of the unit was partially revealed to the model.

LossPred operates using two frozen off-the-shelf HuBERT models, a student model and a teacher model, where the student model has been optimized to predict the quantized representations of the teacher model at layer $L$, and the teacher model comes from the prior iteration of pretraining as described in Hsu et al. ().
Formally, given an input waveform $W$, we extract the teacher labels by passing $W$ into the teacher model and then quantizing the representations at layer $L$ with K-Means. We denote these teacher labels as $Y_{\{1\ldots T\}}$, where $T$ is the number of timesteps. As done during HuBERT pretraining, we give the student model a corrupted version of $W$ where outputs from student CNN feature extraction at select timesteps are replaced with the HuBERT ‘mask’ embedding. We denote these student CNN features as $X^{M}_{\{1\ldots T\}}$ where $M=\{t_{1},\ldots t_{m}\}$ is a set of masked out timesteps. Following the pretext task, the student then predicts the teacher labels at masked timesteps $Y_{t},t\in M$ conditioned on $X^{M}$ and is evaluated using a cross-entropy loss:

$$ ${\mathcal{L}_{\text{HuBERT}}}\coloneq-\log p(Y_{t}\mid X^{M})$ (1) $$

We define a loss prediction matrix $C$, shown in Figure 1, which looks at the losses of the student model as we move a contiguous sliding window of $s$ timesteps to mask. Specifically, $C$ models the raw probabilities of the losses where the column is the timestep being predicted and the row represents the closest unmasked token.

$$ $C_{r,c}\in\mathbb{R}^{T\times T}_{+}=\begin{cases}p(Y_{c}\mid X^{M})\mid M=\{r +1,r+2,\ldots r+s\}\cap[1,T]&\text{if }r<c,|r-c|\leq\lfloor\frac{s}{2}\rfloor, \\ p(Y_{c}\mid X^{M})\mid M=\{r-1,r-2,\ldots r-s\}\cap[1,T]&\text{if }r>c,|r-c| \leq\lfloor\frac{s}{2}\rfloor,\\ 0&\text{otherwise.}\end{cases}$ (2) $$

We separately calculate the upper and lower triangles of $C$, relating to the closest observed waveform being before the mask and after the mask respectively. For simplicity, we assume here that the losses in the first half of the mask span are uncorrelated with information after the mask and vice versa.
In the upper triangle, each entry $C_{\smash{r,c}}$ at row $r$ column $c$ is equal to $p(Y_{\smash{c}}\mid X^{\smash{M}})$ given that the mask span in $X^{\smash{M}}$ starts just after time $r$. Inversely, in the lower triangle, $C_{r,c}$ is equal to $p(Y_{c}\mid X^{M})$ given that the mask span ends just before time $r$. We use a span size $s=50$ corresponding to 1 second as this duration is long enough to mask the majority of spoken words. We calculate the upper triangle using the first 25 timesteps of the mask span, and the lower triangle using the last 25.

To extract $k$ semantic regions with boundaries $B=\{b_{1}=1<b_{2}<\ldots<b_{k+1}=T+1\}$ from $C$, we use the min-cut algorithm from Shi & Malik () discussed further in Peng et al. (), treating $C$ as the input feature-similarity matrix:

$$ $B\coloneq\operatorname*{arg\,max}_{\{b_{1}=1<b_{2}\ldots<b_{k+1}=T+1\}}{{\sum} }\limits_{t=1}^{k}\frac{\sum\limits_{i,j=b_{t}}^{b_{t+1}-1}{C_{i,j}}}{\sum \limits_{i=b_{t}}^{b_{t+1}-1}\sum\limits_{j=1}^{T}(C_{i,j}+C_{j,i})-\sum \limits_{i,j=b_{t}}^{b_{t+1}-1}{C_{i,j}}}$ (3) $$

By choosing $k$ to be proportional to the length of the utterance, we can control the sample rate of our boundaries. We explore modifying this parameter in-depth throughout our experiments. We find that semantic units extracted by LossPred tend to be syllable-like (both via inspection, and also confirmed experimentally in our segmentation and clustering experiments) and so we focus on syllable-resolution units for the rest of the paper.

LossPred is expensive to run due to using one forward pass for each sliding window location. To make this efficient, we extract multiple masked spans simultaneously with a gap between spans of three seconds. This results in roughly 200 forward passes of the student model to calculate $C$ on an arbitrarily-sized audio. Fortunately, we only ever need to run LossPred once to initialize SylBoost training (described in 2), so this does not affect later semantic unit extraction speed.

### 3.2 SylBoost: Bootstrapping Pesudo-Syllabic Units with Iterative Distillation

Figure: Figure 2: Qualitative results on SylBoost controllability for boundary detection. We plot the feature similarity matrix $A$, described in 2 for HuBERT, Data2Vec2, and SylBoost on Data2Vec2 when trained at different unit rates. The number of cuts $k$ is selected dynamically as described in 5.2.
Refer to caption: x2.png

Given the initial boundaries predicted by LossPred, we follow the paradigm of noisy-student-teacher learning (Xie et al., ) to extract semantic representations. Our goal is to “sharpen” the syllabic organization in the feature space of an input student model, as seen in Figure 2. We choose a pretrained HuBERT (Hsu et al., ) or Data2Vec2 (Baevski et al., ) to initialize our SylBoost student and teacher models, with the teacher model parameters held constant.

The SylBoost procedure is depicted in Figure 1. For a set of hypothesized speech segment boundaries $B=\{b_{1}=1<b_{2}<\ldots<b_{k+1}=T+1\}$, we group together all temporal tokens between two boundaries into disjoint groups. $G_{i}=\{t\mid b_{i}\leq t<b_{i+1}\}$. For notation, we let $H_{t}$ be the index of the group that contains $t$: $t\in G_{H_{t}}$. We apply the SylBoost loss (Equation 4) to model features at layer $L$, which we select based on syllabic correlation as explored in detail in Pasad et al. (). This results in student features $X_{\{1\ldots T\}}^{\smash{(L)}}$ and teacher features $Y_{\{1\ldots T\}}^{\smash{(L)}}$. The loss is then the mean squared error between the student features $X_{t}^{\smash{(L)}}$ at each timestep and the mean of the teacher features in the timestep’s corresponding group:

$$ ${\mathcal{L}_{\text{SylBoost}}}\coloneq\frac{1}{T}{{\sum}}\limits_{t=1}^{T} \left(X_{t}^{(L)}-\frac{1}{|G_{H_{i}}|}{\sum}\limits_{s\in G_{H_{i}}}Y_{s}^{(L )}\right)^{2}$ (4) $$

After training, SylBoost results in a student model with sharpened latent features $X_{t}^{\smash{(L)}}$ from which we calculate a matrix $A$ of pairwise frame distances where element $A_{i,j}$ equals the $L_{2}$ distance between the features $X_{i}^{\smash{(L)}}$ and $X_{j}^{\smash{(L)}}$ as depicted in Figure 2. We then extract boundaries from $A$ using a cut algorithm described in the next section (Sec. 3.3). With this, we can generate new pseudolabel boundaries and run SylBoost again to extract even better boundaries, which we perform twice. Finally, we extract discrete clusters by mean pooling the features at layer $L$ over each SylBoost boundary group followed by K-Means and Agglomerative Clustering to a desired number of units.

### 3.3 Efficient Extraction of Unit Boundaries for SylBoost

To extract boundary indices from learned feature representations Peng et al. () proposed adapting the mincut approach in Malioutov & Barzilay (). However, for speech this approach is slow in practice and difficult to parallelize, bottlenecking our ability to extract boundaries in bulk across the large corpora necessary for downstream language modeling. Inspired by the SylBoost objective, we propose a more efficient approach for extraction: given $k+1$ potential boundaries, we seek to choose groups that minimize the sum of the distances from each unit to the mean of its assigned group:

$$ $B\coloneq\operatorname*{arg\,min}_{\{b_{1}=1<b_{2}\ldots<b_{k+1}=T+1\}}{{\sum \limits_{i=1}^{k}\sum\limits_{j=b_{i}}^{b_{i+1}-1}}}\left(X_{j}^{(L)}-\frac{1} {b_{i+1}-b_{i}}\sum\limits_{l=b_{i}}^{b_{i+1}-1}X_{l}^{(L)}\right)^{2}$ (5) $$

We further restrict the setting by choosing a maximum group length of $G$ tokens, where we choose $G=50$ to correspond to one second of tokens, as syllables or words longer than this are fairly rare. With this, we can then split our algorithm into 1) calculating a distance array $D\in\mathbb{R}^{T\times G}$, where $D_{t,g}$ is the cost of the group of length $g$ ending at token $t$ and then 2) solving the minimal interval cover from this distance array with dynamic programming. An efficient implementation using PyTorch (Paszke et al., ) on CUDA (NVIDIA et al., ) runs in $O(k)$ data-aware sequential steps.

## 4 SyllableLM: Speech Language Modeling on Coarse Units

GSLM (Lakhotia et al., ) defines a pipeline for modeling raw audio as three stages: 1) Audio-to-unit Tokenization, 2) Running a language model on these units, and 3) Decoding the tokens back into a waveform.
For Audio-to-unit Tokenization, we use the second iteration of SylBoost as described in Section 3. In this section, we lay out our language modeling and resynthesis.

### 4.1 Language Model

Like AudioLM and TWIST, we use an autoregressive transformer decoder language model to approximate $p(x_{t}\mid x_{t-1},\ldots,x_{1})$ given an input token sequence $x_{1},\dots,x_{T}$. We refer to this language model trained on the discrete clusters output by SylBoost as the main SpeechLM. Like TWIST, we prepend a <BOS> token and make no other special changes.

### 4.2 Token to Speech Decoding

To convert SylBoost tokens back into audio, we adopt the interleaved decoding strategy from Song et al. () to output the units from TWIST Hassid et al. (), obtaining a waveform by cascading this output into their provided vocoder. This interleaving strategy demonstrates superior performance in high-difficulty settings compared to other text-to-speech models like VALL-E (Wang et al., ), and so we use it for all resynthesis experiments. To interleave our units, we sort on the start-timestep of every SylBoost unit and TWIST-unit in ascending order.
We subtract 0.08s (the length of two TWIST units) from each SylBoost unit start time before sorting to account for noisy SylBoost boundaries.
For the rest of the pipeline, we follow Song et al. () without global advance using our syllables as a drop-in replacement for phones and use greedy decoding. We call this model the Interleaved-Vocoder-LM.

Although incorporating this additional resynthesis model slows down generation compared to TWIST, most model scaling happens in the SpeechLM. For example, the TWIST paper still observes scaling improvements in semantic understanding with a SpeechLM of 13B parameters while current SotA speech synthesis models such as Ju et al. () operate with fewer than 1B parameters. Taking the same approach as TWIST, our work focuses purely on improving the semantics of generated audio and is entirely orthogonal to work on audio synthesis quality and voice cloning.

We then generate continuations for a sample by 1) Extracting syllable-units and TWIST units from the sample, 2) Sampling syllable-unit continuations from the SpeechLM, 3) Continuing TWIST units with our interleaved model conditioned on sample TWIST units, sample syllable-units, and continued syllable-units, and 4) Resynthesizing these into speech using the vocoder.

## 5 Experiments

### 5.1 Training Datasets

We train our tokenizer using LibriSpeech (Panayotov et al., ), which contains 960 hours of audio books. We noticed that SylBoost converges before all data is used, and so we randomly subsample LibriSpeech to a 100 hour train set and train for five epochs and two iterations for all experiments. We train our SpeechLMs using all of LibriLight (Kahn et al., ), which provides roughly 55k hours of speech. As a note on fair comparison, although AudioLM uses exactly this split of LibriLight, TWIST collects an additional 100k hours of data, totaling to 155k hours.

### 5.2 SylBoost Unit Configurations

By varying the number of boundaries input to our cut algorithm at each stage in the SylBoost pipeline, we can arbitrarily control our rate of temporal tokenization. We evaluate three main unit rates at 8.33Hz, 6.25Hz, and 5.00Hz, the latter which matches the empirical rate of SD-HuBERT units on LibriSpeech dev-clean. To account for differences between slow and fast speakers, we dynamically choose $k$ during SylBoost to best match the number of distinct feature groups in the embedding space (seen as blocks in the feature self-similarity matrix $A$, Figure 2). We do this by setting a threshold $\delta$ and taking the fewest boundaries $k$ such that the loss per timestep is less than $\delta$ for 75% of timesteps. We choose $\delta$ so that average unit rates over LibriSpeech dev-clean match a desired frequency. We also explore modifying the number of clusters generated by K-Means and Agglomerative Clustering, which combined with changing the unit rate results in fine-grained bitrate control. We note that although prior approaches like SD-HuBERT and VG-HuBERT apply a cut algorithm with $k$ cuts, there is no way to control the actual number of distinct feature groups in the self-similarity matrix during training. As a result, we cannot increase the frequency of SD-HuBERT units by changing $k$: Additional cuts result in close-to-identical representations that map to the same quantized clusters.

### 5.3 Results: Evaluating Unit Quality

**Table 1: Unsupervised syllable boundary detection and clustering accuracy on LibriSpeech (Panayotov et al., ) test. For F1 scores, the superscript is tolerance threshold in ms. All other metrics use 50ms. Higher is better.**
| Approach | Backbone | Training | F1^50 | F1^20 | Pr. | Re. | R | CPur | SPur |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Feat-Sim(Peng et al., ) | HuBERT | no | 47.3 | 24.7 | 46.6 | 48.0 | 54.5 | 28.0 | 30.0 |
| LossPred 5.0Hz (Ours) | HuBERT | no | 59.6 | 31.4 | 54.9 | 66.7 | 56.3 | - | - |
| SD-HuBERT(Cho et al., ) | HuBERT | yes | 66.1 | 32.2 | 64.9 | 67.4 | 70.7 | 43.2 | 45.0 |
| SylBoost 5.0Hz (Ours) | HuBERT | yes | 70.9 | 40.1 | 70.8 | 71.4 | 75.1 | 28.9 | 47.8 |
| SylBoost 5.0Hz (Ours) | Data2Vec2 | yes | 73.2 | 44.6 | 72.1 | 74.4 | 76.9 | 33.6 | 54.9 |

We evaluate the quality of our semantic units with two approaches 1) measuring correspondence with syllables and 2) running speech resynthesis followed by ASR. To measure correspondence with syllables, we use the development and test sets of LibriSpeech (Panayotov et al., ) and follow the approach from Peng et al. (), extracting timesteps for phones using the Montreal Forced Aligner (McAuliffe et al., ) and then converting these phones into syllables with a rule-based method (Gorman, ). We evaluate the quality of syllable boundary detection with a ground truth boundary marked as hit if a proposed boundary is present within a tolerance threshold. We report F1, Precision, Recall, and R score (Räsänen et al., ). We ablate F1 scores with tolerance windows of 20ms and 50ms. Given boundaries, we also evaluate the purity of our clusters at 4096 units. Syllable Purity measures the probability that a syllable is mapped to its most corresponding cluster unit, and Cluster Purity measures the probability that a cluster is mapped to its most corresponding syllable unit.

Even if units do not correspond with syllables, they can still be useful to SpeechLMs if they can resynthesize back into speech that matches the original text. Additionally, training a resynthesis model provides a stronger description of the semantic information contained in units than purity metrics, which may be unreliable because SD-HuBERT does not provide a unit at every timestep while our methods do. To evaluate resynthesized speech, we follow AudioLM and measure Word Error Rate (WER) and Character Error Rate (CER) on the set of 4-10 second segments from LibriSpeech test-clean. For ASR, we follow VALL-E (Wang et al., ) and use the public HuBERT-base CTC ASR model provided by Hsu et al. ().

**Table 2: Boundary detaction with different initialization using HuBERT on LS dev-clean**
| Model | F1 | Pr. | Re. |
| --- | --- | --- | --- |
| Feat-Sim | 46.7 | 48 | 45 |
| -Iter 1 | 51.1 | 50 | 52 |
| -Iter 2 | 50.4 | 51 | 50 |
| LossPred | 60.1 | 53 | 68 |
| -Iter 1 | 67.1 | 67 | 68 |
| -Iter 2 | 70.2 | 70 | 70 |

Table 1 shows our syllabic correspondence results against the prior-state-of-the-art SD-HuBERT (Cho et al., )
and the HuBERT-based feature similarity strategy from Peng et al. (). Applying SylBoost to either HuBERT or Data2Vec2 improves correspondence across-the-board except for in cluster purity. Although LossPred trails SD-HuBERT in performance, it pushes the boundary for syllable recognition using HuBERT without additional training. Improvement across iterations and with different loss initialization can be found in Table 4, which shows that using LossPred as a bootstrapping source instead of HuBERT similarity (Peng et al., ) makes SylBoost converge to much higher segmentation accuracy. We discuss initialization more in Appendix A.2. In Table 4, we demonstrate the controllability of SylBoost to extract boundaries at different granularities: higher unit rates successfully correspond more with phones and lower unit rates correspond more with syllables and words. In Table 4 we show the speedup of our efficient unit extraction.

**Table 5: Unit Resynthesis. WER/CER results on 4-10 second examples on LibriSpeech (Panayotov et al., ) test-clean. Hz and Bitrate are measured post deduplication on LibriSpeech dev-clean.**
| Model | Changes | Hz | #Units | BPS | WER$\downarrow$ | CER$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| SD-HuBERT (Cho et al., ) |  | 5.0 | 4096 | 60 | 37.3 | 22.7 |
| SylBoost (HuBERT) | +Our Clustering | 5.0 | 4096 | 60 | 18.5 | 10.2 |
| SylBoost (Data2Vec2) | +Use Data2Vec2 | 5.0 | 4096 | 60 | 12.8 | 6.4 |
| SylBoost (Data2Vec2) | +Increase #Units | 5.0 | 16384 | 70 | 9.1 | 4.3 |
| SylBoost (Data2Vec2) | +Tune unit-rate, #Units | 8.33 | 2048 | 91 | 8.0 | 3.7 |
| SylBoost (Data2Vec2) | +Tune unit-rate, #Units | 6.25 | 8192 | 81 | 7.0 | 3.2 |
| []     TWIST Tokenizer (upper bound) (Hassid et al., ) | 19.5 | 500 | 175 | 6.3 | 2.5 |  |

### 5.4 Speech Lauguage Model Configuration

All of the SpeechLMs we implement, as well as our Interleaved-Vocoder-LM, follow the OPT (Zhang et al., ) architecture and default to using 12 Transformer layers, an embedding dimension of 768, and learned positional embeddings. This totals to 90M non-embedding parameters, and represents an identical model architecture to TWIST-125M. We also experiment with a larger 24 layer 1024 dimension model totaling to 300M non-embedding parameters, the same as AudioLM and TWIST-350M. For all language model pretraining experiments, we randomly crop files to 25 seconds, use a batch size of 80000 tokens, and train for 200k steps, which amounts to the same compute as in TWIST. To make our approach entirely textless, we do not use TWIST initialization. We tokenize using SylBoost on Data2Vec2 (Baevski et al., ), which provided the best results in low-bitrate WER resynthesis (Table 5.3). We discuss the base encoder further in Appendix A.3. Additional hyperparameters and hardware details are in Appendix A.4.

### 5.5 Speech Language Model Baselines

We compare against a range of SotA SpeechLMs: AudioLM (Borsos et al., ), TWIST (Hassid et al., ), GSLM (Lakhotia et al., ), and the audio-only version of Moshi (Défossez et al., ). As a tokenizer, AudioLM uses w2v-BERT tokens (Chung et al., ), TWIST operates on tokens from a HuBERT model that the TWIST authors pretrain for an additional iteration on multilingual data, Lakhotia et al. () uses HuBERT-base and K-Means, and Moshi operates on a residual codec with the first code distilled to match WavLM (Chen et al., ). AudioLM and TWIST are the most directly comparable models to SyllableLM with respect to dataset and transformer size and output units at 25Hz followed by consecutive deduplication.

We reimplement a 90M parameter model using the TWIST tokenizer units without textually-pretrained initialization (Cold-Init in the TWIST paper) on our data split for an all-else-held equal comparison on unit type. We additionally reimplement Byte Pair Encoding (BPE) to train a SpeechLM as done in Shen et al. (), resulting in the lowest bitrate encoding of speech outside of our model. We run BPE on deduplicated TWIST units and we grid search to find that the minimum bitrate for BPE occurs at 4k units, which we use for all experiments (Shen et al. () originally operated on 50Hz units, meaning that the 118bps rate obtained here is also new). For textual toplines, we train on corresponding LibriLight text transcripts from Kang et al. () and convert text to phones and syllables using the same methods as in Section 5.3.

### 5.6 Results: Spoken Language Modeling

**Table 6: Main SyllableLM results. We evaluate on sWUGGY (In-Vocab, All, Out-of-Vocab), sBLIMP from Nguyen et al. (), and tStoryCloze from Hassid et al. (). Higher is better for all metrics. *Estimated GPU Hours, see Appendix A.4 for details. Bold is best, underlined is second-best.**
|  |  |  |  |  | #Data | #Data | GPU- | sWUGGY | Semantics |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Params | #Units | Hz | BPS | Hours | Toks | Hours | All | IV | OOV | sBLIMP | tSC |
| Phone Topline | 90M | 70 | 12.5 | 76 | 55K | 2.5B | 70 | 81.4 | 95.2 | 67.7 | 68.8 | 80.6 |
| Syllable Topline | 90M | 28k | 5.0 | 74 | 55K | 1B | 70 | 79.5 | 93.1 | 65.9 | 69.3 | 76.6 |
| Lakhotia et al. () | 150M | 100 | 50 | 332 | 6K | 1.1B | - | 64.8 | - | - | 54.2 | 66.6 |
| AudioLM (Borsos et al., ) | 300M | 1K | 25 | 250 | 55K | 5B | 2.9k* | 71.5 | 83.7 | 59.3 | 64.7 | - |
| TWIST (Hassid et al., ) | 300M | 500 | 19.5 | 175 | 155K | 9B | 295 | 70.6 | 80.3 | 61.0 | 56.2 | 69.9 |
| TWIST | 1.3B | 500 | 19.5 | 175 | 155K | 9B | 1.1k* | 71.8 | 81.1 | 62.3 | 57.0 | 70.6 |
| [] TWIST | 7B | 500 | 19.5 | 175 | 155K | 9B | 5.9k* | 72.7 | 83.6 | 61.8 | 59.0 | 74.1 |
| [] TWIST | 13B | 500 | 19.5 | 175 | 155K | 9B | 10k* | 73.9 | 84.1 | 63.7 | 59.2 | 76.4 |
| [] Moshi (Défossez et al., ) | 7B | 2K | 12.5$\times$8 | 1.1K | 7M | 2.5T | - | 74.8 | - | - | 59.9 | 80.9 |
| TWIST-CI | 90M | 500 | 19.5 | 175 | 55K | 3.9B | 84 | 69.7 | 79.8 | 59.7 | 55.5 | 69.0 |
| BPE (Shen et al., ) | 90M | 4k | 9.8 | 118 | 55K | 2B | 84 | 61.8 | 66.7 | 56.8 | 54.5 | 56.2 |
| SyllableLM | 90M | 2k | 8.3 | 91 | 55K | 1.6B | 70 | 72.2 | 81.7 | 62.6 | 62.4 | 71.4 |
| SyllableLM | 90M | 8k | 6.25 | 81 | 55K | 1.2B | 75 | 72.1 | 82.2 | 61.9 | 62.9 | 70.2 |
| SyllableLM | 90M | 16k | 5.0 | 70 | 55K | 1B | 82 | 67.6 | 76.9 | 58.3 | 63.2 | 69.0 |
| SyllableLM | 300M | 8k | 6.25 | 81 | 55K | 1.2B | 290 | 72.2 | 82.2 | 62.0 | 63.7 | 75.4 |

The end-to-end GSLM pipeline is deep, and so it is essential to have metrics to independently evaluate different stages. To evaluate our SpeechLM stage, we follow Lakhotia et al. () and use the ZeroSpeech (Nguyen et al., ) sWUGGY and sBLIMP evaluation. The sWUGGY dataset tasks the model with outputting a higher perplexity on similar but fake spoken words (e.g. brick vs blick). Similarly, the sBLIMP dataset checks syntactic correctness (e.g. the dog sleeps vs the dogs sleeps). We also evaluate the SpeechLM on the tSC set from Hassid et al. (), which operates like the ZeroSpeech metrics on a spoken version of the StoryCloze dataset (Mostafazadeh et al., ) with the last sentence in negative samples randomly chosen. For all metrics we follow prior work and output the mean perplexity per token.

The results for SpeechLM metrics are depicted in Table 6. We find that training with our syllable units improves performance across-the-board on speech understanding tasks. With under 90 hours of training, SyllableLM outperforms the 13B parameter TWIST and 7B Moshi on sBLIMP, which best evaluates semantic understanding (Lakhotia et al., ). Measured on all of sWUGGY, we also beat AudioLM with 30x less GPU compute and TWIST model sizes up to 1.3B parameters in lexical understanding. On tSC, we observe that SyllableLM benefits from scaling with our large model approaching the performance of our textual topline. Because our units are much lower frequency, we see a 4.5x speedup at inference time with the same parameter count in Table 4. Due to compute requirements, we are unable to scale further.

**Table 7: Holding number of SylBoost units and unit rate constant. ZeroSpeech development set.**
| Hz | #Units | sWUGGY | sBLIMP |
| --- | --- | --- | --- |
| 8.33 | 4k | 72.9 | 61.8 |
| 6.25 | 4k | 69.3 | 63.3 |
| 5.00 | 4k | 65.7 | 62.8 |
| 8.33 | 2k | 72.1 | 62.0 |
| 8.33 | 4k | 72.9 | 61.8 |
| 8.33 | 8k | 72.9 | 61.2 |

We notice a decrease in sWUGGY quality with our 5.0Hz units, which we suspect is in part caused by the short length of the dataset audios making input tokenization excessively short. We further ablate changing the unit rate and unit frequency in Table 8. We also find that BPE, despite having the lowest bitrate outside of our approach, does not approach the quality gains created by our syllable-like units.

To measure the quality of end-to-end continuations, we use the VERT@O-PPX and PPX@O-VERT metrics proposed in Lakhotia et al. (), which are shown to be the automatic metrics correlating best with human meaningfulness judgements.
VERT@O-PPX measures the diversity of output at the sampling temperature where the perplexity of generated audio transcriptions matches that of the ground truth text, and PPX@O-VERT performs the inverse.
Like Lakhotia et al. (), we generate 10-second continuations from 1000 randomly sampled 3-second crops from LibriSpeech test-clean, and measure results using their provided environment and parameters.
To prevent errors from randomly cutting off audio in the middle of a syllable-like unit, we discard the last encoded unit for all models when generating continuations. We report these in Table 8 with two sigma error bars and find that SyllableLM outperforms TWIST 300M and 1.3B.

## 6 Discussion and Limitations

Though speech is a very general medium, there are a number of challenges in adapting our methods to generate low-bitrate units angled towards other audio tasks or other domains such as vision. Our LossPred technique assumes that the semantic units to learn are separable across time, one-dimensional, and contiguous. In audio tasks or settings with multiple speakers, sounds or words can occur simultaneously. Images and video have partially occluded or overlapping objects.

Despite this, LossPred is unique in that it finds boundaries using the values (losses) that the pretrained model is directly trained to optimize. Meanwhile, other approaches such as DINO (Caron et al., ) and SD-HuBERT rely on difficult-to-control intermediate features for semantic discovery. SylBoost then demonstrates that we can use LossPred to extract emergent and interpretable features even when they are not clear in the original model. Because of this, we believe that LossPred and SylBoost provide a very promising direction for controllable and scalable semantic feature clustering and extraction in domains like speech, audio, music, and video.

Low-frequency units such as ours also provide a significantly more computationally tractable path toward the scaling that has seen such success in textual language models. On the other hand, SylBoost units may be losing out on useful paralinguistic features like tone whose impact is only salient on non-audiobooks or at scale. We suspect that hybrid models operating on low-bitrate and high-bitrate units could be constructed to balance these considerations and demonstrate significant quality improvements in multimodal speech-text language modeling.

## 7 Conclusion

We introduce a new method to tokenize speech for use in SpeechLMs. We do this by proposing a method to elicit syllabic organization in pretrained speech encoder models, bootstrapping a feature-space clustering algorithm from a static analysis of correlations in off-the-shelf SSL encoder model losses across time. We demonstrate the success of our technique both in having strong associations with syllables and as an extremlely low-bitrate codec of speech for textual resynthesis. Using this tokenization strategy, we successfully train SyllableLM, a SpeechLM that out-performs comparable state-of-the-art approaches across a diverse range of metrics with a significant training and inference speedup. We further ablate several design decisions such as quantization strategy, loss initialization, and the effects of controllability for downstream usecases. Compression is a crucial aspect of learning, and we hope that these significant improvements in the unsupervised learning of low-bitrate speech units can serve as a foundation for approaches towards understanding spoken language and general representation learning.

## 8 Reproducibility

We focused heavily on reproducibility throughout our work, and we commit to releasing all parameters and code used upon deanonymization. We provide the exact equations used to calculate LossPred, SylBoost, and our Efficient Extraction Pipeline, allowing for reimplementation solely from the main text. The SyllableLM architecture and Interleaved-Vocoder-LM are well-defined and follow publicly sourced prior work. All datasets used for training and evaluation are listed explicitly in the main text, and all evaluation procedures are described as would be needed to calculate them.

## 9 Ethics

It is important to note that large textual language models can have harmful effects, such as enabling the generation of misinformation in mass. Although generative spoken language models have not yet caught up to their textual counterparts, it is still necessary to be aware of potential misuses that could arise in the future.

## Appendix A Appendix / supplemental material

### A.1 Randomly Sampled Example Segmentations

We provide randomly sampled example segmentations from the LibriSpeech (Panayotov et al., ) dev-clean set. All models are the second iteration of Data2Vec2, which we use for our SyllableLM experiments in Section 6. Top: Feature Self-Similarity matrix, darker green is closer. Segmented cuts span vertically in blue from the top, ground truth boundaries span vertically in red at the bottom. Bottom: time-aligned Mel-Spectrogram. We call attention to the interesting behavior of global correspondences appearing when words or syllables are repeated. Best viewed zoomed in.

Figure: Figure 3: 8.33Hz
Refer to caption: extracted/5903259/example_cuts/833_116-288048-0010.png

[Uncaptioned image]: extracted/5903259/example_cuts/833_251-118436-0002.png

[Uncaptioned image]: extracted/5903259/example_cuts/833_251-137823-0009.png

[Uncaptioned image]: extracted/5903259/example_cuts/833_700-122868-0038.png

[Uncaptioned image]: extracted/5903259/example_cuts/833_251-118436-0005.png

### A.2 Discussion: Other Bootstrapping Strategies

Of course, there already exist several strategies for unsupervised syllable and word segmentation such as in Fuchs & Hoshen (), Pasad et al. () and Peng et al. () that could be used to bootstrap our first pseudolabels. We have not conducted an exhaustive search over initializing with these strategies, however it is intriguing that the approach from Peng et al. () achieves significantly lower quality, relatively and absolutely, from SylBoost bootstrapping than LossPred. We suspect that this may be caused by the fact that although the representations of these models correlate with boundaries, there is no modeling in the pretraining loss pushing the representations to linearly separate across semantic differences. Meanwhile, the loss is forced to change across semantic boundaries due to the difficulty of language modeling, albeit noisily.

### A.3 Discussion: Base Encoder

Because we want to use a 50Hz base encoder to match SD-HuBERT and have fine-grained boundary control during syllable segmentation, we cannot use the 25hz encoder from TWIST. Unfortunately, this means that the quality of the base encoder may be a confounding factor in our SpeechLM evaluation. We choose Data2Vec2-base (Baevski et al., ) as a middleground for training SpeechLMs on syllable-like units because we find its quality enables lower bitrates than HuBERT, but it is older and trains on less-data than the TWIST tokenizer, and it has 6x fewer parameters than w2v-BERT, used by AudioLM. We suspect that applying newer encoders like w2v-BERT 2 from Barrault et al. () could enable even better performance, which we leave to future work.
Because the Data2Vec2 loss function doesn’t natively work with LossPred, we initialize Data2Vec2 SylBoost from the same HuBERT loss boundaries as discussed in 3.1.

### A.4 Hardware And Hyperparameters

We implement all experiments using NVIDIA A40 46GB GPUS with a Intel Xeon Gold 6226R CPU @ 2.90GHz. Estimated speeds are made using these results as well as scaling from Zhang et al. ().

Hyperparameters for pretraining our models are below. We note that the Batch Size is in terms of tokens, which means that higher unit rates will have fewer seconds of raw audio per batch to keep GPU compute roughly equal per model.

For LossPred, we use a HuBERT-large student and a HuBERT-base teacher as they are the only public checkpoints for a corresponding student and teacher model. We preprocess the audio input to LossPred and Feature Similarity (Peng et al., ) using an unsupervised voice activity dection model (Tan et al., ).

**Table 9: Speech pre-training hyper-parameters.**
|  | SyllableLM Base | SyllableLM Large |
| --- | --- | --- |
| Layers | 12 | 24 |
| Embed Dim | 768 | 1024 |
| MLP Dim | 3072 | 4096 |
| GPUs | 2 | 4 |
| Learning rate | $2\times 10^{-4}$ | $2\times 10^{-4}$ |
| Adam $\beta_{1}$ / $\beta_{2}$ | 0.9 / 0.98 | 0.9 / 0.98 |
| Weight decay | 0.01 | 0.01 |
| Learning rate schedule | Linear Decay | Linear Decay |
| Dropout | 0.1 | 0.1 |
| LayerDrop | 0.0 | 0.0 |
| Warmup updates | 8,000 | 16,000 |
| Batch size (tokens) | 80,000 | 80,000 |
| Updates | 200,000 | 200,000 |
| Position Embeddings | Learned | Learned |

**Table 10: Sylboost Pretraining Prarameters.**
|  | HuBERT Base | Data2Vec2 Base |
| --- | --- | --- |
| $L$ (One-Indexed) | 9 | 11 |
| Learning rate | 5e-5 | 5e-5 |
| Epochs | 5 | 5 |
| LibriSpeech Data | 100 hours | 100 hours |
| Batch Size | 32 | 32 |
| Iterations | 2 | 2 |

### A.5 Sample Continuations

Below are sample continuations generated with a temperature sampling parameter chosen to best match Oracle VERT diversity scores. We provide continuations of roughly 3 seconds of audio, sampled randomly from LibriSpeech test-clean. This text is given as output by our HuBERT ASR Model from Hsu et al. (), with transcription errors present and with no additional modifications. The source text is bolded, and sometimes cuts off mid-word, which can behave differently per sample based on unit rate and quantization artifacts.

| Model | Continuation |
| --- | --- |
| TWIST 300M | HE FILLED FOR AND FOUND THE WIZARD’S BLACK CLOTH THE SQUIRE WAS QUITE DEVOTED TO CARRYING IT IN THE PARKS I COULD TELL YOU THAT HE RATHER LOVED ALL THE WIZARDS IN THE MATTER OF HONEST FIGUROUS TEMPERED MEN THEN IMMEDIATELY AND BECAUSE HE DID |
| TWIST 1.3B | HE FILLET FOR AND FOUND THE WIZARD’S BLACK CLOTH THE SQUIRE WAS QUICK IN MOVEMENT A WHEEL OF HIS JOURNEY MADE HIM THE IMPRESSION OF HARE’S BLOODSHED THERE WAS NO HELP FOR IT HE SURMISED AND HE DID SO |
| SyllableLM 2k 8.33Hz 90M | HE FILLED FOR AND FOUND THE WIZARD’S BLACK CLOTH THE SQUIRE WAS GLAD THAT HIS LAD HAD NOT RETURNED TO KING LENO FOR IT IS PROBABLE THAT HE WOULD HAVE BECOME SO UNWELL THAT HE COULD NOT HAVE SPOKEN |
| SyllableLM 8k 6.25Hz 300M | HE FILT FOR AND FOUND THE WIZARD’S BLACK CLOTH THE SQUIRE WAS HALF WAY DOWN THE PATH WITH THE WIZARD BUT HE STOPPED SHORT BEFORE THE GATE CLUTCHED HIS FISTS SEIZED HIS GOWN AND |

| Model | Continuation |
| --- | --- |
| TWIST 300M | WHEN THEY WERE OUTSIDE UM SIMPLY LATCHED THE DOOR AND STARTED UP THE STAIRS AND EVEN WENT INTO THE LITTLE HOUSE AROSE IN THE SETTING SUN SET FORTH THE PLACES WHERE PEOPLE LIVED FROM MON HONEYSUCKLE HANNEY |
| TWIST 1.3B | WHEN THEY WERE OUTSIDE UM SIMPLY LATCHED THE DOOR AND STARTED UP THE TURNPIKE OL HAT DON KILL ME THE SLING DE IN YORN ME WEAVING OUT CHARLEYS SENSE EXAMINED WHAT HE MADE EXAMINES |
| SyllableLM 2k 8.33Hz 90M | WHEN THEY WERE OUTSIDE UM SIMPLY LATCHED THE DOOR AND STARTED WALKING IN THEY WERE TOO OLD TO CARE MUCH ABOUT GOING HOME THEIR RELATIVES LEFT |
| SyllableLM 8k 6.25Hz 300M | WHEN THEY WERE OUTSIDE UM SIMPLY LATCHED THE DOOR AND STARTED SLOWLY DOWN THE CORRIDOR AND MISSUS BAKER WALKED BESIDE THEODORA THEY WERE NEAR THE OUTER DOOR WHERE |

| Model | Continuation |
| --- | --- |
| TWIST 300M | DO BE OR NOT TO BE THAT IS THE QUESTION WHETHER TIS NOBODY SIBL LINE IN OTHER SHIRTS OR CHOCOLATE NOS MICOTTON BUTTER WHAT WE WERE DO WE SEE THESE HITS WE’VE GOT THE GHOST HERE THEY’RE LOOKING |
| TWIST 1.3B | DO BE OR NOT TO BE THAT IS THE QUESTION WHETHER TIS NO GOOD EITHER THAN TO GO THROUGH THE JUDGMENT OF GAUL AND YOUR DOCTRINES THE LORD YOUR GOD AND YOUR GOSPEL IN RESPECT OF THE POWER OF THIS |
| SyllableLM 2k 8.33Hz 90M | DO BE OR NOT TO BE THAT IS THE QUESTION WHETHER TIS NO OTHER THAN ESO’S OWN DESTINY YOU SEE IT IS A LA MISTER PRIOR THAT THIS IS THE CASE |
| SyllableLM 8k 6.25Hz 300M | DO BE OR NOT TO BE THAT IS THE QUESTION WHETHER TIS NO REGRET OR NO PLEASURE THAT MAY BE RUSHED INTO ACTION AT ONCE WITH THE GREATEST EAGERNESS OF IMPULSE AND ELASTICITY OF HEART |

| Model | Continuation |
| --- | --- |
| TWIST 300M | HE IS CALLED AS YOU KNOW THE APOSTLE OF THE INDIAN KING WHO IS SO GLORIOUS AND ACTING WHY IS THE OTHER PRINCE NOT BELIEVED BY HIM IN EVERY FAITH THAT IS FREE WILL EXCEPT WHEN HE |
| TWIST 1.3B | HE IS CALLED AS YOU KNOW THE APOSTLE OF THE INDIES SAW WHAT HAD PASSED THROUGH HIM LATER IN ANOTHER BOOK AMONG THOSE WHO HAD ENGRAVED IT THIS VOLUME MISTER PICKWICK THOUGHT IT RIGHT NOT TO INSULT YOU |
| SyllableLM 2k 8.33Hz 90M | HE IS CALLED AS YOU KNOW THE APOSTLE OF THE INVIDISIBLE BEFORE THEY RECEIVED THE GRACE OF GODAD CHRIST THEN HAD IN THE FAITH OF HIS SON |
| SyllableLM 8k 6.25Hz 300M | HE IS CALLED AS YOU KNOW THE APOSTLE OF THE INDIES HE IS THE FORERUNNER OF TEACHING AND FAR BEYOND IT HE IS THE EXACT SCIENTIST WHO MEASURES THE MOVE |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: see weaknesses section for suggestions on what to add to the paper
soundness: 3
strengths: - the paper describes the method in sufficient detail to reproduce it
- this appears to be a novel way (although based on much prior work) to discover unit boundaries in speech data, which turn out to be syllables.
- the results are compelling in light of much concurrent work on speech-unit language models, especially as they allow to significantly reduce the cost of training such models due to higher compression ratio
summary: This paper proposes a method to extract speech units from existing pre-trained models such as HuBERT or Data2vec2 at syllable boundaries. They show that they are able to significantly lower the number of units used to encode a particular utterance while maintaining and improving on the quality of re-synthesized speech, or continuation of the speech if used with a language model trained on these new speech units. The method is relatively straight forward and, if it works, allows significant savings in the number of tokens needed to accurately model speech data.
weaknesses: - while using speech units from HuBERT and other models has been widely explored, one downside of such units is that they tend to mostly capture only the semantic content. Presumably, TWIST units may capture some of the prosodic information, but it is not clear how the authors measure the prosody and expressiveness of this method. An alternative for generative tasks is to simply model speech as text (e.g. via an ASR model) and add additional style vectors. Speech understanding tasks may simply use the pre-trained models which this work bootstraps on directly. It would be very interesting to see some quantitative and qualitative comparisons on this topic.
- "Language models require tokenized inputs" - this is not strictly true depending on how one defines "tokens". While discrete tokens are a popular choice, one can certainly build a language model that uses continuous inputs ("tokenized" by splitting it into distinct representations on the time axis or through methods like what is described in this paper) with discrete or continuous targets on the output.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: In the intro, what is evidence for “language models on spoken language still struggle to output semantically meaningful speech.”? 

In method, “We apply the SylBoost loss (Equation 4) to model features at layer L, which we select based on syllabic correlation as explored in detail in Pasad et al. (2024).” The cited paper is about words not syllables, where I couldn’t locate a particular mention of syllabic correlation. Please let me know in which part of the reference paper demonstrates it.

1. Provide motivations/reasons for some design choices: Why 75% of timesteps in Section 5.2? Why 8.33, 6.25, and 5.00Hz?

2. On the statement "training a resynthesis model provides a stronger description of the semantic information contained in units than purity metrics”, please provide the rationale for this. It is counter intuitive since resynthesis will be better if the input embedding is closer to low-level, like acoustics. For example, VQ-VAE based models like EnCodec show great performance in resynthesis but are claimed to lack higher-level information like semantics.

3. Typo in Table 2: Boundary detaction —> detection.

4. In Table 6, sBLIMP is designed for evaluating syntax (or grammar), which should not be categorized under “semantics”. Also, sWUGGY-all and tSC are the best in Moshi but bold annotations are erroneously given. Please fix them if the numbers are correct.
soundness: 2
strengths: LossPred and Sylboost are well-motivated, novel ways to learn speech representation beyond phonemes, allowing different model training with different frequency units. The method is meticulously explained. Authors depict real speed up in using these units compared to TWIST. Furthermore, results shown achieve near SOTA results on syllable boundary detection and discovery. Table 5 well ablates their design choices and results shown in Table 6 show clear improvements over existing baselines. In particular, the gain in length of tokens and bitrate is outstanding compared to previous speech unit LMs, which is an important factor in LM.
summary: The authors propose a novel method to extract rough syllabic segment boundaries utilizing loss patterns in HuBERT induced by different input mask spans. With those rough segments, a learning method, SylBoost, is proposed to learn a segmented representation by fine-tuning pretrained SSL models. By controlling the granularity of the initial segments, the authors train multiple models with different frequency of speech segments, ranging from phoneme to word level. Syllables are more accurately and efficiently detected than the previous model, SD-HuBERT. By training LM on those syllable-like units from clustering, the authors suggest SyllableLM and demonstrate that spoken LMs at syllabic granularity can outperform or on par with previous SOTA spoken LMs using smaller model and training data.
weaknesses: There are several concerns regarding some claims/analysis, as detailed below.

1. Lack of verification of syllabic correspondence of higher frequency models: The higher frequency models (6.25 HZ or 8.33 Hz) seem sub-syllabic or representing di- or tri-phones based on Figure 2 and Table 3. For example, there is a huge gain in phoneme boundaries correspondence (F1-P) by changing from 5Hz to 6.25Hz. Though, F1-S is still higher than F1-P, but this would be due to the usage of monophones in evaluation. This evokes a question whether the 6.25Hz GSLM – the best one beating SOTA, is a “syllable” LM. The 5Hz version seems the most syllable-like (and the only evaluated frequency in Table 1), but it is the worst among different frequency versions according to Table 5 and 6, showing high WER and no outstanding performance in LM.  Detailed analysis of the linguistic characteristics of the units at different frequencies, perhaps including examples and a discussion of how they relate to syllables, phonemes, or other linguistic units will make this clear.

2. Overclaiming controllability: The controllability of the segment frequency is given during training not at inference. However, the statement in the abstract, ”Our method produces controllable-rate semantic units” is misleading as it strongly implies that the controllability is given at the inference time. It requires rephrasing or a significant improvement in clarity. Besides, the authors claim a lack of controllability of previous studies, but the mincut approach cited by the authors has knobs that can control the frequency of resulting segments. The previous method may have a controllability though the prior works didn’t leverage the functionality. If true, the claimed novelty on controllability should be reassessed, may be provide additional experiments as evidence; and the manuscript must be accordingly updated.

3. Adaptability on different speaking rate at inference: The audiobook datasets have relatively consistent speaking rate as the speech is very carefully delivered. However, in the real world, people may use very different speaking rates based on contexts or based on accents or dialects. Can a model work on speech with a different speaking rate? For example, if we put speech elongated by two times, would the model output the same number of syllables or doubled? The authors explain that the k is determined by optimizing timestep losses but not clear how this loss is calculated at the inference time. 

4. Lack of validation of semantic unit discovery: As the authors are citing Choi et al. (2024), I assume semantics here means concepts, e.g., “dog” is similar to “puppy” though they are phonetically distant. However, syllables by definition are not conveying such conceptual meaning, rather structured phonetic information. While the authors are frequently claiming semantic unit discovery, the paper has no supporting evidence that the discovered units or embeddings are semantics. The low error rates in resynthesis cannot verify whether the feature encodes semantic information. The approach suggested by Choi et al. (2024) may be useful to verify the claims.

5. Lack of valid comparison/evaluation of LossPred: The proposed method for initial boundary extraction, LossPred, is novel and makes a lot of sense. Table 2 shows comparisons using Feat-Sim, but why not VG-HuBERT or SD-HuBERT? Those prior works already showed that the Feat-Sim on the original HuBERT is not accurate in terms of syllabic segmentation. As Feat-Sim works best with VG-HuBERT or SD-HuBERT, a fair comparison should involve Feat-Sim on them, not on HuBERT.

6. Absence of audio quality evaluation of generated speech: The paper severely lacks evaluation of the quality of generated audio which is a very crucial aspect of generative model or codec language model of speech. Though it is not claimed to be the main focus, some quality metrics like Mean Opinion Score (MOS) should be reported.  The WER/CER metrics are not sufficient to show the audio quality.

7. Erroneous calculation of RTF: As defined by sources like https://www.openvoice-tech.net/index.php/Real-time-factor and https://link.springer.com/article/10.1007/s11042-020-10073-7, real time factor is defined as the ratio between the time to process an input over the duration of the input. However, it seems like here, the authors chose to report the reciprocal, which is inadequate. The scores should be corrected to comply with the RTF definition. Also, the RTF comparison over TWIST should be calculated end-to-end, not from cached speech units. This is because RTF is aimed to measure efficiency in “real-world” scenarios, so the calculation should start from raw waveform, simulating real-world inference as much as possible. Moreover, this correction is critical as SyllableLM has an additional layer of segmentation in comparison to TWIST, which is not reflected.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Comments:
- It would be interesting to develop a joint algorithm for segmenting and clustering together. 
- Table 5: When training your tokenizer on smaller datasets with a single language and low input variability, it is easier to compress them and reduce the bitrate, as you have less information to distill into the discrete units.  

Writing/Typos:
Lines 14-17 (abstract) can be simpler and more concise

Line 172: Can you clarify: “C models the raw probabilities of the losses”

Line 158: “Two frozen Hubert models” - you used the same single model, right? (otherwise, their token space wouldn’t match..) 

Line 280: TWIST citet->citep

Table 2: Detaction->Detection

line 462: Regarding “the E2E GSLM pipeline is deep” — how about rephrasing it to “consists of several independently trained components”?
soundness: 3
strengths: - The problem of extracting variable-length representations for audio is important, as the amount of information within different audio segments changes. 
- The approach reduces the token rate by x2-3 and SpeechLMs trained on these units obtain better results compared to the fixed-rate VQ/K-means discrete unit extraction.
- The idea of segmenting the audio before clustering is interesting, and seems to be quite efficient.
summary: This paper suggests a discrete unit extraction method for speech, where each unit has a variable length, unlike most discrete audio units with a fixed rate (often 25/50 Hz).
To segment variable length units, they use a pre-trained self-supervised audio model trained on masked language modeling.
They first run the SSL model on the raw sequence to obtain the GT discrete units (teacher predictions).
They then mask a fixed-sized segment and use the SSL model again to measure the likelihood of every teacher token. They repeat this over sliding windows throughout the input, resulting in a square asymmetric “similarity” matrix.
As in previous work, they apply a min-cut algorithm to segment the audio into variable-length units.
They then train a neural model for feature matching, which leverages the produced boundaries in its loss function (average feature teacher feature across the segment). 
This is done to make the unit extraction more efficient (resolve the computational-heavy sliding windows) and hopefully generalize better.
To finally derive discrete units, they extract features from the boosted neural model and use a more efficient cut algorithm to segment the sequence. 
They then calculate the mean feature over each segment, and cluster using K-means and agglomerative clustering to discretize the units. 
Using their novel variable-length discrete units, they train speechLMs and see benefits in several metrics (e.g. sWUGGY, sBLIMP, and tSC).

Edit: I increased my score from 5 to 6 (weak-accept) and the contribution from 2 to 3 (good).
weaknesses: Weaknesses:
- The approach for variable-length unit extraction is quite complicated, with a wide range of hyperparameters to control (span size, number of segments, different ways to combine two clustering algorithms, etc). 
I’m a bit concerned that this method is overfitted to boost the specific metrics evaluated.  I find the core idea to be grouping before clustering - would you agree? 
A simpler algorithm can be: (1) compute a similarity matrix with SSL embeddings (maybe K-band matrix as you did to reduce complexity) (2) segment the audio using the similarity matrix (3) run K-means on the average representation per segment. (no training, no sliding windows)  .

- The metrics calculated to evaluate speechLMs mostly include likelihood ranking and perplexity calculations. However, generative models are mostly used to generate content. 
A sample page with audio continuations would help qualitatively check the differences in the generative capabilities of the models. ([1] may be useful to quantify those generative capabilities).  
I admit that likelihood ranking is the common practice for evaluating speechLMs, but IMO they don’t measure the thing we’re interested in (I’ll use the model mainly to generate, not to measure likelihood), so evaluating generated outputs is also important and should be added.  

- One reason to have variable-rate units is to dynamically distinguish between high-info segments and low-info segments. 
In your approach, the number of semantic regions is proportional to the length. In that case, 10 seconds of silence and 10 seconds of high-speed speech would result in the same number of tokens. 
The units have a variable length, but each sample has a fixed sample rate. With BPE, in contrast, we might get fewer tokens for 10 seconds of silence compared to 10 seconds of fast speech.  

- Table 6: What is the reason for the gray shade on several TWIST? In several metrics, the bolded result is not the best (in contrast to what is stated in the caption). I assume you want to state it is unfair to compare 7/13B models with 300M models (and I agree). In that case, it should be clear from the caption. 

- The tokenizer is trained on 960 hours of audio, which got me concerned about scalability in the unit extraction process. Can your algorithm scale so we can compute the units based on (e.g.) 55K hours of data? Can you elaborate on the sample complexity of the process (w.r.t. the length of each sample and the number of samples) 

- Table 6: The results suggest BPE (an alternative approach to producing a variable-length representation) degrades the results. This contradicts the findings of other several papers reporting benefits from BPE (additional references attached): 
[1] Exploring the Benefits of Tokenization of Discrete Acoustic Units (Interspeech 2024) 
[2] On the Effectiveness of Acoustic BPE in Decoder-Only TTS (Interspeech 2024) 

- Section 5.2: The explanations on the final unit extraction using clustering are vague. How did you combine K-means and Agglomerative clustering? 

- I find the task of unsupervised syllable segmentation is more suitable for speech-oriented venues such as ICASSP and Interspeech.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 1
questions: See Weaknesses.
soundness: 3
strengths: Overall, I believe this is a valuable work, featuring a clear motivation, good technical contributions, and impressive experimental results. The use of semantic units as low as 5Hz undoubtedly holds significant value for training SpeechLLMs. Additionally, with under 90 hours of training, SyllableLM outperforms both the 13B parameter TWIST and the 7B Moshi on sBLIMP, indicating that this paper contributes meaningfully to both the speech and multimodal research communities.
summary: This work introduces a controllable self-supervised technique to merge speech representations into coarser syllable-like units while still preserving semantic information. The propsed method produces controllable-rate semantic units at as low as 5Hz and 60bps and
achieves SotA in syllabic segmentation and clustering. Using these coarse tokens, authors successfully train SyllableLM, a Speech Language Model that matches or outperforms current SotA SpeechLMs on a range of spoken language modeling tasks.
weaknesses: The reason I gave a "boadline rej" is that I still have several concerns due to the somewhat unclear presentation of this paper. If my main questions can be adequately addressed during the rebuttal phase, I would be willing to raise my score.

1) If I understand correctly, SylBoost tokens are extracted from the waveform and include not only a sequence of tokens but also the start and end timestamps for each token. However, it’s unclear how SpeechLLM decodes these tokens when performing speech continuation, particularly in terms of predicting timestamps. Specifically, how exactly are timestamps handled during decoding? Is there a step-by-step process the authors could provide to clarify this? Additionally, the frequent mention of TWIST in Section 4.2 is somewhat confusing. My personal suggestion would be to focus more on the proposed method itself in the methods section, and simply add appropriate citations for references to other works where relevant.

2) I noticed that the authors used a cold-init approach for the 90M model. What are the potential advantages and disadvantages of using a larger text pre-trained model compared to the cold-init approach? How might this affect performance on specific downstream tasks?

3) In addition to semantic information, can SylBoost tokens also model acoustic information? Furthermore, regarding SyllableLM, what specific modifications to SyllableLM might be needed to improve its zero-shot TTS voice cloning capabilities?


Question (3) is intended solely for discussion and does not require corresponding experiments to demonstrate these capabilities.

Meta-review: This paper proposes a method to discover coarse, variable-length speech units—roughly corresponding to syllables—by leveraging a two-stage approach. The authors first apply a masking strategy on top of a pre-trained model (e.g., HuBERT/Data2vec2) to estimate segment boundaries from a “loss pattern” perspective. After computing segment-level features, the method clusters them into discrete units, achieving a lower token rate (e.g., around 5Hz to 8Hz), which can be used to train a speech-language model (SyllableLM) achieving better or on-par performance of prior discrete-unit models.

**Strengths** (1) The method provides a novel approach to control the compression rate by learning discrete units to the syllable level. (2) Empirical results show clear improvements on multiple benchmarks (3) The reduced length will improve both training and inference.

**Weaknesses** (1) Some reviewers found the method complex and hard to follow, where the writing can be improved; (2) Some reviewers suggested including qualitative measures (e.g., MOS) and semantic validation of the learned units in the paper, which can strengthen the claim of learned units; (3) Several reviewers also flagged an RTF (real-time factor) calculation issue and recommended measuring true end-to-end efficiency from raw audio to discrete tokens.

**Decision** Overall, the reviewers are positive about the novelty and empirical utility, and the discussion period makes an impact that the reviewers reached a uniform agreement of acceptance.

Decision: Accept (Poster)