SyllableLM: Learning Coarse Semantic Units for Speech Language Models

Authors: student  hubert
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 01:59:48

Abstract: Self-Supervised Transformer Models are the backbone of much of the recent progress in deep learning. However, these models require their inputs to be tokenized, and tokenization strategies for continuous data like audio and vision are often based on simple heuristics such as fixed sized convolutions or discrete clustering. For speech and audio models in particular, the high resolution of waveforms (16,000 samples/second or more) presents a significant challenge, as several times more tokens are used per word than in textual language modeling. In this work, we introduce a controllable, fully-self-supervised technique to dynamically merge speech representations across time to as low as 5 Hz at 60 bits per second while still preserving semantic information. We do this by 1) extracting noisy boundaries through analyzing correlations between mask spans and model losses and 2) iteratively improving these representations with a novel agglomeration technique. Using these new feature representations, we successfully train SyllableLM, a Neural Codec Language Model (NCLM) competitive with current SoTA NCLMs on a range of common benchmarks with a 30x reduction in pretraining compute, 5x reduction in inference compute, and 2.5x reduction in bitrate.

Main paper: 
 introduction: 
 Self-Supervised Learning (SSL) seeks to learn powerful, abstract representations of data without external labels. These representations can then be used in downstream tasks to achieve high performance even when modest amounts of supervised fine-tuning data are available. In audio and speech processing, a key motivation for this learning paradigm is the fact that young children learn to listen and speak well before they can read or write. While current textual language models [52, 59, 9] can compose highly realistic text, the research community has not yet developed similarly performant models that learn solely from spoken language. An increasing focus has coalesced around Generative Spoken Language Modeling (GSLM) [34] , which sets out to achieve this goal.The most successful of these approaches are autoregressive decoder transformer models [53] such as AudioLM [8] and TWIST [26] , which operate on tokens learned through quantizing the output of SSL encoder models [28, 14] . However, these self-supervised tokenizations are much denser than their textual counterparts with the token rates typically between 25 and 50 tokens per second for speech models, as opposed to the typical human speaking rate of 2-5 words per second. The long context lengths that result from high temporal resolution tokenizations in speech models substantially impair both pretraining and inference speed, and it is additionally unclear to what extent modeling speech with a high granularity harms more abstract semantic understanding.Very recently, there has been significant progress in extracting coarser speech unit representations from raw audio. In particular, SD-HuBERT [12] distills HuBERT [28] using only audio with a DINOlike distillation objective, and VG-HuBERT [45, 46] uses a contrastive loss against cross-modal Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.Figure 1 : Left-Top: The loss prediction matrix C, where brighter is higher likelihood placed on the teacher label. A time-aligned transcript is on the bottom, and predicted cluster unit boundaries span vertically as dashed-lines. Left-Bottom: A Mel-Spectrogram of the input waveform with an example masked timespan in gray. The losses on tokens at timesteps covered by the solid blue and dotted red spans are mapped to their corresponding rows and columns in C as described in Section 3.1. Right: Visual of our agglomeration procedure. We train a student to match intermediate teacher features pooled over regions generated by pseudo-syllable-boundaries. We use a min-cut algorithm to extract boundaries, and then apply K-Means and Agglomerative clustering to obtain discrete units. visual inputs. We continue and significantly improve upon this line of research, resulting in the first syllable-like units suitable for high-quality GSLM. Specifically, we demonstrate breakthrough improvements in textual reconstruction from low-bitrate units of SSL models, reducing the worderror-rate (WER) from 37% using SD-HuBERT units to 7%, and more than halving realized bitrate of previous SpeechLM units from 175Bps to as low as 60Bps. We additionally find that our units correlate strongly with syllables both in boundary detection and in cluster quality. Furthermore, we evaluate the effects of training SpeechLMs on these new units and obtain state-ofthe-art results across a wide-variety of metrics, competitive with or outperforming AudioLM (350M parameters) and all TWIST model sizes (125M-13B parameters) with fewer parameters and fewer GPU-Hours. We commit to making our code open-source and plan to release our tokenizer and SpeechLM parameters. Our contributions are as follows:1. We propose a novel training-free algorithm named LossPred that reveals noisy syllabiclike segmentation of unannotated speech signals by analyzing the loss of a pretrained self-supervised model (e.g. HuBERT) across different masking spans.2. We propose a novel bootstrapping framework for speech unit quantization named SylBoost that achieves SotA unsupervised syllabic segmentation, categorization, and low-bitrate unit-to-audio resynthesis.3. Using quantized SylBoost units as a basis for tokenization, we train SyllableLM, a generative spoken language model that outperforms or matches AudioLM and TWIST on a range of tasks while being 30x faster to train, 5x faster for inference, and having a 2.5x reduction in unit bitrate. 
related work: 
 Self-Supervised Encoder Models There has been a great amount of work in learning high-level representations from data by reconstructing corrupted inputs across speech [3, 28, 6] , audio [24] , text [20, 15] , and vision [10, 27] . To navigate the lack of simple discrete targets in speech, much work has been placed in finding high-quality targets, such as iterative clustering [28] and by predicting the feature representations of a teacher network based on a running average of student model weights [5, 6] . An alternate but similar line of work has been placed into learning low-bitrate units for the task of resynthesis [19, 58, 56, 33, 60, 21] , which include losses focused on reconstruction and use an information bottleneck to enforce compression.The discrete units generated by these self-supervised encoders are versatile and fundamental to much of the recent progress in speech research such as Text-To-Speech [54, 29, 50, 47] , joint audio-text foundation models [57, 13, 38] , unsupervised speech recognition [4] , discrete unit resynthesis [48, 19, 58] , text-to-audio [32, 1, 17] , and generative spoken language modeling [8, 26, 34] . Each of these methods operates on audio units exclusively greater than or equal to 25Hz, which has been a frequently cited area for future work to improve on [26] . Recent work [22] has also explored training speech encoder models with coarser units as targets.Extracting Semantic Units from Raw Data Also relevant to our work are several approaches, particularly in vision and audio, that generate emergent semantic clusterings from self-supervised transformer [53] models. In particular, the DINO approach in Caron et al. [10] observes object representations in attention maps through student-teacher distillation. Similar techniques have been also applied to audio to discover emergent syllable boundaries [12, 46] . These behaviors can vary heavily with small changes in pretraining strategy as explored in Darcet et al. [18] . Merging similar features has also been shown to produce significant vision model speedups such as in Bolya et al. [7] .Most similar to our work, Algayres et al. [2] extracted coarse continuous representations for GSLM, however these results trail behind NCLM-based approaches.3 Learning Self-Supervised, Syllable-Like Representations from Raw SpeechIn this section, we describe the bootstrapping process by which we extract low-bitrate speech units.We first describe LossPred, our algorithm to analyze outputs of self-supervised speech model loss functions to generate initial unit boundaries. Following this, we define SylBoost, an agglomeration procedure to iteratively refine these boundaries with student-teacher distillation. We also propose a new algorithm for the efficient extraction of boundaries from feature self-similarity matrices to fix the bottleneck slowing down VG-HuBERT and SD-HuBERT extraction.units once for all training runs.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We provide a Limitations Section to address the assumptions we make for our LossPred algorithm and on the nascent nature of SpeechLMs and their relation to scaling.We explicitly describe why we choose the models we do, and acknowledge where in the field it is hard to draw all-else-held-equal experiments.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
losspred: extracting syllable-like segmentation from relations in hubert's loss: 
 HuBERT has previously been shown to learn phone-like units with its K-means clusterings [28] which have formed the basis of subsequent works on GSLM and unsupervised ASR [4, 34, 26] . However, other work [42, 43] has shown that the representations learned by these models also correlate with higher level structure such as words, despite these structures not immediately appearing during clustering. Our goal in this section is to propose a method that can be applied to a pre-trained HuBERT model in order to automatically extract unit boundaries at the level of syllables or words, rather than phones. Although we apply our method to HuBERT, we expect that it could also be applied to other SSL speech models that utilize a similar loss function such as WavLM [11] or wav2vec2.0 [3] . The crucial commonality between these models is that they all utilize a masked language modeling (MLM) training objective, whereby input speech tokens are randomly masked and the model is trained to predict the masked inputs conditioned on the unmasked inputs.We ground our intuition with the following thought experiment: If the input tokens corresponding to an entire word were replaced with mask tokens, we would expect the HuBERT model loss at these timesteps to be relatively high, as HuBERT would have to jointly predict word identity and the underlying acoustics to predict the missing span. On the other hand, if only the latter portion of a word were masked out, infilling this masked region given the word prefix may be easier by comparison. With this, if we iteratively shift a contiguous mask over a span of tokens and look at the loss, we would suspect to see a strong decrease in the loss throughout the timesteps corresponding to a masked semantic unit (word, syllable, or otherwise) as the beginning or end of the unit was partially revealed to the model. In our experiments, we find that semantic units extracted by this method tend to be syllable-like (both via inspection, and also confirmed experimentally in our segmentation and clustering experiments) and so we focus on these units for the rest of the paper.We consider the setting of having a pretrained HuBERT teacher model and a HuBERT student model trained to predict the quantized contextualized representations generated by the teacher at layer L, as described in Hsu et al. [28] . Formally, given an input waveform W , we extract the teacher labels used to train the student by passing W unmodified into the frozen HuBERT teacher and then quantizing the contextualized representations of layer L with K-Means. We denote these teacher labels as Y {1...T } , where T is the number of tokens outputted by the CNN feature encoder stage of HuBERT. During pretraining, the student model is given a corrupted version of W where tokens after CNN extraction at select times are replaced with a learned 'mask' embedding. We denote these tokens input to the student as X M {1...T } where M = {t 1 , . . . t m } is a contiguous span of masked timesteps. The student is then trained to predict these teacher labels at masked timesteps using a cross-entropy loss, which we denote as E X M t for the loss on Y t , t ∈ M given X M :EQUATIONWe look at the losses of the student model at the end of pretraining, and define the loss prediction matrix C with mask span size parameter s to capture the raw probabilities of the losses that would result from all possible temporal locations of the mask span M :C r,c ∈ R T ×T + =    p(Y t | X M ) | M = {r + 1, r + 2, . . . r + s} if r < c, |r -c| ≤ ⌊ s 2 ⌋, p(Y t | X M ) | M = {r -1, r -2, . . . r -s} if r > c, |r -c| ≤ ⌊ s 2 ⌋, 0 otherwise.We separately calculate the upper and lower triangles of C, relating to the observed waveform being before the mask and after the mask respectively. In the upper triangle, each entry C r,c at row r column c is equal to p(Y t | X M ) given that the mask span in X M starts just after time r. Inversely, in the lower triangle, C r,c is equal to p(Y t | X M ) given that the mask span in ends just before time r. We use a span size s = 50 corresponding to 1 second as this duration is long enough to mask the majority of spoken words, and calculate the upper triangle based on the first 25 tokens of the mask span, and the lower triangle based on the last 25. We choose to use a span of tokens instead of masking all information after a timestep to prevent global information such as speaker information available to the model changing with respect to mask location. However, this limits us to only generating a diagonal span of probabilties as seen in 1. To extract k regions with boundariesB = {b 1 = 1 < b 2 < . . . < b k = T + 1} from C, we adopt the min-cut algorithm discussed in Peng et al. [46] , treating C as the input feature-similarity matrix:EQUATIONBy choosing k to be proportional to the length of the utterance, we can control the sample rate of our boundaries. We explore modifying this parameter in-depth throughout our experiments.LossPred is expensive to run due to having repeat forward passes for sliding windows. To make this efficient, we extract multiple masked spans simultaneously with a gap between spans of three seconds.This results in roughly 200 forward passes of the student model to calculate C on an arbitrarily-sized audio. We also preprocess the audio using a unsupervised voice activity dection model [51] .Given the initial boundaries predicted by LossPred, we follow the paradigm of noisy-student-teacher learning [55] to iterate and extract better representations. Our goal is to "sharpen" the syllabic organization in the feature space of an input student model that initially results from LossPred, as seen on the right of Figure 1 . We choose a pretrained HuBERT [28] or Data2Vec2 [6] to initialize our student and teacher models, with the teacher model parameters held constant.For a set of hypothesized speech segment boundariesB = {b 1 = 1 < b 2 < . . . < b k+1 = T + 1},we group together all temporal tokens between two boundaries into disjoint groupsG i = {t | b i ≤ t < b i+1 }.For notation, we let H t map from t to its corresponding group: t ∈ G Ht . We apply our loss to the features at layer L, which we select based on syllabic correlation as explored in detail in Pasad et al. [43] . This results in student featuresX (L) {1...T } ∈ R d and teacher features Y (L) {1...T } ∈ R dwhere d is the feature dimension.Then the loss, which is applied to each token of the student model, is the mean squared error between the student features X (L) t and the mean of the teacher features in the token's corresponding group:Z := 1 T T t=1   X (L) t - 1 |G Hi | s∈G H i Y (L) s    2 (3)This results in a model with a mean-squared-error feature similarity matrix as depicted in the right side of Figure 1 . We then extract boundaries using a cut algorithm described later in Sec. 3.3, although the cut algorithm from Peng et al. [46] also works. With this, we can generate new pseudolabels and iterate the process again to extract better boundaries, which we perform twice.To extract boundary indices from learned feature representations Peng et al. [46] proposed adapting the mincut approach in Malioutov and Barzilay [35] . However, for speech this approach is slow in practice and difficult to parallelize, bottlenecking our ability to extract boundaries in bulk across the large corpora necessary for downstream language modeling. Inspired by the SylBoost objective, we propose a more efficient approach for extraction: given k + 1 potential boundaries, we seek to choose groups that minimize the sum of the distances from each unit to the mean of its assigned group:B := arg min {b1=1<b2...<b k+1 =T +1} k i=1 b i+1 -1 j=b i   X (L) j - 1 b i+1 -b i bi+1-1 l=bi X (L) l   2 (4)We further restrict the setting by choosing a maximum group length of G tokens, where we choose G = 50 to correspond to one second of tokens, as syllables or words longer than this are fairly rare.With this, we can then split our algorithm into 1) calculating a distance array D ∈ R T ×G , where D t,g is the cost of the group of length g ending at token t and then 2) solving the minimal interval cover from this distance array with dynamic programming. An efficient implementation using PyTorch [44] on CUDA [40] runs in O(k) data-aware sequential steps.4 Syllable-LM: Speech Unit Language Modeling Over Syllable-Like Units 4.1 Language Model GSLM [34] defines a pipeline for modeling raw audio as three stages: 1) Audio-to-unit Tokenization, 2) Running a decoder transformer model on these units, and 3) Decoding the tokens back into a waveform. Like AudioLM and TWIST, we use an autoregressive transformer decoder language model to approximate p(x t | x t-1 , . . . , x 1 ) given an input token sequence x 1 , . . . , x T . We refer to this model as SpeechLM. We train it on clusters which we extract by mean pooling features at layer L, chosen as before, over their boundary groups, followed by K-Means and Agglomerative Clustering to a desired number of discrete units. Like TWIST, we prepend a <BOS> token and make no other special changes. Due to current prevalence of this architecture, we refer to [26] for additional details.To interleave our units, we sort on the start-timestep of every pseudo-syllable unit and mHuBERT-unit in ascending order. To decrease the odds of mHuBERT units appearing before the pseudo-syllable unit corresponding to the same ground truth syllable due to errant SylBoost boundaries, we subtract 0.08s (the length of two mHuBERT frames) from each pseudo-syllable start time before sorting. For the rest of the pipeline, we follow [50] with our syllables as a drop-in replacement for phones. We note We then generate continuations for a sample by 1) Extracting syllable-unit and mHuBERT units from the sample, 2) Sampling syllable-unit continuations from the SpeechLM, 3) Continuing mHuBERT units with our interleaved model conditioned on sample mHuBERT units, sample syllable-units, and continued syllable-units, and 4) Resynthesizing these into speech using the vocoder.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: Although we provide new algorithms, we justify these with experiments instead of rigorous theory, as done in prior work like [26] Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
resynthesis and the vocoder: 
 For resynthesis, we adopt the interleaved decoding strategy from Song et al. [50] to output the mHuBERT units from TWIST [26] , obtaining a waveform by cascading this output into their provided mHuBERT-to-speech vocoder. This interleaving strategy demonstrates superior performance in highdifficulty settings compared to other Neural Codec Lanaugae Models like VALL-E [54] , and so we use it for all resynthesis experiments. Although the cascading procedure may produce additional errors, we choose this approach for the following reasons:1. Text-to-speech systems like VALL-E traditionally start by converting text units into phones using rule-based strategies to improve quality. This indicates that traditional unit-to-speech resynthesis methods might be challenging for our low-bitrate units.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We commit throughout our work to being entirely reproducable and open source. We explicitly cite all architectures used, their hyperparameters, all datasets used, and all models used to calculate metrics. We plan on releasing all SyllablLM model checkpoints and code, and all datasets used are fully open-source.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example Justification: All datasets experimented on contain at least several thousand examples, meaning that experimental significance results would be redundant and insignificant relative to their metrics. Because of this, we follow prior published work for each experiment and do not report these significance metrics. We also do not have the GPU resources to attempt significance errors across multiple training runs or random seeds, as even our smallest models taking 80 hours to train.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
training datasets: 
 We train our tokenizer using LibriSpeech [41] , which contains 960 hours of audio books. We noticed that the agglomeration procedure described in 3.2 converges before all data is used, and so we randomly subsample LibriSpeech to a 100 hour train set and train for five epochs and two iterations for all experiments. We train our SpeechLMs using all of LibriLight [30] , which provides roughly 55k hours of speech. As a note on fair comparison, although AudioLM uses exactly this split of LibriLight, TWIST collects an additional 100k hours of data, totaling to 155k hours.We implement using the OPT [59] flavor of models and default to using 12 layers, an embedding dimension of 768, and learned positional embeddings for both our SpeechLM and our Interleaved-Vocoder-LM. This totals to 90M non-embedding parameters, the same as TWIST-125M. We also experiment with a larger 24 layer 1024 dimension model totaling to 300M non-embedding parameters, the same as AudioLM and TWIST-350M. For all pretraining experiments we randomly crop files to 25 seconds, use a batch size of 80000 tokens, and train for 200k steps, which amounts to the same compute as in TWIST. To make our approach entirely textless, we do not use TWIST initialization.Additional hyperparameters and hardware details are in Appendix A.2.By varying the number of boundaries input to our cut algorithm at each stage in the agglomeration pipeline, we can arbitrarily control our rate of temporal tokenization. We evaluate three main unitrates at 8.33Hz, 6.25Hz, and 5.00Hz, the latter which matches the empirical rate of SD-HuBERT units on LibriSpeech dev-clean. Combining unit-rates with changing the number of clusters generated by K-Means and Agglomeration gives us fine-grained control of the model bitrate. We note that although SD-HuBERT applies a cut algorithm, this is done after thresholding low-magnitude features that emerge from pretraining. As a result, we find that we cannot control the frequency of SD-HuBERT units by changing parameters of its mincut algorithm becuase additional cuts result in close-to-identical representations that map to the same quantized clusters.From prior work, we compare against the AudioLM tokenizer w2v-BERT [14] , and the tokenizer from TWIST which is an open-source HuBERT model pretrained for an additional iteration on a large and diverse set of multilingual data, henceforth mHuBERT. Both of these tokenizers operate at 25Hz followed by Run Length Encoding, which deduplicates repeated units. We additionally reimplement Byte Pair Encoding as done in Shen et al. [49] on the deduplicated mHuBERT units, resulting in the lowest bitrate encoding of speech outside of our model. We grid search and find that the minimum bitrate from BPE is obtained from 4k-16k units and choose 4k units for all experiments (Shen et al.[49] originally operated on 50Hz units, meaning that the 117bps rate obtained here is also new).Because we want to use a 50Hz base encoder to match SD-HuBERT and have fine-grained boundary control during syllable segmentation, we cannot use the 25hz mHuBERT encoder from TWIST.Unfortunately, this means that the quality of the base encoder may be a confounding factor in our SpeechLM evaluation. We choose Data2Vec2-base [6] as a middleground for training SpeechLMs on syllable-like units because we find its quality enables lower bitrates than HuBERT, but it is older and trains on less-data than mHuBERT from TWIST, and it has 6x fewer parameters than w2v-BERT, used by AudioLM. We suspect that applying newer encoders like w2v-BERT 2 from Communication et al. [16] could enable even better performance, which we leave to future work. We initialize Data2Vec2 SylBoost from the same HuBERT loss boundaries as discussed in 3.1.We evaluate the quality of our semantic units with two approaches 1) measuring correspondence with syllables and 2) running speech resynthesis followed by ASR. To measure correspondence with syllables, we use the development and test sets of LibriSpeech [41] and follow the approach from Peng et al. [46] , extracting timesteps for phones on using the Montreal Forced Aligner [36] and then converting these phones into syllables with a rule-based method [25] . We evaluate the quality of syllable boundary detection with a ground truth boundary marked as hit if a proposed boundary is present within a tolerance threshold. We report F1, Precision, Recall, and R score. We ablate F1 scores with tolerance windows of 20ms and 50ms. Given boundaries, we also evaluate the purity of [54] and use the public HuBERT-base CTC ASR model provided by [28] .Table 1 shows our syllabic correspondence results against the prior-state-of-the-art SD-HuBERT [12] and the HuBERT-based feature similarity strategy from [46] . Applying our LossPred followed by agglomeration strategy on either HuBERT or Data2Vec2 improves performance across-the-board except for in cluster purity. Although it LossPred SD-HuBERT in performance, it pushes the boundary for syllable recognition using HuBERT without additional training. We justify using LossPred as a bootstrapping source instead of a HuBERT similarity metric [46, 43] in Table 4 , which we discuss more in Appendix A.4. Improvement across iterations and with different loss initialization can be found in Table 4 . We explore the effects of changing the unit rate on boundary predictions in Table 5 .We compare against prior SpeechLMs and demonstrate the step-by-step changes used to improve unit cluster re-synthesis quality as compared to SD-HuBERT in table 2 . We observe over a 50% decrease in WER and CER by applying our method using the SD-HuBERT base parameters. We further decrease WER by a third by using Data2Vec2, and from there by modifying the unit sample rate and number of clusters can reach as low as 2048 clusters and a WER of 7%. These results demonstrate by far the lowest bitrate we are aware of for 'reasonable-quality' self-supervised-unit resynthesis. Resynthesis for all models we train is done back into mHuBERT-25Hz units, bounding potential quality at a WER of 6.3%.The end-to-end GSLM pipeline is deep, and so it is essential to have metrics to independently evaluate different stages. To evaluate our SpeechLM stage, we follow Lakhotia et al. [34] and use the ZeroSpeech [39] sWUGGY and sBLIMP evaluation. The sWUGGY dataset tasks the model with outputting a higher perplexity on similar but fake spoken words (e.g. brick vs blick). Similarly, the sBLIMP dataset checks syntactic correctness (e.g. the dog sleeps vs the dogs sleeps). We also evaluate the SpeechLM on the tSC set from Hassid et al. [26] , which operates like the ZeroSpeech metrics on a spoken version of the StoryCloze dataset [37] with the last sentence in negative samples randomly chosen. For all metrics we follow prior work and output the mean perplexity per token.The results for SpeechLM metrics are in 3. We reimplement a 90M parameter model using the TWIST mHuBERT units without textually-pretrained initialization (Cold-Init in the TWIST paper) on our data split for an all-else-held equal comparison on unit type. We also train on BPE units as described in 5.3, the next-lowest bitrate units outside of our model. For textual toplines, we train on corresponding LibriLight text transcripts from Kang et al. [31] and convert text to phones and syllables using the same methods as in Section 5.4. We find that training with each of our syllable units improves perfromance across-the-board on sBLIMP and tSC versus comparably-sized models and is competitive against larger models. In fact, with under 90 hours of training, SyllableLM outperforms even the 13B parameter TWIST on sBLIMP. We also beat AudioLM on the full split of sWUGGY with 30x less GPU compute and TWIST model sizes up to 1.3B parameters. On tSC, we observe that SyllableLM large approaches performance of the textual topline, outperforming all models except for TWIST 13B. Due to compute requirements, we are unable to scale further. We notice a decrease in sWUGGY quality with our 5.0Hz units, which we suspect is in part caused by the short length of the dataset audios making input tokenization excessively short. We further ablate these differences in table 6 . We also find that BPE, despite having the lowest bitrate outside of our approach, does not approach the quality gains created by our syllable-like units.To measure the quality of end-to-end continuations, we use the VERT@O-PPX and PPX@O-VERT metrics proposed in Lakhotia et al. [34] , which are shown to be the automatic metrics correlating best with human meaningfulness judgements. VERT@O-PPX measures the diversity of output at the sampling temperature where the perplexity of generated audio transcriptions matches that of the ground truth text, and PPX@O-VERT performs the inverse. Like Lakhotia et al. [34] , we generate 10-second continuations from 1000 randomly sampled 3-second crops from LibriSpeech test-clean, and measure results using their provided environment and parameters. We report these in Table 7 with two sigma error bars, outperforming TWIST 300M and 1.3B. 
limitations: 
 Though speech is a very general medium, there are a number of challenges in adapting our methods to generate low-bitrate units angled towards other audio tasks or other domains such as vision.Our LossPred technique assumes that the semantic units to learn are separable across time, onedimensional, and contiguous. In audio tasks or settings with multiple speakers, sounds or words can occur simultaneously and can't be separated across the time dimension. Images and video are multi-dimensional, not allowing a trivial sliding window approach. Images and video can also have partially occluded or overlapping objects, violating continuity. Furthermore, it is still unclear whether our longer units will be better at scaling to larger datasets, such as the 4.5M hours used by Communication et al. [16] . For example, our semantic units may be losing out on useful paralinguistic features like tone whose impact is only salient on non-audiobooks or at scale. It is also important to note that large textual language models can have harmful effects, such as enabling the generation of misinformation in mass. Although generative spoken language models have not yet caught up to their textual counterparts, it is still necessary to be aware of potential misuses that could arise in the future. 
conclusion: 
 We introduce a new method to tokenize speech for use in GSLMs. We do this by proposing a method to elicit syllabic organization in pretrained speech encoder models, bootstrapping a feature-space agglomeration algorithm from a static analysis of correlations in off-the-shelf teacher and student model losses across time. We demonstrate the success of our technique both in having strong associations with syllables and as an extremlely low-bitrate codec for speech resynthesis. Using this tokenization strategy, we successfully train SyllableLM, a SpeechLM that out-performs comparable state-of-the-art approaches across a diverse range of metrics with a significant inference speedup.We further ablate several design decisions such as quantitization strategy, loss initialization, and the effects of controllability for downstream usecases. Compression is a crucial aspect of learning, and we hope that these significant improvements in the unsupervised learning of low-bitrate speech units can serve as a foundation for approaches towards understanding spoken language and general representation learning.A Appendix / supplemental material Justification: We make concrete claims in the abstract and introduction regarding the performance of our low bitrate units and SpeechLM results. We address each of these explicitly within the experiments.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We mention our system setup in the Appendix in A.2. We include GPU hours used for all main SpeechLM results, and make experiments significantly cheaper than prior work like Borsos et al. [8] , Hassid et al. [26] .Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). data. We do not use Human Judges. We focus purely on generating semantic continuations, making our approach entirely orthogonal to generating realistic data that can mimic speaker voices such as Wang et al. [54] Guidelines: 
code of ethics: 
 • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Generative Spoken Language Models have not yet caught up to their textual counterparts, however it is still important to note that with increased scaling and methods research GSLM systems may eventually be able to reach parity with today's systems in terms of quality. Because of this, we address the ethical considerations of generative models in ??Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The current quality of GSLM systems trails far behind language models in terms of their capabilities for misuse, and this work instead focuses toward the direction of discovering better representation learning algorithms. All output audio comes from a single speaker, and our approach is orthogonal to voice conversion methods.Guidelines:• The answer NA means that the paper poses no such risks.Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: **Summary**

The paper proposes a new tokenization, syllable-like units, of speech on top of the SSL representations for decoder-only models training in generative spoken language modeling. It uses iterative process to generate the segmentation and then apply clustering on top of the mean pooled features. Authors claim that this reduces the bitrate significantly which makes the language modeling more efficient — the model achieves similar performance as SOTA models while being significantly more efficient.

**Arguments taken for the decision**
- **Reviewer kvPb, RUrK, zmRG** pointed out that the idea on the process to produce syllable-like segmentation of speech in an unsupervised manner is new and the paper brings strong contribution in terms of a lower inference computational cost.
- **Reviewer kvPb** thinks that the design of experiments is sound. In addition **Reviewers RUrK, zmRG** pointed out that Authors report extensive experimental results.
- **Reviewers RUrK, zmRG** pointed out that LossPred is slow in evaluating the loss prediction matrix, may not be applicable to music, speech with background, and is kind of really heuristic. **In the rebuttal** Authors agreed with these points.
- **Reviewer zj85** found it is hard to follow submission and pointed to a lot of places which were unclear. Authors clarified some of the points **in the rebuttal**.
- **Reviewer kvPb** believes that paper needs substantial revision and thus could not be accepted at the current state. During **rebuttal** Authors provided clarifications and the changes they can make to improve readability.
- **Reviewer zmRG** pointed out that clear demonstration of the method efficiency is needed and also comparison with other unit quality reconstruction baselines. **In the rebuttal** Authors pointed out that efficiency is given in Tables, while the acoustic quality of proposed units reconstruction is not a task best suited for the paper.
	- AC comment: I disagree with Authors on the latter point, as otherwise it means that you need to have in addition very strong acoustic tokens to avoid losing information you lose with lowering bitrate even for semantic HuBERT tokens. Moreover you should be doing more heavy work on the detokenization part then, which can be inefficient.
- **After rebuttal discussion:** both Reviewers kvPb and zj85 still point out that substantial changes are needed for the paper readability  as otherwise it is hard to follow paper despite Authors' points.

**Recommendation**

The reviews for the paper are mixed, and during AC-reviewers discussion period **Reviewers kvPb and zj85** are mainly concerned with the paper writing style and substantial changes needed to improve the readability and clarity of the paper. In addition to reviewers I also did a pass over the paper and I agree with reviewers: in many places it is hard to follow, a lot of things are mixed up and language in the paper should be improved. Based on that **I recommend rejection of the paper in its current form**. I agree with reviewers that the paper has valuable contributions though articulation of results is not clear right now. 

**Recommendation to Authors**

I encourage Authors carefully rework the text with clear definitions, formulas, summaries and ideas formulation, having explanation of all main pieces clearly assuming less background knowledge from the readers (e.g. formulating things exactly rather than just citing - e.g. how you perform even unit to speech mapping), and then resubmit to the next venue. Also I encourage Authors to look into dependence on the acoustic tokens extraction and investigate deeper if the proposed method applicable across different speech tokenization or it is specific to HuBERT ones only. This will strengthen the paper significantly.

Reviews: review: paper_summary: paper_summary: This paper proposes an approach for extracting syllable-like units from speech SSL models for use in a transformer-based language model. The motivation is that, compared to baseline acoustic units, which tend to mimic phonetic units in their time resolution, syllable-like units have lower time resolution, which makes them easier to model using techniques from the language domain. The authors propose an adaptation of the SD-HUBERT approach to extract units that can be used in Generative Spoken Language Modeling.
strength_weakness: strength_weakness: Overall, I found the submission difficult to follow. Please see my additional comments below.

line 2 -> Transformers do not require the inputs to be tokenized. The tokenization step is performed so that we can use language modeling techniques in speech.

line 18 -> Generally speaking, there is no requirement for the SSL representations to be powerful or abstract. 

Line 20 -> I don't see how the example of young children motivates your SSL description from the previous sentence; the transition is incoherent.

line 22 -> What does performant mean in this case? What is the connection between composing highly realistic text and the ability of a model to provide features for a downstream task? You seem to conflate the two goals, even though they are not necessarily the same.

line 23 -> The statement on this line is not clear. Several successful speech language model methods were introduced in the literature, what about previous approaches that make them fail? Please consider clarifying.

line 31 -> The temporal resolution impacts the LM part of the problem. Why is it important if we want to extract features for a downstream task?

line 37 -> What does "syllable-like" mean in this case? Can you elaborate on the time resolution it represents? Why is it important to start with a "syllable-like" unit? What makes it suitable for GSLM? What challenges from prior work are you addressing when using "syllable-like" units?

Line 38 -> I would refrain from using words like "breakthrough" and instead let the reader decide if the improvement is indeed a "breakthrough."

line 48 -> I disagree with labeling your method as "train-free" since it relies on a pre-trained HuBERT model. 

line 51 -> The distinction between the first and second contributions needs to be clarified. If the boundaries from the first contributions are not good on their own, then why mention them as a contribution? 

Line 102 -> It is not clear how/where you do the masking. Do you do it on the raw input, mel-spectrogram, or the extracted features? 

line 113 -> Shouldn't the approach be "train-free"? Why do we have a student/teacher model that we are training?

line 147 -> The authors must refine the motivation for why syllabic units are useful for this application. Why not use word units instead? 

line 189 -> Superior compared to what?

line 198 -> I suggest leaving any experimental details to the experiments sections.

Table 1 -> Can you try any non-neural baselines for boundary detection? What would the performance be if we used heuristics based on energy, zero-crossing rate, or changes in Prosody to get rough boundaries?

Table 1 -> What makes Data2Vec2 better than HuBERT for extracting boundaries?

Table 1 -> What happens if you apply SylBoost to Feat-Sim?

Table 1 -> Please describe the metrics and abbreviations in the captions.

Table 2 -> What does the underline represent?

line 221 -> Implement what exactly? Please re-write the sentence.

line 237 -> typo

Table 3 -> What does the underline represent?

*Estimated.-> What does estimated mean? If prior work does not explicitly give this information, then it is better to leave it out.

line 263 -> What is the R score?

line 281 -> Please present the tables in the order they are referenced in the text; you currently jump from Table 1 to Table 4 and then go back to Tables 2 and 3.

line 340 -> Communication is not the last name of the first author from [16]
questions: questions: How does the performance on the syllable boundary detection task relate to the unit re-synthesis and language modeling performance? In other words, is having well-defined boundaries necessary for the speech language modeling task?

It is difficult to disentangle the effect of the number of units and temporal resolution on the overall performance in Tables 2 and 3. Despite having lower resolutions, the results from the proposed methods are not much better than baseline methods (e.g., AudioLM and TWIST). Can the authors elaborate on this?

What happens if we directly use LossPred units in the LM framework? In other words, how important is the boundary refinement stage for the language modeling task?

Keeping everything fixed, how does the number of units impact the re-synthesis and language modeling results?

How does the masking probability and span used when training the HuBERT model impact the quality of the discovered boundaries, and what impact does it have on your approach?
limitations: limitations: Can the authors comment on the trade-off between resolution and ease of modeling (and quality)? What do we lose/gain using syllable-like speech units in a language modeling paradigm?
paper_summary: paper_summary: This paper studies learning low bitrate speech units that preserves semantic information. As presented in the paper, the proposed approaches achieve SoTA performance on tasks like ASR and ZeroSpeech. The proposed approach also shows benefits in terms of compute resources — as claimed by the authors, 30x faster to train, and also benefits in terms of inference and transmission due to low bitrate.
strength_weakness: strength_weakness: Demonstrating efficiency: As efficiency is also one selling point of the paper, it would be great if the authors can demonstrate the training efficiency and low-bitrate benefits in a more comprehensive way, like visualizing the GPU training time vs Performance, and also bitrate vs unit quality for certain tasks.



Limited use cases: The proposed approach focuses on learning semantic units for speech applications. It’s unclear if the proposed methods can be applied to other important non-speech use cases like understanding acoustic environment, and understanding speaker’s identity and emotion. 



Understanding Unit Quality: To demonstrate the unit quality for synthesizing the audio and for generation, should the author also compare with other related works (like [1] and [2]) in terms of reconstructing the original signal? Like in [1] (see Table 1), the authors compare the different approaches in terms of reconstruction performance using a couple of metrics like MEL, STFT and ViSQOL score, and also semantic task performance.

[1]: https://arxiv.org/abs/2405.00233
[2]: https://arxiv.org/abs/2306.06546
questions: questions: See Weaknesses section
limitations: limitations: Not aware of potential negative societal impacts
paper_summary: paper_summary: This paper first introduces an algorithm named LossPred that generates syllable-level speech segmentation without any training or supervision. The algorithm works by analyzing the prediction loss of speech tokens under different mask positions.

With the initial boundaries proposed by LossPred, the paper proposes further training a pretrained HuBERT / data2vec2 model by minimizing the sum of squared distances between feature vectors of each token and the average of feature vectors within the corresponding segment. This process is called SylBoost, and it further improves syllabic segmentation performance and efficiency.

Finally, the paper proposes training a Generative Spoken Language Model (GSLM) with the speech tokens obtained from quantized SylBoost units. Compared to existing GSLMs trained on other discrete representations, SylBoost encodes speech into much shorter sequences, significantly boosting training and inference efficiency.
strength_weakness: strength_weakness: 1. As pointed out by the authors, the proposed LossPred and SylBoost methods seem to be restricted to speech representation learning. It might be difficult to apply these methods to music, singing voice, speech with noisy backgrounds.
2. LossPred is slow in evaluating the loss prediction matrix. Each sentence requires about 200 Transformer network evaluations.
3. LossPred is highly heuristic. There seems to be no theoretical guarantee that the HuBERT model combined with LossPred reveals syllabic boundaries instead of revealing only phoneme or word boundaries.
questions: questions: 1. The equation above line 127 is rather confusing. The description of the procedure for computing the loss prediction matrix is confusing in general.
2. In LossPred, $k$, the number of segments, is chosen to be proportional to the sequence length. What happens when a speaker is speaking very fast or very slow? It seems that $k$ should be determined by the number of syllables in the utterance.
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: This paper proposes a two-stage speech language model with semantic tokens and acoustic tokens similar to AudioLM ([Borsos et al., 2022]).
-   The semantic tokens come from a speech tokenizer that can group a variable number of frames into a single token. To train such a speech tokenizer,
    1.  This paper first takes inspirations from syllable-like structures uncovered from HuBERT, and produces an initial segmentation (Section 3.1).
    1.  An iterative process is then applied to improve the segmentation (Section 3.2).
    1.  Finally the tokens are obtained by clustering of the mean-pooled frame features (Section 4.1).
-    The acoustic tokens are identical to the HuBERT-based tokens in ([Hassid et al., 2023]), referred to as "mHuBERT" in this paper.

Experiments with the proposed model demonstrate the following when compared to previous work,
-   Better unsupervised syllable segmentation
-   Lower speech reconstruction WER
-   Better or competitive accuracy in speech language modelling tasks (sWUGGY, sBLIMP, tStoryCloze) with lower compute
-   Better speech continuation quality

[Borsos et al., 2022]: https://arxiv.org/pdf/2209.03143 "AudioLM: a Language Modeling Approach to Audio Generation"
[Hassid et al., 2023]: https://proceedings.neurips.cc/paper_files/paper/2023/file/c859b99b5d717c9035e79d43dfd69435-Paper-Conference.pdf "Textually Pretrained Speech Language Models"
strength_weakness: strength_weakness: I think the method and results in this paper would make a good paper for NeurIPS, however I cannot make a recommendation for acceptance because this paper needs substantial revision to improve its readability. A non-exhaustive list of issues making the paper hard to follow includes the following,
-   References to items not yet introduced
    -   Lines 153-154, the phrase "our loss" make it sound like a referrence to the masked language model loss discussed in the previous sub-section, whereas in fact it is referring to Equation (3), a yet-to-be-introduced loss for SylBoost.
    -   Lines 159-162 give a very vague description of the "similarity matrix" and the "cut algorithm" which can only be known if the reader has already seen the subsequent Section 3.3.
    -   Starting at line 188, Section 4.2 makes repeated references to "mHuBERT". "mHuBERT" appears to be name given to the acoustic tokens in ([Hassid et al., 2023]) by this paper (line 241). ([Hassid et al., 2023]) itself does not use this name, so an ordinary reader would not be able to tell what an "mHuBERT" model is when they work through Section 4.2.
-   Confusing terminology
    -   "pretraining": This paper makes a liberal use of the term "pretraining" to the point it's very difficult to tell which is the model being "pretrained". For example,
        -   Line 113 mentions a "pretrained HuBERT teacher model", then line 119 says "during pretraining, the **student** model ...". The teacher and the student are presumably not trained at the same time, yet the use of "pretraining" in this context make it appear that the contrary is happening.
        -   Line 225 says "for all pretraining experiments". A reader will have to look really closely to see this means "training of the speech LM", not "pretraining HuBERT, etc".
    -   "Agglomeration" vs "SylBoost": This paper appears to use these two terms interchangeably. Agglomerative clustering is apparently also used  (line 183). This makes it difficult for the reader to tell when "agglomeration" is mentioned, whether the authors intend to refer to SylBoost or just the clustering.
-   Confusing equation
    -   The unnumbered equation between line 126 and line 127 defines the similarity matrix from MLM probabilities. It makes reference to
$Y_t$ without specifying which $t \in M$ is used to define $C_{r,c}$. As a result, after having read the paper 6 times over, I still do not know how to compute $C_{r,c}$.
-   Writing style
    -   Overall the writing style of this paper is very wordy, inconcise and disorganized. Often the same message can get through with far shorter sentences. Most of the paragraphs read like a dump of the stream of consciousness of the author instead of a technical document intended for actual readers. For example,
        -   Lines 102-112 would be a lot easier to understand with formal notations and a concrete example.
        -   Lines 127-131 appear to be a mere repetition of the equation above, without any new information.
        -   Lines 242-255 contain a large amount of disorganized modelling details.

[Hassid et al., 2023]: https://proceedings.neurips.cc/paper_files/paper/2023/file/c859b99b5d717c9035e79d43dfd69435-Paper-Conference.pdf "Textually Pretrained Speech Language Models"
questions: questions: -   Is LossPred really necessary?
    -   While Table 4 shows that LossPred is a better initialization strategy for SylBoost, would a cheaper initialization strategy (like feature similarity or even random) produce equally good segmentation given more iterations?
    -   Suppose the answer to the previous question is yes, would it make sense to run SylBoost but using the final activation of HuBERT instead of an intermediate layer (line 154)? The final activation may be better semantic features.
limitations: limitations: The authors adequately addressed the limitations.

Comments: comment: Thank you very much for responding to all of my comments and questions.
Thank you for the clarifying comments. I will incorporate this information in my final feedback.
Thank you for taking the time to address my review! I have read your rebuttal and will incorporate the new information in my final feedback.