SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
Authors: Maxime Poli
Mahi Luthra
Youssef Benchekroun
Yosuke Higuchi
Martin Gleize
Jiayi Shen
Robin Algayres
Yu-An Chung
Mido Assran
Juan Pino
Emmanuel Dupoux
Venue: Accepted by TMLR ()

Abstract: The parallel advances in language modeling and speech representation learning have raised the prospect of learning language directly from speech without textual intermediates. This requires extracting semantic representations directly from speech. Our contributions are threefold. First, we introduce SpidR, a self-supervised speech representation model that efficiently learns representations with highly accessible phonetic information, which makes it particularly suited for textless spoken language modeling. It is trained on raw waveforms using a masked prediction objective combined with self-distillation and online clustering. The intermediate layers of the student model learn to predict assignments derived from the teacher's intermediate layers. This learning objective stabilizes the online clustering procedure compared to previous approaches, resulting in higher quality codebooks. SpidR outperforms wav2vec 2.0, HuBERT, WavLM, and DinoSR on downstream language modeling benchmarks (sWUGGY, sBLIMP, tSC). Second, we systematically evaluate across models and layers the correlation between speech unit quality (ABX, PNMI) and language modeling performance, validating these metrics as reliable proxies. Finally, SpidR significantly reduces pretraining time compared to HuBERT, requiring only one day of pretraining on 16 GPUs, instead of a week. This speedup is enabled by the pretraining method and an efficient codebase, which allows faster iteration and easier experimentation. We open-source the training code and model checkpoints at https://github.com/facebookresearch/spidr.

Published in Transactions on Machine Learning Research (12/2025)
SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
Maxime Poli Mahi Luthra Youssef Benchekroun**, Yosuke Higuchi', Martin Gleize, Jiayi Shen?, Robin Algayres', Yu-An Chung', Mido Assran,, Juan Pino", Emmanuel Dupoux! EENS-PSL, EHESS, CNRS, FAIR at Meta
'work done in part while interning at Meta "equal contribution
Reviewed on OpenReview: https://openreviev wttp://eii....eeerrooomom/////aa
Abstract
The parallel advances in language modeling and speech representation learning have raised the prospect of learning language directly from speech without textual intermediates. This requires extracting semantic representations directly from speech. Our contributions are threefold. First, we introduce SpidR, self-supervised speech representation model that efficiently learns representations with highly accessible phonetic information, which makes it particularly suited for textless spoken language modeling. It is trained on raw waveforms using a masked prediction objective combined with self-distillation and online clustering. The intermediate layers of the student model learn predict assignments derived from the teacher's intermediate layers. This learning objective stabilizes the online clustering procedure compared to previous approaches, resulting in higher quality codebooks. SpidR outperforms wav2vec 2.0, HuBERT, WavLM, and DinoSR on downstream language modeling benchmarks (sWUGGY, sBLMMP, (SC). Second, we systemal icalyy evaluate across models and layers the correlation between speech unit quality (ABX, PNMT) and language modeling performance, validating these metrics as reliable proxies. Finally, SpidR significantly reduces pretraining time compared HuBERT, requiring only one day of pretraining on 16 GPUs, instead of a week. This speedup is enabled by the pretraining method and efficient codebase, which allows faster iteration and easier experimentation. We open-source the training code and model checkpoints al https:
1 Introduction
Recent progress in self-supervised learning (SSL) (Mohamed et al., 2022) has opened up the intriguing possibility of learning language models like children do, i.e., directly from audio signals, without any text. This approach substitutes the standard text tokenizer with an SSL speech tokenizer, and trains the language model using next token prediction on speech tokens (Lakhotia et al., 2021; Dunbar et al., 2021). This opens up interesting possibilities for textless NLP systems that could address the large number of human languages that do not have sufficient textual resources allow for the standard text-based or ASR-basod approach (Polyak et al., 2021; Kharitonov et al., 2022; Kreuk et al., 2022; Nguyen et al., 2023; Borsos al., 2023). In this work, use the term spoken language model (SLM) to refer to language models that trained without any text. Other authors have used this term in broad range of situations that typically mix speech and text. We use it to refer to the pure spoken language models of the taxonomy proposed by Arora et al. (2025).
Early research on SLM (Lakhotia et al., 2021; Dunbar et al., 2021) used speech SSL models trained with a predictive objective (van den Oord et al., 2019; Hsu et al., 2021) as speoch tokenizers. Since then, SSL models have grown in diversity and coverage, revolutionizing many aspects of speech processing for large variety of downstream tasks (few-shot ASR, few-shot speech classification, speech compression, etc.) (wen Yang et al.,

---

Published in Transactions on Machine Learning Research (12/2025)
Student
Masking - x
Downsamplig
Head 6
For all layers - kS
Cross-entropy
Codebook C
y
Teacher
Figure 1: Architecture of SpidR. The downsampling module, a stack of convolutional layers, transforms the speech waveform into 20ms frames. The student and teacher are Transformers with L 12 layers. For every layer in the last. K ones, the student predicts-throughh a prediction head t the nearest neighbor codebook assignment on the masked positions from the teacher at the same layer. The downsampling module, the student and the prediction heads are updated by gradient descent (in blue). The teacher is an exponential moving average (EMA) of the student, and the codebooks are updated with an EMA of the embeddings of the teacher (in gray).
2021). Yet, progress SSL models for downstream SLM has been slow. We still have little understanding of what constitutes good speech units, and current SLMs still lag behind their textual counterpart in terms of performance and scaling laws (Cuervo & Marxer, 2024).
One popular hypothesis regarding speech units for SLM is that they should be abstract, in the sense of removing non-linguistic information like speaker identily or expressive variation, to be close to existing linguistic units like phonemes. This is why they are typically evaluated by phoneme classification metrics like ABX discriminability (Schatz et al., 2013; Schatz, 2016) PNMI (Hsu et al., 2021). Existing speech units capture phonetic information (Choi et al., 2024), but they lack robustness to both acoustic variations (Gat et al., 2023) and contextual variations caused by coarticulation (Hallap et al., 2023). As result, these units align more closely with contextual phone states (Young et al., 1994) than with actual linguistic units (Dunbar et al., 2022). To address these limitations, some approaches further process units derived from SSL models to make them larger and closer to syllables or words (Algayres et al., 2023; Baade et al., 2025; Visser & Kamper, 2025). However, training new SSL models from scratch costly. For example, HuBERT is trained in several passes, and requires alternating between model training and clustering, with some manual decisions to be made between each pass to select the intermediate layer used to compute the targets. DinoSR (Liu al,, 2023a) is recent single-pass alternative to HuBERT that demonstrates better phonetic discriminability, making it suitable candidate for spoken language modeling. Its original implementation still requires a week of training, which limits the exploration of its training properties.
The contribution of this work has three aspects: technical, scientific, and methodological. On the tochnical front, we provide a minimal pure PyTorch codebase for training speech SSL models that accelerates training by order of magnitude: full pretraining on LibriSpeech now requires only one day 16 GPUs. This
2

---

Published in Transactions on Machine Learning Research (12/2025)
contribution should unlock research on speech SSL models by enabling faster iterations and casier experi- mentation. Scientifically, we leverage this accelerated codebase to develop SpidR, a new architecture that learns strong representations for spoken language modeling in a single pass. While inspired by DinoSR's 's architecture, SpidR's learning objective makes pretraining significantly more resistant to codcbook collapse. Our approach incorporates self-distillation and online clustering with pseudo-labels derived from codebooks at the intermediate layers of the teacher encoder. However, it differs from DinoSR a key way: instead of using only the student's final layer to predict the assignments for each teacher intermediate layer, we use the student's ' own intermediate representations. Our experimental results demonstrate that SpidR outperforms both HuBERT and DinoSR on zero-shot spoken language modeling metrics. Finally, on methodological note, we systematically evaluate across models and layers the correlation between speech unit quality (ABX, PNMI) and rownstream LM performance al several levels (lexical, grammatical, and semantic). We find that unit quality strongly predicts downstream scores, validating these metrics as reliable proxies for SLM performance that enable rapid model development.
2 Related Work
Self-supervised speech representation learning. Modern self-supervised learning for speech emerged, one hand, from the desire to leverage large amounts of unlabeled speech data to learn representations that could then be fine-tuned on smaller labeled datasets for a variety of downstream tasks, mainly ASR. On the other hand, it also follows a long history of approaches for unsupervised pattern discovery in speech (Park & Glass, 2008; Jansen et al., 2010; Kamper et al., 2015; 2016; Dunbar et al., 2022). Current mot have evolved from early autoregressive models (van den Oord et al., 2019; Schneider et al., 2019; Chung & Glass, 2020) to predominantly bidircctional masked prediction approaches (Devlin et al., 2019) that leverage surrounding unmasked context. The wav2vec 2.0 (Baevski al., 2020) model is trained with contrastive learning wetn the contextual representations and quantized units. Tts architecture also established standard backbone that has been adopted by most subsequent approaches, the key differentiation between models lying in how they compute the self-supervised loss and derive training targets. HuBERT (Hsu et al., 2021) introduced an iterative approach where pseudo-targets are obtained from a previous iteration of the model, alternating between clustering and pretraining. Baevski et al. (2022) use self-distillation to derive the targets, an approach followed by DinoSR (Liu et al., 2023a) but with discrete targets instead of continuous embeddings. In eeffdistillation, the teacher and the student architectures are identical, and the teacher is usually exponential moving average of the student (Grill et al., 2020; Caron et al., 2021). Using discrete targets is also the dominant approach in SSL for speech, employed by all models cited above except data2vec, as well as other architectures such as BEST-RQ (Chiu et al., 2022) and w2v-BERT (Chung et al., 2021). ). Although designed primarily for ASR, these models have also been repurposed for spoken language modeling: their reprosentations capture linguistic content, allowing them to serve as discrete specch tokens, a usage more rooted in the unsupervised pattern discovery tradition. Our work builds directly on DinoSR, maintaining the established architecture while focusing specifically on improving the stability of the learning objective rather than architectural innovations. The pseudo-targets targets in data2vec, DinoSR and our work are derived from intermediate layers, an approach also explored by Chung et al. (2021); Wang et al. (2022). Another line of research has focused on enhancing robustness through additional training objectives addressing acoustic or speaker variations, with approaches like WavLM (Chen et al., 2022), Spin, or R-Spin (Chang et al., 2023; Chang & Glass, 2024). Our target application in this work is not ASR or other supervised downstream tasks, but spoken language modeling, which requires different properties from representations. Word or phoneme information should be directly accessible from them, without any additional training. Particularly relevant to our goals, Chang et al. (2025) fine-tune HuBERT to learn codebooks optimized for spoken language modeling. We focus in this work on single pass pretraining without additional fine- tne-tunnng steps, making our approach complementary to these specialized adaptation methods.
Efficient speech representation learning. With the increasing cost to train self-supervised speech models, researchers have explored various approaches to simplify the training procedure and accelerate training time. For instance, Bacvski et al. (2023) improve the sample efficiency of data2vec by training with multiple masked versions of the same sample. For HuBERT specifically, several efficiency improvements have been proposed: Lin et al. (2023a) and Yang et al. (2023) replace the learned downsampling module
3

---

Published in Transactions on Machine Learning Research (12/2025)
by mel-filterbanks and use a cross-entropy loss, while Chen et al. (2023a) use an existing ASR model to extract the targets for the first training iteration instead of MFCC features. Yang et al. (2025) take a different approach by replacing the encoder with a Zipformer (Yao et al., 2024). It's worth noting that training HuBERT also requires extracting features and training K-means between each iteration, which can consume a substantial portion of the total training time (Zanon Boito el al., 2024) a inherent limitation that architectural changes alone cannot address. Additionally, most models derived from wav2vec 2.0, including HuBERT, were originally pretrained using the fairseq library (Ott et al., 2019). While fairseq initially provided essential solutions for distributed training, mixed precision, etc., these features now exist natively in PyTorrh (Ansel et al., 2024), and fairseq is no longer maintained. Our streamlined Py PyTocc--native implementation of SpidR and DinoSR reduces compute requirements, enables faster iteration during development, and provides a hackable foundation for future research.
Spoken Language Modeling. Generative text pretraining has inspired a new family of speech generation models. By proposing to quantize self-supervised representations, Lakhotia et al. (2021) rephrased speech generation as a language modcling task. The discrete tokens function as phonetic units, due to their accessible phonetic information (Nguyen et al., 2022; Sicherman & Adi, 2023; Yeh & Tang, 2024), and serve as inputs to train a Transformer decoder. Borsos et al. (2023) combined these units with audio codec tokens (Zeghidour et al., 2022; Défossez et al., 2023) to capture finer acoustic details. Non-phonetic information has also been incorporated with phonetic units to capture style or prosody (Kharitonov et al., 2022; Nguyen et al., 2025). Chen et al. (2025); Zhang et al. (2024) and Défossez et al. (2024) only use units from audio codecs. Notably, both Speech and Moshi distill HuBERT and WavLM representations into their first quantizer to guide it loward capluring linguistic information; without this, they would only encode local acoustic content. Moshi even uses a split quantizer to disentangle semantic and acoustic tokens inside the codec. Despite their ability to learn linguistic structures (Dunbar et al., 2021), purely speech-based models have exhibited limited factual knowledge and reasoning abilities. This prompted the development of hybrid speech-text models (Hassid et al., 2023; Nguyen et al., 2025; Défosses et al., 2024; Cuervo el al., 2025; Maimon et al., 2025b). A parallel research direction focuses on improving the speech units themselves (Algayres et al., 2023; Baade et al., 2025), as speech representations with more accessible phonetic information significantly improves linguistic knowledge (Poli el al., 2024). In our work, we deliberately focus on pure spoken language modeling from raw audio to isolate and evaluate the specific contributions of our speech encoder.
3 Method
As illustrated in figure 1, SpidR leverages self-distillation and online clustering, making predictions at multiple layers of the network. It is based on DinoSR, but with a novel learning objective. The student layers directly predict the assigument given by the corresponding codebook, instead of having multiple prediction heads at the end of the student encoder, which avoids codebook collapse.
We first extract fcature frames x (2I,..., .n.) from a speech utterance, with x; E Rdd using a convolutional block. We sample a random mask M c (1,..,, n), with the sampling procedure from Baevski et al. (2020), and build x, corrupted version of where for each E M, H, has been replaced by a learned mask embedding. The student encoder is Transformer (Vaswani et al., 2017) with L layers, trained to predict the psoudo-labels derived from a teacher at the masked positions. Let 2² = (zf...., .... xx) be the output of the student encoder at layer k from x As in previous works, this is the output of the feed-forward net work in the Transformer block, before the final residual connection and layer normalization. The prediction is done at the last K layers of the encoder. The label prediction at frame and intermediate layer i is
पे = = E 10,
(1)
where ** is the prediction head at layer k with L K < < L, and V is the number of labels. The prediction head is made of a single linear projection followed by softmax. To derive the pscudo-labels, we first food the unmasked frames x to the teacher. Let z be the output of the teacher encoder at intermediate layer k after instance normalization.
4

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: Summary of evaluation metrics. In section 4.3, we evaluate speech representations with ABX discriminability over triphones and MAP over words. We then compute in section 4.4 the quality of the derived discrete units and the downstream SLM performance the lexical, syntactic, and semantic levels.
Metrie
Level
Task
Example Interpretation
Contimuous representations
d(æ, < d(x,b)) b)?
Within speaker: (batæ, bets, bats, Across speaker: (bals betsi: batsz)
ABX
Phonetic
a E A,b E B,
x a A.
Average precision of retricving same-word embeddings.
MAP words
Lexical
# # P(i)
Discrete units quality
d(z,a) < d(z,b)? a a A,b B, x #
ABX
Phonetic
Same as ABX continuous representations but with one-hots.
How much knowing the units reduces uncertainty about the gold phonemes y.
PNMI
Phonetic
2) (H())
Spoken language modeling
sWUGGY
Lexical Syntactic
p(a) > p(b)?
(brick, blick)
sBLIMP
p(a) > p(b)?
(dogs eat meat, dogs eats meal) (paragraph with coherent ending, same but with uncoherent ending)
tSC
Semantic
p(a) p(b)?
The one-hot target label at frame i and layer l is
if u = arg min Isugy 1
c
v E (0, 1) where for 1se v u (2
(2)
otherwise
where C is the codebook associated to layer k, with V codewords. The model is trained predict the target labels from the teacher the masked positions by minimizing the cross-entropy
Σ
y logg!. gf
(3)
M
REM L-KSLEL
The teacher is updated with an exponential moving average (EMA) the student: the update at step & is freacher + 3.0reacher + (1 Studdet Following Liu et al. (2023a) and Bacvski et al. (2022), the positional embeddings of the teacher copied from the student, not updated by EMA. All activated codewords are updated with EMA of the teacher output embeddings:
Ts + +1-17) - r) Eitoot 2 if fi 11
8
otherwise,
T + (1 - T) Eicgg
if ( (wt), = 13 0, otherwise,
k x-
(4)
n1
8
Ch
n
where st is initialized randomly and n to 1, and is constant decay parameter.

---

Published in Transactions on Machine Learning Research (12/2025)
Layer 5
Layer
Layer 7
Layer 8
Perplexity
200
175
150
Layer 9
Layer 10
Layer 11
Layer 12
Perplexity
200
175
150
200k
400k
0
200k
400k
200k
400k
0
200k
400k
Training steps
Model
Perplexity
RpidR
DinoSR
From codebook
From prediction
Figure 2: Codebook and prediction perplexities during training for SpidR and DinoSR on LibriSpeech dev-clean, with K codebooks. For each layer k, the codebook perplexity is computed over each batch with p x and then averaged across the dataset. The prediction perplexity uses p 5²
Note that with this update procedure, all embeddings 2 are used to update the codewords, but the non- activated codewords do not move. The main change from Liu et al. (2023a. is that the predictions are now aligned with the target layer. The output of layer h of the student is used to predict the label derived from layer & of the teacher, whereas DinoSR, uses only the output of the last layer of the student encoder with ú!¹ = =E,)). Bacvski et al. (2022; 2023) also used the intermodiate representations to train a SSL speech model, but their case there was only one prediction head the top of the student, trained to predict the average of the representations of the last K layers of the teacher. Our approach is reminiscent of the deep supervision literature (Lee et al., 2015), but in self-supervised learning context.
4 Experiments
We pretrain SpidR, compare its training stability to DinoSR section 4.2, and evaluate the phonetic and word discriminability of its representations in section 4.3. We then extract discrete tokens and train spoken language models. In section 4.4, show the improvement of SpidR on zero-shot spoken language modeling task over other SSL encoders in identical conditions. Finally, in section 4.5 we compare the training time of SpidR to that of HuBERT, DinoSR, and previous work on efficient SSL.
4.1 Setup
Pretraining. The architecture follows the standard backbone from Baevski et al. (2020), and make minimal changes from DinoSR. The model has a feature extractor with seven temporal convolutions and a projection layer, downsampling the 16 kHz input speech to 50Hz features of dimension d = 768. The student and teacher are BASE size Transformer encoders with L = 12 layers. The prediction is done at the top K = 8 layers, using codebooks with V = 256 codewords. We protrain with 960 hours of speech from LibriSpeech (Panayotov et al., 2015). To maintain a fair comparison with DinoSR, we keep the same total batch size of minutes of audio across 16 GPUs. The codebook decay parameter is kept constant: == 0.9. The student encoder and feature extractor are optimized with A (Loshchilov & Hutter, 2019) for 400k steps. We use the same learning rate schoduler as Liu al. (2023a), with a warmup from x 10-6 to 5 10-4 within
Prrevious work in SSL for speech (Baevski al., 2020; Hsu al., 2021; Liu al., 2023a) reported using Adam, but the Adam optimizer in fairseq that was used is actually implemented as AdamW.

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: Zero-shot evaluation of self-supervised speech representations (in %, chance level 50% for ABX). All models are trained on LibriSpeech 960h. For each model, the selected layer the one with the lowest average ABX. The best scores are in bold and second best are underlined.
ABX within speaker +
ABX across speaker +
MAP words f
Model
Layer
dev-clean
dev-other
dev-clean
dev-other
dev-clean
dev-other
wav2vec 2.0
6
4.47
5.63
5.25
7.82
44.81
31.92
HuBERT
11
3.38
4.26
4.01
6.49
46.07
33.37
data2vec
4
4.41
5.49
5.07
7.40
39.34
27.90
data2vec 2.0
1
5.13
5.77
5.72
7.53
69.38
53.49
DinoSR
5
4.05
5.11
4.72
7.29
63.02
45.86
DinoSR
5
4.29
5.56
5.22
8.56
51.35
33.74
SpidR
6
3.32
3.74
3.66
4.95
66.50
55.26
t Our re-implementation.
the first 12k steps, held constant until mid-training, and then exponentially decayed to 5 10-6, We freeze the feature extractor after 200k steps. See appendix A in appendix for more details on the model and the hyperparameters.
We found during preliminary experiments that the norm of the weights of the Q, K, projections in the attention layers could increase along training, and potentially lead to spikes in the loss and model collapsc. Removing the biascs in those layers fixed this issuc, with no negative impact. We also modify the schedule of the decay parameter of the teacher Be. Instead of the harmup-and-constane schedule of Baevski et al. (2022) and Liu et al. (2023a),
take smoother approach and set the decay at step t to be 3, = 1- (1 - 86) exp(-t/TT), where T 10000 is timescale parameter and 30 0.999. See appendix C in appendix for an ablation from DinoSR to SpidR.
Discrete units. We extract the embeddings from the layer with the best phonetic discriminability. The output representations this layer are then quantized to derive the discrete units. We consider two quantization methods. We first use vector quantization with K-means clustering (Nguyen et al., 2020; Lakhotia al., 2021), training it with the train-clean-100 subset of LibriSpeech. For DinoSR and SpidR, also consider using the codebook predictions, by taking the assignment. made by the prediction heads from the student encoder 6x and selecting the label for which the probability is maximum. We deduplicate the tokens before passing them to the language model.
Spoken language models. The SLMs are 0PT-125M models (Zhang et al., 2022), trained on the 6k hours subset of Libri-Light (Kahn et al., 2020) using fairseq2 (Balioglu et al., 2023). The architocture choice follows previous work (Hassid al., 2023; Maimon al., 2025a). We train on one A100 node with GPUs, with a batch of at most 81920 tokens, and a context length of 2048 for 25k steps. The learning rate increases linearly to le-2 over 1000 steps, then follows cosine annealing schedule. The other training parameters follow the defaults of 0PT-125M. The selected checkpoint is the one with the lowest validation loss.
4.2 Preventing codebook collapse
Our molivation for changing DinoSR's learning objective was stabilize the training procedure. We found in preliminary studies that the online clustering of DinoSR tended to collapse, as tracked by the codebook and prediction head perplexities. The perplexity HHp)) with H(p) the diversity of codewords used by the model, with Ps being the probability of the assignment v. The codebook perplexity at layer & is measured with p y e (0, 1),, and the prediction head perplexity with p = JZ E (0, 1).. With a perplexity of V, all codewords are used equally.
Enev Pe log, Pe the entropy, measures
In figure 2, we compare the codebook and prediction perplexities of DinoSR and SpidR during training. The perplexities are computed on LibriSpeech dev-clean over each batch, using the same batch size in
7

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: Zero-shot discrete units quality and spoken language modeling metrics from wav2vec 2.0, HuBERT, WavLM BASE, DinoSR, and SpidR. (in %, chance level 50%, except for PNMT). The speech encoders are trained on LibriSpeech 960h and the language models on Ligri-Light 6k. The vocabulary size V 256. For each model, the selected layer is the one with the lowest average ABX on continuous embeddings. The best scores are in bold and second best are underlined.
Discrete units quality
Language modeling
Model
Layer
Units
ABX +
PNMI 1
SWUGGY 1
sBLIMP
tSC f
all
in-vocab
wav2vec 2.0
6
K-means
9.33
0.609
62.29
68.50
53.34
65.97
HuBERT
11
K-means
7.32
0.637
65.50
73.67
55.60
68.75
WavLM BASE
11
K-means
7.01
0.667
69.74
79.88
56.60
70.35
DinoSR
5
Codebook
7.92
0.620
60.10
64.56
57.04
69.44
K-means
10.87
0.558
56.69
59.42
54.32
65.76
DinoSR
5
Codebook
7.73
0.614
58.93
62.07
55.79
66.08
K-means
10.54
0.591
55.73
57.58
54.04
63.68
SpidR
6
Codebook
6.31
0.602
69.78
79.98
58.10
70.14
K-means
7.20
0.636
71.89
82.46
59.48
70.46
Our re-implementation.
pretraining, and then averaged across the datasct. Liu et al. (2023a) report that DinoSR has a much higher perplexity than other online clustering methods, such as VQ-APC (Chung et al., 2020) and Co-training APC (Yeh Tang, 2022). However, DinoSR still prone codebook collapse, especially in the last layers. In DinoSR, the output of the last layer = given to all heads o² to derive the pseudo-labels from the intermediate layers of the teacher. The codebook assignments information for all K layers must be linearly extractable from g4. SpidR is more straightforward: s* is used to predict the assigmments from layer k. This result suggests that our training objective reduces the distribution shift between the embeddings and the codebooks, a challenge frequently encountered in neural networks with vector quantization (Huh et al., 2023).
4.3 Evaluation of the learned speech representations
In order to train a spoken language model, we derive discrete units from the represenlations of the SSL model. For successful language modeling, the units need to encode the underlying linguistic content, not the speaker information or the acoustic background. Therefore, we want the model to have highly accossible phonetic and word information in its representations, and a well clustered representation space. Following previous work (Nguyen et al., 2020), we evaluate the SST, models with metrics computing the discriminability of the embeddings. This evaluation is then used to select the target layer for spoken language modeling (Lakhotia et al., 2021). We summarize the metrics used in this work in table 1.
The first metric of interest is the ABX discriminability over phonemes (Schatz, 2016). It measures how well triphones differing only by the central phone (like /bag/ and /beg/) are discriminated in the embedding space by comparing the distances between two instances x and a the same triphone to the distance between æ and another triphone b. The test successful if the representations of x and a are closer than those of x and b. In the within speaker task, a, and x are from the same speaker, whercas the across speaker task, a and b from the same speaker and & from another one. We use the implementation of Poli al. (2025) to compute ABX scores. It fixes issues with the slicing of features that existed in the Libri-Light version, which explains the differences with the scores reported by Liu et al. (2023a).
In addition to the ABX, which operates al the triphone level, we evaluate embedding discriminability at the word level. An ABX task where and X are instances the same word and B is from a different word would be too easy most cases. Instead, we opt for a more challenging metric: Mean Average Precision (MAP) over words (Carlin et al., 2011). This retrieval task requires that, for each word, the closest embeddings correspond to other instances of the same word. Unlike ABX, which uses Dynamic Time Warping to handle
8

---

Published in Transactions on Machine Learning Research (12/2025)
WUGGY all
BLIMP
tsC
Accuracy (%)
65
70-
60
75
70
60000
600
6000
60000
600
6000
60000
Training hours
SpidR (Codebooks)
SpidR (K- (K-means)
HuBERT (K-means)
Text (BPE)
Data scaling results for a 125M parameters OPT model trained on Libri-Light, with different discrete units encoders. Zero-shot accuracy in %, chance level 50%. The speech encoders have V 256 units. The log-likelihoods are normalized by the nunber of lokens, except for WUGGY with text.
Figure 3:
duration differences between speech segments, we average word representations over the time axis. Following Algayres et al. (2020), we use MAP@R (Musgrave et al., 2020) and get the final score by averaging over all words, where is the number other instances given query word, and
R MAP@R = E(()) where P(i) precision at i if the i-th retrieval is correct, R
1
(5)
otherwise.
The intermediate layer chosen for cach model is the with the lowest average ABX error rate, which is not necessarily the best layer in terms of MAP (see figure 7 in appendix). As shown in table 2, SpidR outperforms baseline SSL models on both metrics. For all models, we computed the ABX using the angular distance on the representations of the intermediate layers, contrary to Liu et al. (2023a) who used the prediction heads with the KL-symmetric distance for DinoSR. See appondix B.1 in appendix for additional discriminability results and appendix B.2 for a visualization of the learned embeddings.
4.4 Evaluation of downstream spoken language modeling
Evaluation metrics. In order to assess the role of the speech encoder in spoken language modeling, we consider three standard tasks. At the lexical level, sWUGGY (Nguyen et al., 2020) evaluates the ability of the network assign higher probability results for "in-vocab" pairs, keeping only the words present in LibriSpeech. At the syntactic level, in sBLMMP, the network has to decide which sentence is grammatically correct, given minimal sentence pairs. Spoken StoryCloze (Mostafazadeh et al., 2017; Hassid et al., 2023) measures the ability of the model to choose the correct continuation of the beginning of a short story. We report the results for the "Topic" version (tSC), based on simpler negative examples. Following previous works, the log-likelihoods are normalized by the number of tokens. These metrics provide zero-shot evaluations of the model's linguistic knowledge; however they do not capture aspects related to non-linguistic or paralinguistic information (see de Seyssel et al. (2023); Maimon et al. (2025c) for complementary metrics). Note that we focus on the linguistic capabilities of the model, and our ovaluation framework only uses log-likchoods of sequences of discrete units. We do not train any vocoder in this work. Previous studies have shown that GANs can be trained to syuthesize speech from discrete units of self-supervised speech encoders (Polyak et al., 2021), and can be conditioned on speaker, pitch, or style tokens (Nguyen et al., 2023; 2025).
the true word than to a matching non-word. We also report
Comparison against other speech encoders. To evaluate the contribution of units from SpidR for spoken language modeling, we compare table 3 SLMs trained with units from wav2vee 2.0, HuBERT, DinoSR or SpidR on those three metrics. We also inchude WavLM BASE, even though uses an additional robustness loss, and belongs to slightly different class of SSL models. We keep vocabulary size of V = 256
9

---

Published in Transactions on Machine Learning Research (12/2025)
Table Zero-shot spoken language modeling results (in %, chance level 50%) for ~150M parameters models trained on Libri-Light 6k from HuBERT or SpidR. discrete units, across different number of units. Results for models based on HuBERT are from Chang et al. (2025); Messica & Adi (2024). The best scores are in bold and second best underlined.
sWUGGY 1 in-vocab
sBLIMP
tSC 1
Num.
units
Model
Units
all
50
HuBERT
K-means
67.48
52.42
66.27
HuBERT
Spinso
58.90
63.52
59.38
65.85
HuBERT
DC-Spinso,4086
65.05
73.51
60.15
69.91
SpidR
K-means
68.51
77.63
61.20
73.13
100
HuBERT
K-means
67.75
51.96
67.18
HuBERT
Spin1o0
65.28
73.25
59.97
68.25
HuBERT
DC-Spin200,
68.04
78.47
61.35
70.18
SpidR
K-means
71.27
82.32
58.70
70.78
200
HuBERT
K-means
71.88
52.43
67.55
HuBERT
Spingo0
68.95
78.
62.55
69.64
HuBERT
DC-Sping00.
70.79
80.59
63.13
69.21
SpidR
K-means
70.98
81.65
58.72
70.35
500
HuBERT
K-means
66.74
74.72
55.54
63.23
HuBERT
Spinsoo
70.03
79.31
60.08
67.45
HuBERT DC-Spins00,4096
71.48
81.38
60.84
67.50
SpidR
K-means
70.94
81.50
57.08
70.03
for all models to allow for exact comparison between units derived from K-means and from the codebook predictions. We also add analysis of the discrete units' quality with the ABX one-hot tokens, as well the Phone Normalized Mutual Information (PNMI) (Hsu et al., 2021). The alignments used for PNMI are those from the ZeroSpeech 2021 challenge (Nguyen et al., 2020). Both metrics indicate how well the units correlate with the underlying phonemes. See appendix B.3 in appendix for a layer-wine wise analysis of the discrcte units quality and downstream spoken language modcling SpidR outperforms all standard encoders on SLM metrics, and even outperforms WavLM Base when using units from K-means.
Data scaling analysis. To assess how the advantage of SpidR over other SSL models generalizes across different training conditions, we compare the scaling properties of SLMs trained with HuBERT SpidR across varying data quantities in figure 3. We train SLMs three dataset sizes: the 600h subset of Libri-Light, the 6k subset, or the full 60k dataset. We maintain the same hyperparameters as before, and we train for 150k steps when using the Ligri-Light dataset instead of 25k steps, and for 15k steps on Libri-Light 600h. Additionally, we train a topline text LM using BPE tokens from the original books read, ensuring exact dataset. matching between text and spoken LMs. The tranccriptions are from Kang al. (2024); on which we train a BPE tokenizer with 4096 tokens. Apart from the vocabulary size, all training hyperparameters match those of the SLMs. We evaluate the text LM on the original text versions of WUGGY, BLIMP and tSC. On WUGGY, do not normalize log- likelihoods for the text LM since non-words are segmented into more tokens by the tokenizer. See appendix B.4 for DinoSR scaling properties.
Across all conditions, SpidR consistently outperforms HuBERT on all metrics, whether using codebook predictions or K-means clustering. However, it does not change the scaling properties: the scaling slopes are similar but SpidR has constant advantage over HuBERT. Furthermore, text LMs trained under the same conditions achieve both better performance and superior scaling, particularly on tSC. Better performance on larger datasets could potentially be achieved with larger models.
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table Pretraining compute footprint SpidR against other SSL models operating at 50Hz. We report the pretraining times and total effective batch sizes in the default settings given in the corresponding papers. k2SSL Zipformer is trained using labels from the first iteration of HuBERT, and Academic HuBERT with labels from E-branchformer (Kim et al., 2023).
Model
GPUs
Steps
Batch size
Pretraining time
GPU hours
HuBERT (Hsu et al., 2021)
A100 x32
650k
47 min
62 hr
1984
DinoSR (Liu et al., 2023a)
V100 x16
400k
63 min
180 hr
2880
data2vec 2.0 (Baevski et al., 2023)
A100 x16
50k
17 min
43 hr
688
MelHuBERT (Lin et al., 2023a)
RTX 3090 x1
630k
8 min
300 hr
300
Academic HuBERT (Chen et al., 2023a)
A100 x8
1760k
47 min
240 hr
1920
k2SST, Zipformer (Yang et al., 2025)
V100 x8
225k
80 min
64 hr
513
SpidR and DinoSR (our reimplem.)
A100 x16
400k
63 min
23 hr
369
Across number of units. Finally, we investigate the role the number of units in the spoken LM in table 4. We train SLMs on SpidR units derived from K-means with vocabulary sizes in {50, 100, 200, 500} under the same conditions above. We compare the zero-shot scores to HuBERT-basee models from Chang et al. (2025); Messica & Adi (2024). Those works use transformer in_big from fairseq (Ott et al., 2019) with 150M parameters, whereas use the OPT-125M architecture. All language models are trained on Libri-Light 6k. The advantage of SpidR over HuBERT remains consistent across different vocabulary sizes. We also compare against units derived from HuBERT with Spin or DC-Spin. These approaches aim to improve speaker invariance and speech tokenization by learning auxiliary codebooks using swapped prediction. SpidR with standard K-means clustering matches the performance of HuBERT with DC-Spin units, with the latter showing advantages on sBLMMP, while SpidR performs better on the other metrics.
4.5 Codebase and pretraining time
In addition to learning strong phonetic reprosentations, SpidR, designed with practical considerations in mind: reducing computational costs and simplifying the training pipeline. We developed a minimal PyTorch codebase compatible with the latest PyTorch features, with model implementations based on HuBERT from torchaudio (Hwang et al., 2023).
We re-implemented DinoSR in this codebase, reducing training time from the reported 180 hours to just 70 hours on V100 GPUs under identical settings Liu et al. (2023a). We further opti- mized the codebase for full compatibility with torch. compile (Ansel et al., 2024) and minimized host-device synchronization points. Since torch. compile merges native PyTorrc modules and functions into optimized kernels, this results in significant throughput improve- ments. As shown in table 5, SpidR can be pretrained in under a day on 16 A100 GPUs. With 14 hours (maintaining the same total batch size), compared to 62 hours for HuBERT. The single-pass magle-pass training of SpidR. also eliminates the feature extraction and label computation steps required by Hu- BERT, removing common engineering challenges. Figure shows pretraining times for SpidR across different hardware configurations (4, 8, and 16 A100 or H100 GPUs) with constant total batch size. Using torch. compile provides approximately 20% speedup pre- training time. We open-source both the final checkpoints and the codebase.
A 100
100
Pretraining time (hours)
53
42
29 23 7
H100
W/o compile
63 48
W/ compile
A100 GPUs, training SpidR only takes
33 26
19 15
K
8
16
Number of GPUs
Figure 4: Approximate pretraining time for various hardware configura- tions with constant total batch size.
11

---

Published in Transactions on Machine Learning Research (12/2025)
5 Conclusion
We presented SpidR: self-supervieed speech representation model that efficiently learns strong reprosentations for spoken language modeling. We demonstrated that its learning objective, adapted from DinoSR, enables stable training and produces representations with salient phonetic information. Spoken language models using units derived from SpidR consistently outperform those based on HuBERT and DinoSR.
We developed SpidR by implementing minimal modifications to DinoSR to prevent codebook collapse. Future work may explore new architectures by iterating on the pairing between the student and the teacher, while preserving the core self-distillation and online clustering components. Our work focused improving the linguistic capabilities of spoken language models through units that provide more accessible phonetic information. However, real-world speech systems require more than linguistic understanding. They must also preserve acoustic content and speaker information, capabilities that remain beyond the scope of our current approach. A possible next step would be to improve the encoder so that it learns not only linguistic units, but also disentangled representations for complementary aspects of the speech signal: prosodic (pitch, energy, duration); expressive (whispered, shouted, angry, sad, etc.), and speaker units. However, achieving disentanglement in a purely self-supervised approach remains significant challenge (Polyak et al., 2021; Qian et al., 2022; Kharitonov et al., 2022; Lin el al., 2023b; Tu et al., 2024).
On a broader level, this work has implications for the design of spoken language models. Despite the emergence of generalist benchmarks such as SUPERB, modern self-supervised representation models have been designed with ASR as the primary objective, with every hyperparameter tuned accordingly. However, spoken language modeling has fundamentally different requirements, more closely aligned with the unsupervised term discovery tradition. Our findings demonstrate that textless SLM performs best when semantic information is readily accessible in the representations, rather than when ASR performance is maximized. Works that try to add speoch modality to text-based LLMs often plug representations from the Whisper encoder into the LLM (Tang et al., 2024; Held et al., 2025; Xu et al., 2025) through an adapter. Our study suggests that this may be suboptimal, and that representations specifically designed for spoken language modeling deserve further exploration.
This work addressed exclusively English, and only with data from LibriVox audiobooks. Major multilingual SSL models are based on either wav2vec 2.0 (Conneau et al., 2021; Babu et al., 2022; Pratap et al., 2024) HuBERT/Wavd / (Chen et al., 2023b; 2024; Zanon Boito et al., 2024), and require massive computational resources for training. SpidR offers solution for Icarning strong representations much faster, serving as foundation for future models and making approaches in other languages or multilingual settings more accessible due to reduced computational cost. Future work will focus on scaling the speech encoder to more data and languages while ensuring robustness to diverse acoustic conditions, with the goal of building a speech encoder capable of learning linguistic representations from ecological speech.
Acknowledgments
This work was performed using HPC resources from GENCI-IDRIS (Grant 002-AD001014368 and was supported in part by the Agence Nationale pour la Recherche (ANR-I7-EURE-OO) Frontcog, ANRIO-IDEX- 0001-02 PST.*). M. P. acknowledges Ph.D. funding from Agence de I'Innovation de Défense. F.D. in his EHESS role was funded by an ERC grant (InfantSimulator).. Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union or the Europcan Rescarch Council. Neither the European Union nor the granting authority can be held responsible for them.
12

---

Published in Transactions on Machine Learning Research (12/2025)
References
Robin Algayres, Mohamed Salah Zaiem, Benoît Sagot, and Emmanuel Dupoux. Exaluating the reliability of acoustic speech embeddings. In Interspeech 2020, pp. 4621 4625, 2020. 0...02114377/nterrppeee wetpeeech.200222550 2020-2362.
Robin Algayres, Yossi Adi, Tu Nguyen, Jade Copet, Gabriel Synnaeve, Benoit Sagot, and Emmanuel Dupoux. Generative spoken language model based on continuous word-sized audio tokens. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3008-3022 dec 2023. 101:10.18653/1 /2202222mmmm. 20133.emmlp-min.11
Jason Ansl Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, C. K. Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Shunting Zhang, Michael Suo, Phil Tillet, Xu Zhao, Eikan Wang, Keren Zhou, Richard Zou, Xiaodong Wang, Ajit Mathews, William Wen, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operaling Systems, Volume 2, pp. 929 947, 2024. 1et::10.115/3620060
Siddhant Arora, Kai-Wei Chang, Chung-Ming Chien, Yifan Peng, Haibin Wu, Yossi Adi, Emmanuel Dupoux, Hung- Yi Lee, Karen Livescu, and Shinji Watanabe. On The Landscape of Spoken Language Models: A Comprehensive Survey, apr 2025. URL https:://artiiii wwreiiiv.rr///o org/aboa/a abs/ //////5544..222. 08528.
Alan Baade, Puyuan Peng, and David Harwath. SyllableLM: Learning coarse semantic units for speech language models. In The Thirteenth International Conference on Learning Representations, 2025. URL
https://openrevvie patt/forumiidddSSn//
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Comneau, and Michael Auli XLS-R: Self-supervised cross-lingual speech representation learning at scale. In Interspeech 2022, pp. 2278-222722 77822 2022. doi: 10h11...143//// /Inntesspeereehhh http:/ppeechh.222144.ee222 2022-143.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli Wav2vee 2.0: A Framework for Self-Supervised Learning of Speech Representations. In Advances in Neural Information Processing Systems, volume 33, pp. 12449 12460, 2020. URL sttps://proceeding neurips. c2/p /hash// Abstract html.
Alexci Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language. In Proceedings of the 40th International Conference on Machine Learning, pp. 1416 1429, jul 2023. URL https://proceeeee html.
mlr. terres/v202/bavveee
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for sulf-supervised learning in speech, vision and language. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1298 1312, 17 23 Jul 2022. URL https://proocediiggs. mlr watsss/vve/bbbevsstt
html.
Can Balioglu, Martin Gleize, Artyom Kozhevnikov, Ilia Kulikov, Tuan Tran, and Julien Yao. fairseq2, 2023. URL http:/awww...c.eeeeeeeeeerrrrr/a
Zalán Borsos, Raphaal Marinier, Damien Vincent, Fugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audi- oLM: A language modeling approach to audio generation. IEEE/ACM TASLP, 31:2523 2533, 2023.
doi:10.110$ 110.111199/TP.... MAAALPP20223328.20
Michael A. Carlin, Samuel Thomas, Aren Jansen, and Hynek Hermansky. Rapid evaluation of speech represen- tations for spoken term discovery. Tn Interspeech 2011, pp. 821-824 2011. doi:10.21437 /Interppeechh 201 1-304.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sell-supervised vision transformers. In TCCV, pp. 1650-9660, October 2021. CRL https: http://c/ppacccce///////// thecvf. wtt.cooccccc/c www./connteet//ccc2222222 htt.://///tmmm/mmmmmmaaamaaa paairon_deerriinn_Ppprrress in Self-ellflllll Saaat--uprrviie___aaee Vision Vissio_TTaasseeeee Transformers_ Becee__221__pppp___ html..html. html.
Heng-Jui Chang and James Glass. R-spin: Efficient speaker and noise-invariant representation learning with acoustic pieces. In Proceedings of the 2024 Conference of the North American Chapler of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Popers), pp. 642 662, jun
2024. Noi:10018653// /00.01/6/////222..aaaaa/ 2.0.2.4aall-oonn....
Heng-Jui Chang, Alexander H. Liu, and James Glass. Self-supervised fine-tuning for improved con- tent representations by
invariant clustering. In Interspeech 2023, pp. 2983 2987, 2023.
doi: http:////nttrrppcc222222222
Heng-Jui Chang, Hongyu Gong, Changhan Wang, James Glass, and Yu-An Chung. DC-Spin: A speaker- invariant speech lokenizer for spoken language models. In Interspeech 2025, pp. 5723 5727, 2025.
doi:10.21437/ 11111.221447//ttespeeeee 2025-244..
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. WavLM: Large-Scale Self-Supervised Pre- vre-Trainng for Full Stack Speech Processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505-BB 1518, oct 2022. ISSN 1941-0484. doi:10.1109/JSTSSS ISS.09///SSPPP.. T02.:/SSSSSPPS..2...22 2022.318811..
Sanyuan Chen, Chengyi Wang, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers. IEEE TASLP, 33:705 718, 2025.
William Chen, Xuankai Chang, Yifan Peng, Zhaoheng Ni, Soumi Maiti, and Shinji Watanabe. Reducing barriers to self-supervised learning: HuBERT pre-training with academic compute. In Interspeech 2023, pp. 4404-4408, 2023a. doi:10.21437 htt.a///nnneeseechh2222222 2023-1176.
William Chen, Jiatong Shi, Brian Yan, Dan Berrebbi, Wangyou Zhang, Yifan Peng, Xuankai Chang, Soumi Maiti, and Shinji Watanabe. Joint prediction and denoising for large-scale multilingual self-supervised learning. In 2023 IEEE Aulomatic Speech Recogrition und Understanding Workshop (ASRU), pp. 8, 2023b. doi:10.1109/ /SRSS/SRUU.9.99 202.2002223.32 10339935.
William Chen, Wangyou Zhang, Yifan Peng, Xinjian Li, Jinchuan Tian, Jiatong Shi, Xuankai Chang, Soumi Maiti, Karen Livescu, and Shinji Watanabe. Towards robust speech representation learning for thousands of languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 10224, nov 2024. 1oi:10.1865:) v1 11/2222.emma.. w.w.llppmmppee... www.l-mmin..5555 main.570.
Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with caudom-projectto quantizer for speech recognition. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Reseorch, pp. 3915 1924,,944, 17-23 Jul 2022. URL htttps:/prooceedings mlr. press/v162. chiu22a. wttps///2..////......... html.
Kwanghee Choi, Ankita Pasad, Tomohiko Nakamura, Satoru Fukayama, Karen Livescu, and Shinji Watanabe. Self-supervised speech representations are more phonetic than semantic. In Interspeech 2024, pp. 4578 4582, 2024. doi: doi:10.21437 21t.1222.8./1ttssseecc22211111 2054-1157
Yu-An Chung and James Glass. Generative pre-training for speech with autoregressive predictive coding. Tn ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3497 3501, 2020.
Yu-An Chung, Hao Tang, and James Glass, Vector-quantized autoregressive predictive coding. In Interspeech 2020, pp. 3760-30 3764, 2020. 1oi:10.21437/11l 4
14

---

Published in Transactions on Machine Learning Research (12/2025)
Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. w2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp- 244-250, 2021. doi:10.1109/ W.SSRRR555...221990060
Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross cross-lingual representation learning for speech recognition. In Interspeech 2021, pp. 2426-2430, 2430, 2021. 11000221447//nnterppeeeee 2021-333 2021-329.
Santiago Cuervo and Ricard Marxer. Scaling properties of speech language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 351-361, nov 2024.
Santiago Cuervo, Adel Moumen, Yanis Labrak, Sameer Khurana, Antoine Laurent, Mickael Rouvier, and Ricard Marxer. Text-speech language models with improved cross modal transfer by aligning abstraction levels, 2025. URL ssttps:/arriv..oom arg/abs/2003 .02211.
Maureen de Seyssel, Marvin Lavechin, Yossi Adi, Emmanuel Dupoux, and Guillaume Wisniewski. Probing phoneme, language and speaker information in unsupervised speech representations. In Interspeech 2022, pp. 1402 1406, 2022. doi:10.214437/ wwwpleesseeech.2222223222
Maureen de Scyssel, Marvin Lavechin, Hadrien Titeux, Arthur Thomas, Gwendal Virlet, Andrea Santos Revilla, Guillaume Wisniewski, Bogdan Ludusan, and Fammanuel Dupoux. Prosaudit, prosodic benchmark for self-supervised speech models. In Interspeech 2023, pp. 2963 2967, 2023. 1oi:10.21437/ 0tt.1..24377/nterspppe htt////eeeeeeehh.0001 2023-438
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi High fidelity neural audio compression. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:/ /atpss////ppeeeeeeeeeeee net/ forum?id=ivCaBz2222 Featured Certification, Reproducibility Certification.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. Tn Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171 4186, jun 2019. doi: Net.10.1865//vll 1
Ewan Dunbar, Mathieu Bernard, Nicolas Hamilakis, Tu Anh Nguyen, Maureen De Seyssel, Patricia Rozé, Morgane Rivière, Eugene Kharitonov, and Emmanuel Dupoux. The Zero Resource Speech Challenge 2021: Spoken Language Modelling. In Interspeech 2021, pp. 1574 1578, aug 2021. doi:10.22137/ 11w.////Ineepeech.22.. 2021- 1755.
Ewan Dunbar, Nicolas Hamilakis, and Emmanuel Dupoux. Self-Supervised Language Learning From Raw Audio: Lessons From the Zero Resource Speech Challenge. IEEE Journal of Selected Topics in Signal Processing, 16(6):1211-1226, oct 2022. ISSN 1941-0484.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: A speech-text foundation model for real-time dialogue, 2024. URL attps://arxiv. wwt//bs////aiiieeee 0200000003.......
Itai Gat, Felix Kreuk, Tu Anh Nguyen, Ann Lee, Jade Copet, Gabriel Synnaeve, Emmanuel Dupoux, and Yossi Adi. Augmentation invariant discrete representation for generative spoken language modeling. In Proceedings of the 20th International Conference on Spoken Language Tronslation (IWSLT 2023), pp-
465 477, jul 2023. doi:10.18653/ wtw//22..wsslt.....
Jean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, ko- ray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent - a new approach sclf-supervised learning. In Advances Neural Information Processing Systems, volume 33, pp. 21271-21284, 2020. URL Paper (araaa80d5.eee774222771111119199 Pateee--aaprrppp....... påf.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Anmol Gulati, James Qin, Chung- Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolatio--angmeneed transformer for speech recognition. In Interspeech 2020, pp. 5036 5040, 2020.
Mark Hallap, Emmanuel Dupoux, and Ewan Dunbar. Evaluating context-invariance in unsupervised speech representations. In INTERSPEECH 2023, pp. 2973 2977, aug 2023. 0in.10.21337/nneerspee 2023-1862
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, et al. Textually pre- trained speech language models. Advances in Newral Information Processing Systems, 36: 63483-63501, 2023. URL https://proceediin httppooceednggs.euuriss.... neurips. www.pps..ccpppprrrrrrr.... 68599996471119905777444366 Saraaa---ttttaaat--Ceeeeeee
html.
William Held, Yanzhe Zhang, Minzhi Li, Weiyan Shi, Michael J Ryan, and Diyi Yang. Distilling an end-to- end voice assistant without instruction training data. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7876 7891, Vienna, Austria, July 2025. doi:10.18653/vl /2025.acc--oog.. 388.. URL sttps://aclanthhologg org/2025.... acl acl-long..88))
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. HuBERT: Sell-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. IEEE/ACM TASLP, 09:3451-3400 2021. ISSN 2329-9304. 1oi: 111000099///.. TTT.TTA.........1.1112 2122291
Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight-through estimator: Overcoming optimization challenges in vector quantized networks. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 14096-14111 14113, 23-29 Jul 2023. URL https://proceedinng mlr. wttp:/sm.w...pee/eeeeee pwwss/v202 http//hhh2aaaahaaee
Jeff Hwang, Moto Hira, Caroline Chen, Xiaohui Zhang, Zhaoheng Ni, Guangzhi Sun, Pingchuan Ma, Ruizhe Huang, Vineel Pratap, Yuekai Zhang, Anurag Kumar, Chin-Yun Yu, Chuang Zhu, Chunxi Liu, Jacob Kahn, Mirco Ravanelli, Peng Sun, Shinji Watanabe, Yangyang Shi, and Yumeng Tao. Torchau- dio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for pytorch. In 2023 JEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-9, 2023. doi:10.1109/ RAA0///SRUS9994200000 2UBRU377962202331389664
Aren Jansen, Kenneth Church, and Hynek Hermansky. Towards spoken term discovery at scale with zero resources. In Interspeech 2010, pp. 1676-1679, September 2010. doi:10.21437/ http///teeppehh2211111 2010-483.
J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P.E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux. Libri-light: A benchmark for ASR with limited or no supervision. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669 7673, 2020.
Herman Kamper, Micha Elsner, Aren Jansen, and Sharon Goldwater. Unsupervised neural network based feature extraction using weak top-down constraints. In 2015 IEFF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5818-5822, April 2015. 011:10.110///CASSP2005....3
Herman Kamper, Aren Jansen, and Sharon Goldwater. Unsupervised word segmentation and lexicon discovery using acoustic word embeddings. IEEE/ACM TASLP, 24(4):669 679, 2016. 16t:100.109TAAALPP.... TASLL. 2016.2517567..
Wei Kang, Xiaoyu Yang, Zengwei Yao, Fangjun Kuang, Yifan Yang, Liyong Guo, Long Lin, and Daniel Povey. Libriheavy: A 50,000 hours ASR corpus with punctuation casing and context. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10991-10995, 2024. 06::000000/CCAASPP8845220.0..0
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. Text-free prosody-aware
16

---

Published in Transactions on Machine Learning Research (12/2025)
generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp- 8666 8681, may 2022. long.593.
Kwangyoun Kim, Felix Wu, Yifan Peng, Jing Pan, Prashant Sridhar, Kyu Han, and Shinji Watanabe. E-branchformer: Branchformer with enhanced merging for speech recognition. In 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 84 91, 2023. doi:10.1109 100.1119//S.T588922200000 202. 10022666.
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu Anh Nguyen, Morgan Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. Textless speech emotion conversion using discrete & decomposed representations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11200-11244 dec 2022. 6oi:10.18653/vl. w20.222..22//o/2 2et.0.2e.e22mmm..m www.eemminnnn.....mmmmmmm
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. On Generative Spoken Language Modeling from Raw Audio. Transactions of the Association for Computational Linguistics, 9:1336-1354, 2021. doi:10.1162 /////aaclaa/ 1
Chen-Yu Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceediings of Machine Learning Research, pp. 562 570, 09 12 May 2015. URL https: // //proceedings. mlr. press/v38/ v38/lee15a, h5m.. html.
Tzu-Quan Lin, Hung-yi Lee, and Hao Tang. Melhubert: simplified hubert on mel spectrograms. In 2023 IEEE Automatic Speech Recognétion and Understanding Workshop (ASRU), pp. 1-8. IEEE, 2023a. URL https:// htapps://arxiv.../ org/ abs/ 2abbs220 2291..9944.
Weiwei Lin, Chenhang He, Man-Wa Wai Mak, and Youzhi Tu. Self-supervised neural factor analysis for disentangling utterance-level speech representations. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 21065-21077, 23-29 Jul 2023b. URL. https: /ppp://///s/cc.gs........ mlr. wwwpppeeeeeesinneeeeee hhmml..
Alexander H. Liu, Heng-Jui Chang, Michael Auli, Wei-Ning Hsu, and Jim Glass. DinoSR: Self- Doff-Distillation and Online Clustering for Self-supervised Speech Representation Learning. Advances in Newral Information Processing Systems, 36:58346-58362 dec 2023a. URL https://procoedingg neurips. cc/paper files/
Oli Danyi Liu, Hao Tang, and Sharon Goldwater. Self-supervised predictive coding models encode speaker and phonetic information in orthogonal subspaces. In Interspeech 2029, pp. doi:10.21437/ 2tt.://///ntttrpccch2222223
2023b.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL http:://ppparrrrrre..cr/mmmmtaaa
Gallil Maimon, Avishai Elmakies, and Yossi Adi. Slamming: Training speech language model on one GPU in day. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 12216, Vienna, Austria, July 2025a. doi:10. 100.188633vlll votp://vv2222...eee 222....iniinnn--cl......
Gallil Maimon, Michael Hassid, Amit Roth, and Yossi Adi Scaling analysis of interleaved speech-text language models. In Second Conference on Language Modeling, 2025b. URL https: / opwprevieeeeeeeeeeeeeee wwwpeeeeiee.eeeeeoooriiiee net/forummid= IxægE8hyJs.
Gallil Maimon, Amit Roth, and Yossi Adi. Salmon: A suite for acoustic language model evaluation. In ICASSP 2025 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1 5, 2025c. doi:10.
202.. 10888861..
Shoval Messica and Yossi Adi. NAST: Noise aware speech tokenization for speech language models. In Interspeech 2024, pp. 4169 4173, 2024. doi:10.21437, /Tntersseeeeehhh... 2024-288.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D. Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maalø Tara N. Sainath, and Shinji Watanabe. Self-Supervised Speech Representation Learning: A Review. IEEE Journal of Selected Topics in Signal Processing, 16(6):1179-2110, oct 2022. ISSN 1941-0484. 1o1:10.1109/STTSPP200 /T2.SSSSSPPP22.2...22 32117200.
Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. LSDSem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lezical, Sentential and Discourse-level Semanlics, pp. 46 51, apr 2017. (ec10018653/ve /1//3///1/1-000066
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In Computer Vision ECCV 2020: 16th Europeon Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pp. 681-699. Springer, 2020. URL https:://ww..t ecva. wet/papers/eccy satpll22000pp000 wwwp//22e2..a....ep 5170_ECCV_ _acCe__2200___0 Sa1r2o_ppeerpppp
/html/ htap/pppprs__e__ /CCth/tm/CC///
Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, L'wan Dunbar, and Emmanuel Dupoux. The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling, 2020. URL https://arxiv.or wtt.:oo///aaaai////// /b.1///...1118..
Tu Anh Nguyen, Benoit Sagot, and Emmanuel Dupoux. Are discrete units necessary for spoken lan- guage modeling? IEEE Journal of Selected Topics in Signal Processing, 16(6):1415 1423, 2022. JSL.1/9/ITSSP.222...2..
Tu Anh Nguyen, Wei-Ning Hsu, Antony D' D'Aviroo, Bowen Shi, Itai Gat, Maryam Fazel-Zarani, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, Felix Kreuk, Yossi Adi, and Emnanuel Dupoux. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. In Interspeech 2023, pp. 4823-4821 2023. doi: Noi:100211477/ /Intee/etteeeehh...mer /
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, and Emmanuel Dupoux. SpiRit-LM: Interleaved spoken and writ- ten language model. Transactions of the Association for Compulational Linguistics, 13:30 52, 2025. doi::001162/terr //cc a 00728.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairsoq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp. 48 53, jun 2019. doi:10.18653/ /1/655/v///i/ /1/119-409.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispoech: An ASR corpus basod on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2206-5210, 2015. 1o0:10.110//TCASSPP220....
Alex S. Park and James R. Glass. Unsupervised paltern discovery in speech. IEEE TASLP, 16(1):186 197, 2008. doi:10.1199/TASIII TA92200079902.9922
Maxime Poli, Emmanuel Chemla, and Emmanuel Dupoux. Improving spoken language modeling with phoneme classification: A simple fine-tuning approach. In Proceedings of the 2024 Conference on Empirioal Methods in Natural Language Processing, pp. 5284-5292, nov 2024. 0oi:10.18653/11 20053//112222// wt112224.cmmm. cmnlp-main.3000
Maxime Poli, Emmanuel Chemla, and Emmanuel Dupoux. fastabx: A library for efficient computation of ABX discriminability, 2025. URL
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdeirah- man Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. In Interspeech 2021, pp. 3615 3619, 2021. doi:10.21437/Inn 211143//Inersspecch222222
Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazal-Zarandi, Alexei Bacvski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(97):1-522 2024. URL http://jmlr. htt.:://////lll.ooog/aaaas////// p hmml.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Kaizhi Qian, Yang Zhang, Heting Gao, Junrui Ni, Cheng-I Lai, David Cox, Mark Hasegawa-Johnson, and Shiyu Chang. VontentVec: An improved self-supervieed speech representation by disentangling speakers. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18003-18017, 17-23 Jul 2022. URL //tppe/s///ccceinngsss mlr. press/ v162/qian22b. html.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Πlya Sutskever. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 28492 28518, 23 29 Jul 2023. URL https://proceedings. mlr. press/v202rracford22 htmll.
Mirco Ravanelli, Titouan Parcollet, Adel Moumen, Sylvain de Langen, Cem Subakan, Peter Plantinga, Yingzhi Wang, Poonch Mousavi, Luca Della Libera, Artem Ploujnikov, Francesco Paissan, Davide Borra, Salah Zaiem, Zeyu Zhao, Shucong Zhang, Georgios Karakasidis, Sung-Lin Yeh, Pierre Champion, Aku Rouhe, Rudolf Braun, Florian Mai, Juan Zuluaaa-Gomme, Seyed Mahed Mousavi, Andreas Nautsch, Ha Nguyen, Xuechen Liu, Sangeet Sagar, Jarod Duret, Salima Mdhallar, Gaëlle Laperrière, Mickael Rouvier, Renato De Mori, and Yannick Estève. Open-source conversational AI with speechbrain 1.0. Journal of Machine Learning Research, 25(333):1-11, 2024. URL http:// wrg. http:/ppprs///5/2-00000 .hmml.
Thomas Schatz. ABX-Discriminablii Measures and Applications. Theses, Université Paris 6 (UPMC), Sep 2016. URL.
Thomas Schatz, Vijayaditya Peddinti, Francis Bach, Aren Jansen, Hynek Hermansky, and Emmanuel Dupoux. Evaluating speech features with the minimal-pair ABX task: Analysis of the classical MFC/PLP pipeline. In Interspeech 2013, pp. 1781-1785, 2013. doi:10.214371 Tnterspeech.2013-5 2013-441.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli wav2vec: Unsupervised pre-training for speech recognition. In Interspeech 2019, pp. 3465 3469, 2019.
Ashish Seth, Ramaneswaran Selvakumar, S Sakshi, Sonal Kumar, Sreyan Ghosh, and Dinesh Manocha. EH-MAM: Easy-to-Hard masked acoustic modeling for self-supervised speech representation learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 6386-6400, nov 2024. doi: 10.118663/111 //t/////222...mm/// 200.2...mnp-mmai......
Jiatong Shi, Hirofumi Inaguma, Xutai Ma, Tia Kulikov, and Anna Sun. Multi-resolution huBERT: Multi- resolution speech self-supervised learning with masked unit prediction. In The Twelfth International Conference on Learning Representations, 2024. URL https: http:////eeeeeeeeeeeeteeee wtt.://toooorrddddduootte.
Amitay Sicherman and Yossi Adi. Analysing discrete self supervised speech representation for spoken language modeling. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023. dot:10.1109/CAASSPPP...
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang, SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: / http://pppneeeeeeeeee.e/ http///rrunid11r/////aa
Youzhi Tu, Man-Wai Mak, and Jen- TaanTanng& Chien. Contrastive self-supervised speaker embedding with sequential disentanglement. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 32:2704 2715, May 2024. ISSN 2329-9290. doi:10.1109) 111111199TTAPPP 202. URL https://doi. 1109/TASLP 2024. 3402077.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding, jan 2019. URL https: //apxii///xxvvv org/abs// httg//bb/1100.0055/ 03748.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(8):22579 2205, 2008. URL http://www..rrr.ppppp///////aaaa
html.
19

---

Published in Transactions on Machine Learning Research (12/2025)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia. Polosukhin. Attention is all you need. In Aduances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL ppf.....ff....... http://.ffffa2222eeeeeeaeaaaas..
www.f/lllppeee paper/
Nicol Visser and Herman Kamper. Spoken Language Modeling with Duration-Penalized Self- Selervuuee Units. In Interspeech 2025, pp. 1968-1972, 2025. doi:10.2143)) 100.22133//Innsssseee /Ittersnerpppchccccc 2025-340.
Chengyi Wang, Yu Wu, Sanyuan Chen, Shujie Liu, Jinyu Li, Yao Qian, and Zhenglu Yang. Improving self-supervieed learning for speech recognition with intermediate layer supervision. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7092 7096, 2022. N00.10.1199ICAASSPPPP /CAASSAS3SS..2..... 2021
doi:10.1109
Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang- Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung yi Lee. SUPERB: Speoch processing universal PERformance benchmark. In Interspeech 2021, pp. 1194-1198, 2021. doi:10.21437/ cttprrspeeech222117777..
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-ommi technical report, 2025.
URL
20215..
Guanrou Yang, Ziyang Ma, Zhisheng Zheng, Yakun Song, Zhikang Niu, and Xie Chen. Fast-hubert: An efficient training framework for self-supervised speech representation learning. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-7, 2023. doi:10.1109) ASW/STRRUU99999 200.64.........0 0023110389778.
10910.1109//SRU5596..
Yifan Yang, Jianheng Zhuo, Zengrui Jin, Ziyang Ma, Xiaoyu Yang. Zengwei Yao, Liyong Guo, Wei Kang, Fangjun Kuang, Long Lin, Daniel Povey, and Xie Chen. k2SSL: A faster and better framework for suleruupe speech representation learning, 2025. URL httpp://arra: //arxiv. org/abs/ org/abs/2411. //////111.1111110 17100.
Zengwei Yao, Liyong Guo, Xiaoyu Yang, Wei Kang, Fangjun Kuang, Yifan Yang, Zengrui Jin, Long Lin, and Daniel Povey. Zipformer: A faster and better encoder for automatic speech recognition. In The Twelfth International Conference on Learning Representations, 2024. URL /tttps://oppnrweeee wwwprrvviieennn//ffuueeee net/forummidd 9WD9KwsSYT.
Sung-Tin Yeh and Hao Tang. Autoregressive co-training for learning discrete speech representation In Interspeech 2022, pp. 5000 5004, 2022. dof:10.214377/Ineesseee 2022-530.
Sung-Lin Yeh and Hao Tang. Estimating the completeness of discrete speech units. In 2024 IEFE Spoken Language Technology Workshop (SLT), pp. 422, 2024. doi: 10i:1001119/ 111.0///111.6.0022221220
S.J. Young, J.J. Odell, and P.C. Woodland. Tree-based state lying for high accuracy modelling. In Human Language Technology: Proceedings of Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994. URL
Marcely Zanon Boito, Vivek Iyer, Nikolaos Lagos, Laurent Besacier, and Ioan Calapodescu. mHuBERT- In Interspeech 2024, pp. 3939-3943, 2024.
147: A compact multilingual HuBERT model. doi:10.21437 /htt///tttrpechhhcceo 2024-938.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM TASLP, 30:495 507, 2022. 10099/01000//000 2109/AAAAL.P221311299900
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open pre-trained transformer language models, 2022. URL
http://axxvv..o/aabbaaasaa 0106/
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Specchtokenizer: Unified speech tokenizer for speech language models. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://pperreee http://www..eettt///////////////
20

---

Published in Transactions on Machine Learning Research (12/2025)
Table 6: SpidR pretraining hyperparameters. We trained with 16 A100 GPUs in our default setting.
Parameter
Value
Parameter
Value
Model
Optimizer
Convld dimension
512
Name
AdamW
Convld [(kernel size, stride]]
[110,5] [[3, 2) x 4 [(2,)) x
Peak learning rate
5 x 10-4
Convid bins
False
Betas
(0.9, 0.95)
Convld normalization
LayerNorm
Weight decay
0.01
Projection dropout
0
Epsilon
1 x 10-6 10
Positional encoding layers
5
Max. gradient
Positional encoding total kernel size Positional encoding groups Hidden dimension d
95
Warmup steps
12000
16
Hold steps
188 000
768
Decay steps
200 000
Number of Transformer layers L
12
Conv. freeze step
200 000
Number of attention heads Transformer dropout
12
Data
0.1
Min. sequence length
2000
Attention dropout
0.1
Max. sequence length
320 000
Feed-forward dimension
3072
Max. samples batch
800 000
Feed-forward dropout
0
Number of buckets
1000
Layer drop probability LayerNorm mode
5%
Padding
False
After
Random crop
True
Q, K, projection biases Number of codebooks K Codebook decay Codebook V
False
8
Masking
0.9
Start probability
8%
256
Span length
10
Initial decay of teacher Pu
0.999
With overlap
True
Decay timescale T
10000
Decay of teacher step
(1 - (0) exp(- -t/T)
A Implementation details
A.1 SpidR pretraining
Table contains the full list of pretraining hyperparameters and figure illustrates the two schedules that occur during training: the learming rate schedule and the EMA decay schedule of the teacher.
The positional encodings of DinoSR and SpidR the same those used by Baevski et al. (2022), and differ from Baevski et al. (2020); Hsu et al. (2021). Instead of only one convolutional layer with large kernel size, they are made of layers, each with kernel size of 95/5 19.
Batches are sampled using the following procedure. Audio files from LibriSpeech are first sorted and grouped into buckets by length, with only samples within the same bucket shufiled logether. Batches are formed by selecting audio files from given bucket until the target maximum number of samples in a batch is reached. If the target is not met, we continue filling the batch using files from the next bucket. No padding applied, and audio samples longer than the maximum sequence length are randomly cropped.
x 10
1.0000
Learning rate
95,
0.9995
2
SpidR
DinoSR
0.9990
200k
400k
200k
400k
Training steps
Training steps
Figure 5: Learning rate schedule (left) and EMA decay schedule of the teacher for DinoSR and SpidR (right).
21

---

Published in Transactions on Machine Learning Research (12/2025)
Table 7: ABX error rate of the codebook predictions (in %, chance level 50%, KL-symmetrie distance). All models have codebooks of size 256. The best scores are in bold and second best are underlined.
ABX within speaker 4
ABX across speaker
Model
Layer
dev-clean
dev-other
dev-clean
dev-other
DinoSR
3.48
3.89
3.80
4.89
DinoSRt
3.18
3.92
3.48
4.99
HuBERT + Spin256
3.85
5.20
4.32
6.36
WavLM + Spin2se
4.41
4.67
4.80
5.82
SpidR
2.99
3.48
3.35
4.56
Our re-implementatiion.
A.2 Masking procedure
We follow the masking procodure of Bacvski al. (2020), with parameters of Liu al. (2023a), to sample the mask M, as shown in figure 6. We first extract features x of shape (n, d) with d = 768 from the audio signal using the downsampling module. The masking process works as follows: each frame E (1, has an 8% probability of starting a mask span of length 10. Mask spans can overlap, and the proportion of masked frames depends on the total number of frames n.
x
Masking
x
Using the parameters from table 6, the average sequence length in LibriSpeech 960h batch is 216000, corresponding to 13.5 seconds of audio and to = 675 frames. For a typical 13.5- second audio sample, approximately 43% of all time-steps are
Downsampling
masked, with an average span length of 11.9 frames, correspond- ing to 238ms audio, a median of 8 frames, and maximum of about 50 frames. For reference, the average triphone duration in LibriSpeech dev-clean and dev-other is 237ms, based the annotations from Nguyen et al. (2020).
Figure 6: Masking procedure. The masked frames are in black, and the unmasked ones in gray.
B Additional results
B.1 Discriminability of continuous embeddings
In table 7, compute ABX discriminability the softmax outputs from either the prediction heads the Spin codebooks. Instead of using the standard angular distance, we use the symmetrized KL, divergence. This metric was used in Liu et al. (2023a) to evaluate DinoSR. We select the Spin checkpoints from Chang et al. (2023) with the same codebook size as DinoSR and SpidR.
We evaluate the phoneme and word discriminability of continuous embeddings from a wide range of monolingual English speech models table 8. All BASE size models were trained on LibriSpeech and all LARGE models Libri-Light. For each model, we select. the best-performing layer in terms of average ABX score. We distinguish between standard self-supervised models (including SpidR), self-supervised models with additional robustness losses such as WavLM (Chen et al., 2022), and supervised models. We conducted a preliminary experiment where we fine-tuned SpidR with our own implementation of Spin.
Overall, masked prediction using discrete targets produces representations with salient phonetic information, and additional losses promoting invariance acoustic and speaker conditions further improve performance. Supervision does not necessarily Whiiper exhibits poor ABX scores, likely because it learns from multiple tasks simultaneously, making phonetic information less salient in its encoder representations.
22

---

Published in Transactions on Machine Learning Research (12/2025)
Table 8: Evaluation continuous representations from monolingual English speech models. All operate at
50Hz framerate, except MR-HuBERT mono-base 25Hz and Conformer which are at 25Hz. Apart from Whisper, WavLM Base+ and WavLM LARGE, all BASE size models were trained on LibriSpeech and all LARGE ones Libri-Light. The best scores in bold and second best are underlined.
ABX within speaker
ABX across speaker J
MAP words
Model
Layer
dev-clean
dev-other
dev-clean
dev-other
dev-clean
dev-other
Self-supervised models
wav2ver (Baevski ul., 2020)
5.63
5.25
7.82
31.92
wav2vec LARGE (Buevski et al., 2020)
16
4.35
5.20
7:29
34.21
HuBERT (Hau et al., 2021)
11
3.38
4.26
4.01
6.49
46.07
33.37
HuBERT LARGE (Hsu et. al., 2021)
24
3.90
4.30
4.49
5.92
68.99
59.11
HuBERT EXTRA LARGE (Hsu et al., 2021)
48
4.04
4.38
4.76
6.21
64.64
54.64
data2vee (Baevski et al., 2022)
4
4.41
5.49
5.07
7.40
39.34
27.90
data2vee LARGE (Baevski et al., 2022)
7
4.51
5.46
5.20
7.15
38.56
28.70
data2vee 2.0 (Baevski et al., 2023)
1
5.13
5.77
5.72
7.53
69.38
53.49
data2vee 2.0 LARGE (Baevski et al., 2023)
2
6.68
6.55
7.41
8.43
66.85
55.42
ER-MAM (Seth et al., 2024)
4.31
5.36
4.96
7.58
60.63
42.80
MR-HuBERT mono-base 50Hz (Shi al., 2024)
3,61
3,99
4.21
5.39
70.72
63.08
MR-HuuBl HuBERT mono-base 25Hz (Shi al., 2024)
3.47
3.82
4,00
5.14
68.50
61.58
DinoSR (Liu et al., 2023a)
4.05
5.11
4.72
7.29
63.02
45.86
(LnoSR((Liu et al., 2023a)
4.29
5.56
5.22
8.56
51.35
33.74
SpidR
3.32
3.74
3.66
4.95
66.50
55.26
With self-supervised robustness loss
WavLM BASE (Chen ct al., 2022)
11
3.03
3.71
3.50
5.21
71.57
58.09
WavLM BASE (Chen ct al., 2022)
12
3.54
4.08
4.08
5.82
62,77
51.63
WavLM LARGE (Chen et al., 2022)
24
3,94
4.32
4.62
5.99
67.56
57 49
Content (Qian et al., 2022)
12
3.29
4.04
3.83
5.49
63.85
62.17
HoBERT + Spinzeas (Chang et al., 2023)
12
2.70
3.23
3.05
4.08
68.70
61.41
WavLM Spinzeas (Chang al., 2023)
12
3.05
3.51
3.52
4.44
75.20
67.13
SpidR Spin2048
12
2.73
3.51
3.11
4.47
61.33
50.99
With supervision
Whisper smalllen (Radford et 2023)
8.21
8.27
11.81
11.08
Conformer ASRè (Gulati al., 2020)
2.79
3.76
3.21
5.22
45.74
wav2vec ASR (Baevski al., 2020)
6
3.94
4.90
4.50
6.52
56.98
43.52
wav2vec LARGE ASR (Baevski et al., 2020)
10
3.52
4.58
4.09
6.20
54.04
40.06
HuBERT LaRGe ASR (Hsu et al., 2021)
14
4.69
5.38
5.48
7.27
43.25
33.09
HuBERT + phoneme classif, (Poli et al., 2024)
12
0.82
1.54
0.97
2.35
68.48
57.04
t Our re-implementation.
$ Using
(Ravanelli et al., 2024).
We compare in figure 7 the ABX and MAP continuous embeddings by layer for HuBERT, DinoSR (both the original checkpoint and our replication) and SpidR. The ABX scores are averaged across subsets and speaker conditions, and MAP across the two subsets.
ABX error rate (%)
14
60
12
10
40
MAP
8
20
6
2
6
7
10
11
12
3
6
7
8
9
10
12
Layer
Layer
SpidR
DinoSR
DinoSR (ours)
HuBERT
Figure 7: ABX and MAP (in %, chance level 50% for ABX) by layer for SpidR, DinoSR and HuBERT.
23

---

Published in Transactions on Machine Learning Research (12/2025)
Phone class
Gender
Fricative
Plasive
Nasal
Female
Male
Affricate
Approximant
Vowel
Figure 8: t-SNE visualization of phone embeddings from SpidR layer 6 on LibriSpooch dev-clean. Embeddings are colored by phone class (Ieft.) and by speaker gender (right).
B.2 Embeddings visualization
We visualize the embedding space of SpidR in two dimensions using t-SNE (van der Maaten & Hinton, 2008), following de Seyssel et al. (2022). We train t-SNE on phone embeddings of LibriSpeech dev-clean from layer 6 of SpidR. For each speaker, we sample 10 instances per phone and average each embedding along the time dimension, resulting in approximately 15 000 samples.
Fricative
Affricate
Plosive
Ip]
Approximant
Nasal
Vowel
Figure 9: t-SNE visualization of phone embeddings from SpidR layer on LibriSpeech dev-clean, colored by individual phones within each phone class. Embeddings from other classes shown in gray.
24

---

Published in Transactions on Machine Learning Research (12/2025)
S1L
AR
Al
D-
Bn
RI
AE
AR
M
M
K
W
W
AA
HR
BH
DR
UW
UW
AW
AW
SH
8H
NG
NG
CHI
CH
Y
1/1
011
ΟY
DY
(a) With codebook predictions (203 active codes).
(b) With K-means quantization (256 active codes).
Figure 10: P(phone| code) visualization for SpidR layer using either codebook predictions (left) K-means
quantization (right), on LibriSpeech dev-clean and dev-other.
figure 8, we color the embeddings by either the underlying phone class or by the speaker gender. For more fine-grained visualization, we color by individual phones within each phone class in figure Overall, the embedding space is well clustered by phone class, and even by individual phone, whereas the speaker information is not directly extractable from the embeddings.
B.3 Layer-wise analysis
In addition to the discrete units analysis in table we compute in figure 11 the ABX discriminability and PNMI for other intermediate layers of SpidR and HuBERT, with units derived from codebook predictions or K-means quantization. As in section 4.1, the K-means Figure 10 shows the (phhon code), with codes from SpidR layer 6 on LibriSpeech dev-clean and dev-other. The vertical axes are sorted by phone frequency in the annotated data.
trained on LibriSpeech train-clean-100
ABX enror rate
16
0.6
14
РNMI
12
0.5
8
0.4
6
4
1
2
3
4
5
6
7
8
9
10
11
12
1
2
3
4
5
6
7
8
9
10
11
12
Layer
Layer
SpidR (Codebooks)
SpidR (K-means)
HuBERT (K-means)
Figure 11: ABX (in %, chance level 50%) and PNMI by layer on discrete units from SpidR using codebook predictions or K-means, and from HuBERT using K-means, with V 256 units. ABX scores averaged across subsets and speaker conditions, and PNMI computed LibriSpeech dev-clean and dev-other.
25

---

Published in Transactions on Machine Learning Research (12/2025)
72
sWUGGY all (%)
72
68
SBLIMP (%)
57.3
64
55.0
60
52.3
S5C
64
56
50.0
60
52
Layer
Layer
Layer
SpidR. (Codebooks)
SpidR (K-means)
HuBERT (K-means)
Figure 12: Zero-shot spoken language modeling from each layer of HuBERT and SpidR (in %, chance level 50%), with units from codebook predictions or from K-means quantization, with V 256 units.
We also trained spoken language models from the units obtained from each intermediate layer in the same conditions as scction 4.1. Figure 12 shows the accuracies zero-shot spoken language modcling for the three encoders. Finally, to assess how well the zero-shot metrics serve proxy lasks, we compare spoken language modeling scores against phonetic- and word-level metrics in figure 13 (contimuous embeddings) and figure 14 (discrete units). We distinguish between SpidR using K-means units, where ABX is computed on standard embeddings, and SpidR using codebook predictions, where ABX is computed on codebook predictions with symmetric KT. divergence. We compute Pearson correlation coefficients between each proxy metric and downstream evaluation score. Note that this analysis does not capture inter-model differences well, and that correlations are influenced by the fact that SpidR's final layers perform poorly across most metrics.
60.0
aWUGGY all (%)
70
57.5
70.0
0.84
IMP (%)
-0.81
-086
55.0
67.5
63.0
62.5
50.0
40
47.5
60.0
12
6
ABX (%)
ABX (%)
ABX (%)
60.0
13
sWUGGY all (%)
37.5
70.0
60
SBLIMP
55.0
(%)
67.5
65.0
52.5
62.5
50.0
40
0.82
47.5
0.84
60.0
0.63
20
40
60
20
40
60
20
40
60
MAP (%)
MAP (%)
MAP (%)
SpidR (Codebooks)
SpidR (K-means)
HuBERT (K-means)
Figure 13: Spoken language modeling against discriminability of the continuous representations. Dots are labeled by intermediate layer index. ABX for SpidR (Codebooks) is computed over codebook predictions.
26

---

Published in Transactions on Machine Learning Research (12/2025)
60.0
sWUGGY all (%)
70.0
0.85
0.83
-0.76
55.0
67.5
SBEIMP
65.0
50.0
62.5
40
17.5
60.0
20
15
ABX discrete (%)
ABX discrete (%)
ABX discrete (%)
60.0
sWUGGY all (%)
70.0
60
55.0
67.5
1LLIMP
52.5
65.0
4
50.0
62.5
40
0.85
47.5
0.80
60.0
0.75
0.4
0.5
0.6
0.4
0.5
0.6
0.4
0.5
0.6
PNMI
PNMT
PNMI
SpidR (Codebooks)
SpidR (K-means)
HuBERT (K-means)
Spoken language modeling against phonetic evaluation of the discrete units, with V = 256 units. Dots are labeled by intermediate layer index.
Figure 14:
B.4 Data scaling analysis
We report in figure 15 the scaling properties of SLMs across varying data quantities, as in figure 3, with DinoSR in addition of SpidR and HuBERT.
SWUGGY all
sBLIMP
LSC
60.0
Accuracy (%)
70
70
57.5
65
55.0
60
65
52.5
55
600
6000
60000
600
6000
60000
600
6000
60000
Training hours
SpidR (Codebooks)
DinoSR (Codebooks)
DinoSR (Ours) (Codebooks)
SpidR (K-means)
DinoSR (K-means)
DinoSR (Ours) (K-means)
HuBERT (K-means)
Figure 15: Data scaling results for 125M parameters OPT model trained on Libri-Light, with different
discrete units speech encoders. Zero-shot accuracy in %, chance level 50%. The speech encoders have V = 256 units. The log-likeli are normalized by the number of tokens.
27

---

Published in Transactions on Machine Learning Research (12/2025)
C
Ablation study
Table 9: Ablation study from DinoSR to SpidR (in % except for PNMI, chance level 50% for ABX). The
discrete units are derived from the codcbook predictions. The ABX scores are averaged across subsets and speaker conditions, MAP across the two subsets, and PNMT is computed LibriSpeech dev-clean and dev-other.
Continuous embeddings
Discrete units quality
Model
Layer
ABX +
MAP 1
ABX
PNMI f
DinoSR
5.91
42.55
7.73
0.614
DinoSR Heads
7
4.22
63.39
7.87
0.610
DinoSR + Exp. EMA
6
5.86
51.85
8.77
0.609
DinoSR = Heads + Exp. EMA- = SpidR
6
3.92
60.88
6.31
0.602
Our re- ce-implementation.
We developed SpidR by making two key changes from DinoSR. First, we modified the learning objective by adding prediction heads to the student's intermediate layers instead of using only the final layer. This showed promising results, but we noticed that the training loss would slightly increase mid-training, suggesting that the student was struggling to keep with the teacher. To solve this problem, we modified the teacher's EMA decay schedule to follow a smoother trajectory that approaches faster without, plateauing at 0.9999.
In table we ablate these two changes: "Heads" refers the new learning objective, and "Exp. EMA" " refers to the new shape of the EMA decay schedule. We evaluate both in terms of continuous embedding discriminability, and discrete units quality using the prediction heads.
D Downstream supervised evaluation
To provide comprehensive evaluation of SpidR beyond our primary task of spoken language modeling, we also assess its performance ASR benchmarks. We finc-tune both DinoSR and SpidR, for ASR using the 1h and 10h labeled subsets of Libri-Light. (Kahn et al., 2020), as well as the train-clean-100 subset. of LibriSpeech (Panayotov et al., 2015). We adopt the exact fine-tuning configuration from wav2vec 2.0 without additional hyperparameter tuning. We also run the phoneme recognition and ASR without LM tasks of SUPERB (wen Yang et al., 2021) with the default hyperparameters.
As shown table 10, SpidR performs comparably to wav2vec 2.0 on clean evaluation sets but lags behind the other two, while DinoSR surpasses all other models. This pattern persists in the ASR and phoneme recognition tasks from SUPERB, shown in table 11. While both SpidR and WavLM have representations that discriminate phonemes the most (as shown in table 8), they are outperformed by data2vee or DinoSR on phoneme recognition with linear probe over frozen features. Similarly, the advantage in terms of ABX discriminability does not translate into superior ASR performance, even after fine-tuning. This divergence between supervised classification and unsupervised clustering may due to the existence of subspaces dedicated different types of information within the embeddings (Liu et al., 2023b). We illustrate the discrepancy between SLM and supervised evaluations of speech encoders in figure 16.
28

---

Published in Transactions on Machine Learning Research (12/2025)
Table 10: Word Error Rate (in %) on LibriSpeech dev and test sets after finetuning Libri-Light low-resource labeled splits. All models are pretrained on LibriSpeech 960h, and decoded greedily, without language model. The bost scores are in bold and second best are underlined.
Model
dev-clean dev-other
test-clean
test-other
1h labeled
wav2vec 2.0 (Baevski et al., 2020)
24.1
29.6
24.5
29.7
HuBERT (Hsu et al., 2021)*
20.2
28.1
20.6
28.9
WavLM Base (Chen al., 2022)
24.5
29.2
MR-HuBERT mono- base (Shi et al., 2024)
18.8
23.7
19.3
24.5
DinoSR (Liu et al., 2023a)*
18.1
23.3
18.3
23.6
DinoSR (Liu et al., 2023a)*
18.4
23.7
18.6
23.7
SpidR
29.1
35.9
29.5
36.1
10h labeled
wav2vec 2.0 (Baevski et al., 2020)
10.9
17.4
11.1
17.6
HuBERT (Hsu et al., 2021)*
9.6
16.6
9.7
17.0
WavLM Base (Chen al., 2022)
9.8
16.0
MR-HuBERT mono- base (Shi et al., 2024)
8.5
13.2
8.5
13.5
DinoSR (Liu et al., 2023a)*
7.1
12.1
7.2
12.5
DinoSR (Liu et al., 2023a)
7.3
12.7
7.4
12.8
SpidR
10.7
19.4
11.0
19.8
100h labeled
wav2vec 2.0 (Baevski et al., 2020)
6.1
13.5
6.1
13.3
HuBERT (Hsu et al., 2021)*
5.8
12.9
5.8
12.8
WavLM Base (Chen et al., 2022)
5.7
12.0
MR-HuBERT mono- base (Shi et al., 2024)
4.9
9.0
4.9
9.2
DinoSR (Liu et al., 2023a)*
4.0
10.1
4.2
9.9
DinoSR (Liu et al., 2023a)
4.2
10.3
4.4
10.2
SpidR
6.1
15.8
6.3
15.9
From Shi et al. (2024).
Original pretrained model that we fino-tuned.
Our re-implemnentation (both pretraining and fine-tuning).
Table 11: Results of SSL models on SUPERB ASR and phoneme recognition tasks. All models are pretrained on LibriSpeech 960h. The best scores are bold and second best are underlined.
Content
Model
PR PER |
ASR
WER
wav2vec 2.0 (Baevski et al., 2020)
5.74
6.43
HuBERT (Hsu et al., 2021)
5.41
6.42
data2vec (Baevski al., 2022)
4.69
4.94
data2vec 2.0 (Baevski et al., 2023)
3.93
4.91
WavLM Base (Chen et al., 2022)
4.84
6.21
MRR-HBBERT mono-bas base (Shi et al., 2024)
4.16
5.76
DinoSR (Liu et al., 2023a)
3.21
4.71
DinoSR (Liu ct al., 2023a)
3.54
5.01
SpidR
4.86
7.47
Our re-implementation.
29

---

Published in Transactions on Machine Learning Research (12/2025)
60
70.0
SBLMMP
tSC
67.5
58
65.0
54
62.5
5.0
5.5
6.0
6.5
7.0
7.5
SWUGGY in-vocab
5.0
5.5
6.0
6.5
7.0
7.5
sWUGGY all
70
65
70
60
55
5.0
5.5
6.0
6.5
7.0
7.5
5.0
5.5
6.0
6.5
7.0
7.5
SUPERB ASR (WER in %)
SUPERB ASR (WER in %)
Speech encoder
wav2vec 2.0
HuBERT
WavLM base
DinoSR
DinoSR (ours)
SpidR
70.0
sBLIMP
tSC
67.5
58
56
65.0
62.5
3.5
4.0
4.5
5.0
5.5
SWUGGY in-vocab
3.5
4,0
4.5
5.0
5.5
SWGGGY all
80
65
70
60
60
35
5.5
3.5
SUPERB PR (PER in %)
SUPERB PR. (PER in %)
Figure 16: Spoken language modeling against supervised evaluation of speech encoders. The speech encoders are all trained on LibriSpeech, and the LMs are evaluated using V = 256 speech units from the best layer of each encoder in terms of ABX discriminability. The SUPERB evaluation is done via a linear probe learned on an average of the intermediate representations. This figure comprises the results from tables and
30


Reviews:
[{"review_id": "PQdXtP1Pvx", "reviewer": "Reviewer_nrt3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The paper is well-written, and the proposed method is explained clearly and rigorously. The key differences from DinoSR are highlighted properly.\n* While the technique is a simple modification of DinoSR, the performance improvements are significant, making this a valuable contribution.\n* The authors worked on an efficient implementation, allowing the model to be trained with reasonable computational resources.\n\nWeaknesses:\n\n* Limited Evaluation: The evaluation primarily focuses on phonetic content and consistency. Speech, however, can capture richer features such as emotion, accent, prosody, and speaker identity, which are not fully considered here. By focusing mostly on phonemes, I think we are not fully evaluating the potentialities of spoken language models. A broader evaluation, including acoustic, gender, and emotion consistency, would make the results of the paper more valuable for the community.\n* Downstream tasks: Connected to the point above, it looks like the authors do not report results on popular downstream benchmarks (e.g., SUPERB). While DinoSR is also limited in this aspect, at least it reports some results for ASR. I think that, while evaluating spoken language modeling is totally fine, assessing performance on downstream tasks can provide deeper insights into the actual model performance.\n* Comparison with audio tokens: There is no comparison with modern audio token methods (e.g., Encodec, Mimi), which I think are relevant alternatives to the proposed approach.\n* WavLM: Hubert is a strong baseline, but WavLM has been shown to outperform it in many tasks (see the SUPERB Benchmark). I would have expected a comparison with WavLM.\n* Scaling laws: Figure 3 and Table 3 do not report scaling laws for DinoSR. It is not clear why to me.\n* Related work coverage: The Self-Supervised Speech Representation Learning section could be broader, including additional relevant works such as WavLM, BesRQ, and others.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes SpidR, a self-supervised method for learning speech representations based on an improvement of the previously proposed DinoSR. The key idea is a modification of the DinoSR framework, based on taking advantage of the student’s own intermediate representation. Compared to DinoSR, SpidR turned out to be more stable and to better capture phonetic information when its learned units are used in spoken language modeling. The authors also focus on training efficiency, showing that SpidR can be trained with reasonable computational resources, making it more accessible.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Based on the weaknesses I noticed, I would recommend that the authors consider the following changes:\n\n* Include metrics beyond phonetic consistency (e.g., acoustic consisentcy, gender consistency, emotion consistency, etc). If not feasible, provide a clear justification.\n* Assess the performance on different downstream tasks, such as those in SUPERB or similar benchmarks. While I understand it may not be feasible to cover all downstream tasks, even one or two additional evaluations would strengthen the paper. If not applicable or relevant, explain why.\n* Compare the proposed method with popular audio codes (e.g., Encodec, Mimi). If not relevant, justify why.\n* Include a comparison with WavLM, or explain why the it is omitted, as WavLM often outperforms HuBERT on many SUPERB tasks.\n* Report the scaling laws for DinoSR.\n* Expand the Self-Supervised Speech Representation Learning section with additional works from the literature.", "broader_impact_concerns": "No ethical concerns for this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eggasyAfM4", "reviewer": "Reviewer_KX3n", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1、The layer-aligned prediction mechanism is well-motivated (addressing distribution shift in DinoSR) and rigorously validated through codebook stability metrics (Figure 2) and ablation studies (Table 8).\n\n2、The study provides a thorough experimental analysis, systematically investigating scaling laws (Figure 3), rigorously evaluating vocabulary size effects (Table 3), and conducting detailed examinations of layer-wise discriminability (Figures 7, 11-12).\n\nWeakness:\n\n1、Limited novelty: the work offers limited novelty, as its core idea—intermediate-layer prediction—is an incremental extension of DinoSR. While SpidR reorganizes prediction heads (student layer *k* → teacher layer *k*), it retains DinoSR's key components like cross-entropy loss with online codebooks and the teacher-student framework. \n\n2、Insufficient downstream validation: While spoken language modeling (SLM) metrics (sWUGGY/sBLIMP/tSC) are reported, critical speech tasks like ASR are absent. HuBERT are widely used in ASR, yet SpidR’s utility for transcription remains unverified. Without ASR results (e.g., WER on LibriSpeech), the claim of \"learning linguistic units\" is weakly supported.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes SpidR, a self-supervised speech representation model that modifies DinoSR’s architecture by aligning intermediate student layers with corresponding teacher layers for masked prediction. Key contributions include the architectural modification, SpidR uses each student intermediate layer to predict the target instead of using the final layer. According to the experiments, the proposed method improves the training stability and efficiency, and demonstrates gains on zero-shot spoken language modeling(sWUGGY, sBLIMP, tSC) and phonetic discriminability (ABX, PNMI) over HuBERT and DinoSR.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Include ASR downstream tasks: Evaluate SpidR on standard ASR benchmarks (e.g., LibriSpeech test sets) using frozen features + linear probe or E2E fine-tuning. Compare WERs against HuBERT/DinoSR to validate practical utility.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eQ275nJlTb", "reviewer": "Reviewer_3tfg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n- This appears to be a good engineering effort. If you do not have 16x A100 but say only 4 then you could build one in 4 days. With access to just 1 GPU you could get somewhere comparable in 16 days, which is not too bad. \n- Although layer-level teacher student is not new, the authors suggest that in this specific context it is. Given popularity of SSLs, this might be of interest to other people working in this area.\n\nWeaknesses: \n- The way the manuscript reads is that this is a speech centric work with limited theoretical contribution and largely an engineering effort. In my mind the target audience of this work might lie elsewhere.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript shows and will provide open source implementation of how in about 1 day of time and with access to 16 A100 GPUs you can build a competitive SSL model on 960 hours of publicly available read speech (LibriSpeech).\n\nSpecific contributions are:\n\n1) layer-level teacher student training where each affected teacher layer provides targets to the corresponding student layer\n\n2) code engineering\n\n3) extensive evaluation\n\nRegarding 1), this is not new (google shows https://arxiv.org/pdf/2210.01351 as an example) but appears not to be done with models explored in this work. Please add a bit info on what kind of layer-level training has been done prior to you. \n\nRegarding 2), only limited details are provided but they sound quite reasonable. \n\nRegarding 3), this is common in this line of work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) It appears that you are following https://arxiv.org/pdf/2504.08528 to position your work as \"speech language modelling\". Without a proper introduction and explanation, I find this terminology/taxonomy of models confusing. Given how much nuances are there and variants to mix speech and language, could you please clearly define what they are and how they are different from models definitely known to the audience of these proceedings.\n\n2) Could you please check your manuscript for grammar/semantics. I find phrasing used in many places to be borderline at best. For example,  \"some approaches added a new step on the units\" where I struggle to understand what this mean. Another example is \"The selected checkpoint in the one with the lowest validation loss\". There may be more than that. \n\n3) I find discussion of terabytes (storage) and weeks (time) to be confusing and somewhat misleading as these units are directly linked with quantities of data used and amount of GPU resource available. Could you please present them with reference to some standard (per hour of speech, per 1xGPU)?\n\n4) One and perhaps the most important claim in this manuscript is that it \"makes pretraining significantly more stable and resistant to codebook collapse\". I believe you present evidence for codebook collapse albeit it is averaged out in PPL across the data used to compute it. I do not believe I see any evidence presented of the previous methodologies displaying it. I also do not see how you measure stability and how it manifests itself in your experiments. I would also love to see some examples of its lack in existing methodologies. Simply saying that does not suffice. \n\n5) Please make it very clear that your methodology requires a certain restriction on the nature of students/teachers (at least as many layers? do you require any other topological and other constraints?)\n\n6) You have presented an evidence that pairing all layers of students and teachers is a good idea. Due to model-centric view (everything is a model), I am not fully certain if you have presented your model where only top layers are paired. I do not believe you have presented any evidence of using pairing for some layers only and it is not clear which layers are important to pair and which are not. Could you please explain why none (some) of that investigation has not been done?\n\n7) I struggle to understand what \"using the latest advancements of PyTorch\" means.\n\n8) I would be careful making statements such as \"hackable foundation for future research without legacy dependencies\" as torch itself is a legacy dependency.\n\n9) In addition to obvious statements such as that your model/methodology outperforms other models/methodologies on some/all metrics, could you please comment on cases where it did not (there are quite a few) as understanding is informative. \n\n10) Given how many different tasks/metrics have been employed in this work, could you please comment on if they are the most important tasks/metrics used by all other authors in the area. It was not always clear to me how common, important they are. It is also unclear to me what they do not show. Please comment on limitations of what you are presenting.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)