# Towards Early Prediction of Self-Supervised Speech Model Performance

Ryan Whetten, Lucas Maison, Titouan Parcollet, Marco Dinarelli, Yannick Estève

► To cite this version:

Ryan Whetten, Lucas Maison, Titouan Parcollet, Marco Dinarelli, Yannick Estève. Towards Early Prediction of Self-Supervised Speech Model Performance. Interspeech 2025, ISCA, Aug 2025, Rotterdam, Netherlands. hal-05091409

HAL Id: hal-05091409

https://hal.science/hal-05091409v1

Submitted on 31 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Towards Early Prediction of Self-Supervised Speech Model Performance

Ryan Whetten<fnref n="1" />, Lucas Maison<fnref n="1" />,², Titouan Parcollet³, Marco Dinarelli⁴, Yannick Estève<fnref n="1" />

¹Laboratoire Informatique d'Avignon, Avignon Université, France

²Thales SIX, France

³Samsung AI Center Cambridge, United Kingdom

⁴Laboratoire d'Informatique de Grenoble, Université Grenoble Alpes, France

ryan.whetten@univ-avignon.fr

## Abstract

In Self-Supervised Learning (SSL), pre-training and evaluation are resource intensive. In the speech domain, current indicators of the quality of SSL models during pre-training, such as the loss, do not correlate well with downstream performance. Consequently, it is often difficult to gauge the final downstream performance in a cost efficient manner during pre-training. In this work, we propose unsupervised efficient methods that give insights into the pre-training quality of SSL speech models, namely, measuring the cluster quality and rank of the embeddings produced by the SSL model. Results show that measures of cluster quality and rank correlate better with downstream performance than the pre-training loss, reducing the need for GPU hours and labeled data in SSL model evaluation.

**Index Terms:** self-supervised learning, speech, evaluation, rank, clustering

## 1. Introduction

Self-Supervised Learning (SSL) is the process of training a model where the targets are derived directly from the input data itself, decreasing the need of human-labeled data. SSL models are usually trained in two stages: *pre-training*, where the model is trained using unlabelled data, and subsequently, *fine-tuning*, where the model is trained for a specific task using labeled data [1]. In the speech processing domain, SSL models have performed well on a variety of tasks such as Automatic Speech Recognition (ASR) or Speaker Verification (SV) [1, 2].

Although performant, SSL speech models are computationally and memory expensive, usually requiring anywhere from 8 - 128 high-end GPUs [3, 4], and up to tens of millions of hours of data [5]. While recent work has been done to reduce the computation and memory requirements necessary to train these models [3, 6, 7, 8], limited research has focused on the efficient evaluation of these models during pre-training. Nevertheless, if it were possible to evaluate a model's performance effectively during pre-training, the total number of models requiring full training could be significantly reduced, leading to significant energy and compute savings.

While it is common practice to monitor the SSL pre-training loss, it does not necessarily correlate well with the downstream performance [3]. Thus, the most reliable method of evaluation remains to simply test each model on the downstream tasks. This can be done by fine-tuning for a specific task, or by using benchmarks such as SUPERB or the Multi-Probe Speech Self-Supervision benchmark (MP3S) [2, 9], in which the pre-trained SSL weights are frozen, and only a light downstream model, sometimes called a *probe*, is trained. Though these evaluation methods are feasible, they require labeled data and additional GPU hours.

To overcome the need for labeled data and additional GPU hours, researchers in the field of computer vision proposed using the effective rank to evaluate the quality of SSL representations [10, 11]. This methodology has shown promising results in the speech domain on the tasks of Phoneme Recognition, Keyword Spotting, and Speaker Identification [12]. However, whether this methodology extends to other tasks such as ASR and SV remains an open question. Additionally, no work has explored other measures like clustering methods. Furthermore, whether this approach can be used early in pre-training to predict final downstream performance remains unclear.

To address these gaps, we propose two compute- and data-efficient unsupervised methods as indicators of final downstream performance at early stages of pre-training: measuring the clusters quality and the rank of the embeddings outputted by SSL models. To test these methods, we experiment with the BERT-based Speech pre-Training with Random-projection Quantizer (BEST-RQ) architecture [13], training 30 BEST-RQ models varying the batch size and the proportion of audio masked and focus on ASR and SV tasks.<fnref n="1" /> Results show that, for ASR, clustering measures and the rank of SSL embeddings correlate better with the final performance than the pre-training loss, even at early stages in training, having the potential to save thousands of GPU hours in future experiments.

## 2. Background

In this section, we give an overview of BEST-RQ, explain how the pre-training loss is calculated, and show theoretically why it is not a good indicator of downstream performance. This is supported empirically in Section 4.

### 2.1. Architecture of BEST-RQ

We use BEST-RQ due to its efficiency, performance, and open source implementation in the SpeechBrain library [14, 15]. In BEST-RQ there are three main components: the quantizer, the acoustic feature extractor (AFE) consisting of two CNN layers, and the context encoder made up of a series of Conformer layers [16]. Given an audio, Mel filterbanks features are calculated and then given to the quantizer. The AFE also receives the Mel filterbanks features, except a random portion of Mel filterbanks are *masked* (replaced with random noise). The output of the AFE is then passed to the context encoder.

Using a random frozen linear projection and a codebook lookup, the quantizer generates discrete pseudo-targets and the context encoder is trained to predict these targets for the masked portions. The loss is calculated by taking the cross entropy between the predicted targets and the pseudo-targets. Due to these

<footnote n="1">¹code at https://github.com/whettenr/ssl-speech-evaluation</footnote>
training conditions, the context encoder must use the surround-
ing unmasked context in order to make accurate predictions on
the masked portions [13].

## 2.2. Pre-training loss vs downstream performance

In this section, we show theoretically how certain hyper-
parameters changes can affect the loss. Let us consider two SSL
BEST-RQ models M_a and M_b, with respective masking hyper-
parameters m_a and m_b, which represent the percentage of audio
that is masked during pre-training. Let m_a < m_b and m_a ≈ 0
meaning almost none of the input is masked. Let both models
use the same quantizer Q. Finally, let x = [x_1, x_2, ..., x_T] be an
input sequence of vectors of length T.

As the context encoder uses the global context to predict the
targets of the masked sections, the larger the mask, the smaller
amount of uncorrupted input available to the model. Thus, the
model M_b, having a larger mask m_b, will have more uncertainty
in its prediction, resulting in higher entropy, and consequently, a
higher loss. However, as will be explained subsequently, having
a higher loss is not always a negative factor in this scenario.

Since $m_a$ is very small, $M_a$ would be trained to mimic $Q$, essentially learning a single linear projection—a simple task for Conformers—resulting in a low loss. In other words, as $M_a$ is trained, the output of $M_a(x_i) \approx Q(x_i)$. Conversely, because $M_b$ has a higher mask percentage, it does more than simply replicate $Q$. Instead, it uses the unmasked context to make accurate predictions of $Q$, compelling $M_b$ to learn the statistical properties of speech and audio. As a result, while $M_b$ incurs a higher loss than $M_a$, it learns important statistical properties that contribute to a strong initialization for downstream tasks.

As a consequence, while the pre-training loss can be used
as an overall guide, due to the fact that certain hyper-parameters
and architectural changes have a direct effect on the difficulty of
the pre-training task, and subsequently the pre-training loss, the
pre-training loss cannot be used as an indicator of downstream
performance. To overcome this issue, recent studies have pro-
posed using the rank of the output of SSL models for evaluation.

## 2.3. Using rank for unsupervised evaluation

The rank of a matrix is the number of linearly independent rows or columns and is a proxy for the information content of the matrix. Thus, the higher the rank, the less redundancy, the more information is contained in the matrix. The classical way to compute the rank of a matrix is to count the number of non-zero singular values. However, when working with real-world matrices which contain non-integer values, many, if not all of the singular values will be non-zero. This necessitates the use of a threshold. However, in practice this threshold is quite application-dependent and difficult to set. To solve this issue, researchers proposed using the effective rank (E-Rank) [11], which is a real-valued extension of the rank:

$$ \text{E-Rank}(A) = \exp \left( - \sum_{i=1}^{\min(N,M)} p_i \log p_i \right), \quad (1) $$

where A is a matrix of size (N, M), and p_i are the normalized singular values. The effective rank, coined RankMe, was shown to be useful in the evaluation of self-supervised representations in computer vision [10].

In the speech domain, researchers used the effective rank
by adapting it to temporal data with RankMe-t [12] which can

be calculated as follows. Given a dataset of n audio files, we
use an SSL model to generate series of embedding sequences
S. Each eᵢ ∈ S is an embedding sequence with a respec-
tive length Tᵢ, i.e. eᵢ = [eᵢ¹, eᵢ², ..., eᵢTᵢ]. Each sequence is
summed along the time dimension resulting in one vector per
audio file, eᵢsum = ΣTᵢ eᵢt. These vectors are then concate-
nated together to form the matrix Z = [e₁sum, e₂sum, ..., eₙsum].
RankMe-t is then calculated by taking the E-Rank of Z or
RankMe-t(Z) = E-Rank(Z).

Although RankMe-t was shown to correlate well with
downstream performance [12], correlations with the pre-
training loss values were not reported, leaving unanswered
whether RankMe and RankMe-t serve as better indicators.
Moreover, RankMe-t operates at the utterance level as each se-
quence of embeddings is summed together, resulting in one vec-
tor per utterance. While this might capture useful information
as to whether or not the model is pre-training well, we hypoth-
esis that RankMe-t looses useful information.

We build on and expand prior research by experimenting
with cluster measures, investigating the use of rank and cluster
measures in the early stages of pre-training, and focusing on
two different tasks which are ASR and SV.

# 3. Methods

In this section, we introduce the unsupervised measures used
in our experiments and then describe the experimental settings,
including the models trained and the downstream tasks.

## 3.1. Unsupervised measures for SSL model evaluation

We hypothesize that the dispersion (or lack thereof) of the out-
put embeddings in the latent space is correlated with the down-
stream performance of the model. In order to test this hypothe-
sis, we use the following clustering and rank-based measures.

### 3.1.1. Clustering measures

For clustering measures, we train KMeans [17, 18] clustering
models on the SSL embeddings from a given layer. Specifically,
we use MiniBatchKmeans [19], a faster variant of KMeans im-
plemented in the scikit-learn [20] library, with a KMeans++ ini-
tialization [21]. We set k = 1024 as the number of clusters.

Once a clustering model is trained, we use two metrics for
evaluating the clustering quality: inertia, also called the Within-
Cluster Sum of Squares (WCSS), and the Davies-Bouldin (DB)
index [22]. The WCSS measures the average intra-cluster dis-
tance, or the mean distance between each data point and its as-
signed cluster centroid. This can be expressed by the following:

$$ \text{WCSS} = \sum_{k=1}^{K} \sum_{i \in C_k} \|x_i - \mu_k\|^2, \quad (2) $$

where K is the number of clusters, Ck is the set of data points
assigned to cluster k, x_i is a single data point and μ_k is the
centroid of cluster k. In contrast to the WCSS, the DB index
takes into the intra- and inter-cluster distance and is calculated
by the following:

$$ \text{DB} = \frac{1}{K} \sum_{i=1}^{K} \max_{i \neq j} \frac{s_i + s_j}{d_{ij}}, \quad (3) $$

where K is the number of clusters, s_i is the diameter of cluster
i and d_{ij} is the distance between centroids of clusters i and j.
### 3.1.2. Rank-based metrics

For our rank based methods, we use RankMe-t as described in Section 2.3 as a baseline, and propose an additional rank based measure that we call the Global Effective Rank (GER).

Given the series of *n* embedding sequences *S* (same as Section 2.3), let *C* represent the concatenation of all the embeddings in *S*. We propose the Global Effective Rank as the Effective rank of *C*. This can be expressed as follows:

$$C = \|_{i=0}^{n} S_i = e_0 \| e_1 \| e_2 \| \cdots \| e_n. \quad (4)$$

$$\text{GER} = \text{E-Rank}(C). \quad (5)$$

The motivation behind GER is to attempt to capture better the true rank of the raw representations instead of computing the rank of a compressed version as is done in RankMe-t.

## 3.2. Experimental settings

As previously noted, for our experiments we work with BEST-RQ due to its state-of-the-art performance, efficiency, and open source implementation. The efficiency is particularly important as we need to train many models to evaluate the methodology. We create various hyper-parameters configurations by varying the batch size from 15 to 120 minutes and the proportion of audio masked from 40% to 80%. We train all models for 200k steps on the 960 hours set of LibriSpeech [23]. We focus on two main checkpoints: 50k steps and 200k steps. While most SSL speech models are trained for longer [24, 25, 13], previous work has shown 200k steps to be sufficient for evaluation [7, 14].

For our first task, we evaluate the frozen SSL models on in-domain ASR using the ContextNet LibriSpeech-100 task from the MP3S benchmark [9]. In this task, a minimal ContextNet [26] probe is trained on the *train-clean-100* subset of LibriSpeech, and performance is measured by word error rate (WER) on the *dev-clean* set using greedy decoding without a language model. Since fine-tuning is common in practice, we also fine-tune the SSL models on in-domain ASR data using the SpeechBrain fine-tuning recipe. Evaluation follows the same setup as the frozen model, measuring WER on the *dev-clean* set with greedy decoding and no language model. For our third task, we evaluate the frozen SSL models on out-of-domain ASR using the LSTM CommonVoice 11.0 Basque ASR task from MP3S. For this task, a 2-layer BiLSTM probe is trained on the provided train splits. Evaluation is done using the WER on the provided development set. Lastly, to assess performance on a non-ASR task, we follow the MP3S benchmark to evaluate the frozen SSL models on Speaker Verification using VoxCeleb1 [27]. In this task, an ECAPA-TDNN [28] probe is trained and models are evaluated using error rate on the development set.

# 4. Results

We first report on preliminary experiments that informed our choices on the amount of audio to use for the unsupervised evaluation methods and the number of epochs for the downstream task. Then we present results for each task.

## 4.1. Preliminary experiments

For generating the output embeddings, we randomly sample about 1 hour of audio from each dataset. This stems from preliminary experiments in which we find that the global effective rank plateaus quickly, increasing minimally when using more

<figure><img src="image_2.png" /><figcaption>Figure 1: Correlations between 4 selected measures (calculated on embeddings at 50k pre-training steps) and the future WER on the LibriSpeech dev-clean set (at 200k pre-training steps and 20 epochs fine-tuning). Each point represents a model with different hyper-parameters (batch size and mask percentage). Fig. 1a, show that some of the models with the lowest WER (y-axis) have a relatively high pre-training loss (x-axis) resulting in a low correlation, whereas unsupervised measures have a stronger correlation with WER.</figcaption></figure>

than 1 hour of audio. Computing all embeddings and metrics for a single layer of one model takes about 20 minutes on CPUs, with the clustering accounting for most of the time.

Another preliminary finding is a strong correlation between the WER early on in pre-training and the final WER when working with in-domain data. For example, the Pearson correlations between WER from minimally trained models (50k pre-training steps and 5 downstream epochs) and their final counterparts (200k pre-training steps, 20 downstream epochs) were 0.951 and 0.967 when freezing and fine-tuning the SSL model, respectively. This means that in order to gauge performance with labeled data, one does not need to fully pre-train nor fully fine-tune when working with in-domain ASR. Thus, for the ASR experiments we only run the downstream for 5 epochs for the models at 50k checkpoints to save computation.

## 4.2. Automatic speech recognition

For in-domain ASR experiments, results were similar whether the SSL models were fine-tuned or frozen; both cluster and rank methods correlate with the WER better than the pre-training loss baseline (see first four columns of Table 1). Some of the highest correlations came from the GER ranging from -0.63 to -0.96, meaning the higher the rank the better the performance (lower WER). In contrast, the pre-training did not correlated strongly with WER, with correlations ranging from -0.170 to -0.303. Graphs comparing a select few of these measures with the WER are shown in Fig. 1.

Similarly, we find high correlations between clustering and rank measures at 50k steps of pre-training and final WER after 200k steps of pre-training and 20 epochs of fine-tuning. This suggests that, without any fine-tuning, we can guage the final performance very early on in pre-training for in-domain ASR.
<table><caption>Table 1: Pearson Correlations between downstream performance and various unsupervised measures. Compared to pre-training loss (first row), unsupervised metrics tend to exhibit a much stronger correlation with downstream performance at either 50k or 200k steps of pre-trainin. Highest correlations are found in the in-domain tasks. Measures based on rank are more correlated with downstream performance than clustering measures. LS = LibriSpeech, CV = CommonVoice, VoxCel = VoxCeleb, (f.t.) = SSL Model is fine-tuned.</caption><thead><tr><th>Measure</th><th>LS 50k</th><th>LS 200k</th><th>LS 50k (f.t.)</th><th>LS 200k (f.t.)</th><th>CV 50k</th><th>CV 200k</th><th>VoxCel 50k</th><th>VoxCel 200k</th></tr></thead><tbody><tr><td>Pre-train Loss</td><td>-0.183</td><td>-0.303</td><td>-0.170</td><td>-0.332</td><td>0.107</td><td>0.143</td><td>0.084</td><td>-0.598</td></tr><tr><td>Inertia (WCSS)</td><td>0.686</td><td>0.622</td><td>0.710</td><td>0.633</td><td>0.412</td><td>-0.556</td><td>-0.000</td><td>0.367</td></tr><tr><td>Davies-Bouldin</td><td>-0.903</td><td>-0.494</td><td>-0.929</td><td>-0.486</td><td>-0.854</td><td>-0.414</td><td>0.155</td><td>0.356</td></tr><tr><td>RankMe-t</td><td>-0.752</td><td>-0.735</td><td>-0.734</td><td>-0.756</td><td>-0.387</td><td>0.110</td><td>0.376</td><td>0.567</td></tr><tr><td>Global Rank (GER)</td><td>-0.931</td><td>-0.630</td><td>-0.962</td><td>-0.634</td><td>-0.885</td><td>-0.003</td><td>0.163</td><td>0.584</td></tr><tr><td>In-Domain</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>No</td><td>No</td><td>No</td><td>No</td></tr><tr><td>Epochs on DS task</td><td>5</td><td>20</td><td>5</td><td>20</td><td>5</td><td>20</td><td>20</td><td>20</td></tr></tbody></table>

<table><caption>Table 2: Pearson Correlations between various unsupervised measures and future downstream performance. The unsupervised measures are calculated from the embeddings of SSL models at 50k pre-training steps and the correlation is measured with their respective final performances at 200k pre-training steps. Correlations for in-domain ASR remains high, but not for out-of-domain tasks. LS = LibriSpeech, CV = CommonVoice, VoxCel = VoxCeleb, (f.t.) = SSL Model is fine-tuned.</caption><thead><tr><th>Measure</th><th>LS</th><th>LS (f.t.)</th><th>CV</th><th>VoxCel</th></tr></thead><tbody><tr><td>Pre-train Loss</td><td>-0.296</td><td>-0.324</td><td>0.140</td><td>-0.604</td></tr><tr><td>Inertia (WCSS)</td><td>0.692</td><td>0.729</td><td>-0.378</td><td>-0.027</td></tr><tr><td>Davies-Bouldin</td><td>-0.840</td><td>-0.837</td><td>0.095</td><td>0.056</td></tr><tr><td>RankMe-t</td><td>-0.771</td><td>-0.785</td><td>0.201</td><td>0.200</td></tr><tr><td>Global Rank (GER)</td><td>-0.903</td><td>-0.905</td><td>0.159</td><td>0.318</td></tr></tbody></table>

For example, the correlations between the GER and final WER were around -0.9, yet only around -0.3 for the pre-training loss (first two columns of Table 2).

When it comes to out-of-domain (CommonVoice) ASR experiments, correlations between unsupervised measure and WER remain high at 50k steps, with the highest being -0.885 for the GER. However, the correlations at 200k are not strong, nor are the correlations between unsupervised measures at 50k and the final WER at 200k (Table 2). Examining why this might be the case, we observe that at 200k all the models perform similarly, with an absolute difference in WER of only 2.8 between the worst and best performance, suggesting that if model performance is very similar, it becomes hard to judge based on clustering or rank measures.

### 4.3. Speaker verification

With regards to Speaker Verification, we observe correlations between unsupervised measures and the error rate after 20 epochs of fine-tuning. While they are not as strong as with ASR, the unsupervised measures still correlate better than the pre-training loss. Note that for 200k checkpoints, there is a strong negative correlation between the loss and the downstream performance of -0.598 (last column Table 1), indicating that the higher the loss, the lower the error rate which is not useful. We find that, like CommonVoice, the majority of models performed similarly (90% of the models exhibited error rates within a 1% range), leading to low correlations.

### 4.4. Additional observations

Although previous work has demonstrated that the most important layers for ASR are the middles layers as they have been shown to contain the most phonetic and word information [29], our work agrees with [12] where the layers most useful for the downstream task are not the layers that correlated most with the downstream performance. For ASR, we found that the measures computed on the embeddings extracted from the first and last layers correlated the most with downstream performance, indicating that the way the model processes incoming acoustic features (or reconstructs them in the case of the last layers) is an important indicator of downstream ASR performance. For SV, we found that the middle layers tended to have stronger correlations than beginning or final layers. We report the correlations from measures calculated on the outputs of layer 12 for ASR and layer 8 for SV in both Table 1 and 2.

One other finding is that the correlations between inertia and downstream performance are positive except in the case of out-of-domain ASR at 200k steps of pre-training where all the models converged to very similar performance. This indicates that the lower the inertia, the better the clustering, and thus better performance. On the other hand, the correlations with the DB index are negative for ASR indicating the worse the clustering, the better the performance. We believe that this because the DB index penalizes clusters that are close to their nearest neighbors. However, overlapping clusters do not necessarily indicate poor performance; instead, close neighboring clusters suggest efficient use of the latent space.

## 5. Discussion

We propose using clustering and rank measures as efficient, unsupervised methods for evaluating the performance of SSL speech models and show that these measures are better indicators of downstream performance than the pre-training loss. To illustrate the potential compute savings, suppose one pre-trains the 30 models from this work to 400k steps. With our settings this would take around 16,800 GPU hours. If one instead pre-trained all models to 50k and then used the rank to select and train only the top 5 models to 400k steps, this would take approximately 6,000 hours, saving approximately 10,800 GPU hours. We believe that future work could involve exploring these methods further and developing more robust measures to accurately evaluate SSL speech models early on in training. We believe that future work could involve exploring these methods further and developing.
# 6. Acknowledgements

This work received funding from the French ANR E-SSL project N°ANR-22-CE23-0013 and used HPC resources from GENCI-IDRIS projects AD011014732 and A0131013821.

# 7. References

[1] A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløe et al., "Self-supervised speech representation learning: A review," *IEEE Journal of Selected Topics in Signal Processing*, 2022.

[2] S. wen Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, "Superb: Speech processing universal performance benchmark," in *Interspeech 2021*, 2021, pp. 1194–1198.

[3] W. Chen, X. Chang, Y. Peng, Z. Ni, S. Maiti, and S. Watanabe, "Reducing barriers to self-supervised learning: Hubert pre-training with academic compute," in *Interspeech 2023*, 2023, pp. 4404–4408.

[4] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, "Xls-r: Self-supervised cross-lingual speech representation learning at scale," in *Interspeech 2022*, 2022, pp. 2278–2282.

[5] Y. Zhang, W. Han, J. Qin, Y. Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V. Axelrod, G. Wang et al., "Google usm: Scaling automatic speech recognition beyond 100 languages," *arXiv preprint arXiv:2303.01037*, 2023.

[6] A. Baevski, A. Babu, W.-N. Hsu, and M. Auli, "Efficient self-supervised learning with contextualized target representations for vision, speech and language," in *International Conference on Machine Learning*. PMLR, 2023, pp. 1416–1429.

[7] T. Parcollet, S. Zhang, R. van Dalen, A. G. C. P. Ramos, and S. Bhattacharya, "On the (in)efficiency of acoustic feature extractors for self-supervised speech representation learning," in *Interspeech 2023*, 2023, pp. 581–585.

[8] R. Whetten, T. Parcollet, A. Moumen, M. Dinarelli, and Y. Estève, "An analysis of linear complexity attention substitutes with best-rq," in *Spoken Language Technology Workshop*, 2024.

[9] S. Zaiem, Y. Kemiche, T. Parcollet, S. Essid, and M. Ravanelli, "Speech self-supervised representation benchmarking: Are we doing it right?" in *Interspeech 2023*, 2023, pp. 2873–2877.

[10] Q. Garrido, R. Balestriero, L. Najman, and Y. Lecun, "RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank," in *Proceedings of the 40th International Conference on Machine Learning*, ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., vol. 202. PMLR, 23–29 Jul 2023, pp. 10929–10974. [Online]. Available: https://proceedings.mlr.press/v202/garrido23a.html

[11] O. Roy and M. Vetterli, "The effective rank: A measure of effective dimensionality," in *2007 15th European Signal Processing Conference*, 2007, pp. 606–610.

[12] Z. Aldeneh, V. Thilak, T. Higuchi, B.-J. Theobald, and T. Likhomanenko, "Towards automatic assessment of self-supervised speech models using rank," *arXiv preprint arXiv:2409.10787*, 2024.

[13] C.-C. Chiu, J. Qin, Y. Zhang, J. Yu, and Y. Wu, "Self-supervised learning with random-projection quantizer for speech recognition," in *Proceedings of the 39th International Conference on Machine Learning*, ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, Eds., vol. 162. PMLR, 17–23 Jul 2022, pp. 3915–3924. [Online]. Available: https://proceedings.mlr.press/v162/chiu22a.html

[14] R. Whetten, T. Parcollet, M. Dinarelli, and Y. Estève, "Open implementation and study of best-rq for speech processing," in *Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 2024, Workshop on Self-supervision in Audio, Speech and Beyond*, 2024, arXiv preprint available at https://arxiv.org/abs/2405.04296. [Online]. Available: https://arxiv.org/abs/2405.04296

[15] M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, "SpeechBrain: A general-purpose speech toolkit," 2021, arXiv:2106.04624.

[16] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, "Conformer: Convolution-augmented transformer for speech recognition," in *Interspeech 2020*, 2020, pp. 5036–5040.

[17] J. MacQueen, "Some methods for classification and analysis of multivariate observations," Proc. 5th Berkeley Symp. Math. Stat. Probab., Univ. Calif. 1965/66, 1, 281-297 (1967), 1967.

[18] S. Lloyd, "Least squares quantization in pcm," *IEEE Transactions on Information Theory*, vol. 28, no. 2, pp. 129–137, 1982.

[19] D. Sculley, "Web-scale k-means clustering," in *Proceedings of the 19th International Conference on World Wide Web*, ser. WWW '10. New York, NY, USA: Association for Computing Machinery, 2010, p. 1177–1178. [Online]. Available: https://doi.org/10.1145/1772690.1772862

[20] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, "Scikit-learn: Machine learning in Python," *Journal of Machine Learning Research*, vol. 12, pp. 2825–2830, 2011.

[21] D. Arthur and S. Vassilvitskii, "k-means++: the advantages of careful seeding," in *Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms*, ser. SODA '07. USA: Society for Industrial and Applied Mathematics, 2007, p. 1027–1035.

[22] D. L. Davies and D. W. Bouldin, "A cluster separation measure," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. PAMI-1, no. 2, pp. 224–227, 1979.

[23] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, "Librispeech: an asr corpus based on public domain audio books," in *2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)*. IEEE, 2015, pp. 5206–5210.

[24] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations," *Advances in neural information processing systems*, vol. 33, pp. 12449–12460, 2020.

[25] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, "Hubert: Self-supervised speech representation learning by masked prediction of hidden units," *IEEE/ACM transactions on audio, speech, and language processing*, vol. 29, pp. 3451–3460, 2021.

[26] W. Han, Z. Zhang, Y. Zhang, J. Yu, C.-C. Chiu, J. Qin, A. Gulati, R. Pang, and Y. Wu, "ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context," in *Proc. Interspeech 2020*, 2020, pp. 3610–3614.

[27] A. Nagrani, J. S. Chung, and A. Zisserman, "Voxceleb: A large-scale speaker identification dataset," in *Interspeech 2017*, 2017, pp. 2616–2620.

[28] B. Desplanques, J. Thienpondt, and K. Demuynck, "Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification," in *Interspeech 2020*, 2020, pp. 3830–3834.

[29] A. Pasad, J.-C. Chou, and K. Livescu, "Layer-wise analysis of a self-supervised speech representation model," in *2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)*, 2021, pp. 914–921.