# An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training

Yanis Labrak, Richard Dufour, Mickaël Rouvier

► To cite this version:

Yanis Labrak, Richard Dufour, Mickaël Rouvier. An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training. International Conference on Text, Speech, and Dialogue, Kamil Ekštein; Miloslav Konopík; Ondřej Pražák; František Pártl, Aug 2025, Erlangen, Germany. pp.13-24, 10.1007/978-3-032-02548-7_2. hal-05225350

HAL Id: hal-05225350

https://hal.science/hal-05225350v1

Submitted on 2 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC0 - Public Domain Dedication 4.0 International License
# An Empirical Analysis of Discrete Unit Representations
in Speech Language Modeling Pre-training

Yanis Labrak¹,³, Richard Dufour², and Mickaël Rouvier¹

Laboratoire Informatique d'Avignon, Avignon University, Avignon, France
Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, Nantes, France
Zenidoc, Marseille, France

**Abstract.** This paper investigates discrete unit representations in Speech Language Models (SLMs), focusing on optimizing speech modeling during continual pre-training. In this paper, we systematically examine how model architecture, data representation, and training robustness influence the pre-training stage in which we adapt existing pre-trained language models to the speech modality. Our experiments highlight the role of speech encoders and clustering granularity across different model scales, showing how optimal discretization strategies vary with model capacity. By examining cluster distribution and phonemic alignments, we investigate the effective use of discrete vocabulary, uncovering both linguistic and paralinguistic patterns. Additionally, we explore the impact of clustering data selection on model robustness, highlighting the importance of domain matching between discretization training and target applications.

**Keywords:** Speech Language Models, Discrete Units, LLM, Robustness, Phonemes Alignment

## 1 Introduction

The rapid advancement of pre-trained Large Language Models (LLMs) [21, 25] has transformed Natural Language Processing (NLP), enabling systems with remarkable capabilities in text understanding and generation. However, these models remain largely text-based, overlooking the richness of spoken language, which conveys prosody, emotion, and speaker characteristics essential for human communication.

Recent advances in self-supervised learning have made significant strides toward integrating speech into language modeling. Models such as WavLM [6], HuBERT [10] and Wav2Vec 2 [3] have proven particularly effective at learning meaningful speech representations without explicit supervision [24].

The first attempts to bridge this gap between speech and language modeling emerged with Generative Spoken Language Models (GSLM) [13], demonstrating the possibility of learning directly from raw audio without relying on text supervision. This breakthrough was followed by various approaches to integrate speech into language models [19, 18], primarily by incorporating discrete speech representations into their vocabularies [9, 28, 27].

While speech-extended LLMs have demonstrated promising results in downstream tasks [7, 17], their performance remains limited [9] and the fundamental challenge of optimizing speech modeling during continual pre-training remains largely unexplored.
<figure><img src="image_2.png" /><figcaption>Fig. 1: Overview of a Speech Language Model.</figcaption></figure>

This stage is critical, as it determines how models initially learn to process speech input and serves as the foundation for all subsequent speech-related capabilities [18, 15].

In this work, we systematically investigate discrete speech unit representations in language modeling. Through extensive experiments across model scales (135M to 1.7B parameters), encoder architectures, and discretization strategies, we address four fundamental questions: the optimal discretization granularity for different encoders, the impact of model scale on semantic information capture, the robustness of speech units to acoustic perturbations, and the nature of linguistic information embedded in these discrete representations.

Our primary contributions are threefold:

- First, we trained 51 speech language models of varying capacities (135M to 1.7B parameters) on the spoken language modeling objective introduced by SpeechGPT [28] through LoRA fine-tuning of pre-trained textual language models of the SmolLM family.
- Second, we discovered a direct correlation between models' speech modeling capabilities and discrete unit granularity, noting that smaller SLMs struggle to capture semantic information from higher discretization granularity units.
- Finally, we observed that discrete units' information strongly aligns with phonemes while simultaneously capturing other forms of acoustic information.

## 2 Spoken Language Modeling

This section details our methodology for training and comparing our speech-extended language models, with a strong focus on speech representations. The studied SLM architecture follows the approach introduced by SpeechGPT [28] and relies on discrete units and vocabulary expansion.
## 2.1 Model Architecture

We experiment with different variants of SmolLM [4], using three model sizes: 135M, 360M, and 1.7B parameters. The core architecture remains unchanged from the original text models, with the only modification being the expansion of the tokenizer vocabulary to incorporate the newer tokens corresponding to the discrete units (see Section 2.2).

The training objective follows a standard autoregressive language modeling approach with negative log-likelihood loss. For a sequence of tokens $x = (x_1, ..., x_T)$, the loss is computed as:

$$ \mathcal{L} = - \sum_{t=1}^{T} \log p(x_t|x_{<t}) \quad (1) $$

where $p(x_t|x_{<t})$ represents the probability of token $x_t$ given all previous tokens in the sequence.

Our approach does not aim for full acoustic reconstruction but instead prioritizes semantic modeling of speech as we are focusing on the first stage of speech adaptation of pre-trained textual language models. This stage consists exclusively of learning to process speech units alongside its existing text capabilities, as shown in Figure 1.

Training is conducted on 16 Nvidia H100 80GB GPUs with a batch size of 16 and gradient accumulation of 1. Using a context window of 2,048 tokens, we process 524,288 tokens per step. The training runs for 300 steps, processing approximately 157 million tokens in total. To optimize training efficiency and resource utilization, we incorporate several technical improvements such as LoRA adapters [11] (rank 64, alpha 16) for parameter-efficient fine-tuning. We chose BFloat16 precision and Flash Attention 2 to reduce memory overhead. It uses AdamW [14] optimization with a learning rate of $3 \times 10^{-4}$ and applies a weight decay coefficient of 0.1. To ensure reproducible results, the random seed is set to 42.

## 2.2 Speech Encoding and Discretization

To convert the raw speech signal from a continuous form into a discrete one that can be incorporated into the text input of the LLM, we need to have two components: an encoder and a discretizer. Here, we will evaluate four widely used self-supervised speech encoders: WavLM [6], HuBERT [10], XLS-R [2], and Wav2Vec 2 [3]. For all encoders, we extract features from the final hidden layer, as prior work suggests that this layer provides a strong balance between acoustic and linguistic information [26, 23]. No additional fine-tuning of the encoders is performed to maintain a fair comparison of their base capabilities. Each encoder extracts frame-level representations at 50 Hz (20 ms frames), which are then discretized into $k$ clusters that will represent speech units using $k$-means, following standard practices in spoken language modeling [28]. To examine the impact of vocabulary size on modeling performance, we experiment with cluster counts of $k \in \{125, 250, 500, 1000, 2500, 5000\}$.

The k-means clustering used for speech encoders is trained on 2,000 hours of unlabeled speech for each of the following corpora: LibriHeavy [12], GigaSpeech [5], People's Speech [8], or CommonVoice 19 [1]. None of the data selected to build the k-means overlaps with the speech modeling dataset.
## 2.3 Speech modeling dataset

We train the language models to process speech modality using LibriSpeech [22], a widely used speech corpus containing 960 hours of read English speech. This dataset comprises three subsets (100h, 360h, and 500h), providing a diverse range of speakers and recording conditions. Speech segments are processed through our encoding pipeline (Figure 1) and using the newly built discrete speech units that serve as input to the language model.

## 2.4 Evaluation Methodology

The effectiveness of each speech unit configuration is measured using Negative Log-Likelihood (NLL) on the LibriSpeech test-clean set. Lower NLL values indicate better modeling of the speech units by the language model, reflecting more stable and predictable representations of the speech signal. Additionally, prior research [15, 7] suggests a strong correlation between NLL and performance on semantic speech understanding tasks, such as sWUGGY [20]. We maintain consistent frame rates across all models to ensure we can properly compare the NLL. In this case, we use 50 Hz encoders and a shared tokenizer for all large language models.

# 3 Experiments and results

We analyze discrete speech units across four dimensions: encoder and discretization methods (Section 3.1), language model scaling (Section 3.2), acoustic robustness (Section 3.3), and linguistic content (Section 3.4).

## 3.1 Comparing Encoders and Discretization Granularity

Results across varying cluster sizes (see Table 1) show a consistent initial degradation in performance as the number of clusters increases, with NLL values ranging from 4.2-4.7 ($k = 125$) to 7.8-8.1 ($k = 5,000$) at Step 100. Training progression significantly improves performance, particularly between Steps 100-200. Among the evaluated encoders, WavLM achieves the best performance (NLL=2.05, $k = 500$) at Step 300, followed by smaller cluster configurations ($k = 125, k = 250$), which remain competitive (NLL ≈ 2.15). HuBERT shows similar trends with slightly higher NLL across all cluster sizes, while XLS-R and Wav2Vec consistently underperform, particularly at larger $k$ values.

Notably, smaller cluster sizes ($k \le 1,000$) consistently yield better performance. In contrast, models using $k \ge 2,500$ experience substantial degradation, with a sharp increase in NLL. This suggests that larger vocabularies introduce excessive speech unit granularity, potentially leading to noisier token distributions and increased token sparsity. As a result, the model struggles to learn stable speech representations, reinforcing the practical advantage of smaller, more compact cluster sets.
<table><caption>Table 1: Negative log likelihood (↓) comparison of different encoders with varying cluster sizes and built from 2,000 hours of unlabeled speech from LibriHeavy. Results are reported at training steps 100, 200, and 300.</caption><thead><tr><th>Encoder</th><th>Clusters</th><th>Step 100</th><th>Step 200</th><th>Step 300</th></tr></thead><tbody><tr><td rowspan="6">WavLM</td><td>k = 125</td><td>4.681</td><td>2.502</td><td>2.149</td></tr><tr><td>k = 250</td><td>5.356</td><td>2.785</td><td>2.158</td></tr><tr><td>k = 500</td><td>6.040</td><td>2.621</td><td>2.048</td></tr><tr><td>k = 1,000</td><td>6.659</td><td>3.057</td><td>2.189</td></tr><tr><td>k = 2,500</td><td>7.281</td><td>5.073</td><td>4.010</td></tr><tr><td>k = 5,000</td><td>7.869</td><td>5.538</td><td>4.208</td></tr><tr><td rowspan="6">HuBERT</td><td>k = 125</td><td>4.705</td><td>2.596</td><td>2.240</td></tr><tr><td>k = 250</td><td>5.393</td><td>2.825</td><td>2.289</td></tr><tr><td>k = 500</td><td>6.087</td><td>2.909</td><td>2.348</td></tr><tr><td>k = 1,000</td><td>6.711</td><td>3.717</td><td>2.822</td></tr><tr><td>k = 2,500</td><td>7.430</td><td>4.940</td><td>3.827</td></tr><tr><td>k = 5,000</td><td>8.052</td><td>5.759</td><td>4.289</td></tr><tr><td rowspan="6">XLS-R</td><td>k = 125</td><td>4.205</td><td>2.694</td><td>2.433</td></tr><tr><td>k = 250</td><td>4.902</td><td>3.436</td><td>2.916</td></tr><tr><td>k = 500</td><td>5.592</td><td>3.608</td><td>3.034</td></tr><tr><td>k = 1,000</td><td>6.276</td><td>3.964</td><td>3.282</td></tr><tr><td>k = 2,500</td><td>7.201</td><td>5.241</td><td>4.177</td></tr><tr><td>k = 5,000</td><td>7.918</td><td>6.034</td><td>4.959</td></tr><tr><td rowspan="6">Wav2Vec</td><td>k = 125</td><td>4.600</td><td>3.069</td><td>2.534</td></tr><tr><td>k = 250</td><td>5.153</td><td>3.559</td><td>2.880</td></tr><tr><td>k = 500</td><td>5.886</td><td>4.042</td><td>3.251</td></tr><tr><td>k = 1,000</td><td>6.656</td><td>4.712</td><td>3.614</td></tr><tr><td>k = 2,500</td><td>7.647</td><td>5.744</td><td>4.434</td></tr><tr><td>k = 5,000</td><td>8.179</td><td>6.397</td><td>5.057</td></tr></tbody></table>

## 3.2 Impact of Model Scale on Discrete Unit Learning

Table 2 shows the results obtained with the SmolLM model across different training conditions. The larger SmolLM-1.7B model significantly outperforms its smaller counterparts, achieving NLL scores of 1.82-1.95 compared to 2.04-2.24 for the 135M model. This improvement suggests that model capacity strongly influences speech unit modeling quality.

WavLM consistently outperforms HuBERT across all model scales, particularly at lower cluster counts ($k \le 500$). The performance gap between encoders remains relatively stable as model size increases. Larger models show better handling of higher cluster counts, with the 1.7B model demonstrating remarkable stability (NLL 1.83-2.28) within its operational range ($k \le 1,000$), though encountering memory limitations at higher clusters.

Our findings show that the best results are achieved by using larger models with fewer clusters. This approach provides a good balance between model performance and computational efficiency. Additionally, larger models seem to be better at handling both noisy token distributions and sparse token patterns, where smaller models struggle.
<table><caption>Table 2: Negative log-likelihood (↓) comparison of different encoders with varying cluster sizes, built from 2,000 hours of unlabeled speech from LibriHeavy, and trained during 300 steps (approximately 150M tokens).</caption><thead><tr><th rowspan="2">Encoder</th><th rowspan="2">Clusters</th><th colspan="3">SmolLM</th></tr><tr><th>135M</th><th>360M</th><th>1.7B</th></tr></thead><tbody><tr><td rowspan="6">WavLM</td><td>k = 125</td><td>2.149</td><td>2.088</td><td>1.887</td></tr><tr><td>k = 250</td><td>2.158</td><td>2.159</td><td>1.861</td></tr><tr><td>k = 500</td><td>2.048</td><td>2.210</td><td>1.829</td></tr><tr><td>k = 1,000</td><td>2.189</td><td>2.386</td><td>1.937</td></tr><tr><td>k = 2,500</td><td>4.010</td><td>2.674</td><td>OOM</td></tr><tr><td>k = 5,000</td><td>4.208</td><td>2.925</td><td>OOM</td></tr><tr><td rowspan="6">HuBERT</td><td>k = 125</td><td>2.240</td><td>2.158</td><td>1.954</td></tr><tr><td>k = 250</td><td>2.289</td><td>2.278</td><td>2.049</td></tr><tr><td>k = 500</td><td>2.348</td><td>2.499</td><td>2.137</td></tr><tr><td>k = 1,000</td><td>2.822</td><td>2.698</td><td>2.282</td></tr><tr><td>k = 2,500</td><td>3.827</td><td>3.054</td><td>OOM</td></tr><tr><td>k = 5,000</td><td>4.289</td><td>3.377</td><td>OOM</td></tr></tbody></table>

## 3.3 Discrete Unit Stability Under Audio Perturbations

We evaluated discrete unit robustness using a SmolLM-135M model with WavLM encoder ($k = 500$) across k-means built from different datasets. Tests included high-intensity Gaussian noise (Noise-H, SNR 15-20dB), low-intensity Gaussian noise (Noise-L, SNR 5-10dB), and random pitch shifts (±5% range) on the test-clean set of LibriSpeech.

<table><caption>Table 3: Negative log-likelihood (↓) on LibriSpeech test-clean for SmolLM-135M model using WavLM ($k = 500$) built from different speech datasets and trained on LibriSpeech during ≈1 epoch.</caption><thead><tr><th>Source k-means</th><th>Clean</th><th>Noise-H</th><th>Noise-L</th><th>Pitch Shift</th></tr></thead><tbody><tr><td>LibriHeavy</td><td>2.621</td><td>2.692</td><td>2.678</td><td>2.704</td></tr><tr><td>GigaSpeech</td><td>3.073</td><td>3.090</td><td>3.089</td><td>3.111</td></tr><tr><td>People's Speech</td><td>2.739</td><td>2.853</td><td>2.860</td><td>2.866</td></tr><tr><td>CommonVoice</td><td>2.852</td><td>3.090</td><td>2.853</td><td>3.111</td></tr></tbody></table>

LibriHeavy-trained models show superior performance and stability, with NLL increasing only marginally from 2.62 (clean) to 2.70 (perturbed). Other datasets exhibit higher baseline NLL and greater perturbation sensitivity, with GigaSpeech and CommonVoice showing NLL increases up to 0.26 points. These results suggest that domain matching between speech unit k-means construction data and target application is crucial for optimal performance and robustness, as shown on LibriHeavy. Notably, training
on inherently noisy datasets like GigaSpeech and CommonVoice does not improve robustness to perturbations, but rather leads to overall performance degradation, challenging the assumption that exposure to bad acoustic conditions during training necessarily benefits model resilience. Finally, the People's Speech dataset stands out by showing both good overall performance and stability when dealing with noise. This can be attributed to its wide range of audio quality levels and its similarity to the target domain.

## 3.4 Clusters attribution

We analyze cluster usage distribution across encoders and vocabulary sizes to understand their effectiveness in capturing speech phenomena. Using perplexity-based metric:

$$H_{\text{clusters}} = \exp\left(-\sum_{i=1}^{k} p_i \log p_i\right) \qquad (2)$$

where $p_i$ represents each cluster's probability. The resulting value $H_{\text{clusters}}$, expressed as a percentage ($\frac{H_{\text{clusters}}}{k} * 100$), indicates cluster utilization efficiency, with 100% representing uniform usage.

<table><caption>Table 4: Cluster utilization percentage (%) across different models and cluster sizes for test-clean (C) and test-other (O) sets.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">k = 250</th><th colspan="2">k = 1000</th><th colspan="2">k = 2500</th><th colspan="2">k = 5000</th></tr><tr><th>C</th><th>O</th><th>C</th><th>O</th><th>C</th><th>O</th><th>C</th><th>O</th></tr></thead><tbody><tr><td>WavLM</td><td>90.9</td><td>87.3</td><td>83.8</td><td>80.3</td><td>81.8</td><td>78.5</td><td>76.5</td><td>73.9</td></tr><tr><td>HuBERT</td><td>91.9</td><td>89.9</td><td>84.5</td><td>83.2</td><td>83.3</td><td>81.1</td><td>79.7</td><td>77.6</td></tr><tr><td>XLS-R</td><td>82.5</td><td>68.0</td><td>71.4</td><td>57.7</td><td>70.3</td><td>52.1</td><td>72.4</td><td>56.0</td></tr><tr><td>Wav2Vec</td><td>76.4</td><td>66.3</td><td>76.8</td><td>64.0</td><td>80.8</td><td>65.6</td><td>78.2</td><td>63.1</td></tr></tbody></table>

HuBERT and WavLM demonstrate superior cluster utilization (77-92% and 74-91% respectively) while maintaining strong NLL scores, compared to XLS-R (52-68%) and Wav2Vec (63-66%). Lower cluster ranges ($k = 250$) show optimal utilization across all encoders, with HuBERT and WavLM exceeding 90% on clean test sets. Comparing test-clean and test-other utilization reveals varying robustness levels. HuBERT and WavLM show minimal degradation (2-4% drop), while XLS-R and Wav2Vec exhibit larger stability gaps (up to 15-18% drop) in challenging conditions. This pattern persists across all cluster sizes.

## 3.5 Discrete unit alignment with phonemes

To better understand what discrete units represent and to try to understand if they capture phonetic information, we analyze their alignment with phonemes using forced alignment from the Montreal Forced Aligner (MFA) [16] on LibriSpeech test clean.
We compute for each discrete unit its temporal overlap with the aligned phonemes, creating a probability distribution over phonemes for each unit. Figure 2 visualizes this alignment as a matrix where rows represent phonemes and columns represent discrete units, with color intensity indicating the probability of association. The clear diagonal pattern reveals that discrete units learn to specialize in specific phonemes, suggesting the model has captured meaningful phonetic structure. This specialization is particularly strong for distinctive phonemes like vowels (/AH/, /IY/, /UW/), certain consonants (/S/, /F/, /M/) and silence, which show dark regions of high probability along the diagonal for a few sets of units.

Interestingly, we observe some natural clustering of acoustically similar phonemes. For instance, related vowel sounds tend to share similar units, as do phonetically similar consonants. This suggests the discretization process captures not just individual phonemes but also underlying phonetic features. The sparse off-diagonal elements indicate minimal confusion between dissimilar phonemes, demonstrating the model's ability to learn discriminative representations.

<figure><img src="image_3.png" /><figcaption>Fig. 2: Phoneme confusion matrices showing the relationship between predicted discrete units and ground truth phonemes. Each matrix represents discrete units k-means built from a different dataset. All of them are based on WavLM ($k = 125$) and represent LibriSpeech test-clean subset.</figcaption></figure>

The alignment quality remains consistent across different k-means building sources and shows a similar pattern across all the granularities (see Figure 3), but wasn't displayed due to a lack of space. This analysis provides quantitative evidence that self-supervised discrete units can effectively capture phoneme-level distinctions without explicit phonetic supervision, supporting their use as intermediate representations for speech processing tasks.
<figure><img src="image_4.png" /><figcaption>Fig. 3: Phoneme confusion matrices showing the relationship between predicted discrete units and ground truth phonemes. Each matrix represents discrete units k-means built from a different dataset. All of them are based on WavLM ($k = 250$) and represent LibriSpeech test-clean subset.</figcaption></figure>

When we increase the number of clusters such as in the Figure 3, similar phonetic patterns remain clearly visible, with the diagonal structure preserved but becoming more fine-grained. The higher cluster count (250) allows for more specialized unit-to-phoneme mappings while maintaining the overall phonetic organization. This suggests that even at higher granularity, discrete units continue to capture meaningful phonetic distinctions, with each phoneme being represented by a more specific set of units rather than becoming fragmented across unrelated regions.

# 4 Conclusion

This work presents a comprehensive empirical analysis of discrete unit representations in speech language modeling, providing key insights into their behavior and optimization at the pre-training stage. Through extensive experiments across model scales and encoder architectures, we demonstrate that smaller discrete vocabularies ($k \le 1,000$) consistently achieve superior performance, with WavLM-based units showing particular promise. The relationship between model scale and unit learning reveals that larger models (1.7B parameters) exhibit enhanced robustness to vocabulary size and better handle acoustic variations, suggesting more abstract speech representation learning.

Our analysis of cluster utilization and phonemic alignments demonstrates that self-supervised discrete units naturally capture phonetic structure without explicit supervision. The strong correlation between domain matching and model performance, particularly evident in LibriHeavy-trained units, emphasizes the importance of careful data selection for discrete unit training.

These findings have important implications for the design of speech adaptation of existing pre-trained large language models, suggesting that optimal performance may
be achieved through a combination of moderate vocabulary sizes, domain-matched training data, and sufficient model capacity. We release our tokenized datasets and clustering models on GitHub and HuggingFace to facilitate further research in this direction.

Finally, understanding the balance between semantic and paralinguistic information remains crucial, necessitating evaluation across diverse tasks including Spoken Question Answering, Spoken Language Understanding, and ASR.

# 5 Acknowledgements

This work was performed using HPC resources from GENCI-IDRIS (Grant AD011013061R3 and A0161014871). This work was financially supported by ANR MALADES (ANR-23-IAS1-0005), BPI PARTAGES and Zenidoc.

# References

1. Ardila, R., Branson, M., Davis, K., Kohler, M., Meyer, J., Henretty, M., Morais, R., Saunders, L., Tyers, F., Weber, G.: Common voice: A massively-multilingual speech corpus. In: Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., Piperidis, S. (eds.) Proceedings of the Twelfth Language Resources and Evaluation Conference. pp. 4218–4222. European Language Resources Association, Marseille, France (May 2020), https://aclanthology.org/2020.lrec-1.520/

2. Babu, A., Wang, C., Tjandra, A., Lakhotia, K., Xu, Q., Goyal, N., Singh, K., von Platen, P., Saraf, Y., Pino, J., Baevski, A., Conneau, A., Auli, M.: Xls-r: Self-supervised cross-lingual speech representation learning at scale. In: Interspeech 2022. pp. 2278–2282 (2022). https://doi.org/10.21437/Interspeech.2022-143

3. Baevski, A., Zhou, H., Mohamed, A., Auli, M.: wav2vec 2.0: a framework for self-supervised learning of speech representations. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS ’20, Curran Associates Inc., Red Hook, NY, USA (2020)

4. Ben Allal, L., Lozhkov, A., Bakouch, E.: Smollm - blazingly fast and remarkably powerful (July 2024), https://huggingface.co/blog/smollm, accessed: 2025-02-06

5. Chen, G., Chai, S., Wang, G.B., Du, J., Zhang, W.Q., Weng, C., Su, D., Povey, D., Trmal, J., Zhang, J., Jin, M., Khudanpur, S., Watanabe, S., Zhao, S., Zou, W., Li, X., Yao, X., Wang, Y., You, Z., Yan, Z.: Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. In: Interspeech 2021. pp. 3670–3674 (2021). https://doi.org/10.21437/Interspeech.2021-1965

6. Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Wu, J., Zeng, M., Yu, X., Wei, F.: Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing 16(6), 1505–1518 (2022). https://doi.org/10.1109/JSTSP.2022.3188113

7. Cuervo, S., Marxer, R.: Scaling properties of speech language models. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 351–361. Association for Computational Linguistics, Miami, Florida, USA (Nov 2024). https://doi.org/10.18653/v1/2024.emnlp-main.21, https://aclanthology.org/2024.emnlp-main.21/
8. Galvez, D., Diamos, G., Ciro, J., Cerón, J.F., Achorn, K., Gopi, A., Kanter, D., Lam, M., Mazumder, M., Reddi, V.J.: The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage. CoRR abs/2111.09344 (2021), https://arxiv.org/abs/2111.09344

9. Hassid, M., Remez, T., Nguyen, T.A., Gat, I., Conneau, A., Kreuk, F., Copet, J., Defossez, A., Synnaeve, G., Dupoux, E., Schwartz, R., Adi, Y.: Textually pretrained speech language models (2024), https://arxiv.org/abs/2305.13009

10. Hsu, W.N., Bolte, B., Tsai, Y.H.H., Lakhotia, K., Salakhutdinov, R., Mohamed, A.: Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio, Speech and Lang. Proc. 29, 3451–3460 (Oct 2021). https://doi.org/10.1109/TASLP.2021.3122291, https://doi.org/10.1109/TASLP.2021.3122291

11. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models (2021), https://arxiv.org/abs/2106.09685

12. Kang, W., Yang, X., Yao, Z., Kuang, F., Yang, Y., Guo, L., Lin, L., Povey, D.: Libriheavy: a 50,000 hours asr corpus with punctuation casing and context (2023)

13. Lakhotia, K., Kharitonov, E., Hsu, W.N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.A., Copet, J., Baevski, A., Mohamed, A., Dupoux, E.: On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics 9, 1336–1354 (2021). https://doi.org/10.1162/tacl'a'00430, https://aclanthology.org/2021.tacl-1.79/

14. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019), https://openreview.net/forum?id=Bkg6RiCqY7

15. Maiti, S., Peng, Y., Choi, S., weon Jung, J., Chang, X., Watanabe, S.: Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks (2024), https://arxiv.org/abs/2309.07937

16. McAuliffe, M., Socolof, M., Mihuc, S., Wagner, M., Sonderegger, M.: Montreal forced aligner: Trainable text-speech alignment using kaldi. In: Interspeech 2017. pp. 498–502 (2017). https://doi.org/10.21437/Interspeech.2017-1386

17. Mousavi, P., Libera, L.D., Duret, J., Ploujnikov, A., Subakan, C., Ravanelli, M.: Dasb - discrete audio and speech benchmark (2024), https://arxiv.org/abs/2406.14294

18. Nguyen, T.A., Muller, B., Yu, B., Costa-jussa, M.R., Elbayad, M., Popuri, S., Ropers, C., Duquenne, P.A., Algayres, R., Mavlyutov, R., Gat, I., Williamson, M., Synnaeve, G., Pino, J., Sagot, B., Dupoux, E.: Spirit lm: Interleaved spoken and written language model (2024), https://arxiv.org/abs/2402.05755

19. Nguyen, T.A., Sagot, B., Dupoux, E.: Are discrete units necessary for spoken language modeling? IEEE Journal of Selected Topics in Signal Processing 16(6), 1415–1423 (2022). https://doi.org/10.1109/JSTSP.2022.3200909

20. Nguyen, T.A., de Seyssel, M., Rozé, P., Rivière, M., Kharitonov, E., Baevski, A., Dunbar, E., Dupoux, E.: The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling (2020), https://arxiv.org/abs/2011.11588

21. OpenAI, Achiam, J., Adler, S., et al.: Gpt-4 technical report (2024), https://arxiv.org/abs/2303.08774

22. Panayotov, V., Chen, G., Povey, D., Khudanpur, S.: Librispeech: An asr corpus based on public domain audio books. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5206–5210 (2015). https://doi.org/10.1109/ICASSP.2015.7178964

23. Pasad, A., Chou, J.C., Livescu, K.: Layer-wise analysis of a self-supervised speech representation model. In: 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). pp. 914–921 (2021). https://doi.org/10.1109/ASRU51503.2021.9688093
24. Pasad, A., Shi, B., Livescu, K.: Comparative layer-wise analysis of self-supervised speech models. In: ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1-5 (2023). https://doi.org/10.1109/ICASSP49357.2023.10096149

25. Touvron, H., Martin, L., et al.: Llama 2: Open foundation and fine-tuned chat models (2023), https://arxiv.org/abs/2307.09288

26. Yang, H., Zhao, J., Haffari, G., Shareghi, E.: Investigating pre-trained audio encoders in the low-resource condition. In: Interspeech 2023. pp. 1498-1502 (2023). https://doi.org/10.21437/Interspeech.2023-343

27. Zeng, A., Du, Z., Liu, M., Wang, K., Jiang, S., Zhao, L., Dong, Y., Tang, J.: Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot (2024), https://arxiv.org/abs/2412.02612

28. Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., Qiu, X.: Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities (2023), https://arxiv.org/abs/2305.11000