SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We train three different SpeechLMs using the OPT family. For each model size, we increase the magnitudes of training data, considering  $1\%$ ,  $10\%$ , and  $100\%$  of the training data with both TWIST and COLD-INIT. Our results (Fig. 2a and Fig. 5; Appendix A.4) first highlight that, as in our previous experiments, SpeechLMs initialized using TWIST perform consistently better than those with COLD-INIT. We next note that, as expected, increasing the model size and the magnitude of the dataset improves model performance in almost all cases. Further, for all model scales, following the TWIST approach with only  $10\%$  of the data yields comparable or superior performance than the corresponding COLD-INIT approach using  $100\%$  of the data. This is consistent with previous findings that pretraining leads to higher sample efficiency for downstream tasks [Peters et al., 2018]. The full set of results for the corresponding models can be found on Table 7 in Appendix A.4.
TWIST converges faster. Next, we analyze how textual pretraining affects model convergence. Fig. 2b presents the validation loss training curves for OPT-350M. We observe that using TWIST, the model reaches the same PPL in about one quarter of the training updates compared to COLD-INIT.
Not all warm initializations are equally important. We have so far seen that warm initialization from OPT models is consistently beneficial for SpeechLMs. Are OPT models unique in this sense, or could other pre-trained models similarly benefit SpeechLMs?
To address this question, we start by reporting results for different pre-trained text LMs. We consider different textual pre-training approaches—BLOOM [Scao et al., 2022] and Pythia [Biderman et al., 2023]—both of similar size to OPT-350M/1.3B. Tables 8 and 9 in Appendix A.5 shows similar trends to our OPT experiments, in which using textually pretrained LMs yields consistently better results.
We next ask whether initialization of SpeechLMs from pre-trained text LMs is particularly important, or would initialization from other pre-trained modalities similarly benefit SpeechLMs. We consider ImageGPT [Chen et al., 2020], an image-generation pre-trained model. We find (Table 10; Appendix A.6) that unlike the textual initialization, this model not only does not outperform COLD-INIT, it substantially underperforms it.
(a) LS-clean.
(b) LS-other.
(c) LL.
Figure 3: Human evaluation (MMOS) for speech generation, with different models and datasets. TWIST outperforms the COLD-INIT, and the TWIST-7B performs better than smaller models. Full results presented in Appendix A.7.
Speech Large Language Models. Equipped with previous findings, we train the largest SpeechLMs to date (to the best of our knowledge), a 7B/13B parameter SpeechLMs initialized from the LLaMA-7B/13B models [Touvron et al., 2023], denoted as TWIST-7B/13B. We use the same configuration as in Section 3.2 with specific exceptions detailed in Appendix A.1.
Table 3 shows our results for the TWIST-7B/13B models along with several baseline methods. We also include COLD-INIT and TWIST models of size 1.3B for a reference. Here we report sWUGGY scores both for 'in-vocab' samples (words found in the LS corpus) along with 'all' samples (including also words which do not appear in the LS corpus). As expected, scaling benefits SpeechLMs. Initialization from LLaMA-7B/13B leads to additional performance improvement over TWIST-1.3B,  $\sim 8 / 10\%$  relative improvement in PPL and  $\sim 1.7 / 2.5\%$  relative improvement in sWUGGY. When comparing to prior work, TWIST-13B outperforms all evaluated methods on both sWUGGY setups.9 As to sBLIMP, TWIST-13B's results are lower than Borsos et al. [2022], although the proposed method is orthogonal to their method.
Spoken StoryCloze. To better assess the contextual understanding of the SpeechLMs we experiment with our collected Spoken StoryCloze benchmark (see Section 3.3).
Table 4 shows the results for TWIST-1.3B/7B/13B models alongside human performance. As expected, both SpeechLMs and humans exhibit superior performance in terms of continuation coherence (TSTORYCLOZE) compared to more fine-grained relations (SSTORYCLOZE). Interestingly, despite humans achieving high scores on the textual StoryCloze benchmark, their performance on spoken benchmarks is not flawless, suggesting that spoken language understanding tasks are more complicated for humans compared to their written equivalents. The results of SpeechLMs indicate
Table 4: Accuracy (%) results for spoken SSTORYCLOZE (SSC) and TSTORYCLOZE (TSC) benchmarks.
MODEL	SSC↑	TSC↑
Human	79.9	90.2
TWIST-1.3B	52.4	70.6
TWIST-7B	55.3	74.1
TWIST-13B	55.4	76.4
reasonable performance on the TSTORYCLOZE benchmark, with a  $\sim 15\%$  gap compared to human performance. However, this gap widens substantially on the SSTORYCLOZE benchmark. This highlights the opportunities for SpeechLMs' improvements both in continuation coherence and, more significantly, in causal and temporal commonsense relations. Lastly, consistent with our previous results, scaling SpeechLMs yields performance improvements on both benchmarks.
Human evaluation. We conclude with a human evaluation for speech generation. As stated in Section 3.3, for each model and sample we generate speech continuation of  $\sim 10$  seconds (using a  $\sim 3$  seconds prompt). We evaluate the following models: 1.3B parameters model with and without TWIST, and TWIST-7B. We also compare to the original audio and the resynthesized reference, which serve as top-line estimates.
Our result (Fig. 3) suggests that TWIST-7B is superior to TWIST-1.3B, while COLD-INIT-1.3B performs consistently worse. Interestingly, we do not observe noticeable differences between the
<sup>9</sup>In fact, under the 'all' setup even TWIST-1.3B outperforms all evaluated methods.
Prompt: Can we please go swimming at the beach
Cold-Init (1.3B): Can we please go swimming at the beach? Asked Tom. I should like to go swimming he said...
TWIST-1.3B: Can we please go swimming at the beach? Asked Clara. No, replied Dora, who had been swimming...
TWIST-7B: Can we please go swimming at the beach with mom on April 1st? Oh my God, I would love to go somewhere with my kids on April 1st...
Figure 4: Generation samples using 1.3B models, with and without TWIST, along with TWIST-7B.
evaluated datasets. We also provide an example of generated sentences given a pre-defined prompt in Fig. 4. This example illustrates that COLD-INIT-1.3B makes grammatical errors, both COLD-INIT-1.3B and TWIST-1.3B remain on topic, and TWIST-7B provides semantically richer continuations. Audio samples can be found in the project website.
Text Language Models have been long studied under various setups [Bahl et al., 1983, Chiang et al., 2022]. While Brants et al. [2007] showed the benefit of optimizing LLMs back in 2007, recently with the help of neural networks such models became foundational models, serving as a base model for different downstream tasks [Shoeybi et al., 2019, Radford et al., 2019, Brown et al., 2020, Lieber et al., 2021, Zhang et al., 2022, Hoffmann et al., 2022, Chowdhery et al., 2022].
The success of pretrained language models as zero-shot or few-shot learners gave rise to an extensive line of work to apply similar techniques to other input modalities. Chen et al. [2020] proposed generating natural images while optimizing a language model using masked prediction loss over the pixel space of natural images. Other works proposed converting natural images to discrete tokens space and optimizing textually conditioned language model to perform text-to-image generation [Yu et al., 2022, Ramesh et al., 2021, Chang et al., 2023].
In the context of speech and audio, Lakhotia et al. [2021] first demonstrated how raw and uncurated speech data can be leveraged into building a GSLM system. Next, Kharitonov et al. [2021] proposed a multi-stream SpeechLM to jointly process "pseudo-text" tokens together with quantized prosodic features (i.e., duration and F0). Polyak et al. [2021] evaluated the robustness and disentanglement properties of speech-to-tokens models and demonstrated the ability to perform voice conversion as well as a lightweight speech codec. Kreuk et al. [2021] proposed to cast the task of speech emotion conversion as a translation task, hence translating between one emotion to the other in the discrete space, while Maimon and Adi [2022] proposed a similar approach for speaking style conversion. Nguyen et al. [2022] proposed training two SpeechLMs jointly to mimic natural spoken dialogues. Recently, Borsos et al. [2022] proposed cascading several LMs, in which one LM operates over semantic speech tokens [Sicherman and Adi, 2023] while the others operate on acoustic tokens [Zeghidour et al., 2021, Defossez et al., 2022]. Such modeling framework allows generating natural speech while keeping the identity of the speaker and acoustic conditions unchanged.
Another line of relevant related work, demonstrated that sound and music can be generated following a similar modeling paradigm. Kreuk et al. [2022] first proposed optimizing language models over discrete audio representations to construct a text-to-audio generation model. Similarly Agostinelli et al. [2023] proposed optimizing three language models (following the same modeling approach as by Borsos et al. [2022]), operating at different granularity of the input representation for the task of text-to-music generation. Donahue et al. [2023] proposed a similar modeling approach for the task of singing to accompaniment generation. Lee et al. [2021, 2022], Popuri et al. [2022] followed a similar modeling mechanism using a different speech tokenizer and proposed a textless approach for speech-to-speech translation. Hsu et al. [2022] proposed a jointly modeling visual discrete tokens together with speech tokens to perform various speech resynthesis tasks including: silent video to speech generation, speech enhancement, and recovering packet loss.
Finally, perhaps the line of research most related to this work is jointly training speech and text models. Bapna et al. [2021], Cheng et al. [2022] and Bapna et al. [2022] considered speech as another language in multilingual setups, and showed that involving speech and text as part of training data
improves results for speech translation and multilingual text tasks. Chen et al. [2023] and Ao et al. [2021] used the joint training to improve transcriptions tasks as ASR and Text-to-Speech (TTS). None of these prior work focuses on SpeechLMs and its modeling and generation capabilities.
In this work, we studied the effect of textual pretraining on speech-based language models. We empirically demonstrated how such simple model initialization can improve modeling performance considering both automatic and human-generated measures. We conducted extensive evaluation of different speech tokenizers, model and dataset sizes, model architecture, and modality pretraining. Our analysis sheds light on the impact of specific modeling design choices on the overall performance of the system. Equipped with these findings, we presented the largest SpeechLMs to date (7B and 13B parameters). Finally, we generated two spoken versions of the StoryCloze textual benchmark to measure contextual understanding abilities of SpeechLMs. We hope such empirical findings and benchmark releases will be found useful for future research in the field.
Limitations. The biggest limitation of SpeechLMs at large is the lack of semantic understanding, which might lead to ungrammatical, off-topic, or even inaccurate responses. Although we provided a better initialization point for optimizing SpeechLM, we did not observe major semantic knowledge transfer following the TWIST approach. This limitation is a general critique of SpeechLMs, which should be better studied and addressed in future research. Another limitation of the proposed approach is the granularity of the speech tokenizer, such small input resolution (50Hz / 25Hz) yields a relatively long sequence (750 / 500 tokens) for relatively short speech segments (~30 seconds). This translates into long inference time and harder optimization.
Broader impact. SpeechLMs share the same potential benefits for society as text-based LMs as they give access, in the audio modality, to the same downstream applications (search, language generation, summarization, translation, chatbots, etc.). Thereby increasing their reach to more use cases and more languages, including 'unwritten' (or sparsely written) languages [Lakhotia et al., 2021]. As a result, they also share the same potential risks regarding intentionally harmful applications (e.g. fake news, spamming, election manipulation) or unintentionally harmful ones (e.g., unfair or biased results, toxic, regurgitated or untrustful generations).
We thank Miri Varshavsky Hassid for the great feedback and moral support.
Authors note. This paper was submitted in the wake of the tragic terror attack perpetrated by Hamas on October 7, 2023, which has left the Israeli nation profoundly devastated. The assault greatly impacted us at a personal level, thereby significantly impacting the course of this research work. This paper was finalized while we grieve and mourn our friends and family, under great stress, with scientific considerations being the last thing on our minds. It may contain subtle errors.
Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711-1721, 2015.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. Slurp: A spoken language understanding resource package. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7252-7262, 2020.
A Supplemental materials
A.1 Model & hyperparameters
All LM models are trained with a batch size of 64, where each sample is bounded for 25 seconds and 704 tokens. The models are trained for 400k steps ( $\sim$ 1.2 epochs), using an inverse-sqrt scheduler, 100 warmup steps and wADAM as the optimization algorithm. We also tune the learning rate per scenario, i.e: using/not-using pretrained LM, we end up with a maximal learning rate of 4e-4/8e-5 and final learning rate of 8e-5/2.5e-5, respectively. As for the LLaMA-7B/13B model, we use the same configuration except the following: cosine learning rate schedule, 500 warmup steps, a maximum learning rate of 1e-4, a final rate of 1e-5, batch size of 1024 over 32 GPUs for 75k steps ( $\sim$ 4 epochs).
The new the frequency HuBERT speech tokenizer (now available in textless-lib Kharitonov et al. [2022]), is trained for 3 iterations with the default  $50\mathrm{Hz}$  features rate. For the 4-th iteration, we add an additional convolutional layer at the CNN Encoder with the strides  $2/3/4$ , resulting in features of  $25\mathrm{Hz}/16.6\mathrm{Hz}/12.5\mathrm{Hz}$ , respectively. Our early ablations show that  $25\mathrm{Hz}$  features with 500 tokens give the best results in terms of language modeling, we thus train our models on these new tokens and compare them with the rest of the tokens.
A.2 Automatic evaluation of speech generation
As stated, PPL scores over the transcribed speech generations (text-PPL) are sensitive to modification in the sampling parameter, vocoder quality and ASR errors [Lakhotia et al., 2021]. Furthermore, text-PPL tells only one side of the story, we also need to count for diversity in the generation, as suggested by Lakhotia et al. [2021]. For instance, repeated speech gets good text-PPL score, at the cost of bad diversity measures.
To mitigate that, when computing the automatic metrics for generated speech, we threshold the temperature parameter using the ASR model confidence. Next, as there can be trade-offs in terms of sentence coherence and diversity, we calibrated the temperature so that the generated speech matches the transcription of the original utterance (as much as possible) in terms of generation diversity (auto-BLEU from Lakhotia et al. [2021]).
For speech prompts, we randomly sample a collection of 1024 examples from the test sets of three different datasets: LS-clean, LS-other and LL. Each speech prompt is  $\sim 3$  seconds long, while the continuation is  $\sim 10$  seconds. For the calibration procedure, we used randomly selected 128 examples from the corresponding validation datasets. Text transcriptions extracted using Whisper "small" [Radford et al., 2022], while text-PPL measures were computed using a LLaMA-7B (text model) [Touvron et al., 2023]. Table 5 reports both text-PPL and auto-BLEU results for the original continuations, and the generations that produced by our main family of models (OPT architecture coupled with the  $25\mathrm{Hz}$  tokenizer).
Table 5: text-PPL and auto-BLEU results, with and without TWIST.
TWIST	# PARAM.	TEXT-PPL↓	AUTO-BLEU↓
—	Original Cont.	36.35	0.23
X	125M	88.17	0.305
✓	74.95	0.29
X	350M	117.17	0.27
✓	92.69	0.25
X	1.3B	67.68	0.29
✓	47.8	0.29
As can be seen, while the diversity metric is comparable across models (due to calibration), TWIST models outperform the COLD-INIT ones in terms of text-PPL. These measures further support our claims: TWIST models outperform COLD-INIT ones, and models generally improve with scale. Notice, the text-PPL for the 350M models are worse than the 125M models. However, as stated, the diversity of the generations (as computed using auto-BLEU) is better for the 350M models.
Figure 5: PPL as a function of training set and model size, for models trained with the 200 tokens at  $50\mathrm{Hz}$  tokenizer.
A.3 Speech resynthesis results
Resynthesis can be considered as an upper bound for our language modeling setup. It does not involve SpeechLMs, and measures our ability to fully recover the speech content after tokenization [Polyak et al., 2021]. As we additionally evaluate several speech tokenizers, we provide resynthesis metrics in the form of Word Error Rate (WER). We use Whisper "small" [Radford et al., 2022] as our ASR model.
In Table 6, we evaluate the effect of the tokenizer on the resynthesis performance, and can better evaluate the impact of the tokenization process on the generated audio. As can be seen, all tokenizers incur a loss in WER. Using 500 tokens at  $25\mathrm{Hz}$  provides the best performance.
Table 6: Speech Resynthesis. Results are reported for different number of tokens and downsampling factors (Frequency).
#TOKENS	FREQUENCY	WER↓
100	50Hz	0.23
200	50Hz	0.18
500	50Hz	0.17
500	25Hz	0.16
A.4 Model and data scaling results
The full set of results, i.e., PPL, sWUGGY and sBLIMP from Section 4 for model and dataset scaling is presented in Table 7. The equivalent of Fig. 2a using 200 tokens at  $50\mathrm{Hz}$  tokenizer can be found in Fig. 5.
A.5 The effect of LM architecture
To further validate our findings holds for other LM architectures other than OPT. In Table 8, we provide results for the BLOOM [Scao et al., 2022] and Pythia [Biderman et al., 2023] of similar size to OPT-350M, with both  $25\mathrm{Hz}$  and  $50\mathrm{Hz}$  tokenizers. In Table 9 we provide results for BLOOM and Pythia of similar size to OPT-1.3B with the  $25\mathrm{Hz}$  tokenizer.
As before, we observe similar patterns in terms of using a pretrained text LM. SpeechLMs initialize from text reach better performance across all metrics.
A.6 The effect of different modality pretraining
Although having completely different granularity, results suggest training SpeechLMs with model initialization from a text based LMs brings a consistent performance improvement. As a result, a
Table 7: Model and Data Scaling. Results are reported for different models on various size using different magnitude of data, with and without TWIST. We report PPL/ sWUGGY / sBLIMP.
TWIST	# PARAM.	#TOKENS	FREQ.	1% OF DATA	10% OF DATA	100% OF DATA
X	125M	200	50Hz	6.08 / 66.90 / 52.45	5.79 / 68.16 / 52.71	5.79 / 68.26 / 53.02
✓	5.94 / 69.48 / 52.87	5.51 / 70.67 / 54.34	5.49 / 70.75 / 53.92
X	350M	200	50Hz	6.14 / 66.49 / 51.97	5.71 / 68.85 / 53.13	5.61 / 68.95 / 53.48
✓	5.93 / 68.49 / 53.13	5.37 / 72.64 / 55.63	5.29 / 72.92 / 55.91
X	1.3B	200	50Hz	6.29 / 64.91 / 52.18	5.51 / 71.39 / 54.58	5.32 / 72.83 / 55.12
✓	6.05 / 67.89 / 52.83	5.37 / 72.45 / 55.65	5.23 / 73.39 / 55.91
X	125M	500	25Hz	7.22 / 77.58 / 53.74	6.82 / 78.12 / 54.00	6.81 / 77.74 / 54.27
✓	7.06 / 78.99 / 54.12	6.52 / 80.08 / 55.45	6.50 / 80.57 / 55.43
X	350M	500	25Hz	7.37 / 76.96 / 53.07	6.79 / 77.93 / 54.71	6.65 / 79.44 / 54.84
✓	7.26 / 78.67 / 53.95	6.41 / 81.23 / 56.08	6.26 / 79.44 / 56.20
X	1.3B	500	25Hz	7.49 / 75.44 / 52.96	6.40 / 80.80 / 55.95	6.20 / 81.94 / 56.52
✓	7.19 / 78.00 / 53.90	6.19 / 82.28 / 56.81	5.93 / 82.49 / 57.05
Table 8: LM Model Architecture. Results are reported for both Bloom and Pythia model architectures ( $\sim$ 350M parameters), with and without TWIST. We report PPL, sWUGGY and sBLIMP.
TWIST	ARCH.	# TAXENS	FREQ.	PPL↓	SWUGGY↑	SBLIMP↑
X	BLOOM	200	50Hz	5.63	69.38	53.03
✓	5.21	72.48	55.79
X	BLOOM	500	25Hz	6.45	79.46	55.60
✓	6.06	82.01	57.22
X	Pythia	200	50Hz	5.62	70.00	53.07
✓	5.23	72.20	56.00
X	Pythia	500	25Hz	6.45	79.82	55.45
✓	6.12	81.41	56.70
Table 9: LM Model Architecture. Results are reported for both Bloom and Pythia model architectures ( $\sim$ 1.3B parameters), with and without TWIST. We report PPL, sWUGGY and sBLIMP.
TWIST	ARCH.	PPL↓	SWUGGY↑	SBLIMP↑
X	BLOOM	6.09	80.47	56.02
✓	5.80	82.63	57.43
X	Pythia	6.05	81.33	56.34
✓	5.81	81.77	57.02
natural question would be do speech and text tokens have special connection or LMs are just general next token prediction mechanisms?
To evaluate such a hypothesis, we consider a language model trained on a different modality. Specifically, we train ImageGPT [Chen et al., 2020] ("medium" size) models, one from scratch and another one pretrained using next pixel prediction using a transformer language model. For the pretrained model we use the official pre-trained model. Table 10 summarizes the results.
Interestingly, ImageGPT pre-trained models perform much worse than models pretrained on text. For a reference, models trained from scratch achieve comparable performance to previously reported models.
10https://huggingface.co/docs/transformers/model_doc/imagegpt
Table 10: Results for the ImageGPT model (image pretraining), with and without TWIST. We report PPL, sWUGGY and sBLIMP. Unlike textual pretraining, image pretraining not only does not benefit SpeechLMs, but substantially hurts their performance.
TWIST	#TOKENS	FREQ.	PPL↓	SWUGGY↑	SBLIMP↑
×	200	50Hz	5.22	72.22	55.70
✓	8.21	54.26	51.90
×	500	25Hz	6.20	81.85	56.39
✓	7.85	72.50	51.84
A.7 Full human evaluation (MMOS) results
We include the full Human evaluation (MMOS) results, corresponds to Fig. 3 in Table 11.
Table 11: Full human evaluation (MMOS) results. We report MMOS score as: mean (95% confidence-interval).
METHOD	LS-CLEAN	LS-OTHER	LL-OTHER
Original Resynthesis	4.11(±0.04)	4.23(±0.04)	4.05(±0.05)
3.95(±0.07)	3.87(±0.06)	3.96(±0.06)
COLD-INIT-1.3B	3.34(±0.08)	3.37(±0.06)	3.31(±0.07)
TWIST-1.3B	3.51(±0.07)	3.67(±0.07)	3.65(±0.06)
TWIST-7B	3.79(±0.06)	3.85(±0.07)	3.81(±0.06)