SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Michael Hassid $^{\text{心}, \spadesuit, *}$
Tal Remez\*,
Tu Anh Nguyen
Itai Gat
Alexis Conneau
Felix Kreuk
Jade Copet
Emmanuel Dupoux
Roy Schwartz\*
Yossi Adi $^{\text{♥}, \spadesuit, *}$
FAIR Team, Meta
michael.hassid@mail.huji.ac.il
Speech language models (SpeechLMs) process and generate acoustic data only, without textual supervision. In this work, we propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language models. We show using both automatic and human evaluations that TWIST outperforms a cold-start SpeechLM across the board. We empirically analyze the effect of different model design choices such as the speech tokenizer, the pretrained textual model, and the dataset size. We find that model and dataset scale both play an important role in constructing better-performing SpeechLMs. Based on our observations, we present the largest (to the best of our knowledge) SpeechLM both in terms of number of parameters and training data. We additionally introduce two spoken versions of the StoryCloze textual benchmark to further improve model evaluation and advance future research in the field. We make speech samples, code and models publicly available. $^2$
Speech is the earliest form of human language. Although it contains more than just textual content (e.g., intonation, non-verbal vocalizations), most spoken language understanding systems are limited to its textual form [Wen et al., 2015, Bastianelli et al., 2020, Gupta, 2022]; see Qin et al. [2021] for a recent survey. Recent progress in speech language modeling [Touvron et al., 2023], speech synthesis [Kong et al., 2020], and acoustic unit discovery [Hsu et al., 2021] provided the ability to build purely speech-based language models, henceforth, SpeechLMs [Lakhotia et al., 2021]. However, despite the fast-growing presence of speech and audio content, $^{3}$  text is still by far the most dominant language modality on the web. This limits the ability of constructing large SpeechLMs, in contrast to the great success of textual Language Model (LM)s [Devlin et al., 2019, Raffel et al., 2020, Brown et al., 2020, Chowdhery et al., 2022]. The development of large textual LMs trained on massive text corpora allows such models to effectively perform various tasks based on either few examples or textual instructions [Brown et al., 2020, Touvron et al., 2023]. Such LMs often serve as a foundational
*Core Contributor. ◎ Work done while working at Meta.
2https://pages.cs.huji.ac.il/adiyoss-lab/twist/
$^{3}$ E.g., podcasts, local radio, and video games. See https://www.insiderintelligence.com/content/look-us-digital-audio-market-2022-how-big-who-s-listening-what-they-listening.
Figure 1: Generative Spoken Language Modeling: the pipeline is composed of three main components (i) Speech tokenizer; (ii) SpeechLM; and (iii) Token-to-speech. This paper introduces TWIST, which initializes the weights of the SpeechLM from a pretrained text LM.
model to be fine-tuned on other tasks such as text classification [Howard and Ruder, 2018], textual instructions [Ouyang et al., 2022], or code generation [Le et al., 2022, Nijkamp et al., 2022].
This success raises the question whether textual LMs can be leveraged to improve SpeechLMs. On the one hand, as both modalities operate on completely different granularity levels (pseudo phoneme-states level vs. sub-word level), it is unclear whether such transfer will bring any benefit. On the other hand, speech and text are closely connected, and thus it is natural to consider transferring models across these modalities. Indeed—previous work was able to train speech and text LMs jointly, focusing on improving speech translation [Bapna et al., 2021, Cheng et al., 2022, Bapna et al., 2022] or transcription tasks [Ao et al., 2021, Chen et al., 2023].
In this work, we show that textual LMs can benefit SpeechLMs. We propose TWIST, Textually Warm Initialized Speech Transformer Language Models, a method for initializing SpeechLMs from a pretrained textual LMs (see Fig. 1). Our empirical findings suggest that such a simple approach is highly effective and provides a consistent improvement across all the evaluated metrics, both automatic and human evaluations. We provide an extensive empirical analysis studying the effect of model and data scale, model architecture, and speech tokenizer on the overall model performance. Based on our observations, we present the largest SpeechLM to date (to the best of our knowledge), both in terms of size (13B parameters) and training data ( $\sim$ 150k speech hours). Finally, to better evaluate SpeechLMs capabilities to model long contextual spoken sentences, we generate two spoken versions of the StoryCloze benchmark [Mostafazadeh et al., 2016]. We synthesize all the stories from the original StoryCloze test set using either the original distractor or a randomly chosen one. Each of the versions evaluates different properties of SpeechLMs; the former captures fine-grained temporal commonsense relation, while the latter captures coarse global sentence coherence.
Our contributions: (i) We introduce TWIST, a textually warm initialized speech transformer language model. We empirically show how text-based LMs can be adapted into SpeechLMs, yielding consistent improvements across all evaluated metrics, including both automatic and human ones; (ii) We provide extensive analysis considering the different components of SpeechLMs. Our analysis sheds light on the different model design choices and how they affect model performance; (iii) We leverage our finding and train the largest SpeechLM to-date in terms of the number of parameters and the amount of training data; (iv) We provide two spoken versions of the StoryCloze dataset capturing different aspects of the spoken content. We hope such datasets will be helpful for the research community to better evaluate SpeechLMs under different setups.
2 Using Textual LMs to Improve SpeechLMs
We formally describe the proposed method. We start (Section 2.1) by presenting the relevant background, including the Generative Spoken Language Modeling (GSLM) pipeline, which we build on, and relevant LM formulation. We then present TWIST, our proposed method (Section 2.2).
In this work we follow the GSLM framework [Lakhotia et al., 2021]. The general GSLM pipeline is composed of three main modules, each trained separately: (i) a speech tokenizer, (ii) a SpeechLM,
and (iii) a vocoder module (i.e., Token-to-speech). Speech resynthesis can be achieved while ignoring the language model and directly feeding the quantized tokens into the vocoder module [Polyak et al., 2021]. In the following, we provide background for each of the components mentioned above. See Fig. 1 for a visual description.
Speech tokenizers encode raw speech into a discrete representation. The common approach is to first encode the speech into a continuous representation and then quantize the representation to achieve a sequence of discrete tokens [Tjandra et al., 2019, 2020, Lakhotia et al., 2021, Borsos et al., 2022].
Formally, denote the domain of audio samples by  $\mathcal{X} \subset \mathbb{R}$ . The representation for a raw signal is therefore a sequence of samples  $x = (x_{1},\ldots ,x_{T})$ , where  $x_{t} \in \mathcal{X}$  for all  $1 \leq t \leq T$ .
Consider an encoder network,  $f$ , which gets as input the speech utterance and outputs a sequence of spectral representations sampled at a low frequency as follows  $f(x) = (v_{1},\dots,v_{T^{\prime}})$ , where  $T^{\prime}$  is determined by the frame rate of the encoder. Note that we do not assume anything about the structure of the encoder network  $f$ . Lakhotia et al. [2021] evaluated several speech encoders, namely, Mel-spectrogram, Contrastive Predictive Coding [Oord et al., 2018, CPC], wav2vec2 [Baevski et al., 2020], and HuBERT [Hsu et al., 2021]. They found that HuBERT provides the best overall performance, and hence we follow the same setup.
Since the representations learned by such models are usually continuous, a k-means algorithm [MacQueen, 1967] is applied over the models' outputs to generate discrete tokens, denoted as  $z = (z_{1},\ldots ,z_{T^{\prime}})$ . Each element  $z_{i}$  in  $z$  is a positive integer,  $z_{i}\in \{1,\dots ,K\}$  for  $1\leq i\leq T^{\prime}$ , where  $K$  is the number of discrete tokens of the vocabulary  $\mathcal{Z} = \{1,\dots ,K\}$ .
Language models aim to learn the underlying joint probability of token sequences  $p(w_{1},\ldots ,w_{n})$ . Each token  $w_{i}$  belongs to a vocabulary  $\mathcal{W}$ , defined by a tokenizer. Using the chain rule, the joint probability of a sequence can be computed as a product of its conditional probabilities:
$$
p \left(w _ {1}, \dots , w _ {n}\right) = \prod_ {i = 1} ^ {n} p \left(w _ {i} \mid w _ {i - 1}, \dots , w _ {1}\right).
$$
Neural LMs, parameterized by  $\theta$ , aim to model the probability  $p_{\theta}(w_i|c(w_{i-1}, \ldots, w_1))$ , where  $c$  is a representation of the previous tokens. The network parameters  $\theta$  are learned by minimizing the negative log likelihood loss between the predicted and true probability distributions, such that
$$
\ell (\theta , w) = - \sum_ {i = 1} ^ {n} \log p _ {\theta} (w _ {i} | c (w _ {i - 1}, \dots , w _ {1})).
$$
The network parameters  $\theta$  are typically initialized with values sampled from a predefined distribution, e.g., a uniform or a centered Gaussian distributions [Glorot and Bengio, 2010].
Speech Language Models (SpeechLMs) are trained on the extracted discrete speech tokens,  $z$ , using a speech tokenizer. When operating on  $z$ , SpeechLMs enable directly modeling spoken data without accessing textual transcriptions. Such a modeling framework additionally allows for capturing and modeling prosodic features [Kharitonov et al., 2021], as well as speaker identity [Borsos et al., 2022], or even natural dialogues [Nguyen et al., 2022]. This is in contrast to using textual features, which do not encode such information.
Token-to-speech modules convert the speech discrete tokens to a raw waveform. Lakhotia et al. [2021] used a Tacotron2.0 [Shen et al., 2018] based model followed by WaveGlow [Prenger et al., 2019] vocoder. Later, Polyak et al. [2021] proposed a token-based vocoder based on the HiFi-GAN architecture to convert tokens to speech directly. Such a paradigm seems to provide high-quality generations with better efficiency, as it uses only one model rather than two. As such, we use their approach in this work for our token-to-speech models.
2.2 Textually Warm- Initialized Speech Transformer Language Models
We propose TWIST, a method for training SpeechLMs initialized from a pretrained text LM, such as OPT [Zhang et al., 2022], and LLaMA [Touvron et al., 2023]. TWIST first replaces the original text vocabulary  $\mathcal{W}$  with  $\mathcal{Z}$ , the set of speech tokens (see Section 2.1), and sets the tokenizer to be
4Typically for text LMs, the tokenizer considers words or sub-words [Sennrich et al., 2016].
Table 1: Speech datasets statistics. We report both the number of hours in the training data of each dataset together with the number of speech tokens used to train the speech language model.
LIBRISPEECH	LIBRILIGHT	SPOTIFY	PEOPLE	VOXPOPULI	OVERALL
Hours	960	53k	59k	17k	20k	150k
Tokens-50Hz	86M	4.79B	5.34B	1.54B	1.76B	13.52B
Tokens-25Hz	58M	3.19B	3.56B	1.02B	1.17B	9.00B
a speech based tokenizer. We then replace the text lookup table with a new randomly initialized embedding table for the speech tokens. The rest of the body of the network remains unchanged during initialization time. Finally, TWIST continues training the entire SpeechLM using speech data. A visual description can be found in Fig. 1.
One might wonder whether initializing a speech model with a textual one makes sense, as speech tokens operate on 20-40ms windows while text tokenizers span longer concepts (e.g., sub-words).<sup>5</sup> Nonetheless, previous work showed that speech and text LMs can be trained jointly to improve either machine translation [Bapna et al., 2021, Cheng et al., 2022, Bapna et al., 2022], or transcription based speech related tasks [Ao et al., 2021, Chen et al., 2023]. We next show that SpeechLMs can benefit from textual LM initialization. We recognize that more advanced methods for converting speech tokens to word tokens probably exist, and hope this study will motivate researchers to explore them.
We follow the same setup as described in Section 2.1. Similarly to Lakhotia et al. [2021], we consider HuBERT [Hsu et al., 2021] followed by a k-means quantizer as the speech tokenizer. We use a token based HiFi-GAN neural vocoder as our Token-to-speech model, with duration predictor as in Polyak et al. [2021], Lee et al. [2021]. Then we optimize, compare and analyze the performance of various SpeechLMs considering different setups.
All SpeechLMs are optimized using a collection of publicly available academic speech datasets: LibriSpeech (LS) [Panayotov et al., 2015], LibriLight (LL) [Kahn et al., 2020], Spotify podcasts [Clifton et al., 2020], People dataset [Galvez et al., 2021], and VoxPopuli [Wang et al., 2021a]. We filter non-English data using the provided meta-data. This results in  $\sim 150\mathrm{k}$  hours, which translate to  $13.5\mathrm{B}$  tokens using a token frequency of  $50\mathrm{Hz}$  and 9B tokens using a token frequency of  $25\mathrm{Hz}$ . See Table 1 for detailed descriptions of the datasets. To the best of our knowledge, this is the first work that uses this scale of data to train SpeechLMs.
In cases where no pre-defined validation and test sets are available, we randomly sample  $2\%$  of the data serving as the validation set and an additional  $2\%$  for the test set. Unless stated otherwise, in all setups reported results are the average across all different test sets.
3.2 Model & Hyperparameters
Our main family of textual LMs is OPT [Zhang et al., 2022]. We examine three model sizes: OPT-125M, OPT-350M and OPT-1.3B, corresponding to 12/24/24 transformer blocks and 768/1024/2048 hidden dimension, respectively. We also consider other pretraining approaches, e.g., different pretraining corpora, and experiment with BLOOM [Scao et al., 2022] and Pythia [Biderman et al., 2023], both with equivalent sizes to OPT-350M/1.3B. For each model size, we compare two variants: TWIST, i.e., a warm initialization from a textual LM, and a cold (randomly initialized) model following the original GSLM approach (henceforth COLD-INIT). We use 8 GPUs for training, except
<sup>5</sup>Considering a speech rate of  $\sim 120$  words per minute (https://virtualspeech.com/blog/average-speaking-rate-words-per-minute) an average word duration is  $500\mathrm{ms}$ , which means that SpeechLMs represent a single word using 12-25 tokens.
The actual number of parameters of the SpeechLMs is about  $10 - 25\%$  lower than the text ones since we replace the text embedding layer with a smaller speech embedding layer, due to the smaller speech vocabulary.
Table 2: Zero-shot modeling results for different number of tokens and downsampling factors (Frequency), with and without TWIST. We report PPL over speech tokens, sWUGGY, sBLIMP. Bold indicates the best model for each tokenizer, and we underline the best performing model overall for sWUGGY and sBLIMP (PPL results are incomparable across tokenizers).
TWIST	#TOKENS	FREQ.	PPL↓	SWUGGY↑	SBLIMP↑
X	100	50Hz	5.26	68.91	53.80
✓	5.03	71.30	55.96
X	200	50Hz	5.61	69.85	53.48
✓	5.29	72.92	55.91
X	500	50Hz	6.36	66.65	50.79
✓	5.85	70.69	52.71
X	500	25Hz	6.65	79.44	54.84
✓	6.25	81.42	56.20
for the 1.3B models, which use 32 GPUs. In all cases, we choose the best checkpoint by the averaged speech perplexity of the model over the validation sets.
For the speech tokenizers, we quantize HuBERT representations, which operate at  $50\mathrm{Hz}$ , with 100, 200, and 500 clusters using the k-means algorithm. HuBERT is trained on LS 960h, while k-means are trained over the 'clean-100h' part only from the LS corpus. Models and quantizers are obtained from the textless-lib [Kharitonov et al., 2022]. Inspired by Chung et al. [2021] and Borsos et al. [2022], we also trained a new HuBERT tokenizer with a lower sampling rate. We trained the HuBERT model on a varied mixture of datasets: Multilingual LS [Pratap et al., 2020], Vox Populi [Wang et al., 2021a], Common Voice [Ardila et al., 2019], Spotify [Clifton et al., 2020], and Fisher [Cieri et al., 2004]. More details regarding model setups can be found in Appendix A.1.
Evaluating such a complex pipeline comprised of several different components is a challenging task. We follow Lakhotia et al. [2021], who were the first to propose an evaluation setup for this pipeline. We also introduce a novel evaluation framework based on the StoryCloze textual benchmark [Mostafazadeh et al., 2016]. Overall, we consider three different evaluation setups: (i) Zero-shot modeling; (ii) Human evaluation; and (iii) Spoken StoryCloze.
Zero-shot modeling metrics. We use the sWUGGY and sBLIMP [Nguyen et al., 2020] metrics to evaluate lexical and syntactic modeling of the SpeechLMs. In sWUGGY, the network is presented with a pair of utterances, an existing word and a matching non-word, and evaluated on their capacity to attribute a higher probability to the existing word. Unless stated otherwise, we follow the common approach [Lakhotia et al., 2021] and report the "in-vocab" split results. As for sBLIMP, similarly to sWUGGY, the network is presented with a matched pair of speech segments, grammatical and ungrammatical sentences. The task is to decide which of the two is grammatical based on the probability of the sentence. For both measures, we compare the geometric mean over the models' sequence probabilities assigned to each utterance within a pair. We additionally report speech tokens Perplexity (PPL), averaged across all test sets. We note that while PPL is not comparable across different tokenizers, it can be helpful to compare TWIST against COLD-INIT.
Human evaluation. To better assess the quality of the generated outputs, we follow the same setup as the Meaningfulness MOS [MMOS; Lakhotia et al., 2021]. In this setup, human raters are presented with several speech continuations ( $\sim 10$  seconds) following the same speech prompt ( $\sim 3$  seconds), and are instructed to evaluate how natural (considering grammar, meaning, and diversity) a given sample is, on a scale between 1-5 with increments of 0.5. For speech prompts, we randomly sample a collection of 50 examples from the test sets of three different datasets: LL, LS-clean and LS-other to reach overall 150 samples per evaluated method. We enforce 10 raters per sample and use the CrowdMOS [Ribeiro et al., 2011] package with the recommended recipes for detecting and discarding inaccurate scores.
(b)
Figure 2: (a) PPL as a function of training set and model sizes. (b) Validation PPL as a function of training steps. TWIST is both more sample-efficient and converges faster than COLD-INIT.
We note that Lakhotia et al. [2021] also proposed computing PPL and auto-BLEU over the transcription of the generated speech to assess both quality and diversity of the generations. While this metrics are automatic, they are highly influenced by numerous factors such as speech vocoder quality, Automatic Speech Recognition (ASR) transcription errors, sampling, etc. In preliminary results, we observe a high variance when computing this metric, hence we focus on a human study, and report these metrics for our main family of models in Appendix A.2.
Spoken StoryCloze. Finally, to better evaluate the capabilities of SpeechLMs in capturing fine-grained textual nuances and continuation coherence, we provide two spoken versions of the StoryCloze textual benchmark [Mostafazadeh et al., 2016], denoted by Spoken StoryCloze (SSTORYCLOZE) and Topic StoryClose (TSTORYCLOZE). The textual StoryCloze benchmark contains 4k five-sentence commonsense stories (split to validation and test sets). For each story, there is an additional negative sample, composed of the first four sentences followed by a fifth, adversarial sentence. The goal is to distinguish the original fifth sentence from the negative one. To generate the spoken benchmark, we synthesize the stories from the test set using a single speaker TTS system as provided by Wang et al. [2021b], comprised of a FastSpeech2.0 model [Ren et al., 2020] and a HiFi-GAN vocoder [Kong et al., 2020].
For sSTORYCLOZE, we follow the original StoryCloze negative samples. With this benchmark, we evaluate models' capabilities to capture fine-grained causal and temporal commonsense relations. For TSTORYCLOZE, we randomly sample the negative ending sentence from the dataset. The premise behind TSTORYCLOZE is to evaluate continuation coherence given a spoken prompt. This version is far easier, but our experiments show that it is still quite challenging for modern SpeechLMs. Similar to sWUGGY and sBLIMP, we feed both speech segments to the SpeechLM, measure the probability of each spoken sentence, and report the percentage of examples where the probability of the positive sample is higher than the negative one.
We also conduct human evaluation on both datasets in order to measure human performance over the speech benchmarks (which may differ from their text equivalents). We introduce human raters with both options and let them rate each one according to MMOS [Lakhotia et al., 2021], in a scale of 1-5 with 0.5 increments. The human score for this benchmarks is the proportion of samples where the score given by the rater is higher for the correct utterance. We use 10 raters for each pair of utterances and use the CrowdMOS [Ribeiro et al., 2011] package with the recommended recipes for detecting and discarding inaccurate scores.
Both datasets are available at https://github.com/slp-rl/SpokenStoryCloze
<sup>8</sup>We introduce a 100ms of silence between segments to generate naturally spoken sentence.
Table 3: Comparison of TWIST-7B/13B models against prior work. We report sWUGGY, using both 'in-vocab' words and 'all' words, along with sBLIMP.
METHOD	PPL↓	SWUGGY↑
all in-vocab	SBLIMP↑
van Niekerk et al. [2021]	-	64.3	72.3	54.0
Lakhotia et al. [2021]	-	-	68.7	57.1
Borsos et al. [2022]	-	71.5	83.7	64.7
COLD-INIT-1.3B	6.20	72.2	81.9	56.5
TWIST-1.3B	5.93	72.7	82.5	57.0
TWIST-7B	5.47	73.9	83.6	59.0
TWIST-13B	5.34	74.5	84.1	59.2
SpeechLMs benefit from warm initialization using text LMs. We start by evaluating the effect of the warm initialization on SpeechLMs. We compare two versions of OPT-350M, with warm (using TWIST) and cold initialization (COLD-INIT), for different frequencies and different number of tokens. Results are reported in Table 2. We observe a consistent improvement across all metrics following the TWIST approach. Interestingly, using speech tokens with larger downsampling factor (i.e., smaller frequency) leads to substantially better sWUGGY and sBLIMP results. These findings are consistent with Borsos et al. [2022], and also reflected by the speech resynthesis results, see Appendix A.3. For the rest of the paper we report results using the speech tokenizer with 500 tokens at  $25\mathrm{Hz}$ , which gets the best sWUGGY and sBLIMP results. For compatibility with prior work [Lakhotia et al., 2021], results for the speech tokenizer with 200 tokens at  $50\mathrm{Hz}$  are in the Appendix.
Scaling improves SpeechLMs. As prior work mainly considered relatively small SpeechLMs (e.g.,  $\sim 100\mathrm{M}$  parameters in Lakhotia et al. [2021], Kharitonov et al. [2021]) and relatively small training data (e.g., Borsos et al. [2022] use LL only), in the following we evaluate the effect of model and dataset scaling on the overall performance.