Textually Pretrained Speech Language Models
Authors: Michael Hassid
Tal Remez
Tu Anh Nguyen
Itai Gat
Alexis Conneau
Felix Kreuk
Jade Copet
Alexandre Défossez
Gabriel Synnaeve
Emmanuel Dupoux
Roy Schwartz
Yossi Adi
Venue: NeurIPS 2023 poster (2023)
Keywords: LLM
speech
generative
GSLM
TL;DR: We propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language model.

Abstract: Speech language models (SpeechLMs) process and generate acoustic data only, without textual supervision. In this work, we propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language models. We show using both automatic and human evaluations that TWIST outperforms a cold-start SpeechLM across the board. We empirically analyze the effect of different model design choices such as the speech tokenizer, the pretrained textual model, and the dataset size. We find that model and dataset scale both play an important role in constructing better-performing SpeechLMs. Based on our observations, we present the largest (to the best of our knowledge) SpeechLM both in terms of number of parameters and training data. We additionally introduce two spoken versions of the StoryCloze textual benchmark to further improve model evaluation and advance future research in the field. We make speech samples, code and models publicly available.

Textually Pretrained Speech Language Models
Michael Hassid
Tal Remez
Tu Anh Nguyen
Itai Gat
Alexis Conneau
Felix Kreuk
Jade Copet
Alexandre Defossez
Gabriel Synnaeve
Emmanuel Dupoux
Roy Schwartz R
Yossi A
FAIR Team, Meta OpennA The Hebrew University of Jerusalem
michael hassid@mail www...ii........... ac il
Abstract
Speech language models (SpeechLMs) process and generate acoustic data only, without textual supervision. this work, we propose TWIST, a method for training SpeechLMs using warm-start from pretrained textual language models. We show using both automatic and human evaluations that TWIST outperforms cold-start SpeechLM across the board. We empirically analyze the effect of different model design choices such as the speech tokenizer, the pretrained textual model, and the dataset size. We find that model and dataset scale both play an important role in constructing better-performing SpeechL.Ms, Based on our observations, present the largest (to the best of our knowledge) SpeechI.M both in terms number parameters and training data. We additionally introduce two spoken versions of the StoryCloze textual benchmark further improve model evaluation and advance future research in the field. We make speech samples, code and models publicly available.?
Introduction
Speech the earliest form of human language. Although contains more than just textual content (e.g., intonation, non-verbal vocalizations), most spoken language understanding systems are limited to textual form [Wen et al., 2015, Bastianelli et al., 2020, Gupta, 2022]; see Qin et al. [2021] for recent survey. Recent progress in speech language modeling ['ouurron et al., 2023], speech synthesis [Kong al., 2020], and acoustic unit discovery [Hsu al., 2021] provided the ability to build purely mpeech-hased language models, henceforth, SpeechI.Mr .Ms [I [Lakhotia et al., 2021]. However, despite the fast-growing presence of speech and audio content," text is still by far the most dominant language modality the web. This limits the ability of constructing large SpeechLMs, in contrast to the great success textual Language Model (LM)s [Devlin et al., 2019, Raffel et al., 2020, Brown al., 2020, Chowdhery et al., 2022]. The development of large textual LMs trained massive text corpora allows such models to effectively perform various tasks based on either few examples or textual instructions [Brown al., 2020, Touvron et al., 2023]. Such LMs often serve a foundational
"ore Contributor. O Work done while working at Meta. "https://paaae as.huji. a.a...cc.. 1t/adiyoss-lab// lab/twist/ 'E.g., podcasts, local radio, and vidco games. See https //www//wwwwaaaaa/ https::/www.hsiidriieellllllll look-us- book-us-digtte dissdiggtal-- markett-rrtt----e 20222 how- big who listeningtnnngg. wwweeeiinn-wtttttte teeythey-111 lis-teninn.n
wtt.cice..cccooootteotee com/content/
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

TWIST
Pretrained Text
Weight
LM
Speech
Language Modeling
SpeechLM
Speach Takerizer
Speech Encoder
Quantizer
Speach
Token
Resynthesis
Figure Generative Spoken Language Modeling: the pipeline is composed of three main components (i) Speech tokenizer; (ii) SpeechL.;; and (iii) Token-to-speech. This paper introduces TWIST,,hio initializes the weights of the SpeechLM from a pretrained text LM.
model to be fine-tuned other tasks such text classification [Howard and Ruder, 2018], textual instructions [Ouyang et al., 2022], or code generation [Le et al., 2022, Nijkamp et al., 2022].
This success raises the question whether textual LMs can be leveraged to improve SpeechLMs. On the one hand, as both modalities operate on completely different granularity levels (pseudo phoneme-state states level vs. sub-wor uub-word level), it is unclear whether such transfer will bring any benefit. On the other hand, speech and are closely connected, and thus it is natural to consider transferring models across these modalities. Indeed-previous work was able train speech and text LMs jointly, focusing on improving speech translation [Bapna et al., 2021, Cheng et al., 2022, Bapna et al., 2022] or transcription tasks [Ao et al., 2021, Chen et al., 2023].
In this work, we show that textual LMs can benefit SpeechLMs. We propose TWIST, Textually Warm Initialized Speech Transformer Language Models, a method for initializing SpeechLMs from pretrained textual LMs (see Fig. 1). Our empirical findings suggest that such a simple approach is highly effective and provides consistent improvement across all the evaluated metrics, both automatic and human evaluations. We provide an extensive empirical analysis studying the effect of model and data scale, model architecture, and speech tokenizer on the overall model performance. Based on our observations, we present the largest SpeechLM to date (to the best of our knowledge), both in terms of size (13B parameters) and training data (~150k speech hours). Finally, to better evaluate SpeechLMs capabilities to model long contextual spoken sentences, we generate two spoken versions of the StoryCloze benchmark [Mostafazadeh et al., 2016). We synthesize all the stories from the original StoryCloze test set using either the original distractor or a randomly chosen one. Each of the versions evaluates different properties of SpeechLMs; the former captures fine-grained temporal commonsense relation, while the latter captures coarse global sentence coherence.
Our contributions: (i) We introduce TWIST, textually warm initialized speech transformer language model. We empirically show how text- text-based LMs can be adapted into SpeechL.ss yielding consistent improvements across all evaluated metrics, including both automatic and human ones; (ii) provide extensive analysis considering the different components of SpeechLMs. Our analysis sheds light on the different model design choices and how they affect model performance; (iii) We leverage our finding and train the largest SpeechI.M to-date in terms of the number of parameters and the amount of training data; (iv) We provide two spoken versions of the StoryCloze dataset capturing different aspects of the spoken content. We hope such datasets will be helpful for the research community to better evaluate SpeechLMs under different setups.
2 Using Textual LMs to Improve SpeechLMs
We formally describe the proposed method. We start (Section 2.1) by presenting the relevant background, including the Generative Spoken Language Modeling (GSLM) pipeline, which build on, and relevant LM formulation. We then present TWIST, our proposed method (Section 2.2).
2.1 Background
In this work we follow the GSLM framework [Lakhotia al., 2021]. The general GSLM pipeline composed of three main modules, each trained separately: (i) speech tokenizer, (ii) SpeechLM,

---

and (iii) a vocoder module (i.e., loken-to-speede Speech resynthesis can be achieved while ignoring the language model and directly feeding the quantized tokens into the vocoder module [Polyak et al.. 2021]. In the following, we provide background for each of the components mentioned above. See Fig. for a visual description.
Speech tokenizers encode raw speech into a discrete representation. The common approach is to first encode the speech into a continuous representation and then quantize the representation achieve sequence of discrete tokens [Tjandra et al., 2019, 2020, Lakhotia et al., 2021, Borsos et al., 2022].
Formally, denote the domain of audio samples by X C R. The representation for raw signal is where 2, E X for all S T.
therefore a sequence of samples x =
Consider an encoder network, ʃ, which gets as input the speech utterance and outputs sequence of spectral representations sampled at a low frequency as follows [(x) = (v1..... ;......,r))) ) where T' is determined by the frame rate of the encoder. Note that we do not assume anything about the structure of the encoder network f. Lakhotia al. [2021] evaluated several speech encoders, namely, Mel- spectrogram, Contrastive Predictive Coding [Oord et al., 2018, CPCJ, wav2vec2 [Baevski et al., 2020], and HuBERT [Hsu et al., 2021J. They found that HuBERT provides the best overall performance, and hence we follow the same setup.
Since the representations learned by such models are usually continuous, k-means algorithm [MacQueen, 1967] is applied over the models' outputs to generate discrete tokens, denoted as = (z1...., 2T Each element 2; is positive integer, 2; E (1...., (.....,K) K} for s sist T', where K is the number of discrete tokens of the vocabulary 2 -1,..,,K),
Language models aim learn the underlying joint probability of token sequences p(wr .... W,.)- Each token belongs to a vocabulary W, defined by tokenizer." Using the chain rule, the joint probability of sequence can be computed product of its conditional probabilities:
p wx = II
w
Neural LMs, parameterized by 0, aim to model the probability a representation of the previous tokens. The network parameters 0 are learned by minimizing the negative log likelihood loss between the predicted and true probability distributions, such that
Wi where is
E(0, w)
Σ
log po (w: ((:; w1)).
The network parameters are typically initialized with values sampled from a predefined distribution, e.g., uniform centered Gaussian distributions [Glorot and Bengio, 2010].
Speech Language Models (SpeechLMMp MMs) are trained on the extracted discrete speech tokens, x, using speech tokenizer. When operating 2, SpeechLMs enable directly modeling spoken data without accessing textual transcriptions. Such a modeling framework additionally allows for capturing and modeling prosodic features [Kharitonov et al., 2021], as well speaker identity [Borsos et al., 2022], or even natural dialogues INguyen et al., 2022]. This is in contrast to using textual features, which do not encode such information.
Token-to-speech modules convert the speech discrete tokens to a raw waveform. Lakhotia et al. [2021] used Tacotron2.0 [Shen et al., 2018] based model followed by WaveGlow [Prenger et al., 2019] vocoder. Later, Polyak et al. [2021] proposed a token-based vocoder based on the HiFi-GAN architecture to convert tokens to speech directly. Such paradigm seems provide high-quality generations with better efficiency, as it uses only one model rather than two. As such, we use their approach in this work for our tokentto-speech models.
2.2 Textually Warm-Initialized Speech Transformer Language Models
We propose TWIST, method for training Speechl.MP .Ms initialized from a pretrained text LM, such OPT [Zhang et al., 2022], and LLaMA [Touvrron et al., 2023). TWIST first replaces the original text vocabulary W with 2, the set speech tokens (see Section 2. 1), and sets the tokenizer to be
'Typically for text LMs, the tokenizer considers words sub-words [Sennrich al., 2016].

---

Table 1: Speech datasets statistics. We report both the number of hours in the training data of each dataset together with the number of speech tokens used to train the speech language model.
LiBRISPEECH
LIBRILIGHT
SPOTIFY
PEOPLE
VoxPOPULI
OVERALL
Hours
960
53k
59k
17k
20k
150k
Tokens-50Hz
86M
4.79B
5.34B
1.54B
1.76B
13.52B
Tokens-25Hz
58M
3.19B
3.56B
1.02B
1.17B
9.00B
a speech based tokenizer. We then replace the text lookup table with a new randomly initialized embedding table for the speech tokens. The rest of the body of the network remains unchanged during initialization time. Finally, TWIST continues training the entire SpeechLM using speech data. visual description can be found in Fig. 1.
One might wonder whether initializing a speech model with a textual one makes sense, as speech tokens operate 20-40ms windows while text tokenizers span longer concepts (e.g., sub- sub-words). Nonetheless, previous work showed that speech and text LMs can be trained jointly to improve either machine translation [Bapna et al., 2021, Cheng al., 2022, Bapna et al., 2022], transcription based speech related tasks [Ao et al., 2021, Chen et al., 2023]. We next show that Speech can benefit from textual LM initialization. We recognize that more advanced methods for converting speech tokens to word tokens probably exist, and hope this study will motivate researchers to explore them.
3 Experimental Setup
We follow the same setup described in Section 2.1. Similarly to Lakhotia et al. [2021]. we consider HuBERT [Hsu et al., 2021] followed by k-means quantizer the speech tokenizer. We use a token based HiFi-GAN neural vocoder as our Token-to-speech model, with duration predictor as in Polyak et al. [2021], Lee al. [2021]. Then we optimize, compare and analyze the performance of various SpeechLMss considering different setups.
3.1 Datasets
All SpeechLMs are optimized using a collection of publicly available academic speech datasets: LibriSpeech (LS) [Panayotov al., 2015), LibriLight (LL) [Kahn et al., 2020], Spotify pod- casts [Clifton et al., 2020]. People dataset [Galvez et al., 2021], and VoxPopuli [Wang et al., 202 We filter non- non-English data using the provided meta-data. This results in 150k hours, which translate to 13.5B tokens using token frequency of 50Hz and 9B tokens using a token frequency of 25Hz. See Table I for detailed descriptions of the datasets. To the best of our knowledge, this the first work that uses this scale of data to train SpeechLMs.
In cases where no pre-defined validation and test sets available, we randomly sample 2% of the data serving as the validation set and an additional 2% for the test set. Unless stated otherwise, in all setups reported results are the average across all different test sets.
3.2 Model Hyperparameters
Our main family of textual LMs is OPT [Zhang et al., 2022]. We examine three model sizes: OPT- 125M, OPT-350M and OPT-1.3B, corresponding to 12/24/24 transformer blocks and 768/1024/2048 hidden dimension, respectively. 6 We also consider other pretraining approaches, e.g., different pretraining corpora, and experiment with BLOOM [Scao et al., 2022] and Pythia [Biderman et al., 2023], both with equivalent sizes to OPT-350M/1.3 1.... For each model size, compare two variants: TWIST, i.e., a warm initialization from a textual LM, and a cold (randomly initialized) model following the original GSLM approach (henceforth COLD-INIT)) We use GPUs for training, except
'Considering speech rate of 120 words per minute (https: / http://vitttallpeeeeeee com/ cww/blog/////cccccccoooccooowooe average- speaking- rwweraatteeeeee words per-minuted an average word duration is 500ms, which means that SpeechLMs represent single word using 12-25 tokens.
'The actual number of parameters of the SpeechLMs is about 10 25-25% lower than the text ones since we replace the text embedding layer with smaller speech embedding layer, due to the smaller speech vocabulary.

---

Table 2: Zero-shot modeling results for different number of tokens and downsampling factors (Frequency), with and without TWIST. We report PPL over speech tokens, sWUGGY, sBLIMP. Bold indicates the best model for each tokenizer, and we underline the best performing model overall for sWUGGY and sBLIMP (PPI. results are incomparable across tokenizers).
TWIST
# TOKENS
FREQ.
PPL.
SWUGGYt
SBLIMP1
x
5.26
68.91
53.80
100
50Hz
5.03
71.30
55.96
5.61
69.85
53.48
200
50Hz
5.29
72.92
55.91
X
6.36
66.65
50.79
500
50Hz
5.85
70.69
52.71
X
6.65
79.44
54.84
500
25Hz
6.25
81.42
56.20
for the 1.3B models, which use 32 GPUs. In all cases, we choose the best checkpoint by the averaged speech perplexity of the model over the validation sets.
For the speech tokenizers, we quantize HuBERT representations, which operate at 50Hz, with 100, 200, and 500 clusters using the k-means algorithm. HuBERT is trained LS 960h, while k-means are trained over the 'clean-1001 part only from the LS corpus. Models and quantizers are obtained from the textless-lib [Kharitonov al., 2022]. Inspired by Chung al. [2021] and Borsos al. [2022], we also trained a new HuBERT tokenizer with a lower sampling rate. We trained the HuBERT model on varied mixture of datasets: Multilingual LS [Pratap al., 2020]. Vox Populi [Wang et al., 2021a], Common Voice [Ardila et al.. 2019], Spotify [Clifton et al.. 2020], and Fisher [Cieri et al., 2004]. More details regarding model setups can be found in Appendix A.I.
3.3 Evaluation
Evaluating such complex pipeline comprised of several different components is challenging task. We follow Lakhotia al. [2021], who were the first to propose an evaluation setup for this pipeline. We also introduce novel evaluation framework based on the StoryCloze textual benchmark [Mostafazadeh et al., 2016]. Overall, we consider three different evaluation setups: (i) Zero-shot modeling; (ii) Human evaluation; and (iii) Spoken StoryCloze.
Zero-shot modeling metries. We use the sWUGGY and sBLIMP INguyen al., 2020] metrics to evaluate lexical and syntactic modeling of the SpecchLMs. In sUUGGY the network is presented with pair of utterances, an existing word and a matching non-word and evaluated on their capacity to attribute a higher probability to the existing word. Unless stated otherwise, we follow the common approach [Lakhotia et al., 2021] and report the "in-vocab" split results. As for sBLIMP, similarly to sWUGGY, the network is presented with matched pair of speech segments, grammatical and ungrammatical sentences. The task to decide which of the two grammatical based on the probability of the sentence. For both measures, we compare the geometric mean over the models' sequence probabilities assigned each utterance within a pair. additionally report speech tokens Perplexity (PPL.,, averaged across all test sets. We note that while PPI. is not comparable across different tokenizers, it can be helpful to compare TWIST against COLD-INIT.
Human evaluation. To better assess the quality of the generated outputs, we follow the same setup as the Meaningfulness MOS [MMOS; Lakhotia al., 2021]. In this setup, human raters are presented with several speech continuations (~10 seconds) following the same speech prompt (~~3 seconds), and are instructed to evaluate how natural (considering grammar, meaning, and diversity) given sample is, on scale between 1-5 with increments of 0.5. For speech prompts, we randomly sample a collection 50 examples from the test sets three different datasets: LL, LSS-clea and L--other to reach overall 1550 samples per evaluated method. We enforce 10 raters per sample and use the CrowdMOS [Ribeiro et al., 201 package with the recommended recipes for detecting and discarding inaccurate scores.
5

---

7.5
Speech Perplexity
Valid Speech Perplexity
Cold-Init.
TWIST (ours)
7.0
8
6.5
7
6.0
10% Data
Data
Cold-Init. TWIST (ours)
110k
400k
100% Data
125M
350M
1.3B
6
100k
200k
300k
400k
# Parameters
Training steps
(a)
(b)
Figure 2: (a) PPL as a function of training set and model sizes. (b) Validation PPL as function of training steps. TWIST is both more sample-efficient and converges faster than COLD-INIT.
We note that Lakhotia et al. [2021] also proposed computing PPL and auto-BLEU over the tran- scription of the generated speech to assess both quality and diversity of the generations. While this metrics are automatic, they are highly influenced by numerous factors such as speech vocoder quality, Automatic Speech Recognition (ASR) transcription errors, sampling, etc. In preliminary results, we observe a high variance when computing this metric, hence we focus on human study, and report these metrics for our main family of models in Appendix A.2.
Spoken StoryCloze. Finally, better evaluate the capabilities of SpeechLMs in capturing fine-grained textual nuances and continuation coherence, we provide two spoken versions of the StoryCloze textual benchmark [Mostafazadeh al., 2016], denoted by Spoken StoryCloze (SSTORYCLOZE) and Topic StoryClose (TSTORYCLOZE)
The textual StoryCloze benchmark contains 4k five-sentence commonsense stories (split to validation and test sets). For each story, there additional negative sample, composed of the first four sentences followed by a fifth, adversarial sentence. The goal is to distinguish the original fifth sentence from the negative one. generate spoken benchmark, we synthesize the stories from the test set using single speaker TTS system as provided by Wang et al. [2021b] 1b), comprised of FastSpeech2.0 model [Ren et al., 2020] and HiFi-GAN vocoder [Kong al., 2020].
For SLTORYCLOZE, we follow the original StoryCloze negative samples. With this benchmark, we evaluate models' capabilities to capture fine-grained causal and temporal commonsense relations. For TSTORYCLOZE, we randomly sample the negative ending sentence from the dataset. The premise behind TSTORYCLOZE is to evaluate continuation coherence given spoken prompt. This version is far easier, but our experiments show that it still quite challenging for modern SpeechLMs. Similar to sWUGGY and SBLIMP, we feed both speech segments to the SpeechLM, measure the probability of each spoken sentence, and report the percentage of examples where the probability of the positive sample is higher than the negative one.
We also conduct human evaluation on both datasets in order measure human performance over the speech benchmarks (which may differ from their text equivalents). We introduce human raters with both options and let them rate each one according to MMOS [Lakhotia et al., 2021], in scale of 1-5 with 0.5 increments. The human score for this benchmarks is the proportion of samples where the score given by the rater higher for the correct utterance. We use raters for each pair of utterances and use the CrowdMOS [Ribeiro et al., 2011] package with the recommended recipes for detecting and discarding inaccurate scores.
Both datasets are available at https: htttp:///ithult http/a/pp.///poeestttrll..lc.
We introduce 100ms of silence between segments to generate naturally spoken sentence.

---

Table 3: Comparison of TWIST-7B/L3B models against prior work. We report sWUGGY, using both 'in-vocab' words and 'all' words, along with sBLIMP.
PPL1
SWUGGYT in-vocab
SBLIMP1
METHOD
all
van Niekerk et [2021]
64.3
72.3
54.0
Lakhotia et al. [2021]
68.7
57.1
Borsos al. [2022]
71.5
83.7
64.7
COLD-INIT-L.3B
6.20
72.2
81.9
56.5
TWIST-1338
5.93
72.7
82.5
57.0
TWIST-7B
5.47
73.9
83.6
59.0
TWIST-13B
5.34
74.5
84.1
59.2
4 Results
MpeechLMs benefit from warm initialization using text LMs. We start by evaluating the effect of the warm initialization on SpeechLMs We compare two versions of OPT-350M, with warm (using TWIST) and cold initialization (COLD-INIT), for different frequencies and different number of tokens. Results reported in Table 2. We observe consistent improvement across all metrics following the TWIST approach. Interestingly, using speech tokens with larger downsampling factor (i.e., smaller frequency) leads substantially better sWUGGY and sBLIMP results. These findings are consistent with Borsos et al. [2022], and also reflected by the speech resynthesis results, see Appendix A.3. For the rest of the paper report results using the speech tokenizer with 500 tokens at 25Hz, which gets the best s SWUGGY and sBLIMP results. For compatibility with prior work [Lakhotia et al., 2021], results for the speech tokenizer with 200 tokens at 50Hz are in the Appendix.
Scaling improves SpeechLMs. As prior work mainly considered relatively small SpeechLMs (e.g.. ~ ~100M parameters in Lakhotia et al. [2021], Kharitonov et al. [2021]) and relatively small training data (c.g., Borsos et al. [2022] use LL only), in the following we evaluate the effect of model and dataset scaling the overall performance.
We train three different SpeechL using the OPT family, For each model size, we increase the magnitudes of training data, considering 1%, 10%, and 100% of the training data with both TWIST and COLD- INIT. Our results (Fig. 2a and Fig. 5; Appendix A.4) first highlight that, as our previous experiments, SpeechLMs initialized using TWIST perform consistently better than those with COLD-INIT. We next note that, as expected, increasing the model size and the magnitude of the dataset improves model performance in almost all cases. Further, for all model scales, following TWIST approach with only 10% of the data yields comparable or superior performance than the corresponding COIDD-NIT approach using 100% of the data. This is consistent with previous findings that pretraining leads to higher sample efficiency for downstream tasks [Peters et al., 2018]. The full set of results for the corresponding models can be found on Table 7 in Appendix A.4.
TWIST converges faster. Next, we analyze how textual pretraining affects model convergence. Fig. 2b presents the validation loss training curves for OPT-350M. We observe that using TWIST, the model reaches the same PPL in about one quarter of the training updates compared COLD-INIT.
Not all warm initializations are equally important. We have so far seen that warm initialization from OPT models is consistently beneficial for SpeechL Are OPT models unique in this sense, could other pre-trained models similarly benefit SpeechI.Ms?
To address this question, start by reporting results for different pre-trained text LMs. We consider different textual pre-training approaches- BLOOM [Scao et al., 2022] and Pythia [Biderman et al., 2023]-bool both of similar size to DPT-350M/1.3B Tables and 9 in Appendix A.5 shows similar trends to our OPT experiments, in which using textually pretrained LMS yields consistently better results.
We next ask whether initialization of SpeechLMs from pre- pre-trained text LMs is particularly important, or would initialization from other pre-trained modalities similarly benefit SpeechL We consider ImageGPT [Chen et al., 2020], image-generation pre-trained model. We find (Table 10; Ap- pendix A.6) that unlike the textual initialization, this model not only does not outperform COLD-INIT, it substantially underperforms it.
7

---

Meaningfulne MOS
Meaningfulneee MOS
Meaningfulnes MOS
4.0
4.0
4.0
3.5
3.5
3.5
3.0
3.0
3.0
Original
Resyath
Cold-Irit
TWIST
1,38
TWIST 778
Original
Resyatt
Cold-Init
1.30
TWIST
1:38
TWIST
Original
Resyath
Cold-Irit.
TWIST
TWIST 718
(a) LS-clean.
(b) LS-other.
(c) LL.
Figure 3: Human evaluation (MMOS) for speech generation, with different models and datasets.
TWIST outperforms the COLD-INIT, and the TWIST-7B performs better than smaller models. Full results presented in Appendix A.7.
Speech Large Language Models. Equipped with previous findings, we train the largest SpeechLMMs date (to the best of our knowledge), 7B/13B parameter SpeechLMs initialized from the LLaMA- 7B/13B models [TTuvron et al., 2023], denoted as TWIST-7B/13B We use the same configuration as in Section 3.2 with specific exceptions detailed Appendix A.I.
Table 3 shows our results for the TWIST-7B/13B models along with several baseline methods. We also include COLD-INIT and TWIST models of size 1.3B for reference. Here we report SWUGGY scores both for 'in-vocab' samples (words found in the LS corpus) along with 'all' samples (including also words which do not appear the LS corpus). As expected, scaling benefits SpeechLMs. Initialization from LLaMA-7B// -BMM-BBB/1BBB leads to additional performance improvement over TWIST-1.2 ~8/10% relative improvement in PPL and ~1.7/2 2.5% relative improvement in sWUGGY When comparing to prior work, TWIST-13B outperforms all evaluated methods on both sWUGGY setups.' As to sBIIMM, TWIST-13B's results are lower than Borsos al. [2022], although the proposed method is orthogonal to their method.
Spoken StoryCloze. To better asses the contextual under- standing of the SpeechLMs we experiment with our collected Spoken StoryCloze benchmark (see Section 3.3).
Table 4: Accuracy (%) results for spoken SSTORYCLOrE (SSC) and TSTORYCLOZE (TSC) benchmarks.
Table shows the results for WWWIST-1.3B7B/11BB mod- els alongside human performance. As expected, both Ms and humans exhibit superior performance terms of continuation coherence (TSTORYCLLOZE) compared to more fine-grained relations (SSTORYCI.OZES Interest- ingly, despite humans achieving high scores the textual StoryCloze benchmark, their performance on spoken bench- marks is not flawless, suggesting that spoken language under- standing tasks are more complicated for humans compared to their written equivalents. The results of SpeechLMs indicate equivalee reasonable performance on the STTOYCLOZE benchmark, with 155% gap compared to human performance. However, this gap widens substantially on the sSTORYCLOzE benchmark. This highlights the opportunities for SpeechLMs' improvements both in continuation coherence and, more significantly, in causal and temporal commonsense relations. Lastly, consistent with our previous results, scaling SpeechI.Ss yields performance improvements on both benchmarks.
MODEL
SSC
TSCT
Human
79.9
90.2
TWIST-1.3B
52.4
70.6
TWIST-7B
55.3
74.1
TWIST-13B
55.4
76.4
Human evaluation. We conclude with human evaluation for speech generation. As stated in Section 3.3, for each model and sample we generate speech continuation of ~00 seconds (using ~3 seconds prompt). We evaluate the following models: 1.3B parameters model with and without TWIST, and TWIST-7B. We also compare to the original audio and the resynthesized reference, which serve as top-line estimates.
Our result (Fig. 3) suggest that TWIST-7B is superior to TWIST-L.3B, while COLD-INIT-I.3B performs consistently worse. Interestingly, do not observe noticeable differences between the
In fact, under the 'all' setup TWIST- outperforms all evaluated methods.

---

Prompt: Can please go swimming at the beach
Cold-Init (1.39); Can we please go swimming at the beach? Asked Tom. I should like to go swimming he said...
TWIST-1.
please go awimming at the beach? Asked Clara. No, replied
Doza, had been swimming...
TWIST-7B: Can we please go swimming at the beach with mom on April 1st? Oh my
God, would love to somewbere with kids April 1st...
Figure 4: Generation samples using 1.3B models, with and without TWIST, along with TWIST-7B.
evaluated datasets. We also provide an example of generated sentences given pre-defined prompt in Fig. 4. This example illustrates that COLD-INIT-L,, 1.3B makes grammatical errors, both COLD-INIT- 1.3B and TWIST-1 ,3B remain on topic, and TWIST-7B provides semantically richer continuations. Audio samples can be found in the project website.
Related Work
Text Language Models have been long studied under various setups [Bahl et al., 1983, Chiang et al., 2022]. While Brants et al. [2007] showed the benefit of optimizing LLMs back in 2007, recently with the help of neural networks such models became foundational models, serving as a base model for different downstream tasks [Shoeybi et al., 2019, Radford et al., 2019, Brown et al., 2020, Lieber et al., 2021, Zhang et al., 2022, Hoffmann et al., 2022, Chowdhery et al., 2022].
The success of pretrained language models as zero-shot or few-shot learners gave rise to an extensive line of work to apply similar technique to other input modalities. Chen et al. [2020] proposed generating natural images while optimizing language model using masked prediction loss over the pixel space of natural images. Other works proposed converting natural images to discrete tokens space and optimizing textually conditioned language model to perform text-to-image generation [Yu et al., 2022, Ramesh et al., 2021, Chang et al., 2023].
In the context of speech and audio, Lakhotia et al. [2021] first demonstrated how raw and uncurated speech data can be leveraged into building a GSLM system. Next, Kharitonov et al. [2021] proposed multi-stream SpeechLM to jointly process "pseudo-text" tokens together with quantized prosodic features (i.e., duration and FO). Polyak et al. [2021] evaluated the robustness and disentanglement models and demonstrated the ability to perform voice conversion
properties of speech-to-tokee as well as a lightweight speech codec. Kreuk al. [2021] proposed to cast the task of speech emotion conversion as a translation task, hence translating between one emotion to the other in the discrete space, while Maimon and Adi [2022] proposed similar approach for speaking style conversion. Nguyen et al. [2022] proposed training two SpeechLMs jointly to mimic natural spoken dialogues. Recently, Borsos et al. [2022] proposed cascading several LMs, in which one LM operates semantic speech tokens [Sicherman and Adi, 2023] while the others operate acoustic tokens [Zeghidour et al., 2021, Défossez et al., 2022]. Such modeling framework allows generating natural speech while keeping the identity of the speaker and acoustic conditions unchanged.
Another line relevant related work, demonstrated that sound and music can be generated following similar modeling paradigm. Kreuk al. [2022] first proposed optimizing language models over discrete audio representations to construct a text-to-audio generation model. Similarly Agostinelli et al. [2023] proposed optimizing three language models (following the same modeling approach as by Borsos et al. [2022]), operating at different granularity of the input representation for the task of text-to-music generation. Donahue et al. [2023] proposed similar modeling approach for the task of singing to accompaniment generation. Lee et al. [2021, 2022], Popuri et al. [2022] followed a similar modeling mechanism using different speech tokenizer and proposed textless approach for speech-to-speede translation. Hsu et al. [2022] proposed a jointly modeling visual discrete tokens together with speech tokens to perform various speech resynthesis tasks including: silent video to speech generation, speech enhancement, and recovering packet loss.
Finally, perhaps the line of research most related to this work is jointly training speech and text models. Bapna et al. [2021], Cheng et al. [2022] and Bapna et al. [2022] considered speech as another language in multilingual setups, and showed that involving speech and text part of training data

---

improves results for speech translation and multilingual text tasks. Chen et al. [2023] and Ao et al. [2021] used the joint training to improve transcriptions tasks as ASR and -exttth-Speech (TTS). None of these prior work focus on SpeechI.Ms and its modeling and generation capabilities.
Conclusion
In this work, we studied the effect of textual pretraining on speech-based language models. We empirically demonstrated how such simple model initialization can improve modeling performance considering both automatic and human- genannatdd measures. We conducted extensive evaluation of different speech tokenizers, model and dataset sizes, model architecture, and modality pretraining. Our analysis sheds light on the impact of specific modeling design choices on the overall performance of the system. Equipped with these findings, we presented the largest SpeechL.Ms to date (7B and 13B parameters). Finally, we generated two spoken versions of the StoryCloze textual benchmark to measure contextual understanding abilities of SpeechLMs. We hope such empirical findings and benchmark releases will be found useful for future research in the field.
7 Limitations and Broader Impact
Limitations. The biggest limitation of SpeechLMs at large is the lack of semantic understanding. which might lead to ungrammatical, off-topic, or even inaccurate responses. Although we provided a better initialization point for optimizing SpeechL.M, we did not observe major semantic knowledge transfer following the TWIST approach. This limitation is general critique Speechl .Ms, which should be better studied and addressed in future research. Another limitation of the proposed approach the granularity of the speech tokenizer, such small input resolution (50Hz/ 25Hz) yields relatively long sequence (750/ 500 tokens) for relatively short speech segments (~30 seconds). This translates into long inference time and harder optimization.
Broader impact. SpeechLMs share the same potential benefits for society as text-based LMs as they give access, in the audio modality, to the same downstream applications (search, language generation, summarization, translation, chatbots, etc.). Thereby increasing their reach to more use cases and more languages, including 'unwritten' (or sparsely written) languages [Lakhotia et al., 2021]. As result, they also share the same potential risks regarding intentionally harmful applications (e.g. fake news, spamming, election manipulation) or unintentionally harmful ones (c.g., unfair or biased results, toxic, regurgitated or untrustful generations).
Acknowledgements
We thank Miri Varshavsky Hassid for the great feedback and moral support.
Authors note. This paper was submitted the wake of the tragic terror attack perpetrated by Hamas on October 7, 2023, which has left the Israeli nation profoundly devastated. The assault greatly impacted us at a personal level, thereby significantly impacting the course of this research work. This paper was finalized while we grieve and mourn our friends and family, under great stress, with scientific considerations being the last thing on our minds. It may contain subtle errors.
In memory of the countless lives shattered by Hamas actions.
References
T'sung-Hsien Wen, Milica Gasic, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711-1721, 2015.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. Slurp: A spoken language understanding resource package. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7252-7262, 2020.
10

---

Akshat Gupta. On building spoken language understanding systems for low resourced languages. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 1-11,201 2022.
Libo Qin, Tianbao Xie, Wanxiang Che, and Ting Liu. A survey on spoken language understanding: Recent advances and new frontiers. arXiv preprint arXiv:2103.0303 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 1xXxv:2302.13377.. 2023.
Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-17033, 2020.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, AbdeIrahman Mohamed, and Emmanuel Dupoux. On Generative Spoken Language Modeling from Raw Audio. TACL, 2021.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: (0.18653/v1/N19--10 URL https: //aclanthology www.aantooooooo/ooolloeo otg/
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL html.
http://mmmlr..gg/pppeppeeee httg//apeer//21/22--0 http://jmlr.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901,202
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint Xrviv: 188000016666..1111111110111 2018
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 2022.
Hung Le, Yue Wang. Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314-21322 2022.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. arXiv e-prints, pages arXiv-2203, 2022.
Ankur Bapna, Yu-an Chung, Nan Wu, Anmol Gulati, Ye Jia, Jonathan H Clark, Melvin Johnson, Jason Riesa, Alexis Conneau, and Yu Zhang. Slam: unified encoder for speech and language modeling via speech-text joint pre-training, arXiv preprint 100:::211001030000 2021.
11

---

Yong Cheng, Yu Zhang, Melvin Johnson, Wolfgang Macherey, and Ankur Bapna. Mu² slam: Multitask, multilingual speech and language models. arXiv preprint arXiv:2212.09553, 2022.
Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. mslam: Massively multilingual joint pre-tra pre-training for speech and text. arXiv preprini 2rx 2022.
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. Speecht5: Unified-modal encoder- decoder pre-training for spoken language processing. 2021.
Zhehuai Chen, Ankur Bapna, Andrew Rosenberg, Yu Zhang, Bhuvana Ramabhadran, Pedro Moreno, and Nanxin Chen. Maestro-u: Leveraging joint speech-text representation learning for zero supervised speech asr. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 68-75. IEEE, 2023.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv: 1604.000490999 2016.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprini 200:::2004000355, 2021.
Andros Tjandra, Berrak Sisman, Mingyang Zhang, Sakriani Sakti, Haizhou Li, and Satoshi Nakamura. Vqvae unsupervised unit discovery and multi-scale code2spec inverter for zerospeech challenge 2019. In Interspeech, 2019.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Transformer vq-vae for unsupervised unit discovery and speech synthesis: Zerospeech 2020 challenge. In Interspeech, 2020.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: a language modeling approach to audio generation. arXiv preprint arXiv:2209.031: 200.00333,, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807:1807.03744 2018
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vee 2.0: A framework for self-supervised learning of speech representations. NeurIPS, 2020.
J MacQueen. Classification and analysis multivariate observations. 5th Berkeley Symp. Math. Statist. Probability, pages 281-297. University of California Los Angeles LA USA, 1967.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Anmual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany, Au- gust 2016. Association for Computational Linguistics. doi: 10.18653/ 1162. URL httpp:///ccaathhology) - otg/P16-11/2.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL
.ongss...... mlr. press/v9/ glorot10aaaa.. html.
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, et al. Text-free prosody- aware generative spoken language modeling. arXiv preprint arXiv:2109.03266 2021.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. Generative spoken 2022.
dialogue language modeling. arXiv preprint
12

---

Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitll, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang. Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In ICASSP, 2018.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: flow-based generative network for speech synthesis. In ICASSP, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint
0ae::2205.01066 2022.
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, al. Direct speech-to-specch translation with discrete units. arXiv preprini arXiv:2107.05604, 2021.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In ICASSP, 2015.
Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Ferre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri-light: A benchmark for with limited supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7669-7673. IEEE, 2020.
Ann Clifton, Aasish Pappu, Sravana Reddy, Yongze Yu, Jussi Karlgren, Ben Carterette, and Rosie Jones. The spotify podcast dataset. arXiv preprint arXiv:2004.042770 2020.
Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón, Keith Achom, Anjali Gopi, David Kanter, Maximilian Lam, Mark Mazumder, and Vijay Janapa Reddi. The people's 's speech: A large-scale diverse english speech recognition dataset for commercial usage. arXiv preprint arXiv: (xv::2111009444 1.09344, 2021.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech cor- pus for representation learning, semi-supervised learning and interpretation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 Ithh International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 993-1003, 2021a.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. BLOOM: A 176b- parameter open-access multilingual language model. arXiv preprint ar arXiv:2211.05100, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprins 1.XY::2304.01373. 2023.
Eugene Kharitonov, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Paden Tomasello, Ann Lee, Ali Elkahky, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, et al. rextless-lib: library for textless spoken language processing, arXiv preprint arXiv:2202.0733: 2022.
Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244-250. IEEE, 2021.
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.034111220 2020.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively- multilingual speech corpus. arXiv preprint 192.v:191. 2019.
13

---

Christopher Cieri, David Graff, Owen Kimball, Dave Miller, and Kevin Walker. Fisher english training speech part transcripts. Philadelphia: Linguistic Data Consortium, 2004.
Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling, In NeurIPS Self-Supervised Learning for Speech and Audio Processing Workshop, 2020.
Flávio Ribeiro, Dinei Florêncio, Cha Zhang, and Michael Seltzer. Crowdmos: An approach for crowdsoureing mean opinion score studies. In 2011 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 2416-2419. IEEE, 2011.
Changhan Wang. Wei-Ning Hsu, Yossi Adi, Adam Polyak, Ann Lee, Peng-Jen Chen, Jiatao Gu, and Juan Pino. fairseq s^2: A scalable and integrable speech synthesis toolkit. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 143-152, 2021b.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006. 04558, 2020.
Matthew F. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Pupers), pages 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/vI/N18- 1202. URL org/N18. N118/N181111 1202.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020.
Benjamin van Niekerk, Leanne Nortje, Matthew Baas, and Herman Kamper. Analyzing speaker information in self-supervised models to improve zero-resource speech processing. arXiv preprint arXiv:2/08.00917, 2021.
Lalit R Bahl, Frederick Jelinek, and Robert L Mercer. A maximum likelihood approach continuous speech recognition. IEEE transactions on pallern analysis and machine intelligence, 12):179-190, 1983.
Cheng-Hau Han Chiang. Yung-Sung Chuang, and Hung- Yi Lee. Recent advances in pre-trained language models: Why do they work and how do they work. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing: Tutorial Abstracts, pages 8-15, 2022.
Thorsten Brants, Ashok Popat, Peng Xu, Franz Josef Och, and Jeffrey Dean. Large language models in machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Langugee Processing and Computational Natural Language Learning DSMNLP-CCNL.L pages 858- 867, 2007.
Mohammad Shoeybi. Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Ime Im: Training multi- multi-billion parameter language models using model parallelism. arXiv preprint arXiv: /90.0089000033 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Opher Lieber, Or Sharir, Barak I Lenz, and Yoav Shoham. Jurassic-1: Technical details and evaluation. White Paper. A121 Labs, 1, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendrieks, Johannes Welbl, Aidan Clark, et al. Training computc-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
14

---

Jiahui Yu, Yuanzhong Xu. Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821 8831. PMLR, 2021.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.0000 000004, 2023.
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei- Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. Textless speech emotion conversion using decomposed and discrete representations. arXiv preprint arXiv:2111.07402, 2021.
Gallil Maimon and Yossi Adi. Speaking style conversion with discrete self-supervised units. arXiv preprint arXiv:2212.097300 2022.
Amitay Sicherman and Yossi Adi. Analysing discrete self supervised speech representation for spoken language modeling. arXiv preprint arXiv:2301.00591, 2023.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint urXiv:2210. 13438, 2022.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209.15335 2022.
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325. 2023.
Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli, Mauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghidour, et al. Singsong: Generating musical accompaniments from singing. arXiv preprint arXiv:2301.12662 2023.
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. Textless speech-to- speech translation on real data. In NAACL, 2022.
Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lec. Enhanced direct speech-to-speech translation using self-supervised pre-training and data augmentation. arXiv preprint arXiv:2204.02990 222677277. 2022.
Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, and Yossi Adi. Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech enhancement, arXiv preprint arXiv:2212.11377. 2022.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.044 2022.
15

---

A Supplemental materials
A.1 Model & hyperparameters
All LM models are trained with a batch size of 64, where each sample is bounded for 25 seconds and 704 tokens. The models are trained for 400k steps (~11.2 epochs), using an inverse-sque sqrr scheduler, 100 warmup steps and wADAM as the optimization algorithm. We also tune the learning rate per scenario, i.e: using/not-using pretrained LM, we end up with a maximal learning rate of 4e-4/8e-5 and final learning rate of 8e-5/2.5e-5, respectively. As for the L.L.A-
model, we use the same configuration except the following: cosine learning rate schedule, 500 warmup steps, maximum learning rate of le-4, final rate of le-5, batch size 1024 over GPUs for 75k steps (~4 epochs).
The new the frequency HuBERT speech tokenizer (now available in textless-lib Kharitonov et al. [2022]), is trained for 3 iterations with the default 50Hz features rate. For the 4-th iteration, we add an additional convolutional layer at the CNN Encoder with the strides 2/3/4, resulting in features of 25Hz/16 (ttt://6..H//1.... 6Hz/12.5Hz respectively. Our early ablations show that 25Hz features with 500 tokens give the best results in terms of language modeling, we thus train our models on these new tokens and compare them with the rest of the tokens.
A.2 Automatic evaluation of speech generation
As stated, PPL scores over the transcribed speech generations (text-PPL. are sensitive to modification in the sampling parameter, vocoder quality and ASR errors [Lakhotia al., 2021]. Furthermore, text-PPL tells only one side of the story, we also need to count for diversity in the generation, as suggested by Lakhotia et al. [2021]. For instance, repeated speech gets good text-PPL score, the cost of bad diversity measures.
To mitigate that, when computing the automatic metrics for generated speech, we threshold the temperature parameter using the ASR model confidence. Next, as there can be trade-offss in terms of sentence coherence and diversity, we calibrated the temperature so that the generated speech matches the transcription the original utterance (as much as possible) terms of generation diversity (auto-BLEU from Lakhotia et al. [2021]).
For speech prompts. we randomly sample collection of 1024 examples from the test sets of three different datasets: LS-clean, LS-other and LL. Each speech prompt is ~-3 seconds long, while the continuation is ~10 seconds. For the calibration procedure, we used randomly selected 128 examples from the corresponding validation datasets. Text transcriptions extracted using Whisper "small" [Radford et al., 2022], while text-PPL measures were computed using LLaMAA-7B (text model) [TTuvrro et al., 2023]. Table 5 reports both text-PPL and auto-B BLEU results for the original continuations, and the generations that produced by our main family models (OPT architecture coupled with the 25Hz tokenizer).
Table 5: text-PPL and auto-BLEU results, with and without TWIST.
TWIST
# PARAM.
TEXT-PPLJ
AUTO-BLEUL
Original Cont.
36.35
0.23
88.17
0.305
125M
74.95
0.29
117.17
0.27
350M
V
92.69
0.25
67.68
0.29
1.3B
47.8
0.29
As can be seen, while the diversity metric is comparable across models (due to calibration), TWIST models outperform the COLD-INIT ones in terms of text-PPL. These measures further support our claims: TWIST models outperform COLD-INIT ones, and models generally improve with scale. Notice, the text-PPI. for the 350M models are worse than the 125M models. However, as stated, the diversity of the generations (as computed using auto-BLEU) is better for the 350M models.
16

---

6.5
10% Dala
TWIST (ours)
Speech Porplexity
100% Data
6.0
5.5
5.0
125M
350M
1.3B
#
Parameters
Figure 5: PPL as function of training set and model size, for models trained with the 200 tokens at 50Hz tokenizer.
A.3 Speech resynthesis results
Resynthesis can be considered as an upper bound for our language modeling setup. It does not involve Mps and measures our ability to fully recover the speech content after tokenization [Polyak et al., 2021]. As we additionally evaluate several speech tokenizers, we provide resynthesis metrics in the form of Word Error Rate (WER). We use Whisper "small" [Radford al., 2022] as our ASR model.
In Table 6, evaluate the effect of the tokenizer the resynthesis performance, and better evaluate the impact of the tokenization process on the generated audio. As can be seen, all tokenizers incur loss in WER. Using 500 tokens 25Hz provides the best performance.
Table 6: Speech Resynthesis. Results are reported for different number of tokens and downsampling factors (Frequency).
# TOKENS
FREQUENCY
WER
100
50Hz
0.23
200
50Hz
0.18
500
50Hz
0.17
500
25Hz
0.16
A.4
Model and data scaling results
The full set of results, i.e., PPI., sWUGGY and sBLIMP from Section 4 for model and dataset scaling presented Table The equivalent of Fig. 2a using 200 tokens at 50Hz tokenizer can be found in Fig.
A.5 The effect of LM architecture
To further validate our findings holds for other LM architectures other than OPT. In Table 8, we provide results for the BLOOM [Scao et al., 2022] and Pythia [Biderman al., 2023] of similar size to OPT-350M, with both 25Hz and 50Hz tokenizers. In Table 9 we provide results for BLOOM and Pythia of similar size OPT-1.3B with the 25Hz tokenizer.
As before, we observe similar patterns in terms of using a pretrained text LM. SpeechLMs initialize from text reach better performance across all metrics.
A.6 The effect of different modality pretraining
Although having completely different granularity, results suggest training SpeechL with model initialization from text based LMs brings consistent performance improvement. As result,
17

---

Table Model and Data Scaling, Results are reported for different models on various size using different magnitude of data, with and without TWIST. We report PPL/ sWUGGY sBLIMP.
TWIST # PARAM.
# TOKENS
FREQ.
1% OF DATA
10% OF DATA
100% DATA
x
6.08 66.90 $2.45
5.79/ 68.16/6 52.71
5.79 68.26/ 53,02
/
125M
200
50HZ
5.94 69.48 52.87
5.51 70.67 54.34
5.49 / 70.75 53.92
X
51.97 6.14 66.49
5.71 68.85 53.13
5.61 68.95 / 53,48
/
350M
200
50Hz
5.93 68.49 53.13
5.37 /7.64/ 55.63
5.29 72.92 55.91
6.29 64,91 52, 18
5.51 /71.39 54.58
5.32 72.83 55.12
1.3B
200
50Hz
67.89
52.83
5.37 72.45 55.65
5.23 73.39 /55.9
X
7.22 77.58 / 53.74
6.82 18.12// 54.00
6.81 77.74/ 54.27
/
125M
500
25Hz
7.06
78.99
54.12
6.52 80.08 55.45
6.50 / 80.57 55,43
7.37
53.07 78.67
76.96
6.79/ 77.93 54.71
6.65 79.44
350M
500
25Hz
7.26
53.95
6.41 /1.23/ 56.08
6.26 79.44 56.20
x
7,49/ / 75.44
52,96
6.40 80.80 55.95
6.20 81, / 56.52
/
1.3B
500
25Hz
7.19 78.00 53.90
6.19 82.28 56.81
5.93 / 82.49 / 57.05
Table 8: LM Model Architecture. Results are reported for both Bloom and Pythia model architectures (~350M parameters), with and without TWIST. We report PPI., sWUGGY and sBLIMP.
TWIST ARCH.
# TOKENS
FREQ.
PPL4
SWUGGYt SBLIMP1
X
5.63
69.38
53.03
BLOOM
200
50Hz
5.21
72.48
55.79
X
6,45
79.46
55.60
BLOOM
500
25Hz
/
6.06
82.01
57.22
X
5.62
70.00
53.07
Pythia
200
50Hz
/
5.23
72.20
56.00
X
6.45
79.82
55.45
Pythia
500
25Hz
/
6.12
81.41
56.70
Table 9: LM Model Architecture. Results reported for both Bloom and Pythia model architectures
1~BB parameters), with and without TWIST. We report PPL, sWUGGY and sBLIMP.
TWIST
ARCH.
PPL.
SWUGGY!
SBLIMP1
X
6.09
80.47
56.02
BLOOM
/
5.80
82.63
57.43
x
6.05
81.33
56.34
Pythia
5.81
81.77
57.02
natural question would be do speech and text tokens have special connection or LMS are just general next token prediction mechanisms?
To evaluate such hypothesis, consider language model trained on different modality. Specifi- cally, we train ImageGPT [Chen et al., 2020] ("mediumm" size) models, one from scratch and another one pretrained using next pixel prediction using transformer language model. For the pretrained model we use the official pre-traine pre-trained model." 10 Table summarizes the results.
Interestingly, ImageGPT pre- pre-trained models perform much worse than models pretrained on text. For a reference, models trained from seratch achieve comparable performance previously reported models.
18

---

Table 10: Results for the ImageGPT model (image pretraining), with and without TWIST. We report PPL. sWUGGY and sBLIMP. Unlike textual pretraining, image pretraining not only does not benefit Mps substantially hurts their performance.
TWIST # TOKENS
FREQ.
PPL|
SWUGGY1
SBLIMP1
5.22
72.22
55.70
200
50Hz
8.21
54.26
51.90
6.20
81.85
56.39
500
25Hz
7.85
72.50
51.84
A.7
Full human evaluation (MMOS) results
We include the full Human evaluation (MMOS) results, corresponds to Fig. 3 in Table 11.
Table 11: Full human evaluation (MMOS) results. We report MMOS score as: mean (95% confidence- interval).
METHOD
LS-CLEAN
LS-OTHER
LL-OTHER
Original
4.11(#0.04)
4.05(#0.05) 4.23(+0.04)
Resynthesis
3.95(=0.07)
3.87(+0.06)
3.96(+0.06)
COLD-INIT-L33B
3.34(#0.08)
3.37(+1.06)
3.31(#0.07)
TWIST-1.38
3.51(+0.07)
3.67(+0.07)
3.65(#0.06)
TWIST-7B
3.79(+1.00)
3.85(10.07)
3.81(#0.06)
19


Reviews:
[{"review_id": "hzKphZHm4t", "reviewer": "Reviewer_qFrm", "summary": "This paper studies the effect of textual LM on SpeechLMs. They propose TWIST, which initializes the SpeechLM model with a pre-trained textual LM, and then finetune with the speech datasets.\nThis paper provides a complementary exploration of generative spoken language modeling, including front-end processing of speech tokenizer, the core component of SpeechLM, and back-end speech synthesis of Vocoder.\nTWIST uses a warm-start for SpeechLM and outperforms the cold-start model in all experiments. In addition, this paper also provides empirical results across various pretrained models, and presents the large-scale 7B-sized Speech language model.\n", "questions": "1. Does HuBERT use different vocoders to resynthesize speech for different clustering tokens and frequencies in Section A.2?\n2. Can the authors provide an explanation for the poor performance of cold-start methods? Is it because there is insufficient training data to fully train the model?\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides insight into the design of SpeechLMs based on rapidly developing large- scale textual pretrained models.\n2. This paper presents many empirical conclusions and findings, such as the HuBERT setting, results on different scales of training data.\n3. This paper introduces new benchmarks for evaluating SpeechLMs.\n", "weaknesses": "1. Some details are missing. For example, warm-start allows better performance compared to cold-start, but training costs for both methods are not reported. These results help researchers estimate the training costs of large-scale SpeechLMs.\n2.  Automatic evaluation results are missed. This paper reports the human evaluation of MMOS for speech generation, but no automatic evaluation results are included, like WER for ASR results. Therefore, it is different to compare results across related work.\n3. Although TWIST shows powerful capability of speech understanding, SpeechLMs still lack deep semantic understanding compared to textual models, as stated in Limitation. Therefore, the proposed method is somewhat of a compromise.\n \nDespite these issues, I still believe that this work is highly valuable and can provide direction for the following researchers.\n \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer believes our work is “highly valuable and can provide direction for following research.” We also thank the reviewer for acknowledging the contributions of our paper including the rigorous empirical results, the insights into the design considerations of SpeechLMs and the introduction of a new speech benchmark based on the StoryCloze benchmark.\n\n**Regarding reporting training costs:** . We highlight that both the cold- and warm-started models are trained using similar resources and have the same computational cost, for example the 1.3B models are each trained on 32 GPUs for 400K steps (about 9 days). See Appendix A.1 for more details. Moreover, in figure 2.b we show that for some models our approach (TWIST) saves about 75% of the training time/cost compared to the cold-init baseline. Finally, we note that these numbers do not include the cost of pretraining the textual models as we assume such models are readily available, and do not require further processing on our behalf. \n\n**Regarding lack of automatic evaluations:** We agree that additional automatic evaluations would make the paper stronger. Unfortunately, automatic evaluations is a weak point of the emerging field of SpeechLMs. In fact, one of our main contributions addresses this particular aspect: the introduction of tStoryCloze and sStoryCloze as other objective metrics. In addition to these metrics, we provided the standard metrics in evaluating speechLMs:  sWUGGY and sBLIMP (see [1-3]). \n\nFurthermore, as discussed in lines 177-182, we tried to add metrics over text transcription (produced by an ASR model) of the generated speech such as sentence coherence (text-PPL) and generation diversity (auto-BLEU) as suggested in [1]. We observe that these metrics can be unstable and sensitive to different hyper-parameters. For instance, errors from the ASR model, such as word corrections for unintelligible speech, can affect the text-PPL metric. To mitigate that, we threshold the temperature parameter using the ASR model confidence. Next, as there can be trade-offs in terms of sentence coherence and diversity, e.g., repeated generations can get good text-PPL but bad diversity measures, we calibrated the temperature so that the generated speech matches the transcription of the original utterance (as much as possible) in terms of generation diversity. Due to all of the above, we decided to drop this evaluation from our manuscript. Nevertheless, as per the reviewer’s request, below we report both text-PPL and auto-BLEU results for our main family of models (OPT architecture). \n\n|     TWIST     |     MODEL-SIZE     |   text-PPL↓   |   auto-BLEU↓   |\n|:-------------:|:------------------:|:-------------:|:--------------:| \n|      ---      | orig. Transcript   |     36.35     |     0.225      |\n|       ✗       | 125M param         |     88.17     |     0.305      |\n|       ✓       | 125M param         |     74.95     |     0.288      |\n|       ✗       | 350M param         |    117.17     |     0.265      |\n|       ✓       | 350M param         |     92.69     |     0.253      |\n|       ✗       | 1.3B param         |     67.68     |     0.298      |\n|       ✓       | 1.3B param         |     47.8      |     0.296      |\n\nAs can be seen, while the diversity metric is comparable across models (due to calibration), TWIST models outperform the cold-init. ones in terms of sentence coherence. These measures further support our claims: TWIST models outperform cold-init, and models generally improve with scale. All text-PPL measures were computed using a LLaMA-7B (text model). \n\nIf the reviewer has additional metric suggestions, we would be happy to incorporate them into the paper. \n\n**Regarding SpeechLMs not on par with TextLMs:** We agree there is still room for improvement for speechLMs when compared to text-based models. While not perfect, we believe our work is a step in the right direction towards closing that gap, and hope it will allow others to build on our approach and make further progress. \n\n**Answers to questions:**\n1. Yes, each HuBERT model comes along with its “own” vocoder. \n2. We can only speculate as to why cold-initialized models perform worse than warm-initialized ones. One option would be, as the reviewer suggests, that the amount of data is still insufficient for a cold-started model. This is backed by Figure 2.a which shows that for larger models using more data makes the gap between cold- and warm-initialized models smaller. This hypothesis is not trivial to test, as we used all publicly available speech data that we are aware of, so simply scaling the models with more data is challenging, which further highlights the need for alternative methods for improving SpeechLMs, such as our proposed method.\nHaving said that, the type of data used for pretraining plays an important role, as can be observed from the modality experiment, which used ImageGPT as warm initialization for TWIST (lines 237-241). \n\n[1] Lakhotia, Kushal, et al. \"On generative spoken language modeling from raw audio.\" Transactions of the Association for Computational Linguistics 9 (2021): 1336-1354.\n[2] Borsos, Zalán, et al. \"Audiolm: a language modeling approach to audio generation.\" IEEE/ACM Transactions on Audio, Speech, and Language Processing (2023).\n[3] Qian, Kaizhi, et al. \"Contentvec: An improved self-supervised speech representation by disentangling speakers.\" International Conference on Machine Learning. PMLR, 2022."}, {"review_id": "86jlBNu9Wo", "reviewer": "Reviewer_6BNW", "summary": "This paper proposes a simple initialization method for speech LMs named, TWIST (Textually Warm-Initialized Speech Transformer Language Models). Instead of cold-initializing a speech LM, twist initializes it with LLM weights (minus the token embeddings, which are replaced with speech vocabulary). The authors show than on OPT models of size 125M to 1.3B (+ BLOOM, Pythia of 350M size), the proposed method of initialization helps in convergence and final perplexity evaluation, as well as shows benefits over cold initialization over human evaluation (MMOS) of generated completions. The authors also create two spoken versions of the StoryCloze dataset.", "questions": "1. Why not include the PPL over speech transcription results in section 3.3? If only one component is varied (cold-init model vs TWIST), then the results could be quite useful?\n2. The data scaling is highlighted as a major contribution, however, except OPT other models aren't varied upto the Billion scale.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is quite simple (just a straight-forward initialization trick) and the empirical results show small benefits across speech LMs of different sizes.", "weaknesses": "1. The experiments could be more comprehensive, e.g., only one size of BLOOM/Pythia is considered. Similarly, the authors do not report the  PPL over the transcription of the generated speech. Even if the results are high-variance, it is an important aspect of evaluation as far as the quality and diversity of speech continuations is considered.\n2. The proposed method is extremely simple, it just initializes the speech LM with LLM weights. There is prior empirical justification to do this, however, the weight spaces as quite incongruent, as acknowledged by the authors, and this problem hasn't been addressed at all. As such, it is hard to consider the contribution very solid in terms of its technical merit. The authors show that text based initialization outperforms Image based initialization, but that again is not a very novel result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s acknowledgement of the new benchmark contributions, the benefit of using TWIST in terms of convergence speed and quality while being simple and easy to use.\n\n**Regarding reporting only one model size for Bloom/Pythia (weakness 1, question 2):** To have a fair comparison in the paper, we report results for different model sizes using the same family of model architecture, OPT. To evaluate the effect of model architecture, we additionally report results for Bloom and Pythia. We also report results for ImageGPT and LLaMA models. Hence, we believe the reported experiments are comprehensive and include a versatile set of models. We highlight that each such experiment is resource intensive (e.g., it takes ~9 days to train the 1.3B parameter model for 400k steps using 32 GPUs), which limits the number of experiments we can run. Nevertheless, we provide cold- and warm-init results for Bloom and Pythia using a 1.3B parameter models. Results show similar trends as we observe with the other reported models. \n\n\n|     TWIST     |     MODEL     |   PPL↓   |   SWUGGY↑   |   SBLIMP↑   |\n|:-------------:|:-------------:|:--------:|:-----------:|:-----------:|\n|       ✗       | BLOOM 1.3B    |   6.16   |    81.88    |    59.40    |\n|       ✓       | BLOOM 1.3B    |   6.02   |    82.87    |    60.47    |\n|       ✗       | Pythia 1.3B   |   6.13   |    81.47    |    59.72    |\n|       ✓       | Pythia 1.3B   |   5.91   |    82.73    |    60.19    |\n\n**Regarding reporting PPL over transcripts (weakness 1, question 1):** \nAs stated in the paper (lines 177-182), text-PPL scores over the transcribed speech generations are  sensitive to modification in the sampling parameter and ASR errors [1]. Furthermore, text-PPL tells only one side of the story, we also need to count for diversity in the generation (such as auto-BLEU, [1]). For instance, repeated speech gets good text-PPL scores, at the cost of bad diversity measures.\n\nTo mitigate that, when computing the automatic metrics for generated speech, we threshold the temperature parameter using the ASR model confidence. Next, as there can be trade-offs in terms of sentence coherence and diversity, we calibrated the temperature so that the generated speech matches the transcription of the original utterance (as much as possible) in terms of generation diversity (auto-BLEU). Due to all of the above, we decided to drop this evaluation from our manuscript. Nevertheless, as per the reviewer’s request, below we report both text-PPL and auto-BLEU results for our main family of models (OPT architecture). \n\n\n|     TWIST     |     MODEL-SIZE     |   text-PPL↓   |   auto-BLEU↓   |\n|:-------------:|:------------------:|:-------------:|:--------------:| \n|      ---      | orig. Transcript   |     36.35     |     0.225      |\n|       ✗       | 125M param         |     88.17     |     0.305      |\n|       ✓       | 125M param         |     74.95     |     0.288      |\n|       ✗       | 350M param         |    117.17     |     0.265      |\n|       ✓       | 350M param         |     92.69     |     0.253      |\n|       ✗       | 1.3B param         |     67.68     |     0.298      |\n|       ✓       | 1.3B param         |     47.8      |     0.296      |\n\nAs can be seen, while the diversity metric is comparable across models (due to calibration), TWIST models outperform the cold-init. ones in terms of sentence coherence. These measures further support our claims: TWIST models outperform cold-init, and models generally improve with scale. All text-PPL measures were computed using a LLaMA-7B (text model). \nNotice, the text-PPL for the 350M param models are worse than the text-PPL of the 125M param models. However, as stated, the diversity of the generations (as computed using auto-BLEU) is better for the 350M models. \n\n**Regarding the simplicity of the proposed method:** As noted by the reviewer themselves, the simplicity of our approach is one of its strengths, especially combined with consistent performance gains that it leads to.\n\n**Regarding prior empirical justification:** To the best of our knowledge, we are the first work to empirically show this phenomenon for SpeechLMs (i.e., text initialization is beneficial for speech based LMs). Similarly, for ImageGPT, we are not aware of any prior work that shows similar findings in the past (see background and method sections). Additionally, the main purpose of the ImageGPT experiment is to better highlight the advantages of using text-based LMs for initialization. We believe such findings are interesting and would be valuable for the community. We would be happy if the reviewer could point to prior work that presents similar findings.\n\n**Regarding data scaling (Question 2):** We believe there is a misunderstanding. Scaling the data does help the model get significantly better performance across all model sizes, except for the 125M param model using 10% or 100% of the data where we observe minor improvements. Which we believe is due to model capacity (see Figure 2a). When considering model scaling, we also observe a significant improvement in performance, when scaling the model. Again, with the exception of only 1% of the data. In this case, the bigger model tends to overfit faster. Can the reviewer please clarify their question / concern? \n\nIn addition, we report the results for BLOOM and Pythia using the 1.3B parameter model and observe similar trends. See the table above.\n\n[1] Lakhotia, Kushal, et al. \"On generative spoken language modeling from raw audio.\" Transactions of the Association for Computational Linguistics 9 (2021): 1336-1354.\n"}, {"review_id": "t5MbRe4nJ3", "reviewer": "Reviewer_pWSQ", "summary": "This work proposes TWIST, Textually Warm-Initialized Speech Transformer-based LMs, a technique to initialize SpeechLMs with pretrained textual LMs. Different textual LMs, tokenizers, models and dataset sizes are evaluated using TWIST. The authors find that a warm start with a textual LM helps compared to a random initialization when evaluated on a number of metrics.", "questions": "- An analysis of the learned speech-based discrete tokens with and without the use of TWIST would be interesting to see. For example:\n    - Visualizing the representations learned via TWIST (i.e., a warm initialization from a textual LM) and those learned via a randomly initialized model. Color-coding representations based on phoneme labels (derived from a forced alignment) might reveal that phoneme distinctions are less pronounced with using TWIST.\n    - Training simple probing classifiers (e.g., to recognize phones or subword textual units) whose inputs are speech representations with and without TWIST and comparing their accuracies.\n- Due to the warm start from textual LMs that model long-term textual dependencies, one would assume that TWIST would do well on measures like sBLIMP that look at the grammaticality of entire sentences. However, from Table 3, TWIST does not fare as well on sBLIMP as prior work from Borsos et al. Could the authors comment on this?  Also, it would be useful to show numbers using a randomly initialized model (in direct comparison to TWIST) in Table 3.\n- Are the results in Table 2 using TWIST with OPT-350M or OPT-1.3B?  Please clarify. Assuming it is 1.3B, why is the sBLIMP score for TWIST-1.3B in Table 3 (60.3) different from that reported in Table 2 (59.3)?", "limitations": "Limitations have been addressed in the submission.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- TWIST shows how a warm start with any pretrained textual LM (OPT, Bloom, Pythia) benefits a speech-based LM.\n- This is one of the first works to use large-scale audio data (i.e., approximately 150K hours of speech) to train a speech-based LM.\n- This work presents a new evaluation benchmark for speech-based LMs based on StoryCloze.", "weaknesses": "Warm start with pretrained textual LMs was found to be effective for speechLMs. A more detailed analysis of the learned representations with and without TWIST would have been useful for the reader. I elaborate on this further in my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the extensive evaluation we conducted across different textual LMs, tokenizers, models and dataset sizes. We also appreciate the reviewer for noting that our work is one of the first ones to work on large-scale speech language modeling and the introduction of new benchmarks for speech language modeling.\n\n**Regarding analysis of the proposed method:** We thank the reviewer for their suggestion, however, we would like to emphasize that both cold- and warm-started models use the same tokenizer (obtained by a HuBERT model). Once analyzing such discrete representation, we already observe a strong correlation to phoneme states (even without any SpeechLM optimization involved). We can consider such input to the model as “pseudo-phonemes”. Similar findings can also be observed in prior work (see [1]). Hence, we do not expect to see strong trends in better-capturing phoneme information between TWIST vs. non-TWIST models.\n\nWe hypothesize the performance improvement we observe while using TWIST is due to better prior weight initialization, which provides better modeling of long sequences. Following the reviewer’s request, we visualize the performance gap between TWIST and non-TWIST model (350M params) while slowly decreasing the context length (see Figure.1 in the attached rebuttal pdf file). We observe that as the bigger the context the bigger the gap between the models. \n\nWe additionally provide below a few textual examples from tStoryCloze in which the TWIST model succeeds in modeling, while the non-TWIST one does not. \\\nEach example is followed with the correct / incorrect suffix.\n\n*Example 123:*\n\n*Harry went to the theme park with his family. His dad and brother rode on a big roller coaster with him. He then rode on some smaller rides with his mom. The family ate at a restaurant at the park. \\\nHarry had a great time at the park. / After counting the cars, we went back home.*\n\n*Example 223:*\n\n*Jennifer forgot to close the front door when she got home. By the time she noticed her pet cat had disappeared. She walked around the neighborhood calling for the cat. She made flyers with her contact information. \\\nJennifer's cat came home the next day, acting very hungry. / She decided to watch her favorite comedy show on Netflix.*\n\n*Example 456:*\n\n*The sea was frightening during a storm. Marcus hated the way the boat rose up and crashed down. It made him sick to his stomach. He cowered in his cabin and shut his eyes but it didn't help. \\\nMarcus then vomited from sea sickness. / The worker thanked Mark for his patronage.*\n\nAs can be seen, the provided examples consistently demonstrate that the accurate suffix incorporates a name that offers a clue to the correctness of the suffix itself. Notice that the last appearance of the name in the prefix is positioned at a considerable distance from the suffix, (initial or second sentences of the prefix). We note that this property does not hold for all examples. This provides further evidence that TWIST models can capture long range dependencies better.\n\n**Regarding sBLIMP scores:** The AudioLM (Borsos et al.) model consists of a cascade of three Transformer models and uses a different speech tokenizer (which is not publicly available). We hypothesize the different tokenizer is the main factor that affects the sBLIMP results, which is why we do a systematic comparison using the same tokenizer and same model, with and without TWIST throughout the paper (e.g., Tables 2 & 6). We also highlight that our contribution is orthogonal to that of AudioLM, which could additionally use TWIST and potentially get a further performance improvement.\n\n**Regarding adding a randomly initialized model for Table 3:** Thank you for your comment, we will add that result to Table 3. \n\n**Regarding results in Table 2:** We use the OPT-350M, we will clarify. \n\n\n[1] Sicherman, Amitay, and Yossi Adi. \"Analysing discrete self supervised speech representation for spoken language modeling.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023."}, {"review_id": "I18ZMvM8fp", "reviewer": "Reviewer_5UoA", "summary": "The authors explain a method to improve the performance of a speech language model by reusing the weights of a language model trained on text. ", "questions": "I would like to ask the authors for additional comments regarding the following points and the weaknesses mentioned.\n\nThe audio samples provided by the authors are not sufficient. While the authors support their claims through various experiments in the manuscript, only some of the samples are included. While source code is the most clear evidence for the experiments and results, it is necessary to present other detailed experimental results when the source code is not available. Furthermore, in the field of speech synthesis, subjective evaluations hold more weight than objective evaluations, so it is essential to provide reviewers with the opportunity to listen to and evaluate the samples directly.\n\nThe method proposed by the authors is a form of transfer learning. However, when observing the training phase's perplexity (PPL) curve presented by the authors, it is difficult to consider the model as having converged at the 400k step where the authors claim to have ended the training. Therefore, it is not clear whether the performance difference claimed by the authors is simply due to the difference in convergence time caused by the initialization of the initial weights or the advantages obtained from the text-based language model (such as having trained on larger data, more diverse contexts, etc.).\n\nThe MMOS results are presented in a table. It is difficult to verify the confidence intervals (CI). If the CI cannot be verified, it should be indicated how many evaluators participated and with how many samples.\n", "limitations": "The limitations have been well described.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Using a well-trained text-based language model as an initial model seems like a good idea.\nIf the authors' claims can be generalized, it can be used as a good starting point for speech language models, which are relatively difficult to train.", "weaknesses": "The authors empirically discovered that they achieved better performance in the same training step, but there seems to be a lack of theoretical analysis. The process of analyzing, estimating, and confirming evidence for why such improvement occurs is missing. This simple empirical discovery without the analysis and validation of why it happens would have limited contribution to the conference-level community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that using text LMs as initialization for SpeechLMs is a good idea, as our empirical results demonstrate.\n\n**Regarding theoretical justification:** We agree that providing theoretical justification of why TWIST outperforms Cold-Init models is interesting. However, such theoretical justification is far from trivial. Furthermore, offering techniques (such as TWIST) that result in consistent and substantial performance improvements, even in the absence of a well-defined theoretical rationale, holds considerable potential to make a meaningful and beneficial contribution to the broader community, for instance, see [1].\n\nWe hypothesize the benefit of TWIST comes from providing a better prior initialization of the model weights for building SpeechLMs. This allows better capturing of long range dependencies in the input sequence (something that we do not observe in image data for instance. See Table 8 in the Appendix). In the attached rebuttal pdf file (Figure.1), we visualize the performance gap between TWIST and Cold-Init. model (350M params), using tStoryCloze, while slowly decreasing the context length. We observe that as the bigger the context, the bigger the gap between the models. We believe such findings will be valuable and of high-interest to the community, especially at the intersection between written and spoken language.\n\n**Regarding audio samples and source-code:** In the appendix material, we provide 15 samples from the proposed method. Per the reviewer's request, we provide an additional 450 samples (link was sent to the AC as per the NeurIPS guidelines), those samples were sent to the MMOS evaluation. As for source-code, we will open source both code and pretrained models. \nHowever, we would like to emphasize that our contribution in this work is not synthesis quality but improved spoken language modeling (i.e., improving what is said rather than how it sounds). Improving speech generation is orthogonal to our method, and future research could improve generation quality on top of TWIST. \n\n**Regarding model convergence:** We agree that if we had infinite compute and infinite data both models might converge to the same point. However, we note that we used the biggest publicly available speech data so far for training speechLMs (that we are aware of), so scaling further is far from trivial, and further, our setup is already quite expensive as it is (training the 1.3B model for 400k steps takes ~9 days on 32 GPUs). Hence, for fair comparison, we decided to limit each run to 400k steps (which is reasonable considering the change in the training loss). Under this setup we do observe better performance across all settings and faster convergence as can be seen in Figure 2a.\nFurthermore, we do train our LLaMA model until convergence, the graphs are presented in the PDF attached to the rebuttal (Figure 2). One can see that, even at full convergence, the TWIST model outperforms the Cold-Init model in terms of speech PPL (similarly to Figure 2b). \n\n**Regarding source of performance difference:** we believe the performance improvement of TWIST is due to both better initialization and the data used to initialize the model: (i) for better initialization: we do observe the model gets better loss values and overall scores (sWUGGY, sBLIMP, etc.) using TWIST from the beginning of the training, hence we conclude TWIST provide better prior weight initialization; (ii) for the type of data used for initialization: we do not observe performance boost when we use ImageGPT (using image data) as initialization for TWIST, which suggests that the type of data also plays an important role in the performance improvement obtained by TWIST. \n\n**Regarding CI and other details for MMOS:** we include the full table of all MMOS scores used for Figure 3 (we will include these results in the manuscript). As for details regarding the number of raters and number of examples, please see line 175, section 3.3 and line 267in section 4, human evaluation paragraph: we use 50 samples per dataset (total of 150 samples), while we enforce 10 raters per sample.\n\n|     Method    |   Mean   | CI@95 |   Mean   | CI@95 | Mean | CI@95 |\n|:-------------:|:--------:|:-----:|:--------:|:-----:|:----:|:-----:|\n|               | LS-clean |       | LS-other |       |  LL  |       |\n| Ref.          |   4.11   |  0.04 |   4.23   |  0.04 | 4.05 |  0.05 |\n| Resynth       |   3.95   |  0.07 |   3.87   |  0.06 | 3.96 |  0.06 |\n| no-TWIST 1.3B |   3.34   |  0.08 |   3.37   |  0.06 | 3.31 |  0.07 |\n| TWIST 1.3B    |   3.51   |  0.07 |   3.67   |  0.07 | 3.65 |  0.06 |\n| TWIST 7B      |   3.79   |  0.06 |   3.85   |  0.07 | 3.81 |  0.06 |\n\n\n\n[1] Peters, Matthew E., et al. \"Deep contextualized word representations.\" Proceedings of NAACL-HLT. 2018.\n"}]

Author rebuttal:
We thank all reviewers for their detailed responses and valuable feedback. We are happy the reviewers acknowledge the use of our method (TWIST) benefits the training and quality of speech language models (SpeechLMs). Reviewer 6BNW mentioned that **TWIST is effective, simple and easy to use**, while reviewer 5UoA agreed that **using a warm initialization from text-LMs is a good idea**. We appreciate reviewers qFrm, 6BNW and pWSQ for acknowledging that **our work is one of the first works to perform large scale experiments on SpeechLMs, both in terms of data scale and model sizes**. We thank reviewers qFrm pWSQ and 5UoA for emphasizing that **our paper includes extensive evaluations across different textual LMs, tokenizers, models and dataset size**. Furthermore, we thank reviewer qFrm for believing that our work is **“highly valuable and can provide direction for following research”** We are grateful for reviewers qFrm, 6BNW and pWSQ noting our contribution to the speechLM community by **presenting new benchmarks (tStoryCloze and sStoryCloze)**.

A specific response for each reviewer appears in the “personal” rebuttal section.

Meta-review: This paper examines the impact of pretraining with text on speech-based language models and shows how this straightforward model initialization can enhance performance, taking into account both automatic and human-generated metrics. The majority of reviewers expressed gratitude to the authors for their rebuttal, and all reviewers ultimately granted acceptance, from borderline accept to accept.
Decision: Accept (poster)