Scaling Speech-Text Pre-training with Synthetic Interleaved Data
Authors: Aohan Zeng
Zhengxiao Du
Mingdao Liu
Lei Zhang
shengmin jiang
Yuxiao Dong
Jie Tang
Venue: ICLR 2025 Poster (2025)
Keywords: large language models; speech language model; spoken chatbots
Primary area: foundation or frontier models, including LLMs

Abstract: Speech language models (SpeechLMs) accept speech input and produce speech output, allowing for more natural human-computer interaction compared to text-based large language models (LLMs).
Traditional approaches for developing SpeechLMs are constrained by the limited availability of unsupervised speech data and parallel speech-text data, which are significantly less abundant compared to text pre-training data, thereby limiting their scalability as LLMs.
We propose a novel approach to scaling speech-text pre-training by leveraging large-scale synthetic interleaved data derived from text corpora, eliminating the need for parallel speech-text datasets.
Our method efficiently constructs speech-text interleaved data by sampling text spans from existing text corpora and synthesizing corresponding speech spans using a text-to-token model, bypassing the need to generate actual speech.
We also employ a supervised speech tokenizer derived from an automatic speech recognition (ASR) model  by incorporating a vector-quantized bottleneck into the encoder. This supervised training approach results in discrete speech tokens with strong semantic preservation even at lower sampling rates (e.g. 12.5Hz), while still maintaining speech reconstruction quality.
Starting from a pre-trained language model and scaling our pre-training to 1 trillion tokens (with 600B synthetic interleaved speech-text data), we achieve state-of-the-art performance in both speech language modeling and spoken question answering, improving performance on spoken questions tasks from the previous SOTA of 13\% (Moshi) to 31\%.
We further demonstrate that by fine-tuning the pre-trained model with speech dialogue data, we can develop an end-to-end spoken chatbot that achieves competitive performance comparable to existing baselines in both conversational abilities and speech quality, even operating exclusively in the speech domain.

Published as a conference paper at ICLR 2025
SCALING SPEECH-TEXT PRE-TRAINING WITH SYN- THETIC INTERLEAVED DATA
Aohan Zeng*s .* Zhengxiao Duts+ Mingdao Liu Lei Zhang', Shengmin Jiang Yuxiao Dong', Jie Tang!
TTinngua University AZ
Code & Models:
con/ THUDM/GLM-4- GLM-G-MM-----
ABSTRACT
Speech language models (SpeechLMs) accept speech input and produce speech output, allowing for more natural human-computer interaction compared to text- based large language models (LLMs). Traditional approaches for developing SpeechLMs are constrained by the limited availability of unsupervised speech data and parallel speech-text data, which significantly less abundant than text pre-training data, thereby limiting their scalability as L.MSs. We propose a novel approach to scaling speech-text pre-training by leveraging large-scale synthetic in- terleaved data derived from text corpora, eliminating the need for parallel speech- datasets. Our method efficiently constructs speech-text interleaved data by sampling text spans from existing text corpora and synthesizing corresponding speech spans using text-to-token model, bypassing the need to generate actual speech. We also employ supervised speech tokenizer derived from an auto- matic speech recognition (ASR) model by incorporating vector-quantized bot- tleneck into the encoder. This supervised training approach results in discrete speech tokens with strong semantic preservation even at lower frame rates (c.g. 12.5Hz), while still maintaining speech reconstruction quality. Starting from pre-trained language model and scaling our pre-training to trillion tokens (with 600B synthetic interleaved speech-text data), we achieve state-of-the-art perfor- mance in speech language modeling and spoken question answering, improving performance on spoken questions tasks from the previous SOTA of 13% (Moshi) to 31%. We further demonstrate that by fine-tuning the pre-trained model with speech dialogue data, can develop an end-to-end spoken chatbot that achieves competitive performance comparable to existing baselines in both conversational abilities and speech quality, even operating exclusively the speech domain.
Accuracy on Spoken QA
31
COMMON
Large Text Corpus FineWeb - 15T tokens
CRAWL
25
Sample Speech Spans
19
Previous SOTA (Moshi)
All NLP tasks
generation tasks.
13
Synethize Speech Tokens
Text-to-Toker LM
Text Tokens
0B 100B 200B
600B
Interleaved Speech- Text Data
Synthetic Interleaved Data (4Tokens)
Figure 1: (Left) The performance on Spoken QA continuously improves as the amount of synthetic interleaved data increases, significantly surpassing the previous SOTA (Moshi). (Right) The pipeline for synthesizing interleaved speech-text data.
"Equal contribution. Email: {zah22, ex-du 1 1iud224 } @ma 1s. t.s.siing tai.nguaa...... edu.cn Work was done when MI., 1.Z. interned Zhipu.AT. Corresponding authors: YD and JT.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Large language models (LLMs) have significantly advanced natural language processing, demon- strating capabilities beyond traditional language tasks. Trained on vast internet corpora, they exhibit emergent abilities such as instruction following (Ouyang al. 2022). logical reasoning (Wei all 2022). and tool utilization (Schick et al. 2023). These advancements have enabled applications like interactive chatbots and personalized digital assistants. However, an ideal AI assistant should not rely solely text. Voice-based interaction offers a more natural and intuitive interface for human- AI interaction. Traditional voice-based systems combine Automatic Speech Recognition (ASR), LLMs, and Text-to-Speed (TTS) models in a cascading manner. This approach, however, suffers from information loss during ASR and TTS processes, limiting the ability to capture and express the rich nuances of speech.
Speech language models (SpeechLMM)) have emerged as a promising approach for building general- purpose voice assistants capable of processing speech input and output end-to-end. Several methods have been explored to construct SpeechLMss, Lakhotia L et al.) (2021) proposed unsupervised learning on speech corpora using discrete semantic tokens. |Hassid al. (2023) improved performance by initializing from pre-trained language models, while Moshi (Défossez et al. 2024) utilized large- scale training on private speech data. However, key challenge remains: the scarcity of speech data compared to text data. While text corpora like Fine (Penedo et al. (0024) offer 15 trillion high-
quality tokens, large unsupervised speech datasets like VoxPopuli (Wang et al. (02I) provide only 400K hours of speech, equivalent to 36 billion tokens at 25Hz. This disparity limits the scalability and performance of SpeechLMMs relative to LLMs.
A straightforward idea to address this limitation is to synthesize speech from text pre-training cor- pora using TTS models. However, this approach faces three major challenges. First, the lower infor- mation density of speech tokens leads to significant token expansion, drastically reducing training efficiency. Second, the process of synthesizing speech for large-scale text corpora is computation- ally expensive. Third, training pure speech data fails to align with the text modality, prevent- ing the model from leveraging the capabilities of existing L.MMs. Recently, Nguyen et al. (2024) has explored the of interleaved speech-text data for training. This approach improves align- ment between speech and text modalities, leading to better speech language modeling performance. However, their method requires parallel speech-text datasets construct the interleaved data, which significantly limits its scalability for large-scale pre-training.
In this paper, we propose a novel approach scaling speech-text pre-training by synthesizing inter- leaved speech-text data from text corpora. The interleaved data is generated by sampling text spans and converting them into speech tokens using text-to-token model. This efficient process bypasses the need to generate actual speech, enabling large-scale pre-training without relying on extensive speech datasets. Inspired by [Du et al.] (2024), we train the tokenizer in supervised manner using ASR models and datasets. Experiments with sampling rates from 6.25Hz to 50Hz revealed trade- offs between semantic retention, model efficiency, speech reconstruction quality, and pre-training performance. We selected 12.5Hz as the optimal rate for balancing these factors. To synthesize large-scale interleaved data, we used existing TTS datasets to train a text-to-token model, generat- ing 600B tokens of interleaved speech-text data and expanding the pre-trat pre-training to I trillion tokens. Finally, through fine-tuning on speech dialogue data, we developed an end-to-end spoken chatbot operating entirely in the speech domain. The main contributions of this paper are as follows:
We propose a novel method to effectively synthesize high-quality interleaved speech-text data from text corpora, addressing data limitation challenges in speech-text pre-training.
We design SpeechL.LM architecture featuring a 12.5Hz single-codehook speech tokenizer trained in a supervised manner, along with fow-matching based decoder for speech reconstruc- tion, achieving both robust semantic preservation and high-quality speech synthesis.
We scale our pre-training to 1 trillion tokens using synthesized interleaved speech-text data, sig- nificantly advancing capabilities in speech language modeling and spoken question answering.
We develop an end-to-ent spoken chatbot by fine-tuning pre-trained models with speech dialogue data, achieving competitive performance in conversational abilities and speech quality while op- erating exclusively in the speech domain.
2

---

Published as a conference paper at ICLR 2025
Wev
Speech Toxentzer
Tokenizo
Interteaved Speach Text
All NLP tasks are generation tasks.
Data
ITS
Text
Text-to- Toxen
Trai
Audio Token
Unsupervised Speech
O]] How you today?
Parallel Speech-Text (ASR, TTS)
Today is Menday
(a) Train Text Token Model using TTS data
Stage I: Large-scale Specch-Text Pro-training
[Oplicnal
Speech Resporisa
generation
Text Corpus
Step Ganerala Speech Tokania
Speech- Text Language Model
Speech Decoder
Text Taken LM
Tat-lo-Tokkn
That Toiera
Interlenteed Spaech-Teoc Data
Speech Input
(Optonal)
Model Output
4|||
All NLP books are generation lasks,
Speech
Stage II: Supervised Fine-tuing for Spoken Chatbot
(b) Construct Interleaved Speech. Text Data From Text Corpus
Input +I|[
Figure 2: Overview of our method. First we train text-to-token model to construct interleaved speech-text data. The speech language model's training contains two stages. In the stage the model pre-trained with synthetic speech-text interleaved data. In the stage 2 the the model is fine-tuned with a speech dialogue dataset.
2
OUR APPROACH
Current approaches for build SpeechLMs typically fall into two categories. One method (Fang et al., 2024 Détossez al., 2024) involves the language model for speech input but outputs embeddings for an additional non- voneeuvoreeressivve (NAR) model to generate speech tokens, which limits the mod- eling capacity and potentially reduces the upper bound of performance. The other method (Xie & Wu 2024) uses inconsistent audio representations for input and output, leading to misalignment between input and output modalitiy.
In this section, we present approach for developing end-to-end spoken chatbot using a unified speech-text modeling framework. Our method integrates a supervised speech tokenizer, a technique for synthesizing interleaved speech-text data, and two-stage training process to extend pre- trained language models to the speech domain. This comprehensive approach enables us to leverage large- scale text data for speech modeling, effectively aligning speech and text modalities within single model.
2.1 SPEECH TOKENIZATION
Supervised Speech Tokenizer Previous methods of discrete speech tokenizers either trained
with reconstraciio//auversaai objectives of speech waveform (Wang et al.] 2023 Chen et al. 2024)
or self- self-supervised learning automatically discovered acoustic units(Hsu et al. 2021). Following
recent advance in text-to-speech synthesis (Du et al. 2024), we train the discrete speech tokenizer by fine-tuning pretrained automatic speech recognition (ASR) model with additional pooling layer and a vector quantization layer in the middle of the encoder.
The pooling layer ID average pooling operator of window size k. which reduces the sampling rate to fraction of 1/k. The vector quantization layer approximates the continuous intermediate representations in the encoder with the closest vectors in the codebook. The selected indices in the codebook are used as the speech token indices. The codebook vectors are learned with exponen- tial moving average (EMA) and we add commitment loss to restrict the volume of continuous representations before quantization. To overcome codebook collapse, we apply the random restart trick (Dhariwal et 2020) to reset vectors whose mean usage falls below a certain threshold.
We also adapt the Whisper architecture to support streaming inference, which is important to re- duce latency for online speech interaction. We replace the convolution layer before the encoder Transformer with the causal convolution layer (van den Oord et al. 2016). We also replace the bidirectional attention in the encoder with block causal attention: the input audios are divided into segments of equal intervals and positions in segment and attend all the positions in the current segment and previous segments, but not positions in the following segments. Empirically we set the segment interval to 2 seconds (100 tokens before the average pooling). We find this can match
3

---

Published as a conference paper at ICLR 2025
Table 1: Speech Reconstruction Results. We evaluate semantic retention with Word Error Rate (WER) and reconstruction quality with VisQOL (Hines et al. 2015) and MOSNet (Lo et al. 2019) for different speech tokenizers across various frame rates. The baseline results are independently evaluated by us.
Frame Rate
LibriSpeech
Model
Bitrate Causal
WER VisQOL1
MOSNerf
Ground Truth RVQGAN
4.62
3.27
75Hz
1.50K
1.74
2.74
SemantiCodec
50Hz
1.30K
X
2.43
3.12
Speech Tokenizer
50Hz
1.50K
X
1.53
2.67
Speech Tokenizer
50Hl
4.00K
X
3.07
3.10
Spirit-Bass
25Hz
225.0
X
11.66
Spirit-Expreseee
38.5Hz
307.0
10.60
Moshi (Mimi)
12.5Hz
1.10K
8.36
2.82
2.89
50Hz
600
6.24
2.67
3.38
25Hz
300
6.80
2.60
3.33
Ours
12.5Hz
175
8.43
2.52
3.39
6.25Hz
100
14.41
2.34
3.24
the ASR performance of bidirectional attention. For more details about speech tokenizer training, please refer Appendix B.1
Speech Decoder Given discrete speech tokens, we synthesize speech through the speech decoder. We follow the decoder architecture Cosy Voice (Du et al. 2024), which consists of speech token encoder, a conditional flow matching model (Mehta et al. 2024). and HiFi-GAN vocoder (Kong cet al. 2020). The speech token encoder converts a sequence of discrete tokens into a sequence of contextual vectors with Transformer encoder. To facilitate the streaming synthesis of speech, adapt the speech token encoder to use the same block causal attention as the speech tokenizer. The flow matching model generates Mel spectrograms conditioned the speech token representations. Finally, the generated Mel spectrograms converted into the speech waveforms through the HiFi- GAN vocoder (Kong al. 2020). To train the speech decoder,
the unsupervised speech data described in Section 2 which consists various speakers. For more details about speech decoder training, please refer to Appendix B.2
We evaluate the content preservation and quality of generated speech by our speech decoder on LibriSpeech (Panayotov et al. 2015). The results are shown in Table []] We measure the content preservation by the Word Error Rate (WER) between the transcription with an ASR model provided in Nguyen al. (2023) and the true transcription. For speech quality, following Défossez et al. 2024), we compute VisQOL. (Hines et al., (015) and MOSNet (Lo et al. 2019) of the reconstructed speech. Our tokenizer performs well across various sampling rates, with the 12.5Hz variant offer- ing optimal balance between efficiency and quality. maintains high quality scores (MOSNet 3.39) and content preservation (WER 8.43) while significantly reducing bitrate (52.7). Our ablation study sampling rates during pre-training (Cf. Section 3.3.2) shows that lower rates improve per- formance, but gains plateau at 12.5Hz. Based on these results, we select the 12.5Hz variant for our subsequent experiments.
2.2 SYNTHESIZE INTERLEAVED SPEECH-TEN DATA
Interleaved speech-text data consists of tokens where speech and text sequences are interleaved at the word level. For example, a sequence might look like: "Today is <Speech..2 4> <Speech_5> <Speech.128 day". We hypothesize that training interleaved speech-text data encourages the model to learn an alignment between speech and text, facilitating the transfer of text-based knowl- edge to speech representations. Previous methods for creating interleaved speech-text data rely aligned speech-text parallel datasets (Nguyen et al. 2024), which are challenging to obtain. We pro- pose novel and efficient approach for constructing interleaved speech-text data using existing text

---

Published as a conference paper at ICLR 2025
datasets. The process consists of two main steps. First, train text-to-token model that directly converts text into corresponding speech tokens, eliminating the need to synthesize actual speech. This approach avoids the eπrrr accumulation associated with text-to-speech- www-s-peechhtooeeeee to-tkken pipelines and significantly improves synthesis efficiency, making it practical and scalable for large-scale data gen- eration. Next, we sample text spans from existing text datasets and transform them into speech spans using the trained tex(-to-token model. This enables the efficient and scalable creation of interleaved speech-text data without requiring aligned speech-text parallel datasets.
Text-to-Token Model We train a 1.5B text-to-token model based on standard transformer archi- tecture to convert text into corresponding speech tokens. While these tokens can be further synthe- sized into actual speech using our speech decoder, this step is unnecessary for constructing inter- leaved speech-tex. data. To prepare the training data, first tokenize speech from text-to-speech datasets into discrete speech tokens. The text-to-token model is then trained to predict these speech token sequences based on the input text. The training objective is to minimize the negative log- likelihood of the predicted speech tokens conditioned on the corresponding text input:
M, log P(ans [T; 01,<3;
f
where T; is the i-th input text, as is the j-th audio token the i-th sample, M; the length of the i-th speech token sequence, A represents the model parameters, and is the number of training samples.
Table 2:
Word Error Rate (WER) of the text- to-token model. The dataset labeled "Interleaved Data" refers to text and speech pairs generated during the construction interleaved speech-text data.
We use multi-speaker text-to-speech datasets
to train this model (see Appendix A.3] for de- tailed data distribution). We also include addi- tional high-quality text-speech pairs generated using the Cosy Voice (Du et al. 2024) model to improve accuracy for short incomplete text spans. The architecture and the training details about the text-to-token model training can be found in Appendix B.3 To speedup the speech
Dataset
Language
WER (%)
VCTK
English
3.20
English
11.62
Interleaved Data
Chinese
9.34
ken generation process, we deployed the model using the SGLang framework (Zheng et al. 2024). achieving generation speed of 25k tokens per second on a single H800 instance.
Interleaved Data Construction construct interleaved data from text document, we apply span corruption technique that randomly selects spans from the text sequence. Span lengths are drawn continuously from a Poisson distribution (A = 10) until the total length of selected spans reaches the predefined ratio 17 of the original text length. Next, text spans corresponding to the drawn lengths are randomly selected from the document. These spans are converted into speech interleaved speech-text sequence. The span
tokens using the text-to-token model, producing corruption ratio plays a crucial role in enabling effective knowledge transfer between the speech and text modalities, demonstrated in our ablation study (Cf. Section (.3.3]. Based on the find- ings of this study, we set 7) to 0.3 for optimal performance. We selected high quality text datasets (OpenCSG Commu- nity) for chinese) to apply the previously mentioned synthesis process, generating total of 600B tokens, with a 2:1 ratio of English to Chinese. Appendix D.3 provides samples of interleaved data constructed.
(FineWeb-Edde (Penedo al. 2024) for English and
We evaluated the performance of the text-to-token model on the VCTK (Yamagishi al. 2019)
dataset and the interleaved data using word error rate (WER) as the evaluation metric. To compute WER, we used our speech decoder to synthesize real speech from the speech tokens generated by the text-to-token model, and then transcribed using whisper-large The results are summarized in Table 2 For the standard VCTK dataset, we observed a lower WER of 3.20. However, the WER for speech spans generated from the text pre-training data was higher. We attribute this discrepancy primarily to some spans in the text pre- vee-training data being difficult to pronounce accurately.
(Radford et al. 2023).
5

---

Published as a conference paper at ICLR 2025
2.3 TRAINING
We initialize the speech language model with a pre-trained large language models' parameters to leverage its existing knowledge. In order to support speech processing. we extend the model's vocabulary and embedding space. Specifically, we augment the original language model vocabulary, Viang. with a discrete speech vocabulary, Vsperch, resulting in a combined vocabulary, V Vinng V "speach- This expansion allows the model to accept both text and speech tokens input and output. However, the ability to effectively understand and generate these tokens relies on subsequent training to align the text and speech modalities.
The training process consists of two stages. the first stage, the model pre-trained on synthetic interleaved data to learn the alignment between text and speech. In the second stage, fine-tuning is performed using speech dialogue data to enable the model to handle speech interactions.
2.3.1 SPEECH-TEXT TRAINING
To extend LLMs' capabilities in speech-text tasks, we introduce a speech-text pre-training stage. This stage enables the model process and represent discrete speech tokens. We utilize four data types, each serving a specific purpose:
Interleaved speech-text data: As described in Section 2.2 these datasets facilitate cross-modal knowledge transfer between text and speech.
Unsupervised text data: We use diverse corpus, similar to GLM et al. (2024), containing 10T tokens from webpages, Wikipedia, books, code, and research papers to maintain the model's language understanding.
Unsupervised speech data: Using the Emilia pipeline (He et al. of high-quality English and Chinese speech data, filtered by DNSMOS P.835 scores above 2.75, ensuring diverse and clean speech inputs.
2024 we collected 700k hours
Supervised speech-text data: This includes ASR and TTS data, teaching the model learn bidirectional relationships between speech and text.
Both text and speech are represented as discrete tokens. The model trained the next-token pre- diction objective with cross-entropy loss function. For text, speech, and interleaved data, the model trained to predict all the tokens. For supervised speech-text data, the model is only trained to predict tokens in the target parts (text in ASR data and speech in TTS data). We set text data at 30% of each batch to preserve language ability. Unsupervised speech and supervised speech-text data were trained for one epoch each, while interleaved data filled the remaining capacity, balancing language comprehension and speech processing. The detailed training data distribution can be found in Appendix A
2.3.2 SUPERVISED FINE-TUNING
Following speech-text pre-training, we fine-tune our model for speech dialogue tasks using a dataset derived from Magpie (Xu et al., 2024). We use GPT-4 to adapt the original text-based dia- logues for speech scenarios by filtering examples, shortening responses, and avoiding outputting text that cannot be read aloud. The detailed prompt for this adaptation process can be found in the Appendix F This curation process results in our SpeechDialog-Pee 90K triplets (Speechl, TextR, SpeechR), where Speechl is the speech instruction, TextR is the text response, and SpeechR is the corresponding speech response synthesized from TextR using MeloTTS (Zhao et al.] (2023). For train Hyper-paramatern we use batch size of 64, a sequence length of 4096 tokens, and train for 10 epochs with a learning rate decaying from 5e-5 to 5e-6. We use the AdamW optimizer.
dataset, which contains
2.4 INFERENCE
During inference, our framework supports two modes: speech-to-speech and text-guided speech generation. In speech-to-speede mode, the model directly processes speech input to generate speech output. Our streaming tokenizer converts the user's speech into discrete tokens, which the model then processes to generate output speech tokens. These output tokens are synthesized into contin- speech by block-wise decoder, operating 2-second blocks (25 tokens 12.5Hz). The

---

Published as a conference paper at ICLR 2025
Table 3: Pre-training Results. 'S'; speech input and output. 'S-T": speech input and text output. TT-+":: text input and speech outpur. Results for Spirit-LM are taken from Nouyen et al. (2024) and other results are from Défossez et al., (2024). We use @ indicate tasks and modalities not supported by the model, and to indicate scores that are not publicly available.
Speech Language Modeling -Topic-StoryClozz sStoryCloze
Spoken Question Anwsering
Model
Web Questions Llama Questions
TriviaQA
S T
S-/T
T->S S->T
S-T
S
S-TT
S
S-T
GSLM
66.6
0
0
53.3
0
1.5
0
4.0
0
AudioLM
0
0
0
0
2.3
0
7.0
0
TWIST
76.4
0
0
55.4
1.1
0
0.5
Spirit-LM
82.9
72.7
88.6
61.0
59.6
64.6
SpeechGPT
0
0
0
6.5
21.6
0
14.8
Spectron
6.1
21.9
0
Moshi
83.0
0
0
60.8
0
9.2
26.6
21.0
62.3
7.3
22.8
Ours (9B)
82.9
85.0
93.6
62.4
63.2
76.3
15.9
32.2
50.7
64.7
26.5
39.1
block-wise encoder computes representations which are then used by the conditional flow match- ing model to generate Mel spectrograms. Finally, these spectrograms are converted into continuous speech using the HiFi-GAN vocoder. For text-guided speech generation, given the speech input (Speechl), the model generates both text response (TextR) and the corresponding speech response (SpeechR) in a single forward pass. The text response is generated as an intermediate step, which then guides the production of the final speech output. The corresponding template for two modes can be found in appendix
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
base LLM for experi- ments. For ablation, we also use a smaller LLM with 1.5 billion parameters detailed in Table [3 Our speech-text pre-training stage processes a total of IT tokens, with a fixed sampling of 30% text data, one epoch each of unsupervised speech and supervised speech-text data, and the remainder consisting of interleaved data. Throughout the pre-training stage, we maintain sequence length of 8192 tokens and use learning rate that linearly decays from 6e-5 to 6e-6. For the fine-tuning phase, we use a batch size of 64, a sequence length of 4096 tokens, and train for 10 epochs on the fine-tuning dataset with learning rate decaying from 5e-5 to 5e-6. We use the AdamW optimizer for both pre-training and fine-tuning stages.
Configuration We employ
(GLM al. 2024)
Baselines For pre- pre-trained trained models, We compare our method with GSLM (Lakhotia et al. 2021 AudioLM (Borsos et al. 2023). TWIST (Hassid al. (023). Spirit-LM (Nguyen al. 2024
SpeechGPT (Zhang et al., 2023) 3, Spectron (Nachmani et al. 2024), and Moshi (Défossez et al. text pretrained language model. Note that Moshi is pretrained on a speech collection of 7 million hours, an order of magnitude larger than our unsupervised speech data. For chat, we only compare end-to-end spoken chatbots supporting speech as both input and output, we choose SpeechGPT (Zhang et al. 2023). Llama- Omni (Fang et al. 2024). Mini-Omin (Xie & Wu 2024) and Moshi (Défossez et al. 2024). Moshi
2024]. Except GSLM and AudioLM, other baselines based
is fine-tnee for full duplex conversations and each conversation must begin with a greeting from the model. Therefore, we wait 3 seconds for the greeting to end before asking the speech query. For Mini-Omni, we use the default AT mode for evaluation.
Speech Language Modeling We first evaluate the pretrained model's ability to model speech by the accuracy of selecting the correct continuation of given context according to the predicted like- lihood. We consider three different settings: from speech context speech continuation (denoted as 'S'). from text context to speech continuation (denoted as 'T++'), and from speech context to text continuation (denoted "'")"". We use two datasets proposed by Hassid et al. (2023). Spoken

---

more Published as a conference paper at ICLR 2025
Table Evaluation results for end-to-end spoken chatbots. All bascline results in this table were obtained through our own evaluation.
w/o Text
Content Quality
Speech Quality
General QAt
Knowledget
UTMOS1
ASR-WER_
SPeechGPT
X
1.40
2.20
3.86
66.57
Mini-Omni
X
2.44
1.10
3.17
25.28
Llama-Ommi
X
3.50
3.90
3.92
9.18
Moshi
X
2.42
3.60
3.90
7.95
9B + Text-guided
X
3.69
4.70
4.33
7.83
No Interleaving
X
2.48
1.00
4.31
10.34
No Pre-training
X
2.20
0.90
4.32
6.11
9B
V
3.18
3.20
4.33
0
No Interlcaving
1.21
0.01
4.33
0
No Pre-training
V
1.10
0.00
4.32
0
StoryCloze and Spoken TopicStoryCloze. The bascline results are taken from Nguyen et al. (0024): Défossez et al. (2024).
Spoken Question Answering Similar closed-book question answering in NLP, spoken ques- tion answering requires the speech-language model to answer spoken questions about broad factual knowledge without access to external knowledge. We consider two settings for the model: from spoken questions to spoken answers (denoted as 'S'), and from spoken questions to textual answers (denoted as 'S->T"). We evaluate our model on datasets in Défossez et al, (2024): Web Ques- tions (Berant et al., 2013), Llama Questions (Nachmani et al. 2024), and TriviaQA (Joshi et al.) 2017)
The baseline results are taken from Défossez et al. (2024).
Evaluating Spoken Chatbots To evaluate the spoken chatbot's capabilities we select two aspects: general question answering and knowledge. For general question answering, we utilized prompts from AlpacaEval'ss (Li al. 2023b) helpful belpful_base _bas and vi cuna categories, which suitable for voice interactions. The knowledge assessment drew 100 questions from Web Questions, Llama Questions, and TriviaQA datasets. The generated speech was transcribed into text using chisper-larger 3. and GPT-4 was used to score the responses on scale of 1 to 10, with the detailed prompt provided Appendix G Additionally, we measured ASR- WER assess the alignment between generated speech and text, as well as UTMOS (Saeki et al. 2022 to evaluate overall speech quality following Fang et (2024).
3.2
MAIN RESULTS
The evaluation results for the pretrained model are shown in Table On speech language modeling. our method outperforms baselines on all the tasks except the 'S' setting of spoken Topic-StoryCloze on which our model achieves comparable accuracy to SpiRit-LM and Moshi. Compared with SpiRit- LM, our method achieves significant improvements on the 'T-S' and 'S-T" setting, indicating that our synthetic interleaved data effectively aligns text and speech modalities. On spoken question answering, our method significant outperforms all the baselines on both the "S' and 'S-TT" setting of three datasets. The improvements are especially substantial on the speech-to-speech setting. On Llama Questions, our method considerably reduces the previous gap between the speech-to-speech and speech-to-text settings, indicating that effectively transfers the knowledge the text modality to the speech modality. Overall, method achieves better performance than the best bascline Moshi, with only a tenth of Moshi's natural speech data.
Table shows the evaluation results for spoken chatbots. Our 9B text-guided model outperforms all baseline models in general question-ansee answering and knowledge-bate based tasks. It also achieves better results in speech quality evaluation compared to others. Notably, even without text guidance, the 9B model still performs comparably with text-guided basclines, highlighting our method's 'ss effectiveness in aligning text and speech modalities.
8

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study on interleaved data scaling and pre-training data composition. 'S':
speech input and output. 'S-+T" speech input and text output. "T-+S": text input and speech output. "Web Q." stands for Web Questions. "Llama Q." stands for Llama Questions.
Speech Language Modeling
Spoken Question Anwsering
Model
Stopic-StoryCloze
sStoryCloze T-+S
Web Q.
Llama Q.
Trivia QA
S
T->S S-+T
S->T
S-+T
S
S-+
S
S-+
9B (600B Interleave)
82.9
85.0
93.6
62.4
63.2
76.3
15.9
32.2
50.7
64.7
26.5
39.1
-oo Interleaving
72.8
53.3
53.3
51.7
51.4
53.7
0.1
0.3
2.3
2.3
0.2
0.5
-100B Interleave
80.9
83.6
93.3
59.4
61.3
73.4
9.3
25.4
37.0
60.0
11.7
26.9
-00BB Interleave
82.1
84.7
93.2
61.5
62.6
76.0
13.3
29.7
44.0
63.0
18.7
31.3
1.5B
77.5
81.4
90.1
55.4
58.6
64.0
5.4
17.6
18.3
42.7
4.6
15.6
No Interleaving
74.6
55.3
51.3
51.7
53.4
52.8
0.0
0.1
1.3
4.3
0.0
0.2
No Speech
78.0
82.1
89.5
55.9
59.3
64.4
4.9
17.3
17.7
38.3
4.6
15.5
No Text
78.5
83.2
89.0
54.8
58.3
63.4
6.3
17.4
20.3
42.3
7.0
16.3
No ASR & TTS
78.7
81.4
89.5
56.7
59.5
68.5
6.1
17.0
23.0
41.0
5.4
14.0
45.0
Average Accuracy
46
4.0
44
3.5
42
Average
35.0
Average
2.0
32.5
1.5
w/ text-guidance
36
30.0
w/o text-guidanc
34
0.5
50Hz
25Hz
12,5Hz
6.25Hz
0.2
0.4
0.6
0.8
1.0
08
100B
2008
600B
Sampling Rate
Span Corruption Ratio
Interleaved Tokens
(a) Sampling Rate
(b) Span Corruption Ratio
(c) Interleaved Data Tokens
Figure 3: (a) Sampling rate average accuracy. (b) Span corruption ratio average accuracy. The
accuracy is averaged over datasets of speech language modeling and spoken question answering. (c) Interleaved data tokens average performance after supervised fine-tuning.
3.3
ABLATION STUDY
3.3.1
DATA SCALING AND COMPOSITION
Our pre-training corpus consists of text, speech data, speech-text interleaved data, and speech-text parallel data (from ASR and TTS tasks). We study the effects of data scaling and composition. First, evaluate how scaling interleaved data impacts model performance. We train the 9B model with interleaved data sizes of O, 100B, and 200B tokens, keeping other parts of the pre-training corpus unchanged. Table [ compares these results with the best model trained on 600B interleaved data. Without interleaved data, the model performs poorly, but interleaved data scales up. performance improves consistently. This demonstrates the effectiveness of scaling synthetic speech-text inter- leaved data. Figure further shows that increasing interleaved data improves chatbot performance after supervised fine-tuning, both with and without text guidance. Next, we analyze the contribu- tions of different parts of the pretraining corpus using 1.5B model. Results in Table 5 show that removing synthetic interleaved data significantly degrades performance. Removing unsupervised speech data slightly reduces spoken question answering accuracy, while removing text speech- text parallel data improves performance on most benchmarks, likely due to capacity competition among modalities in smaller models. For the 9B models, retain all data types they represent essential tasks for downstream applications, and larger models alleviate this competition.
3.3.2 FRAME RATE
The frame rate of the speech tokenizer refers to the number of speech tokens generated per second. Hassid et al. (2023) observed that reducing HuBERT's frame rate from 50Hz to 25Hz improved per- formance on speech language modeling tasks. We trained 1.5B models with tokenizers at different frame rates using the same number of training tokens, excluding ASR and TTS datasets for simplic-

---

Published as a conference paper at ICLR 2025
ity, and analyzed the relationship between sampling rate and accuracy (Figure 3a The results show that lower frame rates improve average accuracy. We hypothesize two reasons: (1) lower sampling rates allow the model to process more speech data within the same training token budget, and (2) shorter token sequences for the same audio reduce modeling difficulty. We selected a 12.5Hz frame rate for our main model, as the 6.25Hz tokenizer showed trade-off where speech information loss outweighed accuracy gains.
3.3.3 SPAN CORRUPTION RATIO
The span corruption ratio decides the proportions of text and speech tokens in interleaved samples. With extreme corruption ratios close or the interleaved samples are dominated by text or speech tokens. To study the effect of the ratio and determine the best value, we train multiple 1 1.5B models with interleaved data from different span corruption ratios and plot the results in Figure 3b Overall, we find that the corruption ratios from 0.2 to 0.4 works well. Larger or smaller ratios result a significant degradation performance. Based on the results, we select 0.3 as the corruption ratio for our main model.
RELATED WORK
Speech Tokenization Speech tokenizers, which transform audio clip into discrete tokens, can be categorized into two directions. The neural acoustic codecs (Zeghidour et al.) [2022] Défossez ctt al. 2023 Kumar et al. 2023 Ji et al.) (024) target at reconstructing high-quality audio at low bitrates. The semantic tokens (Hsu et al. 2021 Chung al., 202T) are extracted from speech representations learned with self-supervised learning on speech data. Speechtokenizer (Zhang et al.) 2024) unifies semantic and acoustic tokens different residual vector quantization (RVQ) layers, but it also suffers from expansion of sequence length. Cosyvoice (Du et al., (024) proposes the supervised semantic tokenizer derived from speech recognition model, and successfully apply the tokenizer to text-to-speech synthesis. The application of the tokenizer on speech language modeling not explored.
Speech- Text Text Pre-training GSLM (Lakhotia et al. 2021) proposes the generative spoken language modeling task, which trains the next-token-predictinn objective on discrete semantic tokens pro- duced by self-supervised learning. AudioLM (Borsos et al. 2023) proposes hybrid tokenization scheme that combines semantic tokens with acoustic tokens from a neural audio codec ZZegidourr et al. 2022). TWIST (Hassid et al. (223) trains the speech language model using a warm-start from a pretrained text language model, specifically OPT (Zhang et al. 2022). Moshi (Défossez et al.) 2024) scales up the size of natural speech data in TWIST to 7 million hours. Spirit-LM (Nguyen ct al. 2024) further extends TWIST by adding speech text interleaving data curated from speech-text parallel corpus. However, the scarcity of parallel corpus restricts the scale of interleaving data.
End-to-End Spoken Chatbots Early works in speech- speech-to-speechh models mainly focus on processing tasks like speech translation (Chen et al., 2021b Ao et al. 2022). Since success of ChatGPT in
text-based chatbots, many works have explored methods to develop speech- chatbots that can understand and respond in speech. Speechgpt (Zhang et al., 2023) proposes combine existing large language models (LLM) with discrete speech representations to obtain speech conversational abilities. Moshi (Défossez al. (2024) proposes full-duplex spoken dialogue framework based on their pretrained speech language model. Llama-omni (Fang et al. (0224) and Mini-omni (Xic)
Wu 2024) both propose light-weight alignment methods that transform an open language model into spoken chatbots.
CONCLUSION
This paper introduced novel approach for scaling speech-text pre-training using supervised se- mantic tokens and synthetic interleaved data. By employing supervised speech tokenizer and generating 600B tokens of interleaved data, we scaled our speech pre-training to trillion tokens, achieving state-of-the-art performance in speech language modeling and spoken question answer- ing tasks. We also developed an end-to-end spoken chatbot by fine-tuning our pre-trained model, demonstrating competitive performance in both conversational abilities and speech quality. Future work could explore more efficient training techniques, investigate larger model sizes, and expand multilingual capabilities.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
This research explores large-scale speech-text pre-training with interleaved speech-text data syn- thesized from text corpora, significantly improving speech modeling capabilities for LLs. Furthermore, by fine-tuning the pre-trained model on speech dialogue data, we develop an end-to- end spoken chatbot with competitive performance in both conversational abilities and speech quality. While our method advances end-to and-to-end spoken interaction, we analyze potential risks and provide possible ways mitigate those issues.
Voice Phishing Voice Phishing is deceptive practice where malicious actors use synthetic voices to imitate trusted individuals for fraudulent purposes. As speech synthesis technology advances, the risk of such abuse increases. In our approach, while the SpeechLLM generates speech tokens that encode semantic information, the speech decoder is responsible for synthesizing the actual speech waveform, with voice identity primarily determined by its training. The decoder's training follows a two-stage process: initial pre-training on multi-speaker data followed by fine-tuning on single-speaker data, resulting in model that generates speech with one fixed voice identity(CI. Appendix B.2). This design significantly reduces the potential for misuse in voice phishing, as the model cannot flexibly adapt to mimic multiple identities, a critical requirement for such fraudulent activities.
Scamming Beyond Voice Phishing, there cxists a risk of Scamming, where the model could po- tentially be exploited generate deceptive fraudulent content for broader malicious purposes. Unlike traditional text-to-speech (TTS) models, which can generate speech from any text input, our model is end-to-end spoken chatbot. This means it takes speech input from the duces speech output as a response, without the ability to generate speech from arbitrary text. This design greatly lowers the risk of misuse for scamming purposes, the model is built for interac- tive speech-basel conversations, not unrestricted audio generation. Additionally, our pre-trained model is fine-tuned on synthesized speech dialogues derived from ethically aligned LLMs (Cf. Sec- tion 2.3.2). This approach inherits the ethical constraints and safety measures already established in these L.MMs. The inherited safeguards enable our model exhibit similar ethical behavior patterns, such as refusing to engage in fraudulent or harmful content generation.
and pro-
Potential Risks of Tunnng and Mitigation Despite the safeguards embedded in our model's design, recognize that fine-tuning could potentially undermine these protections. For instance, fine-tuning the speech decoder on diverse speaker data could enable the model to generate speech with multiple voice identities, therehy circumventing the single-voice limitation and increasing the risk of voice phishing. Similarly, fine-tuning the SpeechLLM on unaligned or malicious data could weaken its ethical alignment, allowing it to generate fraudulent harmful content. To mitigate these risks, we advocate for strict access controls on fine-tuning capabilities, ensuring that only authorized users with ethical oversight can modify the model. Additionally, we recommend the development of robust detection mechanisms for synthetic speech, such as watermarking or forensic analysis tools, to identify and prevent misuse. Ethical guidelines should rigorously enforced during any fine-tuning process, and transparency in model deployment should be prioritized to maintain ac- countability. By acknowledging these risks and implementing proactive measures, we aim to ensure that our LL is used responsibly and ethically.
ACKNOWLEDGEMENN
We would like to thank the anonymous reviewers for their suggestions in refining this work. This work is supported by NSFC 62425601, 62276148, and New Comnestonee Science Foundation through the XPLORER PRIZE, and research fund from Tsinghua Guoqiang Institute.
11

---

Published as a conference paper at ICLR 2025
REFERENCES
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko. Qing Li, Yu Zhang, et al. Speecht5: Unified-modke encoder-decoder pre-training for spoken language processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3723-5738, 2022.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. Common voice: A speech corpus. In Proceedings of The 12th Language Resources and Eval- uation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pp. 4218-4222 European Language Resources Association, 2020.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Wash- ington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1533 133--144. ACL, 2013.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matthew Sharifi, Dominík Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghi- dour. Audiolm: A language modeling approach to audio generation. IEEE ACM Trans. Audio Speech Lang. Process., 31:2523 22033,25332222 2023.
Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. AISHELL-1: an open-source mandarin speech corpus and speech recognition baseline. In 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech 1/O Systems and Assessment, O-COCOSDA 2017, Seoul, South Korea, November 1-3, 2017, pp. 1-5. IEEE, 2017.
Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuai- jiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang. Zhao You, and Zhiyong Yan. Gigaspeech: An evolving, multi-domain ASR corpus with 10, 000 hours of transcribed audio. In 22nd Annual Conference of the International Speech Communication Association, Interspeech 2021, Brno, Czechia, August 30- September 3, 2021, pp. 3670-3674. ISCA, 2021a.
Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. VALL-E 2: Neural codec language models are human parity zero-shot text to speech synthesizers. CoRR, abss/2406.05370. 2024.
Yi-Chen Chen, Po-Han Chi, Shu-wen Yang, Kai-Wei Chang, Jheng-hao Lin, Sung-Feng Huang, Da-Rong Liu, Chi-Liang Liu, Cheng- Kuang Lee, and Hung-yi Lee. Speechnet: A universal modularized model for speech processing tasks. arXiv preprint arXiv:2105.03070. 2021b.
Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. w2y-bert: Combining contrastive learning and masked language modeling for self- supervised speech pre-training. In IEEE Automatic Speech Recognition and Understanding Work- shop, ASRU 2021, Cartagena, Colombia, December 13-17, 2021, pp.
IEEE, 2021.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Trans. Mach. Learn Res., 2023, 2023.
Alexandre Défossez, Laurent Mazaré, Manu Orsini. Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dia- logue. Technical report, Kyutai, September 2024. URL http:httt:///tttt
wrw./ pdf
Prafulla Dhariwal, Heewoo Jun, Christine Bhyne,JJng Wook Kim, Alec Radford, and llya Sutskever. Jukebox: generative model for music. CoRR, abs/2005. 003411, 2020.
Zhihao Du, Qian Chen, Shiliang Zhang. Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijie Yan. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens, 2024. URL :ttttpss//aaaaee ://arxiv. brg/abss/2477 /00p 05407
12

---

Published as a conference paper at ICLR 2025
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-ommi: Seamless speech interaction with large language models, 2024. URL https:: /ttp///aavvvv//. org/
abs/2409 06666
Zhifu Gao. Zerui Li, Jiaming Wang, Haoneng Luo. Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, and Shiliang Zhang. Funasr: A fundamental end-to-end speech recognition toolkit. In 24th Annual Conference of the International Speech Communication Association, Interspeech 2023, Dublin, Ireland, August 20-24, 2023, pp. 1593-1597, ISCA, 2023.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang. Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang. Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang. Jingyu Sun, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm- 1lm- to glm-4 all tools, 2024. URL https: /arpii///axxvv.oo org/
abs/ / 2406. 12793
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Défossez, Gabriel Synnaeve, Emmanuel Dupoux, Roy Schwartz, and Yossi Adi. Tex- tually pretrained speech language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Or- leans. LA, USA, December 10 16, 2023, 2023.
Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang. Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. CoRR, abs/2407.05361. 2024.
Andrew Hines, Jan Skoglund, Anil Kokaram, and Naomi Harte. Visqol: an objective speech quality model. EURASIP Journal on Audio, Speech, and Music Processing. 2015 (13):1-18, 2015.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEE ACM Trans. Audio Speech Lang. Process., 29:3451 134600, 2021.
Shengpeng Ji, Ziyue Jiang, Xize Cheng, Yifu Chen, Minghui Fang, Jialong Zuo. Qian Yang, Ruiqi Li, Ziang Zhang, Xiaoda Yang. Rongjie Huang, Yidi Jiang, Qian Chen, Siqi Zheng, Wen Wang. and Zhou Zhao. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. CoRR, abs/2408. 16532, 2024.
Mandar Joshi, Funsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 1601-1611. Association for Computational Linguistics, 2017.
J. Kahn, M. Riviere, W. Zheng, E. Kharitonov, Q. Xu, P.F. Mazare, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, May 2020. doi: 10.1 10.1099ccap4477720000999222
URL htttp://dk dttp:///xxx.....oo.//oooo org 10. 1
ICASSP40776. 2020. 9052942
Jungil Kong. Jachyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversar- ial networks for efficient and high fidelity speech synthesis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural In- formation Processing Systems. volume 33, pp. 77022-17033. Curran Associates, Inc., 2020. URL huRliiicccss/pprreeeinnneeeeee #1ee//58738807766644422286161163
httpp//pppppeeeeeppeeeeeeeeeeeee ffless/papee//22///
13

---

Published as a conference paper at ICLR 2025
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved RVQGAN. Advancee in Neural Information Process- ing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, IA, USA, December 10 16, 2023, 2023.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336-1354, 2021.
Xinjian Li, Shinnosuke Takamichi, Takaaki Saeki, William Chen, Sayaka Shiota, and Shinji Watan- abe. Yodas: Youtube-oriented dataset for audio and speech. In IEEE Automatic Speech Recog- nition and Understanding Workshop, ASRU 2023, Taipei, Taiwan, December 16-20, 2023, pp. 1-8. IEEE, 2023a. doi: 10.1 AAL109//ASRUS776442233188188 10.1109/ASRTTT 644 2023. 10389689
URL https: /ocoi//o...o org/
Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of mnstruction-followii models. httpt::////t /i thub. cooom/ttts--llbb. lttpalaaabapppp_aaaaaaa__aa
paca_eval 2023b.
Chen-Chou Lo,, Szu-Wer Fu, Wen-Chin Huang, Xin Wang, Junichi Yamagishi, Yu Tsao, and Hsin- Min Wang. Mosnet: Deep learning- based objective assessment for voice conversion. In Gernot Kubin and Zdravko Kacic (eds.), 20th Annual Conference of the International Speech Communi- cation Association, Interspeech 2019, Gruz, Austria, September 15-19, 2019, pp. 1541-1545. ISCA, 2019. doi:
URL https: //tp://oool.........
21437/Interspan
2019-2003
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https:
//aaxivv.rg/aba//ITT 1111000101
Shivam Mehta, Ruibo Tu, Jonas Beskow, Éva Székely, and Gustav Eje Henter. Matcha- TTS: A fast TTS architecture with conditional flow matching. In Proc. ICASSP, 2024.
Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, R. J. Akerry-Ryan, and Michelle Tadmor Ramanovich. Spoken ques- tion answering and speech continuation using spectrogram-povee poweeed LLM. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OvenReview.net, 2024.
Tu Anh Nguyen, Wei-Ning Hsu, Antony D' Avirro, Bowen Shi, Itai Gat, Maryam Fazel-Zarandi,, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, Felix Kreuk, Yossi Adi, and Emmanuel Dupoux. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. In Naomi Harte, Julie Berndsen, and Gareth Jones (eds.), 24th Annual Conference of the International Speech Communication Association, Interspeech 2023, Dublin, Ireland, August 20- 24, 2023, pp. 4823 4827. ISCA, 2023.
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Gabriel Synnaeve, Juan Pino, Benoit Sagot, and Emmanuel Dupoux. Spirit-Im: Interleaved spoken and written language model, 2024. URL lhttps://aaxiitii org/ aaaaabs/2245 2202244220000 05755
OpenCSG Community. Chinese fineweh edu dataset. Settasets/ppencag//cee chinese/ccieeeeeeeeee www.eeseefineeeeeeee.ee fineweb-edul
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ast corpus based on public domain audio books. In 2015 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 2206-5210, 2015. doi: 10. 10.1109/
71 7178964.
14

---

Published as a conference paper at ICLR 2025
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URLhtta:: org. http//aas/2/04 17557
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. MLS: A large-scale multilingual dataset for speech research. In 2 Anmual Conference of the Interna- tional Speech Communication Association, Interspeech 2020, Virtual Event, Shanghai, China, October 25-29, 2020, pp. 2757-2761 ISCA. 2020.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 28492-28518, PMLR, 2023.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022, 2022. URL https:
//aaxxii.rrg/aas//00000 02152
Timo Schick, Jane Dwivedi-Yit Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In The 9th ISCA Speech Synthesis Workshop, SSW 2016, Sunnyvale, CA, USA, September 13-15, 2016, pp. 125. ISCA, 2016.
Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation, 2021. URL
https::/arxiie org/ aww.g//////// 2tt.//211..100001 00390
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers. CoRR, lbs/2301.02111,, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824- 24837, 2022.
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in 16722
streaming, 2024.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing. 2024. URL http:::t/////..tt. /ttpss////aiivi.//aa org/ abs/ /twp////22..22 08464
Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al. Cstr vetk corpus: English multi- speaker corpus for estr voice cloning toolkit (version 0.92). University of Edinburgh. The Centre for Speech Technology Research (CSTR), pp. 271-350, 2019.
Zhuoyuan Yao, Di Wu, Xiong Wang, Binbin Zhang, Fan Yu, Chao Yang, Zhendong Peng, Xiaoyu Chen, Lei Xie, and Xin Lei. Wenet: Production oriented streaming and non-streaming end-to- end speech recognition toolkit. In 22nd Annual Conference of the International Speech Com- munication Association, Interspeech 2021, Brno, Czechia, August 30 - September 3, 2021, pp- 4054-4058. ISCA, 2021.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE ACM Trans. Audio Speech Lang. Process., 30: 495-507, 2022. doi: 00.1099ASLLP2202111299999 URL https: : wttp///c..co/ wtt.:i..og/1111110
TASLP 2021 3129994
15

---

Published as a conference paper at ICLR 2025
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abil- ities, 2023.
abs 300./20000.0000
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Out, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. OPT: open pre-trained transformer language models. CoRR, abs/2205.01068, 2022.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech language models. The Twelfth International Conference on Learn- ing Representations. ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Wenliang Zhao, Xumin Yu, and Zengyi Qin. Melotts: High-quality multi-lingual multi-accem text- to-speech, 2023. Uttsttss:////thutt http://wwwuu.uccoomemeleelllllle http:t/sse/..ttttt....///////aaa -i/MeeloTs
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024. URL /ttps://arxiv. orc/abbs/312.07100
16

---

Published as a conference paper at ICLR 2025
A
TRAINING DATASETS
A.1
INTERLEAVED SPEECH-TEXT DATA
Table Statistics on interleaved pre-training data. Tokens measured in billion.
Corruption Ratio
Text Tokens
Speech Tokens
Speech Ratio
Total Tokens
Dataset
Tokenizer
Text-60k Speech-50Hz
0.30
56.21
343.79
0.86
400
Text-60k
0.30
98.78
301.22
0.75
400
Speech-25Hz
0.10
282.82
117.18
0.29
400
Fineweb-Edu
0.20
209.05
190.95
0.48
400
0.30
158.43
241.57
0.60
400
Text-60k
0.40
121.54
278.46
0.70
400
Speech- 12.5Hz
0.50
93.48
306.52
0.77
400
0.75
46.15
353.85
0.88
400
1.00
0.10
399.90
1.00
400
Text-60k
0.30
226.50
173.50
0.43
400
Speech-6.25Hz 6.25Hz
Text-150k
0.30
150.51
249.49
0.62
400
Mpeech-12.5HL.
Text-60k
0.30
78.80
121.20
0.61
200
Seines-Finnewe-BEEE
Speech- 12.5Hz
Text-1500 150k
0.30
77.59
122.41
0.61
200
Speech- 12.5Hz
A.2 SUPERVISED SPEECH DATA
Table 7: TTS training data of text-to-token LLM with 12.5Hz speech tokenizer. Tokens measured in billions.
Language
Speech Hours
Speech Tokens
Text Tokens
Total Tokens
Chinese
94,980 42,726
4.27B 1.92B
1.18B 0.56B
5.45B
English
2.49B
Table 8: Training data breakdown of ASR task for 12.5Hz speech tokenizer. billion.
Tokens measured in
Language
Speech Hours
Speech Tokens
Text Tokens
Total Tokens
Chinese English
21,624 68,733
0.97B 3.09B
0.42B 1.06B
1.39B
4.16B
17

---

Published as a conference paper at ICLR 2025
A.3 TEXT-TO-TOKEN MODEL DATA
Table 9: Training data of text-to-token LLM with 12.5Hz speech tokenizer synthesized by Cosy Voice model on span-corruption data.
Language Speech Hours
Speech Tokens
Text Tokens
Total Tokens
Chinese 9,895 11,074
0.45B 0.50B
0.13B 0.16B
0,58B
English
0.65B
The data used to train the text-to-token model consists of two parts: the TTS data presented in Table and the synthesized data of incomplete text spans the generated by Cosy Voice, as detailed in Table
B TRAINING DETAILS
B. SPEECH TOKENIZER
We fine-tune the vector- quatt--uaatized Whisper model with collection of ASR datasets, including Lib- riSpeech (Panayotov et al. 20115). GigaSpeech (Chen et al. 202Ia), MLS-Eng (Pratap et al. (020). Wenet Yao al. (021), Common Voice (Ardila al.] (2020), AISHELL-1 (Bu al. (201)). and proprietary Chinese ASR dataset of 10k hours. We also include the unsupervised speech data with pseudo labels generated by Whisper (Radford al. 2023) for English and FunASR (Gao al. 2023 for Chinese.
All of speech tokenizers in Tar fine-tuned from wh sper- rge-v3 3 for 2 epochs with batch size 4096 and learning rate lc-5. The ratio of supervised samples to pseudo-labeled samples 1:3. The codebook vectors updated with exponential moving average with decay coefficient 0.99 and the commitment loss coefficient is 10.0. To reduce the information loss of average pooling, we increase the codebook size the sampling rate decreases.
Table 10: ASR results of Whisper models with pooling layers and vector quantization. The Lib- riSpeech (English) is measured with word-error-rate (WER) and AISHELL-I (Chinese) is measured with character-error-raae (CER). The first model is the original whisper large-v3 without fine- tuning.
Model
Sampling
VQ
Finetuned
LibriSpeech test-clean
AISHELL-I
Rate
Codebook
test-other
test
whisper-large-a
v3
50Hz
No
2.50
4.53
9.31
whisper-large-v3
50Hz
4,096
Yes
1.85
3.78
2,70
whisper-large-v3
25Hz
4,096
Yes
1.94
4.16
2.86
laisper-
12.5Hz
16,384
Yes
2.10
4.90
3.02
lhisper-
6.25Hz
65,536
Yes
2.48
6.34
3.33
During training of speech tokenizers, measure the semantic information with accuracy of automatic speech recognition (ASR) datasets. We evaluate the finetuned Whisper on Lib- riSpeech (Panayotov et al. 2015) and AISHELL-I (Bu et al. 2017), along with the original Whisper model. The results are shown Table TO, Overall all the tokenizers reserve enough semantic infor- mation achieve accurate ASR performance.
18

---

Published as a conference paper at ICLR 2025
Table 11: Ablation study on block sizes in the block causal attention of speech tokenizers.
Model
Attention
Block Size
LibriSpeech test-clean
AISHELI-1
Type
test-other
test
whisper-large-v3
Bidirectional
3.45
5.82
4.71
whisper-large-va
Causal
3.13
7.10
6.27
whisper-large-v3
Block Causal
0.5s
3,85
7.30
5.70
whisper-large-v3
Block Causal
Is
3.37
6.50
5.14
whisper-large-v3
Block Causal
2s
3.39
6.16
4.76
We also conduct an ablation study on the effect of block size in the block causal attention. In the ablation study we fine-tune the Whisper model with only the supervised ASR datasets for 20,000 steps with batch size 1024. For all the models we use VQ codebook size 4096 and sampling rate 25Hz. The results are shown in Table 11
B.2 SPEECH DECODER
The speech decoder uses the same architecture as the flow matching model in Cosy Voice (Du et al., 2024). For simplicity, we exclude the speaker embedding component from the flow model. The decoder is trained from scratch for 2 epochs with dynamic batching of 20000 frames in batch and learning rate 1e-3. The training datasets include Emilia (He al. 2024). Yodas2 (Li et al.) (023a), Ligri-Light (Kahn et al., (020) and proprietary Chinese speech dataset. To achieve consistent speaker identity our end-to-end spoken chatbot, we further fine-tune the speech decoder using speech responses synthesized by MeloTTS in the SpeechDialog-908 dataset.
B.3 TEXT-TO-TOKEN MODEL
The text-to-token model is initialized from 1.5B pre-trained text LM of the same architecture (further experiments indicate that training from scratch yields the same performance). The training dataset consists of the TTS corpus in Table and 9 with sampling ratio proportionate the number of samples in each subset. We the AdamW (Loshchilov & Hutter 2019) optimizer with 31 = 0.9 and Bg = 0.95. The model is trained for 300B tokens with sequence length of 4096 and batch size of 256, learning rate that decays from 4 x 10 to 4 10 and weight decay 0.1. The architecture of the text-to-token model is shown in Table 12 (number speech tokens not included in the vocab size).
Table 12: Model architecture of the text-to-token language model.
Hyper-parameters
Value
Number of layers
28
Hidden size
2048
FFN inter hidden size
6144
Activation function
SwiGLU
Attention heads
16
Attention head size
128
Attention group size
4
Maximum sequence length
8192
Vocab size
59264
19

---

Published as a conference paper at ICLR 2025
B.4 SPEECH LANGUAGE MODEL
Table 13: Model architecture of the speech language model.
9B
1.5B
Number of layers
40
28
Hidden size
4096
2048
FFN inter hidden size
13696
6144
Activation function
SwiGLU
SwiGLU
Attention heads
32
16
Attention head size
128
128
Attention group size
2
Maximum sequence length
8192
8192
Vocab size
151552
59264
C EVALUATION DETAILS
For Spoken StoryCloze and Spoken TopicStoryCloze, synthesize the speech for contexts and continuations with the provided texts with the TTS engine. When selecting the most probable con- tinuation, the likelihood is normalized by the number of tokens in each continuation.
For Llama Questions, we use the audio files provided in the datase We synthesize the speech for Web Questions and TriviaQA with the TTS engine. For TriviaQA, randomply sample 1, 1,000 samples from the test set of the 're' setting to match the size of the other two datasets. For all the three datasets of spoken question answering, we add the text prompt "the answer is" after the question for both the 'S' and 'S-TT" settings. For the 'S' setting the model generates speech of at most 10 seconds, and for the 'S-T' setting the model generates at most 128 tokens.
D CASE STUDY
D.1 SPOKEN QUESTION ANSWERING
Here we provide examples of spoken question answering from Llama Questions, Web Questions, and TriviaQA. Given question in speech, the speech language model predicts speech tokens, which are then fed into the speech decoder to get the output audio. We utilize whisper-large-v3 to transcribe the audio into text.
Dataset: Llama Questions
Question: what is the capital of France? The answer is
Ground Truth: Paris
Prediction: Paris. The capital of France Paris. Paris is the capital
of the, the, the, the,
Dataset: Web Questions
Question: Who governor of ohio 2011? answer is Ground Truth: John Kasich
Prediction: John Kasich. What is the name of the governor? 2012. The answer. John Kasich. What is
Dataset: TriviaQA
Set
20

---

Published as a conference paper at ICLR 2025
Question: who plays the role of Bubble in Absclutely Fabulous? The answer
Ground Truth: Jane Horrocks
Prediction: Julia Steiner Julia Steiner is an American actor, singer and dancer. Be $ he 9 he
For the examples above we can observe that our model can answer questions about world knowledge.
D.2 SPOKEN CHATBOT EVALUATION
Here we provide examples spoken chatbot evaluation. tions of the speech imput and output.
The question and response are transcrip-
Task: General QA
need to get my nephew birthcay gift. I'm thinking of giving him game for his Nintendo Switch. Can you give few opticns that came out recently?
Question:
what a great nephew you are. Giving a game for his Nintendo Switch is fantastic idea. Here are some amazing options that just released. Super Mario Odyssey, The Legend of Zelda Tears of the Kingder, Bookman Sword and Shield, Just Dance, Minecraft, The Elder Scrolls Five Skyrim Anniversary Edition, Portal Knights, Shovel Knight, Celeste, and Overcooked two. These games are highly rated and perfect for any Nintendo Switch player. Choose one that matches his interests and enjey his birthday.
Response:
Score:
Task: Knowledge
Question: How many days does Hanukkah last?
Ground Truth; Night
Hanukkah, also known as the Festival of Lights, is an eight- day Jewish holiday that commemorates the rededication of the Temple in Jerusalem during the Maccabean Revolt. It typically falls in November December, depending observed for eight days and nights, with each day corresponding of the eight brands of the menorah, Traditional activities include lighting the menorah, being frayed foods like lakes and sufkat, exchanging gift, playing dreidel and reciting the Hanukkah prayers. The lighting of the menorah is central ritual. With the first candle lit on the earth's day, accitional candles added each day, A festival includes on the eighth day with the lighting of the final candle and the sabration of Shemini Atzeret and Simkat Terah. Some cuitures celebrate Hanukkah for Nidaz, including the Ishan al-iev, Shemini Atzeret and simkat Torah.
Response:
the Hlbbrew lunar calendar, the festival is one
Judgment: Correct
D.3 INTERLEAVED PRE-TRAINING DATA
Here we provide examples of speech-text interleaved pre-training data. The following examples are sampled from FineWeb interleaved pre-training dataset with 150k text tokenizer and 12.5Hz speech tokenizer. The speech tokens are transcribed by the corresponding ASR model of the speech tokenizer and are displayed in blue and the special token in bold. An extra new line is added before and after the audio segment for clarity.
Eugene Van Reed
Died: 1873
21

---

Published as a conference paper at ICLR 2025
<|begin.of.audio/ [53 speech tokens] originally from francisco, reed first traveled to Japan <f ouuiiio//> 1859, where he established his own trading company, dealing in arms among other goods. Named Consul General of <Ibegin of audio|> [34 speech tokens] the hawadian kingdom in eighteen sixty six <|end of audio|>
>
he played a key role in <|begin..of.uudiol
[43 speech tokens] organizing for the first japanese immigrants to travel to hawaii in </end_of.audio/> 1868. This first group of 148, known the gannenmeno, encountered severely harsh working condition considerable tension between the governments of Japan, Hawaii, <lbegin of audio|> [40 speech tokens] and the united states, resulting in official japanese, <|end.of..udion immigration
the sugar plantations, leading
opegin.of.audion [71 speech tokens] hawaii at not beginning until 1885, after longthy vegotiations <f
Van Reed died in 187 3 aboard ship called Japan which he had been taking home to San Francisco from Japan
According to declarations irom the sector Chambers, Argentina consumed 340 million 1itres of pesticides and heroicides in the last year; and this quantity is increasing 158 to 20% each year. These poisons sprayed, fumigated and applied to areas inhabited by 12 million people. For
<lbegin of audio|> [20 speech tokens] a long time the
residents </end of audio|>
of the affected localities have been denouncing suffer from serious diseases as consequence
their being contaminated by </begin.of arriii.of.auddae audioi [52 speech tokens] the postatites this situation was confirmed at the first <f and 2nd Meeting
[73 speech tokens] of physicians of the fumigated towns, at the cordoba medical cns of faculty, and, <lend audio|> Medica cal Sciences Faculty of the Rosario National University in 2010 and 2011, zespectively. There is
of
<|begin.of.audiol> 162 speech tokens| substantial public demand to reclassify pesticides in arginina. this demand < of is sound: depending on how pesticides
ofbeggn.of..uudio... [77 speech tokens] classified the provincial and municipal regulations determine the distances between fumigated of audio//
(sprayed) and inhabited areas.
Currently, the classification is made according to the quantity in milligrans of poison that, when fed to rats, kills 50% of the population tested (Lethal Dose test or LDS0); the less the quantity of poisenous substance is required,
<|begin.of.audioll [34 speech tokens] the higher the level of toxicity is attributed <|end of audio|>
such, this form measurement ignores medium and long term eifects, including oncogenic, reproductive <|begin.of.audie [43 speech takens] immunological and endocrine ones in the light of on
that substance.
these facts, ofbegin.of.audioo [38 speech tokens] glyphosate should be reclassified as level ib ofudioo
(highly hazardous; the WHO recommended classification of pesticides by hazard), particularly because of the scientific and epidemiological data, showing that its accumulation in the body is connected to <lbegin audiol> 149 speech tokens continenta. naitormat and apontaneous abortions <|
22

---

Published as a conference paper at ICLR 2025
3].
Purthermore, current toxicological classification of acute effects of pesticides
<lbegin.off a ofudiol) [24 speech tokens] done taking to account
<|end audioo|>
of information and scientific data, which shew the acute damage of these poisons for agricultural use on humans, and that are different from <|begin.of.audion 138 speech tokens| the findings in rodents
hi innlighting patterns </end_of.auddo/>
speci fic to humans.
E PROMPT FOR CONSTRUCTING SPEECH DIALOGUE DATASET
You are an AI assistant designed to convert text SET data into SFT data adjusted for speech synthesis tasks. Your task is to generate a medified response suitable for text-to-speech synthesis under the following conditions:
Exclusion of Unrcadable Charactors and Number Conversion: Remove any characters that text-to-speech (TTS) systems cannet synthesize, such as parentheses (), bullet points, or other special symbols. Convert all numbers into their English word equivalents. For example, convert one to one, twenty to twenty, and so on, Do not include lists or line breaks; the response should be single paragraph.
Specificity in Response: Make the response mere specific and to the point, avoiding lengthy explanations. Focus delivering the key message concisely.
Clarity: Ensure that the response is clear and easy to understand when spoken aloud.
Avoidance of Code Content: the prompt suggests writing generating code, return an empty JSON object. If the prompt only inquires about knowledge related to code without requiring code generation, provide a modified response.
Below the conversation input:
[Prompt]: (prompt)
[Kesponse]: (response)
Please output the following JSON format if the conditions are met:
'Jjson
["responss"; "mmodifi '*
If the prompt is filtered out, output: {}
F PROMPT TEMPLATE FOR SPOKEN DIALOGUE
Direct Generation
</sytemm>>
User will provide you with a speech instruction. Think about the instruction and speak the response aloud directly.
</uerr>>
<| beeginoo__uddio (Speech (Speech
Is
beegin_of_audi (Speech Response] < lend_of audiol> >
23

---

Published as a conference paper at ICLR 2025
Text-guided Generation
s|sttem|>
User will provide you with a speech instructicn. Think about the instruction and speak the response aloud directly.


I <
(Text Response)
<| assistant:
begin_of_audio (Speech Response) <|end_of_audic]>
G
PROMPT FOR EVALUATING SPOKEN CHATBOTS
General QA
[Instruction]
Please act as an impartial judge and evaluate the quality of the response provided by an Al assistant to the user question displayed below. Your evaluation should consider der factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response. Begin your evaluation by providing short explanation. Be objective as possible. After providing your explanation, you must rate the response on a scale of 1 to 10 by strictly following this format: "[[rating]]", for example: "Rating: [[5]]".
[Question] (instruction)
[The Start of Assistant's Answer] (response]
[The End of Assistant's Answer]
Knowledge
Your will given question, the reference answers that questicn, and an answer to be juaged. Your tasks is to judge whether answer to be judged is correct, given the question and reference answers. An answer considered correct expresses contains the same meaning as at least **one of** the reference answers. The format and the tone of the matter.
response does
You should respond in JSON format. First provide a one-sentence concise analysis for the judgement in field 'analysis', then your judgment in field 'judgment'. For example, ''json
{ (("nnalysis;; <a onc-sentence concise analysis for the judgement>, judgment"; <your final judgment, "correct" or "incerrect">))
Question
(instruction)
Reference Answer
(targets)
i Answer To Be Judged
Lanswer_to_be_judgee
24


Reviews:
[{"review_id": "wVNV6TZYDY", "reviewer": "Reviewer_9jmQ", "summary": "The paper proposes a speech-text pretraining process for scaling speech-language model training without acquiring large amounts of speech audio data. The process mainly includes an ASR-based low-bitrate speech tokenizer and a text-to-speech-token model to produce large quantities of speech tokens for speech-text pertaining. The pre-trained model is fine-tuned on spoken dialog datasets and shows competitive performance compared to existing SOTA models.", "questions": "See weaknesses.\nHow are the speech and text tokens interleaved to form training samples? What are the details of this data creation process?\nHow does the model benefit from interleaved speech and text modalities? \nHow do you deal with the different sampling rates and information granularities between speech and text tokens during the process?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The ASR-based speech tokenizer achieves semantic information preservation and decent speech audio reproduction at the same time.\n2. The low-bitrate speech tokenizer and the text-to-token model effectively use the existing large amounts of text data to synthesize large amounts of speech tokens, which saves resources to collect large amounts of speech audio data and improves the language model's speech performance after pretraining.", "weaknesses": "The weaknesses are mainly in terms of paper writing and presentation. \n1. The paper mentions \"we are first to use supervised semantic tokens for SpeechLMs\". However, one of the baselines, Mini-Omini also uses a whisper-based speech tokenizer. \n2. The details on how the speech and text modalities are interleaved are missing. \n3. As an important part of the process, the details of the text-to-token model are missing—for example, model architectures, training schemes, etc.\n4. The large amounts of speech tokens generated by the text-to-token model are still from existing datasets and speech-synthesized audio from text. How is this process different from generating speech tokens from synthesized speech audio using large amounts of text? For example, llama-omni also uses cosy-voice to synthesize speech audio to augment training data. What's the innovation here between text-to-speech-to-token and text-to-token?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fNjzjM7EG5", "reviewer": "Reviewer_LeY9", "summary": "This paper is about scaling up data to train large speech language models.  The authors present a method for tokenizing speech using the Whisper encoder and demonstrate their tokenizer retains semantic information as well is fine-grained information for good quality speech generation.  They also describe a method for training a text-to-token model.  With these, they are able to tap into large resources of text data to generate synthetic training data, which they interleave with other conventional text and speech/text sources to pre-train a speech LM.  By fine-tuning the LM on a dialogue corpus they demonstrate a speech chat-like capability.  Extensive experimentation is performed, and the speech pretrain method does quite well on a range of tasks.", "questions": "While the whisper ASR model has achieved excellent performance on a range of tasks, it does have its limitations, especially with regards to unseen or low-resource languages.  That is not an issue for this paper which seems to focus on English (although there was quite a bit of Chinese data used as well).  Have the authors given any thought as to how to extend this work to cover more languages?\n\nAre there any plans to open source the tokenizer, text-to-token model, or the speech LM itself?\n\nAlso, it would be nice if the authors could describe the amount of computation required to pretrain the speech LM.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is a nice contribution to the very hot topic of speech LMs.  By developing an effective speech tokenizer and text-to-tokenizer model the authors are able to create a very large speech language model that produces impressive results on a wide range of tasks.  The authors perform extensive experiments and ablation studies on the speech tokenizer, speech generator (decoder), and the speech LM.  The model is able to achieve strong performance on both spoken language modeling and spoken question answering tasks.  Finally, when fine-tuned on dialogue data, the model does well on a spoken chat-bot task.", "weaknesses": "Although this is not necessarily a weakness, this paper seems very strong on the engineering side and a little weaker on the novelty side of things.  The recipe the authors put forward consists of three separate steps 1) tokenizer, 2) text-to-token model 3) pretrain speech LM.   While the authors build a strong tokenizer based on the Whisper model, the approach is not especially novel as it is built on top of a strong speech recognition model.  Likewise the use of a TTS corpus to learn a text-to-token model is a nice approach, but has been done before to learn similar kinds of models (e.g., Hsu et al., Text-Free Image-to-Speech Synthesis Using Learned Segmental Units, 2020).  Finally, the interleaving of different kinds of text and speech data to pretrain an LLM with an additional token vocabulary is not especially novel.  However, while these points are arguably true, I find it impressive that the authors have put all the pieces together to create a very strong speech LM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n8B52M27fJ", "reviewer": "Reviewer_kEHP", "summary": "This paper tackles the important problem of closing the gap between textLMs and SpeechLMs, to enable a spoken conversation with an AI model.\nThe paper leverages the recently proposed *supervised* semantic tokens (introducing a discrete bottleneck in ASR models) which show better alignment with text. \nMoreover, they train a text to audio tokens to enable the generation of synthetic audio tokens based on high-quality texts. \nThey suggest randomly replacing textual tokens with the corresponding synthetic speech tokens (resulting in an interleaved sequence), which helps to align the text and audio tokens. \nThey train large SpeechLMs on diverse text/audio inputs (audio only, text only, interleaved, [text,audio] and [audio,text]), and show convincing results on SLM, SQA. \nThey perform supervised finetuning on a proprietary (?) spoken dialogue dataset, and evaluate their model as a spoken chatbot using GPT-4 as a judge.", "questions": "Please consider clarifying the following questions in the final revision, (apologies if I missed some details that are in the paper):\n\nSection 2.1 and table 1:\n- I’m assuming that you measured WER in content/semantic retention - is that right?\n- How did you measure the ground truth (Whisper?)? \n- The title says ‘Speech Tokenizer Results” but the Quality also measures your speech decoder.\n- Is the speech decoder single/multi-speaker? Please add information regarding speaker identity preservation. \n- What was the training data of the speech decoder? (single or multi speaker?)\n- Missing citations (e.g. Expresso). \n- LS stands for Librispeech? this isn’t stated.\n\nTable 2:\n- How do you measure WER? (those tokens into the bottleneck layer of the quantized Whisper? Decode the audio using the speech decoder and apply the regular Whisper?)\n\nSection 2.2:\n- Regarding \\eta, please clarify if that is the ratio of text that is replaced or the ratio that audio will take out of the final sequence. I assume it is the first based on section A.1 but better for it to be clear in the main text too.\n\nSection 2.3.2\nIn the main text, please add that you used GPT-4 to filter examples, shorten the responses, and avoid outputting text that cannot be read aloud (help the reader understand it from the main text). Also consider adding optimization details on the finetuning step (lr, batch size, etc')\n\n\nSection 3.1: \n- Please help the reader and add in the main paper that the GPT-4 content quality is on a scale of 1-10.\n- The response of the SpeechLM was converted to text before it moved to GPT-4, right? How was the conversion performed? (Quantized whisper? your SpeechLM? Decoded and then used regular whisper?)\n- Two comments regarding the ASR-WER metric in Table 4. First, it is more of a content quality metric rather than a speech quality metric. Secondly, as there are many ways to answer a question correctly, I suggest moving to ASR-ROUGE or ASR-BLUE instead.\n\nTable 3: \n- what’s the difference between \\emptyset and “-“?\n\n\nComments:\n\n- Fig1a is hard to understand at first glance - specifically, that the yellow tokens are replaced with speech tokens. Adding a color legend (Yellow: SpeechTokens, Cyan: TextTokens) would make it easier to understand.\n\n- The second row in Fig 2a needs to be clarified. “Text—Audio token—>Text-to-token LM” can be perhaps: “Text — (TexttoTokenLM)—> Audio Tokens”. \n- Semantic tokens determine the pacing of speech, which is a part of the speaker’s prosody. Your synthetic audio tokens are not conditioned on a speaker, so you are likely to get the semantic tokens of an average speaker. It is fine overall.\n\n- Regarding audio tokenization - consider reducing the dimension (e.g. from 1024 to 8/16) before quantization to prevent codebook collapse.  \n- Moreover, I suggest applying text-tokenization algorithms (BPE?) on the speech units, to produce variable-length representations with a more balanced distribution, and further shorten the audio sequence.\n\nTypos:\n- Line 514: AudioLLM->AudioLM\n- Line 516: Moshi Citet->citep", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Supervised speech tokenizers are a great way to distill the content from audio.  Audio is high-dimensional, and using text and a low bitrate bottleneck to focus on content is a good idea, suitable for SpeechLMs. \n- Training a “TTS” model to generate synthetic audio tokens is interesting - as it doesn’t require generating the final audio (high-bitrate, compute-intensive, issues with OOD synthetic data).  Instead, they generate latent audio tokens that focus on content. \n- the interleaving (replacing spans of text with its synthetically produced speech tokens) is interesting as it forces the model to learn alignment between text and audio tokens. It was also shown to be effective in practice.\n- The ability to perform text-guided response (which is a kind of chain-of-thought) is interesting.\n- The ablation study was done well.", "weaknesses": "- Several methodological evaluation details are missing (what was measured and how was it computed), mostly in Section 2.1 and Table 1 (See questions).  Whenever you report some metric with an intuitive non-exact name (e.g., Content Preservation - LS), you should explain somewhere it more precisely (e.g., Content Preservation: We run our quantized whisper on the LS (LibriSpeech) dataset to generate text and report the WER to the GT transcript) I understand that there’s a space limitation, but this is important. \nI've listed some specific details I found missing in the Questions section. I suggest adding a short sub-paragraph that describes the evaluation methodology (defines all datasets+metrics being used) or adding those details into the main text within the relevant sections. If space is an issue, you can add those into an appendix section.\n\n- Currently there's no sample page (unless I missed something). Consider creating a sample page with samples on speech continuation (audio prefix, audio GT continuation, and the model's audio continuation). Also consider adding examples of spoken question answering (audio question, audio GT answer, the model's prediction). Examples from the spoken chatbot evaluation would also be great. Moreover, you can visualize how the interleaved samples sound like (paragraph with audio tokens that were decoded in it).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8KQMfxKzJp", "reviewer": "Reviewer_XoEC", "summary": "This paper proposes a method for scaling SpeechLMs using a new pre-training scheme called \"Synthetic Interleaved Data\". In this scheme, a text to token lM is first trained on supervised data, and then used to expand text-based datasets by predicting the speech tokens directly from the text data. For pre-training, only spans of text are converted to speech tokenization and text and speech are interleaved with one another. After this pre-training, the model is trained in the usual SpeechLM format. The strength of this approach is the ability to generate a large-scale dataset from text-corpora. Furthermore, this method shows strong results on speech-understanding and generation datasets.\n\nOverall, the method presented in this paper and novel, and has an interesting contribution. On the other hand, the writing is unclear and the evaluations are somewhat lacking. I thus recommend to borderline reject this paper.", "questions": "1  Why is the method pre-trained on Chinese data?\n2. Why is GPT-4 used for scoring?\n3. Why is whisper used as the encoder? did it perform better than other encoders?\n4. It is stated that the model can do streaming, was this evaluated?\n5. Why are the ablation also done on a 1.5B model?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1. A novel approach for expanding the training corpora of SpeechLMs.\n2. The method is easy to implement, and thus can be expanded to other methods.  \n3. State of the art results on sTopic-StoryCloze, sStoryCloze, Web Questions, Llama Questions and TriviaQA.", "weaknesses": "1. The writing is unclear for most parts of the paper. While Synthetic Interleaved Data is the main contribution of the paper, it is not clear what this means from the abstracts and introduction. Furthermore, the main explanation of this is just a small part of the paper. I would suggest reducing the length and condensing the section regarding speech tokenization (as this is a well established concept) and increasing the amount of detail in the section regarding Synthetic Interleaved Data. I would also suggest adding a better summerization of this concept in the introduction.\n2. While there is a good ablation analysis, there aren't any explanations for why the architectural / training parameters where chosen as they where. I suggest to add a dedicated subsection or add this into the methodology section where the parameters are introducted.\n3. The training datasets are lacking in clarity:\n- For table 1, is it unclear on which dataset it was evaluated and why MOSNet scores are so low. \n- It is unclear what Supervised speech-text data are used to train the model.\n- It is unclear what datasets areused to train the text to speech tokens model. \n- It is unclear what datasets are used to fine-tune the tokenization encoder and decoder.\nThese should be added in the section specifying the training pipeline or in a dedicated table / figure.\n4. The experimental results are lacking in clarity:\n- the origin of the baseline numbers in all tables is lacking, are these from other papers or from independently evaluating? I would suggest adding these directly to the table or in the caption.\n- In table 3, speechGPT and Spectron are speech to speech methods, while the results are stated in speech to text.\n- in Table 1 MOSNet was used while in table 4  UTMOS is used. This reason for this should be explained in the paper or have uniformity between them. \n5. The paper is lacking some evaluations:\n- Human evaluations of speech quality, such as MOS or MUSHRA evaluations where humans will rate the speech quality of the proposed method compared to the baseline.\n- Evaluation on other tasks, such as speech continuation, reconstruction and TTS for the full method. Speech continuation and Reconstruction results at least should be added, while TTS might be left for future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ijTBBYrb92", "reviewer": "Reviewer_LiYj", "summary": "This work  present a novel approach for scaling speech-text pre-training by leveraging large-scale synthetic interleaved data derived from existing high-quality text corpora. This work utilized existing text-to-speech (TTS) datasets to train a text-to-token language model, which is used to synthesize 600B tokens of speech-text interleaved data. Experiments have demonstrated the effectiveness of incorporating interleaved speech-text data, which can effectively align speech and text. Furthermore, this work constructs a speech instruction dataset, SpeechDialog-90K, to fine-tune models into a chatbot model, which can directly generate speech responses without intermediate text response and significantly improve the previous SOTA.", "questions": "1. I'm curious if there is any additional filtering process for the text input into the text-to-tokens model, as there are many texts that cannot be synthesized, such as code or mathematical formulas.\n\n2. Why is it required for the encoder and decoder to be causal when training the speech tokenizer? During the inference stage, speech are segmented into 2s-chunks for inference, which does not require a streaming speech tokenizer.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This work demonstrates the need to use interleaved speech-text data for cross-modal pre-training, paving the way for more effective speech-text pretraining. Previous pretraining methods have typically relied on paired ASR or TTS data, which is limited in scale; or independently utilized unsupervised speech and text data, which cannot model the dependency between the two modalities. I believe that this work makes a great contribution to the filed of speech-text pretraining.\n\n2. Although the use of interleaved data has been proven effective in the field of image-text pretraining [1], it has not been explored in the field of speech-text pretraining. In contrast to the vision field, web data can naturally form interleaved image-text data, but it is difficult to collect real data with interleaved speech and text. This work proposes a novel method to synthesize pseudo-interleaved data using a text-to-tokens model, and through thorough experimentation, demonstrates the effectiveness of synthesized data and observes that scaling synthesized data continues to provide benefits.\n\n3. This work is very solid and well-motivated. The paper is well-structured, with a clear presentation of the methodology, experiments, and results. This work also reports state-of-the-art performance in speech language modeling and spoken question answering.\n\n[1] Chameleon team. Chameleon: Mixed-Modal Early-Fusion Foundation Models. arxiv: 2405.09818.", "weaknesses": "1. There is a lack of performance evaluation for the text-to-tokens model. For example, after converting a piece of text into tokens and then decoding it into speech using a vocoder, what is the ASR-WER of the resulting speech? This result is necessary to demonstrate the semantic representation capability of the tokens generated by the text-to-tokens model.\n\n2. Based on my experience, tokens generated by text-to-models lacks diversity, and the speech instruction dataset SpeechDialog-90K in the SFT stage is also synthesized by TTS, so I am concerned about whether the model can understand real speech input. I checked the evaluation datasets in this work, all of which were synthesized through TTS, lacking evaluation on real speech input (such as AIRBench [2]).\n\n3. The quality of the output speech is not satisfactory, as evidenced by the poor ASR-WER in Table 4. In comparison to llama-omini, which was only trained on 100 hours of speech data, this model was trained on a much larger scale of 700k hours of speech data. The author needs to provide a reasonable explanation for why the ASR-WER is so poor.\n\n[2] Yang, Qian, et al. AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension. arxiv 2402.07729.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)