Scaling Speech-Text Pre-training with Synthetic Interleaved Data

Authors: Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, shengmin jiang, Yuxiao Dong, Jie Tang
Primary area: foundation or frontier models, including LLMs
Keywords: large language models; speech language model; spoken chatbots

Abstract:
Speech language models (SpeechLMs) accept speech input and produce speech output, allowing for more natural human-computer interaction compared to text-based large language models (LLMs).
Traditional approaches for developing SpeechLMs are constrained by the limited availability of unsupervised speech data and parallel speech-text data, which are significantly less abundant compared to text pre-training data, thereby limiting their scalability as LLMs.
We propose a novel approach to scaling speech-text pre-training by leveraging large-scale synthetic interleaved data derived from text corpora, eliminating the need for parallel speech-text datasets.
Our method efficiently constructs speech-text interleaved data by sampling text spans from existing text corpora and synthesizing corresponding speech spans using a text-to-token model, bypassing the need to generate actual speech.
We also employ a supervised speech tokenizer derived from an automatic speech recognition (ASR) model  by incorporating a vector-quantized bottleneck into the encoder. This supervised training approach results in discrete speech tokens with strong semantic preservation even at lower sampling rates (e.g. 12.5Hz), while still maintaining speech reconstruction quality.
Starting from a pre-trained language model and scaling our pre-training to 1 trillion tokens (with 600B synthetic interleaved speech-text data), we achieve state-of-the-art performance in both speech language modeling and spoken question answering, improving performance on spoken questions tasks from the previous SOTA of 13\% (Moshi) to 31\%.
We further demonstrate that by fine-tuning the pre-trained model with speech dialogue data, we can develop an end-to-end spoken chatbot that achieves competitive performance comparable to existing baselines in both conversational abilities and speech quality, even operating exclusively in the speech domain.

Decision: accept (poster)
Reviewer scores: 6, 8, 8, 5, 8

Reviews and discussion:
<thread>
creation date:
1729825864668

thread_type:
official_review

<thread_object>
creator: reviewer_liyj

<content>
summary:
This work  present a novel approach for scaling speech-text pre-training by leveraging large-scale synthetic interleaved data derived from existing high-quality text corpora. This work utilized existing text-to-speech (TTS) datasets to train a text-to-token language model, which is used to synthesize 600B tokens of speech-text interleaved data. Experiments have demonstrated the effectiveness of incorporating interleaved speech-text data, which can effectively align speech and text. Furthermore, this work constructs a speech instruction dataset, SpeechDialog-90K, to fine-tune models into a chatbot model, which can directly generate speech responses without intermediate text response and significantly improve the previous SOTA.

strengths:
1. This work demonstrates the need to use interleaved speech-text data for cross-modal pre-training, paving the way for more effective speech-text pretraining. Previous pretraining methods have typically relied on paired ASR or TTS data, which is limited in scale; or independently utilized unsupervised speech and text data, which cannot model the dependency between the two modalities. I believe that this work makes a great contribution to the filed of speech-text pretraining.

2. Although the use of interleaved data has been proven effective in the field of image-text pretraining [1], it has not been explored in the field of speech-text pretraining. In contrast to the vision field, web data can naturally form interleaved image-text data, but it is difficult to collect real data with interleaved speech and text. This work proposes a novel method to synthesize pseudo-interleaved data using a text-to-tokens model, and through thorough experimentation, demonstrates the effectiveness of synthesized data and observes that scaling synthesized data continues to provide benefits.

3. This work is very solid and well-motivated. The paper is well-structured, with a clear presentation of the methodology, experiments, and results. This work also reports state-of-the-art performance in speech language modeling and spoken question answering.

[1] Chameleon team. Chameleon: Mixed-Modal Early-Fusion Foundation Models. arxiv: 2405.09818.

weaknesses:
1. There is a lack of performance evaluation for the text-to-tokens model. For example, after converting a piece of text into tokens and then decoding it into speech using a vocoder, what is the ASR-WER of the resulting speech? This result is necessary to demonstrate the semantic representation capability of the tokens generated by the text-to-tokens model.

2. Based on my experience, tokens generated by text-to-models lacks diversity, and the speech instruction dataset SpeechDialog-90K in the SFT stage is also synthesized by TTS, so I am concerned about whether the model can understand real speech input. I checked the evaluation datasets in this work, all of which were synthesized through TTS, lacking evaluation on real speech input (such as AIRBench [2]).

3. The quality of the output speech is not satisfactory, as evidenced by the poor ASR-WER in Table 4. In comparison to llama-omini, which was only trained on 100 hours of speech data, this model was trained on a much larger scale of 700k hours of speech data. The author needs to provide a reasonable explanation for why the ASR-WER is so poor.

[2] Yang, Qian, et al. AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension. arxiv 2402.07729.

questions:
1. I'm curious if there is any additional filtering process for the text input into the text-to-tokens model, as there are many texts that cannot be synthesized, such as code or mathematical formulas.

2. Why is it required for the encoder and decoder to be causal when training the speech tokenizer? During the inference stage, speech are segmented into 2s-chunks for inference, which does not require a streaming speech tokenizer.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729931289735

thread_type:
official_review

<thread_object>
creator: reviewer_xoec

<content>
summary:
This paper proposes a method for scaling SpeechLMs using a new pre-training scheme called "Synthetic Interleaved Data". In this scheme, a text to token lM is first trained on supervised data, and then used to expand text-based datasets by predicting the speech tokens directly from the text data. For pre-training, only spans of text are converted to speech tokenization and text and speech are interleaved with one another. After this pre-training, the model is trained in the usual SpeechLM format. The strength of this approach is the ability to generate a large-scale dataset from text-corpora. Furthermore, this method shows strong results on speech-understanding and generation datasets.

Overall, the method presented in this paper and novel, and has an interesting contribution. On the other hand, the writing is unclear and the evaluations are somewhat lacking. I thus recommend to borderline reject this paper.

strengths:
1. A novel approach for expanding the training corpora of SpeechLMs.
2. The method is easy to implement, and thus can be expanded to other methods.  
3. State of the art results on sTopic-StoryCloze, sStoryCloze, Web Questions, Llama Questions and TriviaQA.

weaknesses:
1. The writing is unclear for most parts of the paper. While Synthetic Interleaved Data is the main contribution of the paper, it is not clear what this means from the abstracts and introduction. Furthermore, the main explanation of this is just a small part of the paper. I would suggest reducing the length and condensing the section regarding speech tokenization (as this is a well established concept) and increasing the amount of detail in the section regarding Synthetic Interleaved Data. I would also suggest adding a better summerization of this concept in the introduction.
2. While there is a good ablation analysis, there aren't any explanations for why the architectural / training parameters where chosen as they where. I suggest to add a dedicated subsection or add this into the methodology section where the parameters are introducted.
3. The training datasets are lacking in clarity:
- For table 1, is it unclear on which dataset it was evaluated and why MOSNet scores are so low. 
- It is unclear what Supervised speech-text data are used to train the model.
- It is unclear what datasets areused to train the text to speech tokens model. 
- It is unclear what datasets are used to fine-tune the tokenization encoder and decoder.
These should be added in the section specifying the training pipeline or in a dedicated table / figure.
4. The experimental results are lacking in clarity:
- the origin of the baseline numbers in all tables is lacking, are these from other papers or from independently evaluating? I would suggest adding these directly to the table or in the caption.
- In table 3, speechGPT and Spectron are speech to speech methods, while the results are stated in speech to text.
- in Table 1 MOSNet was used while in table 4  UTMOS is used. This reason for this should be explained in the paper or have uniformity between them. 
5. The paper is lacking some evaluations:
- Human evaluations of speech quality, such as MOS or MUSHRA evaluations where humans will rate the speech quality of the proposed method compared to the baseline.
- Evaluation on other tasks, such as speech continuation, reconstruction and TTS for the full method. Speech continuation and Reconstruction results at least should be added, while TTS might be left for future work.

questions:
1  Why is the method pre-trained on Chinese data?
2. Why is GPT-4 used for scoring?
3. Why is whisper used as the encoder? did it perform better than other encoders?
4. It is stated that the model can do streaming, was this evaluated?
5. Why are the ablation also done on a 1.5B model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730297517492

thread_type:
official_review

<thread_object>
creator: reviewer_kehp

<content>
summary:
This paper tackles the important problem of closing the gap between textLMs and SpeechLMs, to enable a spoken conversation with an AI model.
The paper leverages the recently proposed *supervised* semantic tokens (introducing a discrete bottleneck in ASR models) which show better alignment with text. 
Moreover, they train a text to audio tokens to enable the generation of synthetic audio tokens based on high-quality texts. 
They suggest randomly replacing textual tokens with the corresponding synthetic speech tokens (resulting in an interleaved sequence), which helps to align the text and audio tokens. 
They train large SpeechLMs on diverse text/audio inputs (audio only, text only, interleaved, [text,audio] and [audio,text]), and show convincing results on SLM, SQA. 
They perform supervised finetuning on a proprietary (?) spoken dialogue dataset, and evaluate their model as a spoken chatbot using GPT-4 as a judge.

strengths:
- Supervised speech tokenizers are a great way to distill the content from audio.  Audio is high-dimensional, and using text and a low bitrate bottleneck to focus on content is a good idea, suitable for SpeechLMs. 
- Training a “TTS” model to generate synthetic audio tokens is interesting - as it doesn’t require generating the final audio (high-bitrate, compute-intensive, issues with OOD synthetic data).  Instead, they generate latent audio tokens that focus on content. 
- the interleaving (replacing spans of text with its synthetically produced speech tokens) is interesting as it forces the model to learn alignment between text and audio tokens. It was also shown to be effective in practice.
- The ability to perform text-guided response (which is a kind of chain-of-thought) is interesting.
- The ablation study was done well.

weaknesses:
- Several methodological evaluation details are missing (what was measured and how was it computed), mostly in Section 2.1 and Table 1 (See questions).  Whenever you report some metric with an intuitive non-exact name (e.g., Content Preservation - LS), you should explain somewhere it more precisely (e.g., Content Preservation: We run our quantized whisper on the LS (LibriSpeech) dataset to generate text and report the WER to the GT transcript) I understand that there’s a space limitation, but this is important. 
I've listed some specific details I found missing in the Questions section. I suggest adding a short sub-paragraph that describes the evaluation methodology (defines all datasets+metrics being used) or adding those details into the main text within the relevant sections. If space is an issue, you can add those into an appendix section.

- Currently there's no sample page (unless I missed something). Consider creating a sample page with samples on speech continuation (audio prefix, audio GT continuation, and the model's audio continuation). Also consider adding examples of spoken question answering (audio question, audio GT answer, the model's prediction). Examples from the spoken chatbot evaluation would also be great. Moreover, you can visualize how the interleaved samples sound like (paragraph with audio tokens that were decoded in it).

questions:
Please consider clarifying the following questions in the final revision, (apologies if I missed some details that are in the paper):

Section 2.1 and table 1:
- I’m assuming that you measured WER in content/semantic retention - is that right?
- How did you measure the ground truth (Whisper?)? 
- The title says ‘Speech Tokenizer Results” but the Quality also measures your speech decoder.
- Is the speech decoder single/multi-speaker? Please add information regarding speaker identity preservation. 
- What was the training data of the speech decoder? (single or multi speaker?)
- Missing citations (e.g. Expresso). 
- LS stands for Librispeech? this isn’t stated.

Table 2:
- How do you measure WER? (those tokens into the bottleneck layer of the quantized Whisper? Decode the audio using the speech decoder and apply the regular Whisper?)

Section 2.2:
- Regarding \eta, please clarify if that is the ratio of text that is replaced or the ratio that audio will take out of the final sequence. I assume it is the first based on section A.1 but better for it to be clear in the main text too.

Section 2.3.2
In the main text, please add that you used GPT-4 to filter examples, shorten the responses, and avoid outputting text that cannot be read aloud (help the reader understand it from the main text). Also consider adding optimization details on the finetuning step (lr, batch size, etc')


Section 3.1: 
- Please help the reader and add in the main paper that the GPT-4 content quality is on a scale of 1-10.
- The response of the SpeechLM was converted to text before it moved to GPT-4, right? How was the conversion performed? (Quantized whisper? your SpeechLM? Decoded and then used regular whisper?)
- Two comments regarding the ASR-WER metric in Table 4. First, it is more of a content quality metric rather than a speech quality metric. Secondly, as there are many ways to answer a question correctly, I suggest moving to ASR-ROUGE or ASR-BLUE instead.

Table 3: 
- what’s the difference between \emptyset and “-“?


Comments:

- Fig1a is hard to understand at first glance - specifically, that the yellow tokens are replaced with speech tokens. Adding a color legend (Yellow: SpeechTokens, Cyan: TextTokens) would make it easier to understand.

- The second row in Fig 2a needs to be clarified. “Text—Audio token—>Text-to-token LM” can be perhaps: “Text — (TexttoTokenLM)—> Audio Tokens”. 
- Semantic tokens determine the pacing of speech, which is a part of the speaker’s prosody. Your synthetic audio tokens are not conditioned on a speaker, so you are likely to get the semantic tokens of an average speaker. It is fine overall.

- Regarding audio tokenization - consider reducing the dimension (e.g. from 1024 to 8/16) before quantization to prevent codebook collapse.  
- Moreover, I suggest applying text-tokenization algorithms (BPE?) on the speech units, to produce variable-length representations with a more balanced distribution, and further shorten the audio sequence.

Typos:
- Line 514: AudioLLM->AudioLM
- Line 516: Moshi Citet->citep
</content>
</thread_object>

</thread>
<thread>
creation date:
1730411988391

thread_type:
official_review

<thread_object>
creator: reviewer_ley9

<content>
summary:
This paper is about scaling up data to train large speech language models.  The authors present a method for tokenizing speech using the Whisper encoder and demonstrate their tokenizer retains semantic information as well is fine-grained information for good quality speech generation.  They also describe a method for training a text-to-token model.  With these, they are able to tap into large resources of text data to generate synthetic training data, which they interleave with other conventional text and speech/text sources to pre-train a speech LM.  By fine-tuning the LM on a dialogue corpus they demonstrate a speech chat-like capability.  Extensive experimentation is performed, and the speech pretrain method does quite well on a range of tasks.

strengths:
This paper is a nice contribution to the very hot topic of speech LMs.  By developing an effective speech tokenizer and text-to-tokenizer model the authors are able to create a very large speech language model that produces impressive results on a wide range of tasks.  The authors perform extensive experiments and ablation studies on the speech tokenizer, speech generator (decoder), and the speech LM.  The model is able to achieve strong performance on both spoken language modeling and spoken question answering tasks.  Finally, when fine-tuned on dialogue data, the model does well on a spoken chat-bot task.

weaknesses:
Although this is not necessarily a weakness, this paper seems very strong on the engineering side and a little weaker on the novelty side of things.  The recipe the authors put forward consists of three separate steps 1) tokenizer, 2) text-to-token model 3) pretrain speech LM.   While the authors build a strong tokenizer based on the Whisper model, the approach is not especially novel as it is built on top of a strong speech recognition model.  Likewise the use of a TTS corpus to learn a text-to-token model is a nice approach, but has been done before to learn similar kinds of models (e.g., Hsu et al., Text-Free Image-to-Speech Synthesis Using Learned Segmental Units, 2020).  Finally, the interleaving of different kinds of text and speech data to pretrain an LLM with an additional token vocabulary is not especially novel.  However, while these points are arguably true, I find it impressive that the authors have put all the pieces together to create a very strong speech LM.

questions:
While the whisper ASR model has achieved excellent performance on a range of tasks, it does have its limitations, especially with regards to unseen or low-resource languages.  That is not an issue for this paper which seems to focus on English (although there was quite a bit of Chinese data used as well).  Have the authors given any thought as to how to extend this work to cover more languages?

Are there any plans to open source the tokenizer, text-to-token model, or the speech LM itself?

Also, it would be nice if the authors could describe the amount of computation required to pretrain the speech LM.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661541828

thread_type:
official_review

<thread_object>
creator: reviewer_9jmq

<content>
summary:
The paper proposes a speech-text pretraining process for scaling speech-language model training without acquiring large amounts of speech audio data. The process mainly includes an ASR-based low-bitrate speech tokenizer and a text-to-speech-token model to produce large quantities of speech tokens for speech-text pertaining. The pre-trained model is fine-tuned on spoken dialog datasets and shows competitive performance compared to existing SOTA models.

strengths:
1. The ASR-based speech tokenizer achieves semantic information preservation and decent speech audio reproduction at the same time.
2. The low-bitrate speech tokenizer and the text-to-token model effectively use the existing large amounts of text data to synthesize large amounts of speech tokens, which saves resources to collect large amounts of speech audio data and improves the language model's speech performance after pretraining.

weaknesses:
The weaknesses are mainly in terms of paper writing and presentation. 
1. The paper mentions "we are first to use supervised semantic tokens for SpeechLMs". However, one of the baselines, Mini-Omini also uses a whisper-based speech tokenizer. 
2. The details on how the speech and text modalities are interleaved are missing. 
3. As an important part of the process, the details of the text-to-token model are missing—for example, model architectures, training schemes, etc.
4. The large amounts of speech tokens generated by the text-to-token model are still from existing datasets and speech-synthesized audio from text. How is this process different from generating speech tokens from synthesized speech audio using large amounts of text? For example, llama-omni also uses cosy-voice to synthesize speech audio to augment training data. What's the innovation here between text-to-speech-to-token and text-to-token?

questions:
See weaknesses.
How are the speech and text tokens interleaved to form training samples? What are the details of this data creation process?
How does the model benefit from interleaved speech and text modalities? 
How do you deal with the different sampling rates and information granularities between speech and text tokens during the process?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new speech-text pretraining method without large-scale audio-text pre-training data by leveraging interleaved text-to-token data from text corpus. 

As the importance of multimodal agentic AI increases, speech-text multimodal training is becoming significant and popular to make better agentic AIs. The proposed method using interleaved data address lack of speech-text parallel corpus, which is the main obstacle of speech-LMs. The method design seems effective and novel. 

Althoubh some reviewers pointed out lack of details, those are not main concerns.

With positive scores from all reviewers, AC recommends accepting this paper.

Following the reviewers' comments, AC asks the authors to refine the manuscript to be more clear.