Venue: NAACL 2025

VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning

Yifan Peng\({}^{1*}\), Krishna C. Puvvada\({}^{2*}\), Zhehuai Chen\({}^{2*}\), Piotr Zelasko\({}^{2}\), He Huang\({}^{2}\), Kunal Dhawan\({}^{2}\), Ke Hu\({}^{2}\), Shinji Watanabe\({}^{1}\), Jagadeesh Balam\({}^{2}\), Boris Ginsburg\({}^{2}\)

Equal contribution. Work done while Yifan was an intern at NVIDIA.We will publicly release our data generation scripts, training code, and pre-trained model weights: [https://github.com/py798/NeMo_VoiceTextBlender](https://github.com/py798/NeMo_VoiceTextBlender)

###### Abstract

Recent studies have augmented large language models (LLMs) with speech capabilities, leading to the development of speech language models (SpeechLMs). Earlier SpeechLMs focused on single-turn speech-based question answering (QA), where user input comprised a speech context and a text question. More recent studies have extended this to multi-turn conversations, though they often require complex, multi-stage supervised fine-tuning (SFT) with diverse data. Another critical challenge with SpeechLMs is catastrophic forgetting, where models optimized for speech tasks suffer significant degradation in text-only performance. To mitigate these issues, we propose a novel single-stage joint speech-text SFT approach on the low-rank adaptation (LoRA) of the LLM backbone. Our joint SFT combines text-only SFT data with three types of speech-related data: speech recognition and translation, speech-based QA, and mixed-modal SFT. Compared to previous SpeechLMs with 7B or 13B parameters, our 3B model demonstrates superior performance across various speech benchmarks while preserving the original capabilities on text-only tasks. Furthermore, our model shows emergent abilities of effectively handling previously unseen prompts and tasks, including multi-turn, mixed-modal inputs.1

Footnote 1: We will publicly release our data generation scripts, training code, and pre-trained model weights: [https://github.com/pyf98/NeMo_VoiceTextBlender](https://github.com/pyf98/NeMo_VoiceTextBlender)

**Correspondence:** pengyf21@gmail.com, kpuvvada@nvidia.com, zhehuaic@nvidia.com

## 1 Introduction

Large language models (LLMs) have demonstrated impressive success in natural language processing [4, 5], sparking a surge of research into multimodal foundation models that extend beyond text. Recent studies in speech processing have focused on augmenting pre-trained LLMs with speech capabilities, giving rise to a new class of models known as speech language models (SpeechLMs) [6, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 222, 213, 214, 215, 216, 217, 218, 219, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 282, 284, 286, 287, 288, 289, 291, 285, 289, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 323, 335, 336, 337, 338, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 422, 434, 446, 447, 448, 451, 452, 453, 454, 455, 456, 457, 458, 460, 409, 411, 422, 434, 459, 461, 462, 463, 464, 474, 475, 465, 466, 476, 477, 483, 477, 484, 485, 486, 487, 488, 490, 411, 422, 434, 449, 491, 440, 412, 443, 449, 413, 440, 414, 445, 415, 416, 417, 418, 419, 420, 421, 434, 446, 449, 422, 447, 448, 451, 452, 453, 454, 456, 457, 458, 459, 460, 461, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 510, 511, 523, 535, 540, 512, 540, 513, 541, 556, 557, 561, 562, 563, 576, 578, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 501, 595, 596, 597, 598, 599, 600, 599, 610, 62, 63, 64, 650, 66, 671, 68, 692, 600, 63, 64, 66, 693, 650, 66, 694, 601, 602, 603, 604, 605, 606, 607, 61, 608, 61, 612, 613, 614, 615, 62, 62, 63, 64, 650, 66, 672, 68, 695, 616, 696, 651, 697, 608, 617, 609, 618, 619, 6 

freeze the LLM to preserve text capabilities, but this can lead to degraded speech performance, as shown in Section 4.3 and prior work Wang et al. (2023). Alternatively, models like GPT-4o mix speech and text data during pre-training to create a natively multi-modal model, but this requires access to extensive pre-training data and infrastructure, making it computationally expensive and requiring significant tuning. In this work, we propose single-stage joint speech-text supervised fine-tuning (SFT) with low-rank adaptation (LoRA) Hu et al. (2022), which preserves text-only performance while achieving excellent speech understanding capabilities.

Our contributions are summarized below.

* We propose a single-stage joint speech-text SFT strategy for training SpeechLMs, which simplifies the training process, preserves the LM's original text-only performance, and delivers strong results on speech tasks.2 Our 3B model outperforms previous 7B or 13B SpeechLMs on most evaluated benchmarks. Footnote 2: The current work mainly focuses on the linguistic content of human speech. In the future, we will expand to diverse attributes such as speaker identity Wu et al. (2024).
* We incorporate multiple methods for generating speech-related SFT data, including a novel approach that can construct _mixed-modal interleaving speech-text SFT data_ by applying text-to-speech (TTS) to randomly selected sentences from text SFT data. These diverse training data enable our model to handle multi-turn, mixed-modal conversations and generalize to previously unseen prompts and tasks.
* We will publicly release the pre-trained model weights, along with the code for data generation and training, to support and advance research on SpeechLMs.

## 2 Related Work

**SpeechLM overview.** SpeechLMs integrate language modeling with speech foundation models, and can be broadly categorized into two types. The first category of SpeechLMs directly models the distribution of speech features to facilitate speech generation Lakhotia et al. (2021); Borsos

Figure 1: Our VTBinder 3B with joint SFT enables multi-turn, mixed-modal conversations, allowing user input in the form of pure speech, pure text, or a combination of both. It’s worth noting that our speech-related SFT data consists of only single-turn interactions, while our text SFT data has multiple turns.

 

et al., 2023). In this case, speech signals are typically represented as discrete tokens, which are extracted using self-supervised speech encoders Hsu et al. (2021); Chen et al. (2022). The second category of SpeechLMs aims to augment LLMs with speech understanding capabilities. To preserve as much information from the speech input as possible, these models commonly employ continuous features extracted by supervised pre-trained speech encoders Radford et al. (2023); Zhang et al. (2023); Peng et al. (2023); Puvvada et al. (2024). Earlier works in this area primarily focused on single-turn speech-based QA tasks Gong et al. (2024); Tang et al. (2024), where the input consists of a speech segment and a text-based question. More recent research extends this approach to multi-turn interactions, allowing user input to be entirely speech-based Chu et al. (2024); Dubey et al. (2024). Our work falls within the second category, aiming to support multi-turn mixed-modal interactions in which user input can be pure text, pure speech, or a combination of both.

**SpeechLM training.** SpeechLMs are typically trained in multiple stages with supervised data. Gong et al. find that an appropriate curriculum is crucial to train their LTU models. Specifically, they propose a four-stage training procedure that gradually increases the number of learnable parameters and the complexity of the training tasks. Tang et al. adopt a three-stage training pipeline for their SALMONN models: pre-training, instruction tuning, and activation tuning. SpeechVerse Das et al. (2024) observes that training all learnable parameters from scratch on diverse speech tasks often leads to divergence. Hence, they propose two-stage training with gradually increased learnable modules and speech tasks. While multi-stage curriculum learning methods are intuitive and enhance training stability, they significantly increase the complexity of design choices. These approaches require extensive tuning and heuristics to determine the appropriate order for updating modules and assigning tasks. In this work, we adopt a _single-stage training strategy that combines text-only SFT data with various mixed-modal SFT data_. Our approach streamlines the training pipeline while achieving strong performance across diverse benchmarks.3

Footnote 3: A recent work, AudioChatLama Fathullah et al. (2024), also conducts single-stage training. However, it is not explicitly trained on diverse speech tasks. Instead, it uses ASR data and relies on the modal-invariance trick, which differs greatly from our training objective.

Catastrophic forgetting in SpeechLMMost SpecchLMs are optimized for speech tasks, often at the expense of their original text capabilities, a phenomenon known as catastrophic forgetting. To preserve the text-only performance of instruction-tuned LMs, some studies freeze the backbone LM Wang et al. (2023a); Fathullah et al. (2024); Dubey et al. (2024). However, as discussed in Section 4.3 and by Wang et al., this approach may degrade performance on speech tasks. Many other works use parameter-efficient fine-tuning methods, such as LoRA adapters Hu et al. (2022), to mitigate catastrophic forgetting Gong et al. (2024); Das et al. (2024). However, our experiments in Section 4.3 reveal that merging the LoRA parameters into the original model parameters significantly degrades performance on text-only benchmarks, demonstrating that LoRA alone does not ensure preservation of the model's original capabilities. A recent study in vision language models, VILA Lin et al. (2024), finds that incorporating text SFT data in their multi-stage training paradigm mitigates catastrophic forgetting. However, it only considers the vision modality, not speech. Inspired by this line of work, we propose a joint speech-text SFT approach, which integrates text-only SFT data with speech-related SFT data. Our method preserves text-only performance while achieving strong results on newly added speech tasks.

## 3 Proposed Method

### Model Architecture

Figure 2 shows the overall architecture of our VTBlender, consisting of three components: a speech encoder to extract continuous features from raw speech input, a modality adapter to map speech features into a shared embedding space with text, and

Figure 2: Model architecture. Only a pair of speech and text are depicted for simplicity, but the input can contain multiple segments of speech and text in any order.

 

a language model to generate text responses conditioned on the input. Similar architectures have been commonly used in prior works Chen et al. (2024); Das et al. (2024). The speech encoder is initialized from a pre-trained Canary encoder Puvvada et al. (2024). The LM has undergone text SFT to improve instruction-following capabilities (see Section 4.1 for more details). The modality adapter consists of randomly initialized Conformer layers Gulati et al. (2020). During training, both the speech encoder and modality adapter are fully fine-tuned, whereas the LM is partially fine-tuned with LoRA adapters Hu et al. (2022).

Let \(S\) and \(X\) be the input speech waveform and text tokens, respectively. They are mapped into a shared embedding space as follows:

\[{\bf S}^{\text{enc}}=\text{Enc}(S)~{}~{}~{}\in\mathbb{R}^{T\times D}, \tag{1}\] \[{\bf S}^{\text{adp}}=\text{Adp}({\bf S}^{\text{enc}})~{}~{}~{} \in\mathbb{R}^{T^{\prime}\times D^{\prime}},\] (2) \[{\bf X}^{\text{emb}}=\text{Emb}(X)~{}~{}~{}\in\mathbb{R}^{L\times D ^{\prime}}, \tag{3}\]

where \({\bf S}^{\text{enc}}\) is the output of the speech encoder with length \(T\) and feature size \(D\). \({\bf S}^{\text{adp}}\) is the speech feature sequence after the modality adapter with length \(T^{\prime}\) and size \(D^{\prime}\). \({\bf X}^{\text{emb}}\) is the text feature sequence after the LM embedding layer with length \(L\) and feature size \(D^{\prime}\). Then, the speech and text features are concatenated and fed into the LM to generate the output text \(Y\):

\[{\bf X}^{\text{inp}}=\text{Cat}({\bf S}^{\text{adp}},{\bf X}^{ \text{emb}})~{}~{}\in\mathbb{R}^{(T^{\prime}+L)\times D^{\prime}}, \tag{4}\] \[Y=\text{LM}({\bf X}^{\text{inp}}), \tag{5}\]

where \({\bf X}^{\text{inp}}\) combines both speech and text features and follows the chat template of the pre-trained LM. While only a pair of speech and text are depicted for simplicity, our framework is designed to accommodate any combination of speech and text inputs. For each user turn, the input may consist of speech alone, text alone, or a combination of both.

During training, we minimize the following loss:

\[\mathcal{L}=-\log P(Y~{}|~{}S,X;\Theta), \tag{6}\]

where \(\Theta\) is the set of learnable parameters, including the speech encoder, modality adapter, and LoRA adapter.

### Joint Speech-Text SFT

To preserve the original text capabilities while adding new capabilities of speech understanding, we propose joint speech-text SFT, which mixes text-only SFT data with three types of speech-related SFT data during training. Our training process is single-stage, with different data types sampled at specific probabilities when creating mini-batches.

Specifically, the text-only SFT data consists of multi-turn conversations, commonly used to enhance the instruction-following abilities of LLMs in the "post-training" stage. For speech-related SFT, we employ three types of data to address various use cases and improve the model's performance with mixed-modal inputs, which will be discussed in the following three sections. Note that the text-only SFT dataset has multiple turns, whereas all the speech-related SFT datasets have only one turn. Through joint SFT, our model can generalize to multi-turn mixed-modal conversations (see Section 4.4 and Figure 1).

#### 3.2.1 Multilingual ASR and AST

ASR and AST are foundational tasks that enable the model to understand speech, where each input includes speech and a text instruction describing the task. During training, the same instruction is used for all samples within a specific task and language.

Figure 3: Different types of SFT data are generated for training.

 

However, we observe that our model is capable of understanding and following unseen instructions for ASR and AST tasks (see Section 4.4.2).

#### 3.2.2 Speech-based QA from ASR Data

To enable general QA capabilities about speech, we create speech-based QA data from English ASR data by prompting a pre-trained LLM. As shown in Figure 2(a), we provide the transcript to an LLM4 and prompt it to generate a question-answer pair based on the provided context. During training, the text question and corresponding speech context are given as input, while the model is trained to predict the text answer. This approach has been used in prior studies Tang et al. (2024); Gong et al. (2024); Noroozi et al. (2024), but we scale it up to 20k hours of audio.

Footnote 4: [https://huggingface.co/google/gemma-2-27b-it](https://huggingface.co/google/gemma-2-27b-it)

The SQA data includes large volumes of real audio from diverse acoustic environments, helping mitigate overfitting. However, its limitation lies in the restricted diversity of generated questions, which are less varied than general-purpose instructions, and the answers tend to be short and simple. Additionally, the input always consists of a speech context and a text question, so the model often struggles with spoken instructions or interleaving speech-text inputs. For example, SALMONN Tang et al. (2024) is trained on SQA-style data. When we input a pure spoken instruction into the model, it tends to disregard the instruction and performs ASR instead.

#### 3.2.3 Mixed-Modal SFT Data with TTS

To overcome the limitations of SQA data and enable more flexible mixed-modal input, we create another type of data containing _mixed-modal interleaving speech-text inputs_. Specifically, our mixed-modal SFT data is generated by applying TTS to existing single-turn text SFT data, as illustrated in Figure 2(b). For each text SFT sample, we randomly select a subset of consecutive sentences from the user turn and replace them with synthesized audio. This mixed-modal input is then used for training, resulting in each user input containing one speech segment that may appear at the beginning, middle, or end. If all sentences are replaced by audio, the user input is pure speech without text.

This data has more flexible input formats than SQA, making speech and text inputs interchangeable. Instructions can now be conveyed through speech rather than being limited to text. Additionally, text SFT data typically covers more diverse instructions, and the responses maintain high-quality language and style. This contributes to the overall quality of our generated mixed-modal data. However, a potential limitation of this data type is that the audio is synthetic, reflecting only the limited acoustic conditions provided by the TTS model.5

Footnote 5: For simplicity, this work uses a single TTS model. Future work can explore multiple TTS models with diverse speakers and emotions to enhance robustness.

## 4 Experiments

### Experimental Setups

**Training data.** Table 1 shows the statistics of our training data mixture. Our text-only SFT data is from Nemotron's training data Adler et al. (2024), which consists of multi-turn conversations. During training, the loss is computed only on model turns but not on user turns. The ASR and AST datasets are the same as the training data of Canary Puvvada et al. (2024). ASR has four languages: En, De, Es, and Fr. AST has six language pairs: X-En and En-X, where X is any of De, Es, or Fr. For ASR, the text instruction is: "Transcribe the content to [language], with punctuations and capitalizations." For AST, the instruction is: "Translate

\begin{table}
\begin{tabular}{c c c c c} \hline \hline Task & Dataset & @Samples & @Hours & Sampling Ratio \\ \hline Text-only SFT & Nempton & 94.0k & N/A & 0.1500 \\ ASR, AST & Canary & 32.8M & 85k & 0.7556 \\ Speech-based QA & Canary Subset & 4.1M & 20k & 0.0378 \\ Mixed-modal SFT & Alpaca & 55.3k & 85 & 0.0189 \\  & Magrie & 254.5k & 461 & 0.0378 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of our training data mixture. When creating mini-batches, different types of data are sampled according to the ratio shown in the last column.

\begin{table}
\begin{tabular}{c l c c} \hline \hline Task & Dataset & Languages & Metric \\ \hline ASR & CommonVoice & En, De, Es, Fr & WER \\ \hline AST & FLEURS & En-De, En-Es, En-Fr & BLEUU \\ \hline \multirow{3}{*}{SAQ} & SPGI & \multirow{3}{*}{En} & \multirow{3}{*}{GPT Score} \\  & SOuAD2 & & \\ \cline{1-1}  & AIR-Bench & & \\ \hline \multirow{3}{*}{Speech-only} & IFEval & En & Prompt-level \\ \cline{1-1}  & \multirow{3}{*}{GSMK} & \multirow{3}{*}{En} & 5-shot Exact Match \\ \cline{1-1}  & & & (flexible extract) \\ \cline{1-1}  & IFEval & & \\ \cline{1-1}  & BBH & & Strict Accuracy \\ \cline{1-1}  & MMLU & & 5-shot Accuracy \\ \hline \hline \end{tabular}
\end{table}
Table 2: Summary of our evaluation datasets.

 

the [source language] content to [target language], with punctuations and capitalizations." The SQA data is synthesized using a subset of ASR data (20k hours in total) by prompting gemma-2-27b-it (see Section 3.2.2). The prompt template is provided in Appendix A. Lastly, we use a TTS model6 to synthesize mixed-modal SFT data on two public single-turn text SFT datasets, Alpaca Taori et al. (2023) and Magpie Xu et al. (2024)7.

Footnote 6: [https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/tts_en_multispeaker_fastpitchhifigan](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/tts_en_multispeaker_fastpitchhifigan)

**Model configs.** Our speech encoder is the pre-trained Canary encoder8 with 609M parameters. The modality adapter has 52M parameters and consists of two Conformer layers with hidden size 1024. No subsampling is applied after the speech encoder, resulting in a time resolution of 80 ms for the speech features. For LLM, we use Gemma with 2.5B parameters Mesnard et al. (2024). We begin with the base LLM and perform text-only SFT using our dataset, following Gemma's chat template (see Appendix D). This instruction-tuned LM is then used to initialize our VTBlender.9 The LoRA adapter of the LM has a rank of 32 and 36M parameters. It is applied to the linear layers in self-attention and feed-forward networks.

Footnote 7: [https://huggingface.co/datasets/Magpie-Align/Magpie-Gemma?Pro-200K-Filtered](https://huggingface.co/datasets/Magpie-Align/Magpie-Gemma?Pro-200K-Filtered)

Footnote 8: [https://huggingface.co/vroidai/canary-lb](https://huggingface.co/vroidai/canary-lb)

**Training configs.** Our model is implemented using the NeMo toolkit Kuchaiev et al. (2019) based on PyTorch Paszke et al. (2019). The multimodal data loading is based on Lhotse Zelasko et al. (2021); Zelasko et al. (2024). We use the Adam optimizer Kingma and Ba (2015) with a peak learning rate of \(1e-4\) and a cosine-annealing schedule. The weight decay is \(1e-3\). The model is trained for 100k steps; the first \(2500\) steps are the warmup stage. The final checkpoint is used for evaluation. We use 64 NVIDIA A100 GPUs (80GB) for training. The batch size per device is \(4\) for speech-related SFT data and \(1\) for text-only SFT data. The total training time is 20 hours.

**Evaluation setups.** Greedy decoding is performed for inference. Table 2 summarizes the five types of evaluation tasks and their metrics. For ASR and AST, we use standard multilingual benchmarks, namely Common Voice Ardila et al. (2020) and FLEURS Conneau et al. (2022). We normalize the text using Whisper's normalizers Radford et al. (2023) before computing the Word Error Rate (WER). For SQA, we create data from three sources and evaluate the quality of responses with OpenAI's GPT-4 API (see Appendix B). The first SQA test set consists of real audio recordings from SPGISpeech ASR data O'Neill et al. (2021) and synthetic text questions and answers from Nemotron-4 340B Adler et al. (2024). The second SQA test set is a spoken version of a widely used text QA benchmark, SQuAD 2.0 Rajpurkar et al. (2018), synthesized by NeMo FastPitch TTS10. The third test set is the "Speech Chat" subset from a public benchmark, AIR-Bench Yang et al. (2024). These three SQA test sets span diverse acoustic conditions across various domains, offering a comprehensive evaluation of our models. To assess spoken instruction-following capabilities, we synthesize a spoken IFEval dataset from the original text-based IFEval Zhou et al. (2023) using the aforementioned Neo Mo FastPitch model. Unlike SQA tasks, where the input consists of both an

\begin{table}
\begin{tabular}{l|c c c c|c c c c|c c c|c c c c} Model & \multicolumn{3}{c|}{ASR WER k} & \multicolumn{3}{c|}{En-X BLEU+} & X-E BLEU+ & Speech-based QA t & Speech + & \multicolumn{3}{c}{Text †} \\  & En & De & Es & Fr & De & Es & Fr & De & Es & Fr & SPGI & SQuAD2 & AIR & IFEval & GSMSK & IFEval & BBH & MMLU \\ \hline \multicolumn{13}{l}{_Prior studies_} \\ Whisper-w3 1.5B & 9.92 & 6.17 & 4.94 & 11.18 & N/A & 33.4 & 22.7 & 33.7 & & & N/A & & & & \\ SALMONN 7B & 20.84 & 40.83 & 37.47 & 36.78 & 18.0 & 17.1 & 27.8 & 5.1 & 7.1 & 3.3 & 0.778 & 0.597 & - & 0.147 & - & - & - & - \\ SALMONN 1SB & 17.07 & 44.08 & 28.47 & 38.52 & 19.0 & 18.5 & 29.1 & 6.5 & 3.6 & 3.8 & 0.778 & 0.604 & 6.16 & 0.113 & - & - & - & - \\ Qwen-Azudio 7B! & 8.78 & 7.67 & 5.65 & 9.49 & 24.8 & 18.9 & 27.7 & 30.7 & 22.2 & 29.6 & 0.810 & 0.656 & **7.24** & 0.140 & - & - & - & - \\ \hline \multicolumn{13}{l}{_Text-only baseline_} \\ Gemma 2.5B & & & & & & & N/A & & & & & **0.2479** & 0.2089 & **0.3324** & **0.3554** \\ \hline \multicolumn{13}{l}{_Ours_} \\ VTBlender 3B & **7.90** & **5.53** & **4.52** & **7.09** & **29.6** & **22.5** & **38.6** & **36.3** & **25.6** & **33.8** & **0.828** & **0.684** & 6.31 & **0.191** & 0.2358 & **0.2237** & 0.3003 & 0.3484 \\ \hline \end{tabular}
\end{table}
Table 3: Comparison of our method against prior studies. † Quven2-Audio has two versions: base and instruct models. The instruct model often generates additional text for ASR and AST, leading to much worse performance. Hence, we follow their official evaluation script to use the base model for ASR and AST, and the instruct model for others.

 

audio context and a text question, this task features speech-only user input. Finally, we select four text-only benchmarks to evaluate text-based performance: GSM8K for mathematical reasoning Cobbe et al. (2021), IFEval for instruction following Zhou et al. (2023), BBH for complex reasoning Suzgun et al. (2023), and MMLU for multi-task knowledge assessment Hendrycks et al. (2021). We use lm-evaluation-harness for text-only evaluation.11

Footnote 11: [https://github.com/EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)

### Main Results

Table 3 compares our VTBlender 3B against previous models that are publicly available:

* **Whisper-large-v3**Radford et al. (2023) is trained on 5 million hours of speech data for ASR and X-En AST.
* **SALMONN**Tang et al. (2024) is trained on a few thousand hours of audio SFT data covering diverse audio tasks.
* **Qwen2-Audio**Chu et al. (2024) is pre-trained on 520k hours of general audio data (including 370k hours of speech) and then post-trained with SFT and reinforcement learning. It is one of the state-of-the-art SpeechLMs, but the details of its training data have not been publicly released.

For ASR and AST, our VTBlender 3B achieves the best results among the evaluated models. In speech-based QA, our model outperforms others on SPGI and SQuAD 2.0, demonstrating its strong ability to recognize and comprehend speech. AIR-Bench, however, contains questions about speech attributes beyond linguistic content, such as emotion, speaker identity, and gender. While SALMONN and Qwen2-Audio explicitly incorporate such data in their training, our VTBlender does not utilize specialized data for these attributes. Consequently, VTBlender 3B falls behind Qwen2-Audio 7B in this domain, although it still outperforms SALMONN 13B.

Our VTBlender 3B also outperforms other 7B or 13B models on Spoken IFEval, showing that it better follows spoken instructions.

Compared to the Gemma LM from which our model is initialized, our model shows comparable results on text-only benchmarks, indicating that it successfully preserves the original text capabilities.

### Ablation Studies

Our VTBlender is trained with single-stage joint speech-text SFT in which the LM is updated with LoRA adapters. To investigate the impact of these strategies, we conduct three ablation studies and present our results in Table 4.

**Joint SFT vs. speech-only SFT.** B1 is the same model trained exclusively on speech-related SFT data, without incorporating text-only data. When compared to our VTBlender 3B, B1 achieves similar performance on speech tasks but performs significantly worse on text-only benchmarks. On GSM8K and BBH, B1's performance is nearly zero, and on MMLU, it approaches random chance (25%) given the four-choice format of the questions. This highlights that using LoRA alone does not prevent catastrophic forgetting of the model's original text capabilities. By incorporating both text and speech SFT data, our model preserves its original text performance while excelling on speech tasks.

**LoRA vs. frozen LM.** Our VTBlender 3B updates the LM using LoRA adapters. As discussed in Section 2, a common strategy for preserving text-only performance is to freeze the LM. To compare, we froze the LM backbone and trained another model,

\begin{table}
\begin{tabular}{l|c c c|c c c|c c c|c c c|c c c c} \hline \hline Model & \multicolumn{6}{c|}{ASR WER \(\downarrow\)} & \multicolumn{6}{c|}{En-X BLEU \(\uparrow\)} & \multicolumn{6}{c|}{X-En BLEU \(\downarrow\)} & \multicolumn{6}{c|}{Speech-based QA \(\uparrow\)} & \multicolumn{6}{c}{Speech \(\uparrow\)} & Text \(\uparrow\) \\  & En & De & Es & Fr & De & Es & Fr & De & Es & Fr & SPGI & SQUaD2 & AIR & IFEval & GSM8K & IFEval & BBH & MMLU \\ \hline \multicolumn{9}{l}{_Text-only baseline_} \\ Gemma 2.5B & \multicolumn{9}{c|}{N/A} & & & & & & & & & & & & & & & **0.2479** & 0.2089 & **0.3324** & **0.3554** \\ \hline \multicolumn{9}{l}{_Ours_} \\ VTBlender 3B & 7.90 & 5.53 & 4.52 & **7.09** & 29.6 & 22.5 & 38.6 & **36.3** & **25.6** & **33.8** & **0.828** & 0.684 & **6.31** & **0.191** & 0.2358 & **0.2237** & 0.3003 & 0.3484 \\ B1 & **7.83** & **5.50** & **4.36** & 7.11 & **30.6** & **22.6** & **38.9** & **36.3** & 24.9 & 33.7 & **0.828** & **0.687** & 6.26 & 0.181 & 0.0243 & 0.1294 & 0.0023 & 0.2457 \\ B2 & 9.96 & 8.77 & 7.03 & 9.49 & 21.5 & 18.3 & 29.3 & 31.3 & 21.9 & 29.3 & 0.028 & 0.121 & 3.22 & 0.150 & **0.2479** & 0.2089 & **0.3324** & **0.3554** \\ B3 & 8.92 & 6.67 & 5.39 & 7.97 & 26.3 & 20.5 & 34.4 & 34.5 & 23.2 & 31.4 & 0.666 & 0.529 & 5.29 & 0.135 & 0.0675 & 0.1867 & 0.2622 & 0.2586 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Ablation studies. Our VTBlender uses single-stage joint SFT where the LM is partially updated with LoRA. “$1” is trained with speech-only SFT. “$2” uses a frozen LM with speech-only SFT. ”$3” is trained in two stages with speech-only SFT, where the first stage freezes the LM and the second stage updates the LM with LoRA.

 

82, on all speech-related SFT data--excluding text-only data, since the LM remains unchanged. B2 performs significantly worse than our VTBlender in all speech benchmarks, indicating that the model struggles to understand speech inputs unless the LM itself is also adapted to speech.

**Two-stage vs. single-stage training.** Previous results of B1 and B2 indicate that freezing the LM negatively impacts speech performance, while updating the LM with LoRA from the beginning leads to a loss of text performance. Then, a natural alternative is to use a two-stage training process with speech SFT data. In the first stage, the LM is frozen, and the remaining components are trained for 100k steps. In the second stage, the LM is updated with LoRA for an additional 15k steps. This model is referred to as B3. Compared to B2 (with a frozen LM), B3 shows much better performance on most speech tasks. However, B3 still experiences significant degradation on text-only evaluations. On GSM8K, its performance is nearly zero, and on MMLU, it approaches random chance. These results demonstrate that LoRA alone is insufficient to preserve text-only performance, even when the model is updated for a limited number of steps.

Our ablation studies demonstrate that the proposed joint speech-text SFT is essential for preserving text-only performance while delivering strong results on speech tasks.

### Demonstrations

This section provides examples to demonstrate the various capabilities of our proposed VTBlender 3B.

#### 4.4.1 Standard Speech Tasks

Figure 6 in Appendix E.1 shows that our model performs well for ASR, AST, and SQA tasks.

#### 4.4.2 Generalization to Unseen Conditions

**Multi-turn mixed-modal chat.** As introduced in Section 3.2, our text-only SFT data has multiple turns, whereas the speech-related data has only one turn. Through joint SFT, our model can generalize to multi-turn mixed-modal chat (see Figure 1).

**ASR w/ unseen prompt.** As described in Section 3.2.1 and Section 4.1, the ASR instruction is fixed during training using the verb "transcribe". In Figure 3(a), our model understands a different verb phrase "write down" and performs ASR correctly.

**AST in unseen direction.** Our AST training data consists of X-En and En-X directions where X is one of De, Es, and Fr, but the model can also perform other directions like Es-Fr as shown in Figure 3(b).

**Controlling output format.** Figure 5 shows three examples where we can specify the output text style or format, which can benefit downstream tasks. It works for different speech tasks like ASR or SQA. This demonstrates that our VTBlender achieves good instruction-following capabilities based on mixed-modal input.

Appendix E.2 presents examples of other tasks,

Figure 4: Generalization to unseen instructions.

Figure 5: Output style and format can be controlled.

 

including contextual biasing ASR, math/coding based on information provided in both speech and text inputs, and SQA based on multi-speaker audio despite being trained on single-speaker data only.

## 5 Conclusion

We propose a novel single-stage joint speech-text SFT approach for training SpeechLMs using LoRA adapters. This method simplifies the training process, preserves the text-only performance of the LLM backbone, and achieves excellent speech understanding capabilities. Specifically, we combine multi-turn text-only SFT data with single-turn speech-related SFT data during training. To extend beyond speech-based QA tasks, we propose a novel data generation method that can create mixed-modal interleaving speech-text inputs. Our model achieves excellent performance across various speech benchmarks while retaining performance on text-only benchmarks. Our 3B model even outperforms previous 7B or 13B SpeechLMs on most evaluated benchmarks. Furthermore, our model exhibits emergent capabilities in handling previously unseen instructions and multi-turn mixed-modal conversations. We will publicly release our codebase and pre-trained models to advance research in SpeechLMs.

### Limitations

We primarily employ small-sized LMs with a few billion parameters. While our model demonstrates strong performance across various benchmarks, its overall capacity may be constrained compared to larger models, particularly in terms of world knowledge and complex reasoning abilities.

Our training data and tasks are also limited. The training data focuses on linguistic content and does not encompass specialized speech tasks, such as spoken language understanding, speaker recognition or verification, multi-speaker ASR, or speech enhancement. This restricts the applicability of the current model to certain use cases. Furthermore, our efforts are concentrated on human speech, without addressing general audio processing.

The text pre-training data of Gemma is not publicly released, which might raise concerns. Due to license issues, some of the speech training data cannot be directly released. Instead, we provide statistics about those data and describe the details about our training procedure. For SQA and mixed-modal SFT, we plan to release the data generation scripts.

We introduce speech capabilities at the SFT stage without involving pre-training. Additionally, we do not utilize reinforcement learning from human feedback (RLHF), which may result in hallucinations or unexpected behavior in the model's output. Therefore, it is important to exercise caution and thoroughly verify the output when using this model.

### Broader Impacts and Ethics

We adhere to the ACL ethics policy and there is no violation of privacy in the experiments. We plan to publicly release the data generation scripts, training code, and pre-trained model weights, which can benefit a broader audience within the research community.

## References

* Aglerwal et al. [2024] Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. 2024. Nemotron-4 340b technical report. _arXiv preprint arXiv:2406.11704_.
* Ardila et al. [2020] Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In _Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020_, pages 4218-4222. European Language Resources Association.
* Borosy et al. [2023] Zalan Boriso, Raphael Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matthew Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliascchi, and Neil Zeghidour. 2023. Audiolm: A language modeling approach to audio generation. _IEEE ACM Trans. Audio Speech Lang. Process._, 31:2523-2533.
* Chen et al. [2022] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. _IEEE J. Sel. Top. Signal Process._, 16(6):1505-1518.
* Chen et al. [2024] Zhehuai Chen, He Huang, Andrei Andrusenko, Oleksii Hrinchuk, Krishna C. Puvvada, Jason Li, Subhankar Ghosh, Jagadeesh Balam, and Boris Ginsburg. 2024. SALM: Speech-Augmented Language Model with in-Context Learning for Speech Recognition and Translation. In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 

2024, Seoul, Republic of Korea, April 14-19, 2024_, pages 13521-13525. IEEE.
* Chu et al. (2024) Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-audio technical report. _arXiv preprint arXiv:2407.10759_.
* Cieri et al. (2004) Christopher Cieri, David Miller, and Kevin Walker. 2004. The fisher corpus: a resource for the next generations of speech-to-text. In _Proceedings of the Fourth International Conference on Language Resources and Evaluation, HEC 2004, May 26-28, 2004, Lisbon, Portugal_. European Language Resources Association.
* Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, et al. 2021. Training verifiers to solve math word problems. _arXiv preprint arXiv:2110.14168_.
* Conneau et al. (2022) Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. FLEURS: few-shot learning evaluation of universal representations of speech. In _IEEE Spoken Language Technology Workshop, SLT 2022, Doha, Qatar, January 9-12, 2023_, pages 798-805. IEEE.
* Das et al. (2024) Nilaksh Das, Saket Dingliwal, Srikanth Ronanki, Rohit Paturi, Zhaocheng Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, Xilai Li, Karel Mundnich, Monica Sunkara, Sundararajan Srinivasan, Kyu J. Han, and Katrin Kirchhoff. 2024. Speechverse: A large-scale generalizable audio language model. _CoRR_, abs/2405.08295.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Garciaes Diaz-Munio et al. (2021) Goncal V. Garces Diaz-Munio, Joan-Albert Silvestre-Cerda, Javier Jorge, Adria Gimenez Pastor, Javier Iranzo-Sanchez, Pau Baquero-Arnal, Nahuel Rosello, Alejandro Perez-Gonzalez de Martos, Jorge Civera, Albert Sanchis, and Alfons Juan. 2021. Europarl-ASR: A Large Corpus of Parliamentary Debates for Streaming ASR Benchmarking and Speech Data Filtering/Verbatimization. In _Proc. Interspeech 2021_, pages 3695-3699.
* Fathullah et al. (2024) Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. 2024. AudioChatLlama: Towards general-purpose speech abilities for LLMs. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 5522-5532, Mexico City, Mexico. Association for Computational Linguistics.
* Galvez et al. (2021) Daniel Galvez, Greg Diamos, Juan Torres, Keith Achorn, Juan Felipe Ceron, Anjalti Gopi, David Kanter, Max Lam, Mark Mazumder, and Vijay Janapa Reddi. 2021. The people's speech: A large-scale diverse english speech recognition dataset for commercial usage. In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_.
* Godfrey et al. (1992) John J. Godfrey, Edward Holliman, and Jane McDaniel. 1992. SWITCHBOARD: telephone speech corpus for research and development. In _1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP '92, San Francisco, California, USA, March 23-26, 1992_, pages 517-520. IEEE Computer Society.
* Gong et al. (2024) Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James R. Glass. 2024. Listen, think, and understand. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Gulati et al. (2020) Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augmented transformer for speech recognition. In _21st Annual Conference of the International Speech Communication Association, Interspeech 2020, Virtual Event, Shanghai, China, October 25-29, 2020_, pages 5036-5040. ISCA.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. _IEEE ACM Trans. Audio Speech Lang. Process._, 29:3451-3460.
* Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Kuchnaive et al. (2020) Oleksii Kuchnaive, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Beliaev, Vitaly Lavrukhin, Jack Cook, Patricice Castonguay, Mariya Popova, Joelyn Huang, 

and Jonathan M. Cohen. 2019. Nemo: a toolkit for building AI applications using neural modules. _CoRR_, abs/1909.09577.
* Lakhotia et al. (2021) Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. On generative spoken language modeling from raw audio. _Transactions of the Association for Computational Linguistics_, 9:1336-1354.
* Lin et al. (2024) Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024. VILA: on pre-training for visual language models. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024_, pages 26679-26689. IEEE.
* Maiti et al. (2024) Soumi Maiti, Yifan Peng, Shukjae Choi, Jee-Weon Jung, Xuankai Chang, and Shinji Watanabe. 2024. Voxtlm: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks. In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024_, pages 13326-13330. IEEE.
* Mesnard et al. (2023) Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgenac Riviere, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Leonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barna, Alex Botecy, Alex Castro-Ros, Ambrose Slone, Amelie Heliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clement Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Nolan, Geng Yan, George Tucker, George-Cristian Muraru, Grigory Rozddestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, and et al. 2024. Gemma: Open models based on gemini research and technology. _CoRR_, abs/2403.08295.
* Noroozi et al. (2024) Vahid Noroozi, Zhehuai Chen, Somshubra Majumdar, Steve Huang, Jagadeesh Balam, and Boris Ginsburg. 2024. Instruction data generation and unsupervised adaptation for speech language models. _arXiv preprint arXiv:2406.12946_.
* O'Neill et al. (2021) Patrick K O'Neill, Vitaly Lavrukhin, Somshubra Majumdar, Vahid Noroozi, Yuekai Zhang, Oleksii Kuchaiev, Jagadeesh Balam, Yuliya Dovzhenko, Keenan Freyberg, Michael D Shulman, et al. 2021. Spgispech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition. _arXiv preprint arXiv:2104.02014_.
* OpenAI (2023) OpenAI. 2023. GPT-4 technical report. _CoRR_, abs/2303.08774.
* Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In _2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015_, pages 5206-5210. IEEE.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junje Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 8024-8035.
* Paul and Baker (1992) Douglas B Paul and Janet Baker. 1992. The design for the Wall Street Journal-based CSR corpus. In _Proc. Workshop on Speech and Natural Language_.
* Peng et al. (2023) Yifan Peng, Jinchuan Tian, Brian Yan, Dan Berrebbi, Xuankai Chang, Xinjiang Li, Jiatong Shi, Siddhant Arora, William Chen, Roshan S. Sharma, Wangyou Zhang, Yui Sudo, Muhammad Shakeel, Jee-Weon Jung, Soumi Maiti, and Shinji Watanabe. 2023. Reproducing whisper-style training using an open-source toolkit and publicly available data. In _IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2023, Taipei, Taiwan, December 16-20, 2023_, pages 1-8. IEEE.
* Pratap et al. (2020) Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. MLS: A large-scale multilingual dataset for speech research. In _Interspeech 2020, 21st Annual Conference of the Interspeech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020_, pages 2757-2761. ISCA.
* Puvvada et al. (2024) Krishna C. Puvvada, Piotr Zelasko, He Huang, Oleksii Hirinchuk, Nithin Rao Koluguri, Kunal Thawan, Somshubra Majumdar, Elena Rastorgueva, Zhehuai Chen, Vitaly Lavrukhin, Jagadeesh Balam, and Boris Ginsburg. 2024. Less is more: Accurate speech recognition & translation without web-scale data. In _Interspeech 2024_, pages 3964-3968.
* Radford et al. (2023) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavevey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_, volume 202 of _Proceedings of Machine Learning Research_, pages 28492-28518. PMLR.
* Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SUqAD. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 784-789, Melbourne, Australia. Association for Computational Linguistics.

 

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Sorich, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sotitauux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross McIlroy, Melvin Johnson, Johan Schlakwyk, Eli Collins, Eliza Rutherford, Eric Moreira, Kaereen Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Akanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, and et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _CoRR_, abs/2403.05530.
* Rubenstein et al. (2023) Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalan Borosso, Felix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Padfield, James Qin, Danny Rozenberg, Tara N. Sainath, Johan Schalkwyk, Matthew Sharifi, Michelle Tadmor Ramanovich, Marco Tagliasaccchi, Alexandru Tudor, Mihailoj Usimirovic, Daminer Vincent, Jiahui Yu, Yonggiang Wang, Vicky Zagats, Neil Zeghidour, Yu Zhang, Zhishuai Zhang, Lukas Zilka, and Christian Havno Frank. 2023. Audiopalm: A large language model that can speak and listen. _CoRR_, abs/2306.12925.
* Suzgun et al. (2023) Mirac Suzgun, Nathan Scales, Nathanaael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 13003-13051, Toronto, Canada. Association for Computational Linguistics.
* Tang et al. (2024) Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. SALMONN: towards generic hearing abilities for large language models. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Taori et al. (2023) Rohan Taori, Isaac Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumi B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. _Stamford Center for Research on Foundation Models. https://ccfm. stanford. edu/2023/03/13/alpaca. html_, 3(6):7.
* Wang et al. (2021) Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaithanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 993-1003, Online. Association for Computational Linguistics.
* Wang et al. (2022a) Chen Wang, Minpeng Liao, Zhongqiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong, and Jiajun Zhang. 2022a. BLSP: bootstrapping language-speech pre-training via behavior alignment of continuation writing. _CoRR_, abs/2309.00916.
* Wang et al. (2023b) Mingqiu Wang, Wei Han, Izhak Shafran, Zelin Wu, Chung-Cheng Chiu, Yuan Cao, Nanxin Chen, Yu Zhang, Hagen Soltau, Paul K. Rubenstein, Lukas Zilka, Qian Yu, Golan Pundak, Nikhil Siddharthka, Johan Schalkwyk, and Yonghui Wu. 2023b. SLM: bridge the thin gap between speech and text foundation models. In _IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2023, Taipei, Taiwan, December 16-20, 2023_, pages 1-8. IEEE.
* Wu et al. (2024)Junkai Wu, Xulin Fan, Bo-Ru Lu, Xilin Jiang, Nima Mesgaranti, Mark Hasegawa-Johnson, and Mari Ostendorf. 2024. Just ASR+ LLM? A Study on Speech Large Language Models' Ability to Identify And Understand Speaker in Spoken Dialogue. In _2024 IEEE Spoken Language Technology Workshop (SLT)_, pages 1137-1143. IEEE.
* Xu et al. (2024) Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024. MagpiE: Alignment data synthesis from scratch by prompting aligned lms with nothing. _CoRR_, abs/2406.08464.
* Yang et al. (2024) Qian Yang, Jin Xu, Wennvi Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. 2024. AIR-bench: Benchmarking large audio-language models via generative comprehension. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1979-1998, Bangko, Thailand. Association for Computational Linguistics.
* Zelasko et al. (2024) Piotr Zelasko, Zhehuai Chen, Mengru Wang, Daniel Galvez, Oleksii Hrinchuk, Shuoyang Ding, Ke Hu, Jagadeesh Balam, Vitaly Lavvukhin, and Boris Ginsburg. 2024. Emmett: Efficient multimodal machine translation training. _arXiv preprint arXiv:2409.13523_.
* Zelasko et al. (2021) Piotr Zelasko, Daniel Povey, Jan "Yenda" Trmal, and Sanjeev Khudanpur. 2021. Lnotes: a speech data representation library for the modern deep learning ecosystem. _CoRR_, abs/2110.12561.
* Zhang et al. (2023) Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, Andrew Rosenberg, Rohit Prabhavalkar, Daniel S. Park, Parisa Haghani, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Ramabhadran, 

Tara N. Sainath, Pedro J. Moreno, Chung-Cheng Chiu, Johan Schalkwyk, Francoise Beaufays, and Yonghui Wu. 2023. Google USM: scaling automatic speech recognition beyond 100 languages. _CoRR_, abs/2303.01037.
* Zhou et al. (2023) Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. _arXiv preprint arXiv:2311.07911_.

## Appendix A Prompt Template for SQA Data Generation

As introduced in Section 3.2, we generate speech-based QA training data from ASR data by prompting an LLM. The prompt template is:

I will provide you with several sentences. Please generate **one** question that is closely related to the content of these sentences, along with a corresponding answer. Ensure that your answer is **accurate** and clearly stated. Write your output in a single line in json format:

["question": "xxx", "answer": "xxx"]

If the question and answer contain a double quote, insert backslash before it to ensure the output can be loaded by python library 'json.loads()'. Do not add unnecessary backslash for symbols like dollar $, ampersand &, etc. However, if the sentences are meaningless, please return **none** in those fields.

Here are the sentences:

PROVIDE THE TRANSCRIPT HERE.

## Appendix B Prompt Template for GPT Scoring

As described in Section 4.1, we use OpenAI's GPT APIs to evaluate SQA tasks. For the public benchmark AIR-Bench Yang et al. (2024), we follow the standard evaluation script with gpt-4-0125-preview. For the other two test sets, APGI and SQuAD2, we use a more recent API, gpt-4o-2024-08-06, with the following prompt template.

The system message is:

You are an expert evaluator of questionnaire performance. Your task is to evaluate the "correctness" and "redundancy" of an AI assistant's response to a user question based on the provided context. Provide your output following the schema provided. Here is a description of the required fields:

- correctness_score: either 0 or 1

- Score 0: The AI assistant's answer is incorrect based on the provided context, or the AI assistant's answer simply copies the context.

- Score 1: The AI assistant's answer is correct based on the provided context, and it does not simply copy the context.

- correctness_evaluation: explanation of your score for "correctness".

- redundancy_score: an integer score between 1 and 10, where a higher score indicates that the AI assistant's answer copies more redundant information from the context.

- redundancy_explanation: explanation of your score for "redundancy".

The input is:

[Question] QUESTIONHERE

[Context] CONTEXT HREE

[Start of Reference Answer] REFERENCE ANSWER HERE

[End of Reference Answer]

[Start of Assistant's Answer] MODEL RESPONSE HERE

[End of Assistant's Answer]

Finally, we report the average correctness score for all test samples.

## Appendix C Training Data and Licenses

Our use of various data is consistent with their intended use. The data has been commonly used in this area, which does not contain personally identifying information or offensive content. The Canary training data Puvvada et al. (2024) consists of the following subsets:

* LibriSpeech Panayotov et al. (2015): CC BY 4.0
* Fisher Corpus Cieri et al. (2004): LDC
* Switchboard Godfrey et al. (1992): LDC
* WSJ Paul and Baker (1992): LDC
* National Speech Corpus12: Singapore Open Data Licence 

* VCTK13: CC BY 4.0
2. VocPopuli Wang et al. (2021): Attribution-NonCommercial 4.0 International Footnote 13: [https://huggingface.co/datasets/CSTR=Edinburgh/vctk](https://huggingface.co/datasets/CSTR=Edinburgh/vctk)
3. Europarl-ASR Diaz-Munio et al. (2021): License is described here: [https://www.mllp.upv.es/git-pub/ggarces/Europarl-ASR/#licence](https://www.mllp.upv.es/git-pub/ggarces/Europarl-ASR/#licence)
4. Multilingual LibriSpeech Pratap et al. (2020): CC BY 4.0 Mozilla Common Voice14: CC0-1.0

Footnote 14: [https://commonvoice.mozilla.org/en/datasets](https://commonvoice.mozilla.org/en/datasets)
5. People's Speech Galvez et al. (2021): CCBY-SA and CC-BY 4.0
6. Other in-house data
7. Onto-Ont 

* [10] S. K. Goyal, A. M. R. Smith, and J. L. Rosner. The role of the social network in a social context: an overview. In _Proceedings of the 27th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 35-46, 2009.
* [11] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6812_, 2014.
* [12] C. V. Mnih et al. Learning to predict the distribution of words in words of natural language. _Journal of Machine Learning Research_, 1(1):1-10, 2014.
* [13] H. N. Srivastava and I. Sutskever. Training deep neural networks with applications to machine learning. _CoRR_, abs/1412.03080, 2014.
* [14] Y. Bengio. Generative adversarial nets. In _Advances in Neural Information Processing Systems_, volume 1, pages 1-10, 2015.
* [15] W. Teh and T. Darrell. Deep residual learning for image recognition. In _NIPS Conference on Artificial Intelligence_, pages 1-10, 2015.
* [16] B. E. Hinton and R. Fergus. Visualizing features from tiny images. _IEEE transactions on pattern analysis and machine intelligence_, 1(1):1-1, 2004.

[MISSING_PAGE_POST]

 Workshops (CVPR'01)_, pages 1-10, 2 

* [3] D. A. Bostrov, M. V. K. S. Baburov, and J. E. L. Smith, "A new approach to the design of a single-speaker data set for the use of a single speaker identification system in the presence