VITA-Audio: Fast Interleaved Audio-Text Token Generation for Efficient Large Speech-Language Model
Authors: Zuwei Long
Yunhang Shen
Chaoyou Fu
Heting Gao
lijiang Li
Peixian Chen
Mengdan Zhang
Hang Shao
Jian Li
Jinlong Peng
Haoyu Cao
Ke Li
Rongrong Ji
Xing Sun
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Models
Large Speech Language Models
Primary area: applications

Abstract: With the growing requirement for natural human-computer interaction, speech-based systems receive increasing attention as speech is one of the most common forms of daily communication. However, the existing speech models still experience high latency when generating the first audio token during streaming, which poses a significant bottleneck for deployment. To address this issue, we propose VITA-Audio, an end-to-end large speech model with fast audio-text token generation. Specifically, we introduce a lightweight Multiple Cross-modal Token Prediction (MCTP) module that efficiently generates multiple audio tokens within a single model forward pass, which not only accelerates the inference but also significantly reduces the latency for generating the first audio in streaming scenarios. In addition, a four-stage progressive training strategy is explored to achieve model acceleration with minimal loss of speech quality. To our knowledge, VITA-Audio is the first multi-modal large language model capable of generating audio output during the first forward pass, enabling real-time conversational capabilities with minimal latency. VITA-Audio is fully reproducible and is trained on open-source data only. Experimental results demonstrate that our model achieves an inference speedup of 3~5x at the 7B parameter scale, but also significantly outperforms open-source models of similar model size on multiple benchmarks for automatic speech recognition (ASR), text-to-speech (TTS), and spoken question answering (SQA) tasks.

VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model
Zuwei Long".i, Yunhang Shen 1. Chaoyou Fu?** Heting Gao', Lijiang Li
Peixian Chen', Mengdan Zhang', Hang Shao', Li', Jinlong Peng'
Haoyu Cao', Ke Li', Rongrong Ji", Xing Sun
Tencent Youtu Lab, Naajjnng University, "Xiamen University
Equal Contribution
Project Leader "oorresponding Author
https://github. com/ www.cooo/////// MEITTAMLLMM/VIAAAAAE Audio
Abstract
With the growing requirement for natural human-computer interaction, speech- based systems receive increasing attention as speech is one of the most common forms of daily communication. However, the existing speech models still experi- ence high latency when generating the first audio token during streaming, which poses a significant bottleneck for deployment. To address this issue, we propose VITA-Audio, an end-to-end large speech model with fast audio-text token gen- eration. Specifically, we introduce lightweight Multiple Cross-modal Token Prediction (MCTP) module that efficiently generates multiple audio tokens within a single model forward pass, which not only accelerates the inference but also sig- nificantly reduces the latency for generating the first audio in streaming scenarios. In addition, four-stage progressive training strategy is explored to achieve model acceleration with minimal loss speech quality. To our knowledge, VITA-Audio is the first multi-modal large language model capable of generating audio output during the first forward pass, enabling real-time conversational capabilities with minimal latency. VITA-Audio is fully reproducible and is trained open-source data only. Experimental results demonstrate that our model achieves an inference speedup of ~ 5x at 7B parameter scale, but also significantly outperforms open- source models of similar model size on multiple benchmarks for automatic speech recognition (ASR), text-to-speech (TTS), and spoken question answering (SQA).
Introduction
Real-time speech systems have become a crucial research focus for enabling natural dialogue. Traditional speech systems predominantly adopt modular design that decomposes real-time speech processing into three discrete components: automatic speech recognition (ASR), large language models (LLMs), and text-to-speech (TTS) [54, 32, 69]. However, this cascaded approach suffers from cumulative latency, loss of paralinguistic information (e.g.. emotional prosody, rhythm) during modality conversion, and eтror accumulation between modules, substantially lowering the practical utility of cascaded architectures in real-time interactive scenarios. To address the limitations of traditional methods, many recent studies have adopted an end-to-end approach to handle inputs and outputs of the model [58, 10, 23). These methods directly input speech into L.LMs through an audio encoder and then synthesize speech response with discrete tokens [66] or LLM hidden states [56].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Text Token
days
bi
*H
wook-
Audio Token
(a) The visualization of the attention maps between text tokens and audio tokens.
Ei xinggi
Text Token
+:11
GHI SON
ff tyi xing)
Masked Text Token
No.0 StF1 -0
A:11
SIM
Audio Token
(bu za)
88H
30:00
(b) The transcription results of the generated audio into text under different attention masks.
Figure 1: (a) The audio sequence generated by the speech model exhibits a strong correlation with the corresponding text tokens. (b) With irrelevant text tokens being masked out, the model is still able to generate the correct audio, and the pronunciation remains contextually appropriate. However, all text tokens masked, the model outputs random audio. This suggests that the hidden states from the LLM include sufficient contextual information for generating the corresponding audio tokens. Consequently, the mapping from text hidden states to audio tokens is accomplished using relatively simple modules, without the need for the extensive semantic modeling typically required by LLMs.
While existing end-to-end speech models generate output in streaming fashion to reduce the response latency, their first token delay is still high. Specifically, current speech models cannot directly deliver the first streaming audio chunk upon completing the first LLM forward pass, i.., decoding the first text token. the applications requiring high real-time performance, this delay poses significant bottleneck to the deployment of LLMs for speech processing. This prompts pertinent question:
How
achieve more real-time audio generation within end-to-ent speech models?
To explore this issue, we visualized the hidden states the final decoder layer of the speech model. As shown in Fig. Ia, the audio tokens generated by the speech model show increased attention to the text tokens they correspond to. the generation audio tokens progresses, the text tokens attended by the new audio token advance accordingly. This finding is also reported in many literature on attention-basee speech systems [7, 34]
In Fig. 1b, we show a Chinese sentence with a homograph "IF" as an example. The pronunciation of this character can be /xing/ or /hang/ depending on its context. In Chinese, 'yihang' emphasizes spatial arrangement (e.g. row/line of egrets), while 'yixing' focuses on the concept of group unit (e.g. group/party of people). Our speech model correctly decides the character's 'ss pronunciation be the former, given the hidden states of historical inputs. We then modify the model inference process by masking out all text hidden states, except the one corresponding the token "-fff"(yixing" before generating its corresponding audio tokens. This modification prevents subsequently generated audio tokens from directly attending to other text tokens, although they can still attend to previously generated audio tokens. We find that the subsequently generated audio tokens accurately produce the sounds as "yixing", which remains contextually appropriate. The same observation also holds for other non-homograph tokens. We therefore argue that the hidden states from the LLM include sufficient contextual information for generating its corresponding audio tokens, and attending to additional texts is unnecessary. Finally, experiment with masking out all text tokens. This time, the generated audio fails to align with its text and sounds like random non-speech babbles even though the model has access to the previously generated audio tokens.
These findings suggest that the speech model learns to primarily focus on the small span of cor- responding text hidden states without heavily modeling the semantic space of the entire text and

---

Table I: Comparison of recent speech models, VITA-Audio leverages the hidden state enhance model performance, adopts an end-to-end architecture, and achieves zero audio token delay.
Model
Audio Token Delay
Leveraging Hidden States
End-to-End
Freeze-Omni
[56]
Text Length
MinMo
[9]
5
Mini-Omni
[58]
7
X
Moshi
[19]
X
GLM-4-Vort
[66]
13
X
LUCY
[28]
X
Audio
0
audio sequence. This discovery instills confidence that we can learn the simple mapping relationship between text hidden states and audio tokens with relatively simple modules and without relying on the extensive semantic modeling of L.Ms..
In this paper, we introduce VITA-Audio, lightweight framework that uses separate efficient modules, named Multiple Cross-modal Token Prediction (MCTP), to efficiently generate audio responses from text embeddings and LLM hidden states. This approach enables obtaining both text tokens and an audio chunk a single LLM forward pass, achieving zero delay in audio tokens. A comparison of the delay of the first audio token is presented in Table 1 where define "audio token delay" as the number of additional LLM forward steps required to generate the first audio token after the first L.M forward pass. We distinguish this delay from "audio generation delay" which is the number of additional LLM forward passes to generate a meaningful and consistent chunk of audio. VITA-Audio has both zero audio token delay and zero audio generation delay.
To this end, through a four-stage progressive training strategy, we construct set of lightweight yet powerful MCTP modules, which predict 10 audio tokens directly from historical inputs and LLM hidden states without requiring additional LLM forward passes, thus significantly enhancing the model's 's inference speed without sacrificing audio quality.
In summary, our main contributions are follows.
We introduce VITA-Audio, the first end-to-end speech model capable of generating audio during the first forward pass. Leveraging audio generation without relying on extensive text semantic modeling capabilities, VITA-Audio designs lightweight MCTP modules to generate decodable audio token chunks with zero audio token delay, thus overcoming the real-time limitations in traditional cascaded models and existing end-to-end methods.
VITA-Audio achieves remarkable end-to-end inference acceleration by generating ten audio tokens single forward pass, resulting in 5x speedup when implemented on a 7B LLM while preserving the ability of high-quality speech synthesis.
We fully release VITA-Audio to the open-source community. Although VITA-Audio is trained open-source data only, comprehensive evaluations reveal that VITA-Audio achieves the state-of-the-ar performance on multiple benchmarks for ASR, TTS, and SQA tasks, outperforming existing models in both efficiency and accuracy, especially the open- source ones with similar parameter scale, therefore setting a new standard for real-time speech-to-speech models.
Related Work
Large language models (LLMs) have revolutionized human-computer interaction with advanced natural language processing. Extending these capabilities to speech-a a natural communication modality- has become key research focus. Traditional speech interaction systems [54, 32, 69] typically adopt a cascaded architecture, combining separate ASR, LLM, and TTS modules. However, this approach suffers from increased latency, loss of paralinguistic cues, and error propagation.
Recent work [24, 49] has improved integration by connecting audio encoders L.MSs via trainable adapters, but still relies independent TTS modules. To address this, some methods incorporate
3

---

LLM hidden states into audio decoders. For example, Llama- Omni [23] uses non autoregressivve transformer to predict audio tokens from upsampled LLM states, while Freeze-omni [56] freezes the L.M and combines autoregressive and non-autoregressive decoders. Minmo [9] integrates language model with Cosy Voice2 [22] for mixed speech-text processing.
End-to-end models further unify TTS within LLMs, enabling direct text and speech generation. These models follow either parallel interleaved audio-text modeling. In the parallel modeling paradigm, the model uses different heads to process hidden states, generating both text and multiple audio tokens [10, 19]. Since the input the LLM is altered during autoregression, maintaining the original capabilities of the LLM presents significant challenges. To perform inference without targe-scale pretraining, Mini-Ommi [58] and LUCY [28] rely on batch parallel decoding to preserve the inference capability of the LLM.
Compared to parallel-paradigm models, intereeae-ppardignn models appear better preserve lan- guage capabilities, as suggested by the performance comparison on spoken question-answet -nnweringg benchmarks [66]. We attribute this difference the fact that parallel-paradigm models use an av- erage of text and audio representations as input, which significantly diverges from the inputs used during pretraining. However, interleave-parddigm models face latency issue due to their sequential prediction of audio tokens, especially when the audio token rate is high.
VITA-Audio leverages the strengths of these architectures by adopting the interleaved modeling paradigm and introducing MCTP for audio generation. The former maximally preserves the LLM's language ability, and the latter reduces inference latency by generating multiple audio tokens in single forward pass.
3 Method
3.1
Overview
As illustrated in Fig. 2, VITA-Audio consists of four major components: an audio encoder, an audio decoder, large language model backbone, and a of Cross-modal Token Prediction (MCTP) modules. The audio signal is first processed by the audio encoder, whose output is then fed into the LLM for further processing. During each forward pass, the LLM alternately generates text and audio tokens. The hidden states from the final layer of LLM, along with the embedding of the predicted token, are provided as imput to the MCTP modules. The historical input tokens, the tokens predicted by the LLM, and by the MCTP modules are concatenated to form the inputs to the next LLM forward pass. Finally, the audio tokens generated by both the LLM and the MCTP modules are aggregated and passed to the audio decoder to generate the final audio output.
3.2 Multiple Cross-Modal Token Prediction (MCTP) Module
As described in Section 1, the text and speech modalities exhibit a monotonic alignment pattern. This cross-modal alignment allows us to avoid complex modeling of the semantic latent space and to focus on learning a simple text-to-speech mapping relationship. which we propose to use lightweight modules to learn. the preliminary experiments, we use a few lightweight Transformer blocks to predict multiple audio tokens from LLM hidden states, and embed the predicted tokens into the LLM's autoregressive inference.
Standard autoregressive modeling can be formulated as:
Pr....,,.... Yo) P|Y, Yo),
(1)
where Y, denotes the predicted audio token time step f, and pt represents the conditional probability distribution based on the historical sequence. When extended to multi-step prediction, ie., predicting the i-th audio token at time step 1+ i, the formulation becomes as:
Pt+ x (Y- Yo) P|Y.+ Yo).
(2)
At this point, there is a significant deviation in the consistency of the distribution between P. and P. As increases, the difference between the two distributions will progressively widen, resulting in growing accumulation of errors and leading to poor mapping between text and audio.

---

Text Token
Audio Token
Hidden State
MCTP-10
Audio Decoder
MCTP-2
Ourpor Head
Ourpot Head
MCTP
MCTP
MCTP-1
Outpul Head
Transformer Block
Large Language Model
Lincar Projection
RMSNom
Encoder
RMSNorm
Embeeding Layer
Text
Speech
Figure 2: Architecture overview. (a) VITA-Audio an end-to-end large speech model equipped with 10 lightt Multiple Cross-modal Token Prediction (MCTP) modules that enable speech generation with extremely low latency. As shown in Fig. we observe that the hidden states of certain text tokens in the LLM backbone contain sufficient semantic information for generating the corresponding audio tokens, which means that it is unnecessary to attend to additional text tokens when generating audio. Thus, propose to utilize set of light-weight MCTP modules to model the mapping from LLM hidden states to the audio tokens. (b) The details of the MCTP modules. Our MCTP module has a fight-weight architecture, which enables it to finish one forward pass within 0.0024 seconds 11% of the LLM backbone). The MCTP module is capable of generating 10 audio tokens from the LLM hidden states and the text embedding, and the generated audio tokens can be decoded by the audio decoder directly. The utilization MCTP modules enables VITA-Audio to generate audio responses in one LLM forward pass, which achieves extremely fast generation speed.
To address this issue, we adopt a cascaded prediction architecture. Specifically, the hidden states and output sequence from the preceding modules are employed joint input conditions for the subsequent modules:
Pt::(Y Yo) P[Y YYY++..-.......... Yo, h O1ti-1..... or).
where h and Or+i-1 represent the hidden state and output sequence of the preceding module, respectively. By introducing progressively updated contextual information, modules can achieve incremental optimization of cross-modal mapping, ensuring accurate modality synchronization each time step.
Inspired by DeepSeek V3 [18], we adopt isomorphic Multi-Token Prediction (MTP) framework to construct our MCTP module. Unlike DeepSeek and speculative decoding [38], where former focuses on improving training and the latter requires verification to ensure that the sampling distribution matches exactly with that of the original model, we use the MCTP module for audio-text mapping. presumably simpler task than semantic modeling, As a result, we require comparatively small amount of text data to train our model.
Since the embedding layer and output heads are shared with the LLM, the audio tokens generated by the MCTP module are directly incorporated into the autoregressive process of the LLM. As illustrated in Fig. 2, the hidden states and output token, from the LLM or the preceding MCTP module, are concatenated with the input tokens and fed into Transformer block for next-step processing. The resulting hidden states and token are then passed to the next MCTP module. Upon completion of forward pass, the audio tokens generated by either the L.M the MCTP modules are aggregated as the input sequence for the subsequent LLM forward iteration.
5

---

Stage I
Stage 2
Stage 3
Stage
MCTP
X
MCTP
10
MCTP
10
LLM
LLM
LLM
LLM
Audio Encoder
Audio Encoder
Audio Encoder
Audio
Encoder
i Speech
l|lluool Speech
dilluton Speech
Text
Speech
Text
Text
Text
(5%)
ASR Data
ASR Data (5%)
Text Data
(5%)
Speech Q4 Data
Text Token
Audio Token
Training
Frozen
Figure 3: Training pipeline of VITA-Audio. The first stage (Audio-Tex: Alignment) enhances the
LLM by extending its audio modeling capability through large-scale speech pre-training. The second stage (Single MCTP module Training) connects an MCTP module with the LLM to predict one subsequent token based on the input tokens and the LLM's hidden states. The third stage (Multiple MCTP Modules Training) increases the number of MCTP modules in the model to predict more tokens in each model forward. The last stage (Supervised Fine-tuning) provides the speech-to-speech capability to the model by optimizing it on the large-scale speech QA dataset.
3.3 Training
3.3.1 Data Construction
VITA-Audio is trained exclusively on open-source datasets, integrating multi-domain and multi- language speech data resources. The training dataset encompasses diverse range sources. Detailed descriptions of the datasets used at each stage are provided in Table E1 in the Appendix.
All training data are uniformly packed into sequences of fixed length (8K tokens), an approach that enables effective training samples of varying lengths [47]. We reinitialize the positional embeddings and attention masks for all packed samples to ensure that the model attends exclusively to tokens within the same original sample. This processing strategy not only eliminates potential artifacts introduced by data concatenation but also significantly enhances training stability and reduces computational overhead.
3.3.2 Training Pipline
For VITA-Audio output consistent sequence of audio tokens in a single forward pass, each MCTP module must model a distinct distribution. As result, training all the MCTP modules simultaneously becomes challenging task, especially when the number of modules is large, due to potentially misaligned optimization objectives. We propose four-stage training strategy, as shown in Fig. 3, to progressively equip the MCTP modules with the ability map to its audio, thereby reducing the difficulty of their convergence. Further training details are provided Sec. B of the Appendix.
3.4 Inference
In order to address diverse scenarios, four distinct inference paradigms have been designed as shown in Fig. 4.
For the ASR and TTS tasks, we propose VITA-Audio-Turre In each forward pass, the L.MM generates token, followed by the generation of ten tokens by the MCTP modules. This paradigm

---

(1) Turbo
A
A
A
A
A
A
A
T
T
T
T
A
(2) Boost
A
A
A
A
A
A
A
A
A
T
T
T
A
A
A
(3) Balance
A
T
A
A
A
T
A
(4) Vanilla
token from main model
Audio from main model token from MCTP model
Audio token from MCTP model
Figure 4: The four text-audio interleaved inference modes illustrated as follows: 1) Turbo: As the fastest inference mode, it generates token by the main model and 10 additional tokens via MCTP in each forward pass. To ensure that valid audio chunk is decoded after the first forward pass, the first generated 11 tokens are split into text token and 10 audio tokens. Then, the Turbo mode iteratively generates text tokens and 10 audio tokens in the following forward. 2) Boost: To enhance the quality of text tokens, Boost mode follows the text-audio cyclic pattern of Turbo mode, with the main model generating every text token and MCTP generating every audio token. 3) Balance: To keep a balanced text-audio ratio, i.e., 2, the balance mode further changes the text-audio cyclic pattern of the Boost mode. Specifically, the balance mode sequentially generates 1 text token from the main model, 4 audio tokens (2 tag tokens mark the beginning and end of audios, and common tokens denote the audio content) from MCTP, 3 text tokens from the main model, 8 text tokens tag tokens mark the beginning and end of audios, and 6 common tokens denote the audio content) from MCTP, and then iteratively generates 4 text tokens from the main model and audio tokens tag tokens mark the beginning and end of audios, and 8 common tokens denote the audio content) from MCTP, 4) Vanilla: As the slowest inference mode, Vanilla mode follows the text-audio eyclic pattern of Balance mode, with the main model generating every token.
is the most efficient among the options; however, the performance of speech dialogue tasks degrades due to the need to predict the text token.
For speech dialogue tasks, we introduce VITA-Audio- and VITA-Audio-Bal Balance. Their main difference lies in audio token generation: VITA-Audio-Boost generates eight directly decodable audio tokens in the first forward pass, whereas VITA-Auuio-Ballanc adheres to strict 1 2 text-to-audio token ratio for enhanced speech quality. The latter requires two forward passes to generate enough audio tokens for decoding. To optimize model performance, distinct models were trained for each of the modes.
VITA-Audio- Vanilla designed for scenarios that require higher language performance. It generates tokens solely using the main LLM, sacrificing efficiency but offering slight performance boost.
Experiment
4.1 Experiment Settings
We use the Qwen2.5-7B-Innee Inssruuttt [45] as the pre-trained text LLM. The initial version of VITA-Audio utilizes the speech tokenizer and speech decoder in GLM-4-Voice [66], which effectively captures semantic information at an ultra-low bitrate. In the second version, ie., VITA-Audio-Pln further replaces the GLM-4-Voice tokenizer with Sense VoiceSmall [1] and MLP-based adapter. The detail comparison between VITA-Audio and VITA-Audio-Pin
listed Table E2.
4.2 Evaluation on Spoken Question Answering
We evaluate the spoken question answering capability VITA-Audio on three public English datasets: Web- Reb-Quustiions [5], Llama- Mama-Question [42], and TriviaQA [33]. Two evaluation methods are employed: ST+,, where the text responses generated by the model are evaluated directly, and S-SS where the model's speech responses transcribed using Whisper [46] before evaluation.
7

---

Table 2: Results Spoken Question Answering (SQA) benchmarks. "Sr" denotes the 27-th training stage of speech models.
Llama Question
TriviaQA
Web Question
Mean
Model
#Params
S - T
+
S --7
S - S
-> T
S +
-
T
S 3->
Proprictary Models 64.1
MinMo
[9]
7B
78.9
48.3
37.5
55.0
39.9
60.7
47.2
Open-source
Models
Moshi
[19]
7B
62.3
21.0
22.8
7.3
26.6
9.2
37.2
12.5
GLM-4- Voice
[66]
9B
64.7
50.7
39.1
26.5
55.0
39.9
45.3
31.0
LUCY (S2)
[28]
7B
59.6
51.0
23.2
18.2
26.6
18.2
36.5
29.1
VITA-Audio- Boost
68.7
60.3
30.5
29.3
30.4
44.0
VITA-Audio- Aaniiii---anllla
7B
71.3
66.3
31.9
30.1
33.5
31.4
45.6
42.6
VITA-Audio-Plun Plaiiioo-lluBBeeee Boost
76.3
64.6
43.6
39.5
44.2
40.0
54.7
48.0
VITA-Audio-Plus Vanilla
7B
75.6
68.0
45.9
42.7
45.0
41.7
55.5
50.8
Table 3: Results on Text to Speech (TTS) Benchmarks. "Sx" denotes the æ-th training stage.
Seed-TTS
LibriTTS lest-clean WER (%) 4
Model
lest-zh CER (%) L
lest-en WER (%) 4
test-hard WER (%) +
Seed-TTS
[2]
1.12
2.25
7.59
Cosy Voice
[21]
3.63
4.29
11.75
2.89
Cosy Voice2
[22]
1,45
2.57
6.83
2.47
V1TA-1 (S3)
[25]
8.44
2.63
Gli---Voice
[66]
2.91
2.10
5.64
VITA-Auuio- Turbo (S1)
1.18
1.92
10.58
1.96
VITA-AAuio-Turrie (S2)
0.96
1.92
9.72
1.98
VITA-Audio-Turbae (S3)
1.05
1.77
9.86
1.99
VITA-Auuio-Turbo (S4)
1.07
2.26
10.08
2.08
VITA-Audio-Plus-Boor
1.32
2.21
12.05
2.21
VITA-Audio-Pie Vanilla
1.13
1.85
10.21
1.89
We compare VITA-Audio with the latest speech models that have comparable parameter sizes, and the results are shown in Table 2. Our model demonstrates superior performance in the S-+T task and achieves SOTA results of VITA-Audio ensures minimal degradation between S-+T and S-S, with performance drop of only 9%. This indicates that VITA-Audio achieves high-quality alignment between text and speech modalities, with benefits extending beyond processing speed alone.
S-S setup. is particularly noteworthy that the training approach
4.3 Evaluation on Fundamental Speech Competence
TTS We evaluate the TTS performance of VITA-Audio on Seed-TTS [2] and LibriTTS [65] benchmarks. We use Whisper-Large-V [46] and Paraformer[29] to transcribe into text the generated English and Chinese speech, respectively.
We present the results of VITA-Audio at each stage in Table 3. these results, the output of VITA- Audio's 's first stage (S1) consists of text tokens directly generated by the LLM, while the outputs of the second (S2), third (S3), and fourth (S4) stages are alternately generated by both the LLM and the MCTP module. The experiments demonstrate that VITA-Audio outperforms other open-source models with a similar number parameters. Additionally. should be noted that, despite using ten MCTP modules for accelerated inference, VITA-Audio 's TTS capabilities largely preserved throughout the training process, further validating the effectiveness of the MCTP module in aligning text and audio.
ASR We evaluate the ASR performance of the four stages VITA-Audis on WenetSpeech [67], AIshell [6], LibriSpeech [43], and Fleurs [15], and subset of the results are reported in Table 4.

---

Table 4: Results Automatic Speech Recognition (ASR) Benchmarks. "Sx" denotes the æ-th
training stage. Compared to other methods, VuTA-Audio is trained with open-source data only.
WenetSpeech
AIShell
LibriSpeech
Model
test coss___eetiggg +
jest_net
test 4
test-clean
test-other +
Qwen2-Audio-bar
[12]
8.40
7.64
1.52
1.74
4.04
Baichuan-Audio-ban
[37]
13.28
10.13
1.93
3.02
6.04
VITA-1.5 (S3)
[24]
10.0
8.4
2.2
3.4
7.5
Freeze-Omni
[56]
13.46
11.8
2.48
3.82
9.79
LUCY (S1)
[28]
10.42
8.78
2.40
3.36
8.05
Step-Audio-chat
[49]
10.83
9.47
2.14
3.19
10.67
Qwen2.5-Omni
[59]
7.71
6.04
1.13
2.37
4.21
VITA-Audio-Vaniinn
17.34
13.45
4.46
2.98
8.07
HTTA-AudioPPlus
9.38
8.97
4.72
3.13
7.07
VITA-Audio-Plus-VVee (lnsllaanlll((
6.68
6.59
1.51
1.91
4.29
VITA-Audio-Plus-VVee
7.12
6.90
1.94
2.00
4.60
Table 5: Boostup Ratio under Different Inference Paradigms.
Mode
Model Size
#GPU
Total Second
Token Second Speedup
Vanilla
53,89
76.00
1.00 x
Boost
20.65
198.35
2.61 x
0.5B
Balance
20.71
197.78
2.60 x
Turbo
11.83
346.24
4.56 x
Vanilla
63.38
64.62
1.00 x
Boost
23.97
170.88
2.64 x
7B
1
Balance
23.94
171.09
2.64 X
Turbo
13.43
304.99
4.72 x
Vanilla
255.13
16.05
1.00 x
Boost
84.98
48.20
3.00 x
72B
2
Balance
85.13
48.11
3,00 X
Turbo
39.5
103.60
6.46 X
More detailed results can be found in Table E3 and Table E4 in the Appendix. The results for other works are partially reproduced from their respective original works for comparison.
can be observed that MUTTA-Addi--plu-vaniii demonstrates highly competitive performance across various benchmarks. Moreover, VITA- VTTA-Audio-plus-- Boost achieves remarkably fast inference speed while still maintaining strong overall performance.
4.4 Evaluation of Latency
Inference Speedup Efficient mapping between text and speech is the core of VITA-Audio. To demonstrate its effectiveness, we compare the inference time across different modes of VITA- Audio for various model sizes. Specifically, 148 TFLOPS under bfloat16 precision, with the output fixed at 4096 tokens, and record the total time the model's inference time. All models, regardless of size, are randomly initialized, and this initialization do not affect the inference time measurements. We use Transformers [57] and FlashAttention [17].
evaluate the inference speed GPUs capable of
As mentioned in Section 3.4, VITA-Audio- Vanilla only uses the main model for output; VITA- Audio-Turbo uses both the main model and all the MCTP modules during each forward pass; and VITA-Audio-BI Boost and VITA-Audio- Audio-Baaae progressively increase the number of MCTP modules used to ensure higher accuracy.

---

1000
Vanilla
Balance
Turbo
600
400
200
Nomber Decoded Tokons
Figure 5: Token generation speed curves of four text-audio interleaved modes.
As shown in Table 5, we present comparison of the time consumption for different inference modes and model sizes. Noted that the time consumption does not include the cost of the audio encoder and audio decoder. We observe that in VITA-Audio-Turbo, a speedup of approximately 5x is achieved across models ranging from 0.5B to 72B, greatly enhancing the output token throughput. VITA-Audio-Boww also achieves around 3x acceleration across various sizes, resulting in desirable performance for real-time speech dialogue systems. For example, 72 B VITA-Audio generates approximately 40 tokens per second, which, excluding generated text tokens, corresponds to roughly three seconds of audio and associated text when using a 12.5Hz speech tokenizer. This performance sufficiently fast for human-computer interaction.
Latency In human-computer interaction, latency is crucial metric, as it determines whether users can interact with the model in real-time. Given that most speech models support streaming output, the key to reducing perceived latency lies in shortening the time required to generate the first chunk of audio
We visualize the timeline of model decoding phrase in Fig. 5. The green marks denote the tokens generated by the main model, and the blue marks are the tokens generated by MCTP modules. We set the number of prefill tokens to 32. And Fig. 5 shows that VITA-Audio- Turbo completes the generation of the first audio chunk in about 50 ms, while VITA-Audio- Audio-Vanilla requires about 220 ms. generate fewer audio tokens in the first forward and
VITA-Audio-Bon Boost and VTTA-AudioBaalle more text audio tokens in the following forward. Thus, they are slower than VITA-Audio- Turbo but still significantly faster than VITA-Audio- Vanilla.
Thanks the advantage of zero audio generation delay, VITA-Audio produces multiple audio tokens in the first forward pass, allowing the first audio token chunk to be generated during the initial forward pass, which can then be used for decoding. This significantly reduces the perceived delay. In the experimental environment previously mentioned, VITA-Audio reduces the time to generate the first audio token chunk from 236 to 53 ms, as shown in Table E7.
Conclusion
In this paper, we introduce VITA-Audio, a lightweight framework that uses separate efficient modules, named Multiple Cross-modal Token Prediction (MCTP) modules, to efficiently generate audio re- sponses from text embeddings and LLM hidden states. MCTP learns the simple mapping relationship between text hidden states and audio tokens with relatively simple modules and without relying on the extensive semantic modeling of LLMs. Our model achieves new state-of-the-att performance multiple benchmarks for ASR, TTS, and SQA tasks, outperforming existing models efficiency and accuracy, especially the open-source ones with similar parameter scale. Therefore, it sets standard for real-time speech-to- tee--tt--speeeee models.
Acknowledgments
This work is partially funded by National Natural Science Foundation of China (Grant No. 62506158 and No. 62441234), and CCF-Tencent Chino-Bird Open Research Fund,
10

---

References
[1] Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao. Yue Gu, Ting He, Hangrui Hu, Kai Hu, Shengpeng Ji, Yabin Li. Zerui Li, Heng Lu, Haoneng Luo, Xiang Lv, Bin Ma. Ziyang Ma, Chongjia Ni, Changhe Song, Jiaqi Shi, Xian Shi, Hao Wang. Wen Wang. Yuxuan Wang. Zhangyu Xiao, Zhijie Yan. Yexin Yang. Bin Zhang. Qinglin Zhang, Shiliang Zhang, Nan Zhao, and Siqi Zheng. FunAudiol MM: Voice Understanding and Generation Foundation Models for Natural Interaction Between 2024. 7,4
Humans and M.. (.XXv:2407.000
[2] Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong. Lelai Deng. Chuang Ding, Lu Gao, Mingging Gong, Peisong Huang. Qingqing Huang. Zhiying Huang, Yuanyuan Huo. Dongya Jia, Chumin Li, Feiya Li, Hui Li. Jiaxin Li. Xiaoyang Li, Xingxing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, Yuchen Liu, Zhengxi Liu, Lu Lu, Junjie Pan, Xin Wang. Yuping Wang. Yuxuan Wang. Zhen Wei, Jian Wu, Chao Yao, Yifeng Yang. Yuanhao Yi. Junteng Zhang. Qidi Zhang, Shuo Zhang, Wenjie Zhang, Yang Zhang, Zilin Zhao, Dejian Zhong, and Xiaobin Zhuang. Seed-TTS: A Family of High-Quality Versatile Speech Generation Models. arXiv:2406 02430, 2024. 8
[3] Yushi Bai, Xia Lv, Jiajie Zhang. Yuze He, Ji Qi, Lei Hou, Jie Tang. Yuxiao Dong. and Juanzi Li. LongAlign:
A Recipe for Long Context Aligmment of Large Language Models. 2024. 1,3
[4] Yushi Bai, Jiajie Zhang, Xin Lx, Linzhi Zheng. Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongWriter: Unlcashing 10,000+ Word Generation from Long Context LLMs. 2024. 1,3
[5] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic Parsing on Freebase from Question-Answer Pairs. In Conference on Empirical Methods in Natural Language Processing (EMNLP),
2013. 7
[6] Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. AISHELL-1: An Open-Source Mandarin Speech Corpus and a Speech Recognition Baseline. Conference of the Oriental Chapter of the International Coordimating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA), 2018. 8, 1,3,5
[7] William Chan, Navdeep Jaitly, Quoe V Lc, and Vinyals Google Brain. Listen, Attend and Spell. 2015. 2
[8] Guoguo Chen, Shuzhou Chai, Guanbo Wang. Jiayu Du Wei Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Zhao You, and Zhiyong Yan. GigaSpeech: An Evolving. Multi-Domain ASR Corpus with 10,000 Hours of Transcribed Audio. In Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech). 2021. 1,3
[9] Qian Chen, Yufeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng. Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, Yahin Li, Xiang Lv, Jiaqing Liu, Haoneng Luo, Bin Ma, Chongjia Ni, Xian Shi, Jialong Tang, Hui Wang, Hao Wang, Wen Wang, Yuxuan Wang, Yunlan Xu, Fan Yu, Zhijic Yan, Yexin Yang. Baosong Yang. Xian Yang. Guanrou Yang. Tianyu Zhao, Qinglin Zhang. Shiliang Zhang. Nan Zhao, Pei Zhang, Chong Zhang, and Jinren Zhou. MinMo: A Multimodal Large Language Model for Seamless Voice Interaction. arXiv:2501 200.:22200.028...... 2025. 3. 4,4, 5
[10] Wenxi Chen, Ziyang Ma, Ruiqi Yan, Yuzhe Liang, Xiquan Li, Ruiyang Xu, Zhikang Niu, Yanqiao Zhu, Yifan Yang. Zhanxun Liu, Kai Yu, Yuxuan Hu, Jinyu Li, Yan Lu, Shujie Liu, and Xie Chen. SLAM-Omni: Combrl-Controllable Voice Interaction System with Single-Stage Training. arXiv:2412. 15649, 2024. 1, 4
[11] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. LongLoRA: Efficient Fine-Tuning of Long- Long-Context Large Language Models. 2023. 1,3
[12] Yunfei Chu, Jin Xu, Qian Yang, Haojic Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-Audio Technical Report. arXiv:2407.10755 2024. 5,
[13] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1xXi:22110.14168, 2021.
[14] Together Computer. Long Data Collections, 2023. 1,3
[15] Alexis Conneau, Min Ma, Simran Khanuja. Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. FLEURS: FEw-Shot Learning Evaluation of Universal Representations of Speech. 2022 IEEE Spoken Language Technology Workshop. SLT 2022 Proceedings, 2023. 8, 4
[16] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free Dolly: Introducing the World's First Truly Open Instruction- -uuned LLM, 2023. 1,3
[17] Tri Dao. FlastAttention Faster Attention with Better Parallelism and Work Partitioning. The Interna- tional Conference on Learning Representations (ICLR), 2023.
11

---

[18] DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue. Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng. Chenyu Zhang. Chong Ruan, Damai Dai, Daya Guo. Dejian Yang. Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao. Hui Li, Hui Qu. J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song. Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang. Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang. Qiancheng Wang. Qihao Zhu, Qinyu Chen. Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge. Ruisong Zhang. Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu. Shengfeng Ye. Shengfeng Ye. Shirong Ma. Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang. Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang. Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li. Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang. Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li. Yaohui Wang. Yi Yu, Yi Zheng. Yichao Zhang. Yifan Shi, Yiliang Xiong. Ying Hc, Ying Tang. Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yonggiang Guo. Yu Wu. Yuan Ou Yuchen Zhu, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunkian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, 7. Z. Ren, Zehui Ren, Zhangli Sha. The Fu, Zhean Xu, Zhen Huang. Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang. Zhuoshu Li, Zihui Gu. Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. DeepSeek-VV V3 Technical Report. 2024. 5
[19] Alexandre Défossez. Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou. Edouard Grave, and Neil Zeghidour. Moshi: A Speech-Text Foundation Model for Real-Time Dialogue. arXiv:2410.0000 2024. 3, 4, 8,5
[20] Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu. AISHELL-2: Transforming Mandarin ASR Research into Industrial Scale.
000.//008....,,,. 2018. 3,5
[21] Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijie Yan. Voice: A Scalable Multilinuual Zero-Shot To-eeehh ftt::2407.004007 2024.
Synthesizer Based on Supervised Semantic Tokens.
[22] Zhihao Du Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv. Tianyu Zhao, Zhifu Gao, Yexin Yang. Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang. Zhijie Yan, and Jingren Zhou. CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models. arXiv:2412.10117, 2024. 4,8
[23] Qingkai Fang. Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. ALaMA-Omma Seamless Speech Interaction with Large Language Models. arXiv:2409.06660 2024. 1,4,5
[24] Chaoyou Fu, Haojia I. Lin, Zuwei I Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, Ran He, Rongrong Ji, Yunsheng Wu, Caifeng Shan, and Xing Sun. VITA: Towards Open-Source Interactive Omni Multimodal L.M. arXiv:2408.05211 2024. 3,9
[25] Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao. Ke Li. Xiawu Zheng, Rongrong Ji. Xing Sun, Caifeng Shan, and Ran He. VITA-1.5: Towards GPT-40 Level Real-Time Vision and Speech Interaction. arXiv:2501. 00957. 2025. 8, 5,5
[26] Yihui Fu, Luyao Cheng. Shubo Lv, Yukai Jv. Yuxiang Kong. Zhuo Chen, Yanxin Hu, Lei Xie, Jian Wu, Hui Bu, Xin Xu, Jun Du, and Jingdong Chen. AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario. In Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech), 2021. 1,3
[27] Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón, Keith Achorn, Anjali Gopi, David Kanter, Maximilian Lam, Mark Mazumder. and Vijay Janapa Reddi. the People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage. 2021. 1,3
[28] Heting Gao, Hang Shao, Xiong Wang, Chaofan Qiu, Yunhang Shen, Sigi Cai, Yuchen Shi, Zihan Xu, Zuwei Long. Yike Zhang, Shaoqi Dong, Chaoyou Fu, Ke Li, Long Ma, and Xing Sun. LUCY: Linguistic Understanding and Control Yielding Early Stage of Her. veXiv:2501.16322 2025. 3, 4, 8, 9, 1,5
[29] Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Yan. Paraformer: Fast and Accurate Parallel Transformer for Non- Autoregressive End-To- -nd Speech Recognition. In Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech), 2022.
12

---

[30] Haorui He. Zenggiang Shang, Chaoren Wang. Xuyuan Li. Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang. Jiaqi Li. Peiyang Shi, Yuancheng Wang. Kai Chen, Pengyuan Zhang, and Zhizheng Wu. Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation. arXiv:2501.15907,
2024. 1,3
[31] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021. 6
[32] Rongjie Huang, Mingze Li, Dongchao Yang. Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Yuexian Zou, Zhou Zhao, and Shinji Watanabe. AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head. In AAAI Conference on Artificial Intelligence (AAAI), 2023. 3,
[33] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprchension. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2017. 7
[34] Suyoun Kim, Takaaki Hori, and Shinji Watanabe. Joint CTC-Attention Based End- To-Ed Speech Recognition Using Multi- Multi-Task Learning. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017. 2
[35] Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schütze. LongForm: Optimizing Instruction
Tuning for Long Text Generation with Corpus Extraction. 2023. 1,3
[36] Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Communicative Agents for "Mind" Exploration of I. Large Language Model Society. 2023. 1,
[37] Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang. Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong, Jianhua Xu, Haoze Sun, Zenan Zhou, and Weipeng Chen. Baichuan- Audiio: Unified Framework for End-To- Tn--To-End Speech Interaction. arXiv:2502. 17239, 2025. 9, 9,5,
[38] Yuhui Li. Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. International Conference on Machine Learning (ICMI.), 2024. 5
[39] Linhan Ma, Dake Guo, Kun Song, Yuepeng Jiang, Shuai Wang, Liumeng Xue, Weiming Xu, Huan Zhao, Binbin Zhang, and Lei Xic. WenetSpeech4TTS: A 12,800-Hour Mandarin TTS Corpus for Large Speech Generation Model Benehmark. In Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech), 2024. 1,3
[40] Ltd. Magic Data Technology Co. MAGICDATa Mandarin Chinese Read Speech Corpus, 2019. 1,3
[41] Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-Math: Unlocking the Potential of SLMs in Grade School Math. 2024. 1,3
[42] Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, R. J. Skerry-Ryan, and Michelle Tadmor Ramanovich. Spoken Question Answering and Speech Continuation Using Spectrogram-Povee Powered Llm. In The International Conference on Learning Representattins (ICLR), 2024.7 7
[43] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR Corpus Based on Public Domain Audio Books. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015. 8, 1,
[44] Vincel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. MLS: A Large-Scale Multilingual Dataset for Speech Research. In Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech), 2020. 1,3
[45] Qwen, :, An Yang. Baosong Yang, Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren. Yang Fan. Yang Su. Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui. Zhenru Zhang. and Zihan Qiu. Qwen2.5 Technical Report. 2024. 7.4
[46] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust Speech Recognition Via Large-Scale Weak Supervision. International Conference on Machine Learning (PMLR), 2022. 7,8
[47] Yunhang Shen, Chaoyou Fu, Shaoqi Dong, Xiong Wang. Peixian Chen, Mengdan Zhang, Haoyu Cao, Ke Li, Xiawu Zheng, Yan Zhang, Yiyi Zhou, Rongrong Ji, and Xing Sun. Long- VITA: Scaling Large Multi-Modal Models to Million Tokens with Leading Chort-CCntext Accuracy. arXiv:2502.05177. 2025.
6,
[48] Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li. AISHELL-3: A Multi Spelki-ppeaker Mandarin TTS Corpus and the Baselines. In Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech), 2020. 3
13

---

[49] Step-audio Team. Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction.
arXiv:250.11144, 2024. 3, 5,5
[50] Teknium. OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants, 2023. 1, 3
[51] Liu Tiedong. Goat, 2023. 1,3
[52] Atlas Unified. Atlas math sets, 2023. 1,3
[53] Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. VoxPopuli: Large-Scale Multilingual Speech Corpus for Representation Learning. Semi-Supervinn Learning and Interpretation. In International Joint Conference on Natural Language Processing (IJCNLP), 2021. 1,3
[54] Peng Wang, Songshuo Lu, Yaohua Tang. Sijie Yan, Yuanjun Xiong, Wei Xia, and Mthreads Ai. a Full-Duplex Speech Dialogue Scheme Based on Large Language Models. 1,3
[55] Wenbin Wang. Yang Song. and Sanjay Jha. GLOBE: A High Sigh-Quality English Corpus with Global Accents for Zero-Shot Speaker Adaptive Text-To-Speech. In Proceedings of the Annual Conference of the International Speech Communication Association (Interspeeck) 2024. 1,3
[56] Xiong Wang. Yangze Li, Chaoyou Fu, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-Omni: A Smart and Low Latency Speech-To-Speeee Dialogue Model with Frozen I.I.M vxXi:2211.00072 2024. 1, 3, 4,
9,5
[57] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick Platen, Clara Ma, Yacine Jemnie,, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace's Transformers: State-Of-The-Ar Natural Language Processing. 2019. 9
[58] Zhifei Xie and Changqiao Wu. Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming. arXiv:2408. 16725, 2024. 1, 3,4
[59] Jin Xu, Zhifang Guo, Jinzheng He. Hangrui Hu, Ting He. Shuai Bai, Kegin Chen, Jialin Wang. Yang Fan, Kai Dang. Bin Zhang, Xiong Wang, Yunfci Chu, and Junyang Lin. Qwen2.5-Omni Technical Report. arXiv:2503. 20215, 15, 2025, 9, 5,5
[60] Jianxin Yang. Long Efficient and Effective Method Extend Context Length of Large Language Models. 2023. 1,3
[61] Yuan Yao. Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level mllm your phone. arXiv preprint ar 2iv:: 2024. 5
[62] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap Your Own Mathematical Questions for Large Language Models. The International Conference on Learning Representations (ICLR), 2024. 1,3
[63] Vijiong Yu. "Paraphrasing the Original Text" Makes High Accuracy Long-Context QA. 2023. 1,3
[64] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammothh Building Math Generalist Models Through Hybrid Instruction Tuning. The International Conference on Learning Representations (ICLR), 2024. 1,3
[65] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A Corpus Derived from LibriSpeech for Text-To-Spp To-Speech. In Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech). 2019. 8, 1,3
[66] Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. GLM-4-Voice, Towards Intelligent and Human-like End-To-Enn End Spoken Chatbot, arXiv:24/2.02612. 2024. 1, 3, 4, 8,
[67] Binbin Zhang, Hang Lv, Pengcheng Guo, Qijic Shao, Chao Yang. Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, and Zhendong Peng. WenetSpeech: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition. In International Conference on Acoustics, Speech and Signal Processing
(ICASSP), 2022. 8, 1, 3,5
[68] Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. LongCite: Enabling LLMs to Generate Fine-Grained Citations in Long-Contexi QA. 2024. 1.3
[69] Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, Qipeng Guo, Haodong Duan, Xin Chen, Han Lv, Zheng Nie, Min Zhang. Bin Wang. Wenwei Zhang, Xinyue Zhang. Jiaye Ge, Wei Li, Jingwen Li, Zhongying Tu, Conghui He, Xingcheng Zhang, Kai Chen, Yu Qiao, Dahua Lin, and Jiaqi Wang. InternL XComposer2.5 -iamposer..-Omuiiiiee OmnniLive:: A Comprehensive Multimodal System for Long-Term Term Streaming Video and Audio Interactions. 2024. 1.3
14

---

[70] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoee, and Omer Levy. LIMA: Less Is More Alignment. 2023, 3
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. Our main contributions are also detailed in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, please see Sec.C for limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
We provide detailed assumptions and proofs in Sec. and Sec.3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We use publicly-accessable datasets. upload the codes and instructions to recover the results.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: We use publicly-accessabbe datasets. We upload the codes and instructions to recover the results. Once the blind review period is finished, we'll open-source all codes, instructions, and model checkpoints.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
httpl//gguides/oeeSubiiii otwooddeessiisssooooolooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.co.
wwwpc//gg//////eeeeeeeeece CoodeSubmiiie /utec//ggiddeeoeeeubbbbi wwwssssssiiioooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the data used in the experiments as well as the detailed experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, all our experimental results and latency measurements are averaged over multiple runs.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the computational power of the GPUs used in our experiments in Sec.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https / http///eurrpp..eee ec/ cc/public/ htppulc//tttiseeeeeeeeeeeeeeeeee /Eteee/Eiiieee demuiidellnse?
Answer: [Yes]
Justification:
We followed the NeurlPS Code of Fthics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work involves training and evaluation on academic, publicly available datasets. It is not related to any private personal data, and there are explicit negative social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We do not foresee any high risk for misuse of this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we credited them in appropriate ways.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi wwwode.hhoe...c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
Justification: No, our work does not describe the usage of LLMs as an important, original, or non-standard component of the core methods in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /attps:/on /ttp///erss////// /2e.eeeeeeneee22//////// /LLL/L for what should or should be described.
22

---

Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model
VITA-Audion
Supplementary Material
A Training Data
ASR Data We aggregated approximately 100, 000 hours of open-source Automatic Speech Recog-
nition (ASR) data, including WenetSpeech [67], Librispeech [43], Multilingual LibriSpeech [44], Common Voice 17 [43], MMCRSC [40], GigaSpeech [8], People's Speech [27], VoxPopuli [53], and the AISHELL series (AISHELL-I [6] to AISHELL-4 [26]).
TTS Data Concurrently, integrate approximately 100, 000 hours of open-source Texl-to-Specch (TTS) data, primarily consisting of the Wenetspeech4TTS [39], LibriTTS [65], GLOBE [55], and
Emilia [30] datasets.
Speech QA Data For speech question question-answeriing answering (Speech- -A),, we utilize VoiceAssistant- 400K [58] and AudioQA-1.0M [28], totaling 1.4 million speech QA data, to enhance the model's speech-to-speech dialogue capabilities.
Text-Only Data The pure text data is collected from OpenHermes-2.5 [50] LIMA [70], databricks-
dolly-15k [16]. MetaMathQA [62]. MathInstruct [64], Orca-Math [41], atlas-math-sets [52],
goat [51], and camel-ai-man [36]. Given that discrete audio token sequences exhibit significantly longer lengths compared their textual counterparts,
incorporate several specialized long-context text datasets following the Long- VITA [47] to enhance contextual modeling capabilities. These
include Long-Instruccte Instruction [63], LongForm [35], LongAlign- 10k [3], LongCite-45k [68], Long Writer- 6k [4], LongQLoRA [60], LongAlpaca [11], and LongData-Collectnn [14].
B Training Pipline
Stage 1: Audio-Text Alignment. Building upon pretrained language models, the goal of this stage is extend the audio modeling capabilities to the LLM through large-scale speech pretraining. We freeze the audio encoder and audio decoder, and train the LLM using ASR, TTS, and Text-only data. During this stage, the output of the LLM can be either pure text tokens audio tokens.
Stage 2: Single MCTP Module Training. After Stage 1, the model Icarns both text and audio distributions. The objective of Stage 2 is to train the initial MCTP module to predict one subsequent token based on the output tokens and hidden states from the LLM. This stage employs the same dataset configuration layer of the LLM, with the gradient detached from the LLM.
Stage 1. We initialize the MCTP module using parameters from the final
Stage 3: Multiple MCTP Modules Training. The objective of this stage is to extend the single MCTP module to multiple MCTP modules. Specifically, each MCTP module predicts the token its corresponding position given the output tokens and hidden states of the previous MCTP module. All subsequent MCTP modules are initialized using the weights of the MTCP module from Stage This stage also incorporates gradient detachment to optimize the model training process.
Stage 4: Supervised Fine-tuning. After the previous three training stages, VITA-Audio has acquired the ability to efficiently and accurately map text to audio. To enable speech-te -peechhto-speech dialogue capability, we then conduct supervised fine-tuning using speech QA datasets while maintaining small amount of TTS, ASR, and text-only data to ensure training stability. To optimize training effectiveness, different learning rates are used for the MCTP module and the main LLM. For speech-to- to-spe data, we employ interleaved output format. This design enforces the model to initiate audio token generation during the first forward pass, enabling synchronized decoding of the audio tokens rather than waiting until all text tokens have been generated.

---

C
Limitations
While our approach enables efficient generation of audio tokens, the overall end-to-end latency remains above the theoretical lower bound, primarily due to the constrained generation speed of the audio decoder. Further improving the response speed of the audio decoder in end-to-end speech models worthwhile direction for future exploration.
D
Data Format
Speech QA Interleaved Data Format
"messages":
"role": "user",
"content"; "f audio/> audio_ _iaio_sequene__1 < <| of
audio |>?
"role": "assistant",
" "content": "text_ "textssequence. </ </bggin_ of of_ =f__adio/> audio seeousseuueee__e 2 <|end_of of of__of_a_ddlollo text_ _ext__seuence__ < bbegin_off audiol> o
audio_sequence_3 < <|end_of__ _udioo/
"role": "user",
"content"; ofagiin_o__udii// audio_sequence_ < len__of_audia |>
}
"role": "assistant", "content": "text_ _ext__seqeene__3 < text_sequence_F 4 audio_ aodiiosseuence.. <lend_of of _f__iool_> text_
< <lbegin_or of audio/> > audio_ _erio__squence_6 <|end_of of a
}
]
Prompt for TTS task.
"messages":
"role": "user",
"content"; "Convert the text to speech. \ntext_sequence
}
]

---

Prompt for ASR task.
"messages":
{
"role": "user",
" "content": "Convert the speech to text. \n
< <f _f_iioo/>>>>> audio_sequence </end_ of audio/ >
}
]
E
Figures and Tables
Table E1: Summary of datasets used in VITA-Audio for different stages.
Sampling Ratio
Task
Name
Total Number
Stage
Stage 2
Stage 3
Stage
WenetSpeech
[67]
10,000H
1.0
1.0
1.0
0.05
Librispeech
[43]
1,000H
1.0
1.0
0.05
Multilingual LibriSpeech
[44]
71,506H
1.0
1.0
1.0
Common Voice 17
[43]
2.849H
1.0
1.0
1.0
0.05
MMCRSC
[40]
755H
1.0
1.0
1.0
0.05
GigaSpeech
[8]
10,000H
1.0
1.0
1.0
0.05
ASR
People's ' Speech
[27]
1,000H
1.0
L.O
1.0
0.05
VoxPopuli
[53]
543H
1.0
LO
LO
0.05
AISHELI.-1
[6]
170H
1.0
1.0
1.0
AISHELL-2
[20]
1,000H
10
LO
LO
0.05
AISHELL-3
[48]
85H
1.0
1.0
1.0
0.05
AISHELL-4
[26]
120H
1.0
LO
0.05
Wenetspeech4TTS
[39]
12,800H
1.0
LO
LO
0.05
LibriTTS
[65]
585H
1.0
1.
1.
0.05
TTS
GLOBE
[55]
535H
1.0
1.0
1.0
0.05
Emilia
[30]
96,700H
1.0
1.0
1.0
0.05
VoiceAssistant-4400K
[58]
400K
Speech QA
AudioQA-1.0M
[28]
1M
0.0
0.0
0.0
2.0
OpenHermes-2.55
[50]
1M
1.0
1.0
1.0
0.05
LIMA
[70]
IK
1.0
1.0
1.0
databricks-dolly-I1ss
[16]
15K
LO
LO
LO
0.05
MetaMathQA
[62]
395K
1.0
LO
1.
0.05
Text QA
MathInstruct
[64]
262K
1.0
LO
LO
0.05
Orca-Math
[41]
200K
1.0
1.0
1.0
0.05
atlas-math-sets
[52]
17.8M
1.0
1.0
1.0
0.05
goat
[51]
1.7M
1.0
1.O
L.
camel-ai-math
[36]
50K
1.0
1.0
1.0
0.05
Long Instruction
[63]
16K
LO
LO
LO
0.05
LongForm
[35]
23K
1.0
I.O
1.0
0.05
LongAlign- 10k
[3]
1.0
1.0
0.05
LongCite-45k
[68]
1.0
1.0
1.0
Long Text QA
LongWritee Writer-6k
[4]
6K
1.0
1.0
1.0
0.05
LongQLoRA
[60]
39K
1.0
1.0
1.0
0.05
LongAlpaca
[11]
12K
1.0
1.0
1.0
0.05
Long- Data- Cala-Coolssssooss
[14]
98K
1.0
1.0
1.0
0.05
3

---

Table E2: Comparison of model structures between VITA-Audio and VITA-Audio-
Name
Base LLM
Audio Encoder
Audio Adapter
Audio Decoder
VITA-Audio
Qwen2.5-7B [45]
1MM-4--oice-Toocmiizee [66]
GLM-- M----Vocc--Deoeee [66]
VITA-Audio-Phr
Qwen2.5-7B [45]
Sense VoiceSmall [1]
GLM-4 Voice-Decoder [66]
Table E3: Results on Automatic Speech Recognition (ASR) Benchmarks. "Sx" denotes the æ-th
training stage. Compared to other methods, VITA-Audio is trained with open-source data only.
Datasets
Model
WER (%)
Owen2-Audio-ber
[12]
1.74 4.04
Mhichuan-Audioharn
[37]
3.02 6.04
Freeze-Omni
[56]
3,82 9.79
VITA-1.5 (S3)
[25]
3.40 7.50
LUCY (S1)
[28]
3.36 8.05
LibriSpeech [43]
Step-Audio-chat
[49]
3.19 10.67
test-clean lest-other
Qwen2.5-Omni
[59]
2.37 14.1
VITA-Audio-Turbo
6.29 12.86
VITA-Audio-V Vanilla
2.98 8.07
VITA-Audio-Plls-BBat
3.1317.00
VITA-AAdiio-Plus-VV Vanilla (S1)
1.91 14.29
VITA-AAuiioPPlus-Vain Vanilla
2.00 1.660
Qwen2-Audio-base
[12]
3.63
Baichuan-Audig base
[37]
1.007
Fleurs [15]
Step-Audio-chat
[49]
4.26 8.56
zh en
Qwen2.5-Oma
[59]
2.92 4.17
VITA-Audio-Plus--ann
3.69 | 4.54

---

Table E4: Results on Automatic Speech Recognition (ASR) Benchmarks. "Sr" denotes the z-th
training stage. Compared to other methods, VITA-Audio is trained with open-source data only.
Datasets
Model
WER (%)
Qwen2-Audio base
[12]
1.52
Baichuan- Aanittn--udii-b-baa
[37]
1.93
Freeze-Omni
[56]
2.48
LUCY (S1)
[28]
2.40
Step-Audio-chat
[49]
2.14
AISHELL-1 [6]
Qwen2.5-Omni
[59]
1.13
VITA-Audio-Turbo
7.70
VITA-Audio-Vaniile
4.46
VITA-Audio-Plue Pltsio--PluBBosttt
4.72
VITA-Audio- Saiuudii--Pl------ Vanilla (S1)
1.51
VITA-Audio- Plus-Vanilla Vanilla
1.94
Qwen2-Audio base
[12]
3.08
Meichuan-Audio
[37]
3,87
AISHELL-2 ios [20]
Step-Audio-chat
[49]
3,89
Qwen2.5-Omni
[59]
2.56
VITA-Audio-Plar Vanilla
3.29
Qwen2-Audiobbal
[12]
8,40 | 7.64
Baichuan-Audio-bas
[37]
13.28
10.13
Preeze-Omm
[56]
13.46
11.80
VITA-1.5 (S3)
[25]
10.0 8.40
LUCY (S1)
[28]
10.42 8.78
WenetSpeech [67] lest-meeting test-nel
Step-Audio-chat
[49]
10.83 9.47
Qwen2.5-Omni
[59]
7.71 6.04
VITA-Audio-Turtae
23,97
18.66
VITA-Audio-Val Vanilla
17.34 13.45
VITA-Audio-Plus -ouudiio-llu-oosstt
9.38 8.97
VIT'A-Audi-PPla Vini-ul--lllllallllla (S1)
6.68 6.59
VITA-Audio-P Vanilla
Table E5: More results on Spoken Question Answering (SQA) benchmarks. "Sx" denotes the 2rth training stage of speech models.
Llama Question
TriviaQA
Web Question
Mean S-17
Model
#Params
$ T
S
S
S S
T
S
$ S
Proprictary Models 64.1
MinMo
[9]
78.9
48.3
37.5
39.9
60.7
47.2
Open-source Models
Moshi
[19]
7B
62.3
21.0
22.8
7.3
26.6
9.2
37.2
12.5
LLM-4-Voice
[66]
9B
64.7
50.7
39.1
26.5
55.0
39.9
45.3
31.0
LUCY (S2)
[28]
7B
59.6
51.0
23.2
18.2
26.6
18.2
36.5
29.1
MiniCPM-o2.6
[61]
7B
61.0
40.0
40.2
47.0
Llama-Omni
[23]
45.3
22.9
10.7
26.3
VITA-Auuio-Boon
7B
68.7
60.3
30.5
29.3
32.9
30.4
44.0
40.0
VITA-Audio- Vanilla
7B
71.3
66.3
31.9
30.1
33.5
31.4
45.6
42.6
VITA-Audio- Aaaii-PPooPPP----- Boost
7B
76.3
64.6
43.6
39.5
44.2
40.0
54.7
48.0
VIIA-Audio-Piae Plus- Vanilla
7B
75.6
68.0
45,9
42.7
45.0
41.7
55.5
50.8
5

---

Table E6: We have tested Audioo after aligning with ASR and TTS tasks on several text modality benchmarks. After the ASR and TTS alignment, the model's original text understanding capabilities were indeed affected. However, this might be due to the lack of large-scale, high-quality text data used during training. Interestingly, which may be due to the fact that our training dataset included a significant amount of math-related data.
observed slight improvement in performance GSM8K,
Model
MMLU[31]
GSM8K[13]
Qwen- Been-BB-un Feee-BB-Insee Instruct
74.22
80.06
VITA-Audio- Vanilla
66.92
80.14
Table E7: Generation time (ms) of the first audio segment under different inference modes in streaming inference. To enable more real-time speech generation, we progressively increase the number of steps in the flow matching model during streaming inference. The table shows the decoding time when the sampling step of the flow matching model is set to 1.
Inference Mode
Audio Encoder
First Audio Token Chunk
Audio Decoder
Sum
VITA-Audio-Boost VITA-Audio-Vami
39
53
151
243
39
236
151
426


Reviews:
[{"review_id": "7cBgsNQBby", "reviewer": "Reviewer_4oP8", "summary": "This paper introduces VITA-Audio, an end-to-end speech model designed for real-time audio generation during the first forward pass, addressing latency issues in cascaded and interleaved generation pipelines. The core innovation lies in the MCTP module, which allows the model to generate decodable audio token chunks with zero audio token delay, achieving 3-5x inference speedup while maintaining high-quality output. VITA-Audio is trained solely on open-source data, fully open-sourced, and evaluated comprehensively on ASR, TTS, and speech question answering benchmarks. Results show that VITA-Audio is competitive with existing models of similar size in both efficiency and accuracy, while offering low-latency generation.", "questions": "- MCTP novelty: The MCTP module is central to your approach. Could you clarify how it differs from existing chunked or multi-head generation strategies commonly used in LLMs and TTS systems? For instance, could LoRA adapters or other parameter-efficient methods be used instead of MCTP modules to achieve similar latency and throughput benefits?\n\n- Multispeaker synthesis: How is multispeaker TTS handled in VITA-Audio? Are speaker embeddings used in the decoder, and if so, how are they obtained and integrated during training?\n\n- Training stages: The current setup involves multiple stages. Are all of these steps necessary for good performance? For example, could the MCTP modules be trained directly in Stage 1 while fine-tuning the LLM, without needing the full staged pipeline?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The paper proposes a simple and practical solution to reduce token latency in interleaved speech-language models, which is important for enabling more human-like, real-time conversational systems.\n\n- The model is fully open-source and reproducible, trained entirely on public datasets.\n\n- The evaluation is comprehensive, covering multiple tasks (ASR, TTS, SQA) and speech language models.\n\n- The paper is clearly written and well-structured.\n\n\n**Weaknesses**\n\n- Methodological novelty is limited. The work builds on known ideas such as the fact that speech-text alignment is many-to-one, and adds incremental improvements (e.g., MCTP, adapter-based tuning). While useful, these do not represent a major modeling innovation.\n\n- The evaluation focuses mainly on semantic modeling. There is no acoustic modeling, which limits the system's ability to handle speaker consistency, prosody, or low-level audio fidelity. Tasks like speaker-consistent speech continuation are likely unsupported. Comparisons with acoustic-semantic models (e.g., Moshi) may be unfair, as VITA-Audio only models semantic tokens.\n\n- The paper does not compare against other methods to for parallel tokens generation (e.g., LoRA adapters, Medusa, speculative decoding), which could offer similar benefits in terms of latency compared to using MCTP modules.\n\n**Overall Assessment**\n\nVITA-Audio is a good engineering effort that advances real-time speech generation using open-source models and data. Although the methodological novelty is modest, the model is efficient, well-evaluated, and reproducible. It offers a practical approach to low-latency, semantic token-based speech generation.", "was_revised": true, "final_justification": "The rebuttal clarifications addressed my concerns. While the novelty is limited, I believe it is fair and the contribution is solid. The paper is clearly written, reproducible, and evaluated thoroughly. Therefore I confirm my initial score.", "rebuttal": "We sincerely thanks for your positive recognition of our work. Our point-by-point responses to all your comments are provided below.\n\n------\n\n**Q1: Innovation of the MCTP Module.**\n\n1. Design Motivation: The design motivation for the MCTP module stems from our observation that the hidden states of certain text tokens in the LLM backbone contain sufficient semantic information for generating the corresponding audio tokens. Based on this, we aim to design a lightweight module capable of efficiently generating audio tokens from these hidden states. Potential candidates for this lightweight module include: a set of Transformer layers, a small-scale LLM, or reusing the final layers of the LLM by feeding the last-layer hidden states back to those layers. Ultimately, for the simplicity of the training pipeline, we choose to use a set of sequential Transformer layers as the core of the MCTP module.\n\n2. Advantages over Speculative Decoding: Existing speculative decoding methods (such as Medusa[1]) typically require verifying all candidate tokens to ensure the correctness of the generated results. If the verification step is omitted, the accuracy rate of the generated results tends to be low. Even with optimizations like tree attention, the verification phase still requires inputting longer candidate sequences into the LLM for forward computation, and the input sequence length increases rapidly as the number of parallel predicted tokens grows. In high-concurrency or resource-limited environments, the acceleration benefits of speculative decoding diminish significantly. In contrast, the MCTP module does not require verification and can generate up to ten audio tokens at once. More importantly, this module has a very low computational cost, ensuring stable and significant inference acceleration even in high-concurrency or resource-limited settings.\n\n3. Complementarity: It is important to emphasize that speculative decoding methods and MCTP are not mutually exclusive. In practical applications, speculative decoding could be used to accelerate text generation, while the MCTP module can be employed during speech generation, further enhancing the overall end-to-end inference speed.\n\n------\n\n**Q2: Regarding Multispeaker Synthesis.**\n\nCurrently, our main focus remains on ASR, TTS, and SQA tasks. Although speaker embeddings are used in the audio decoder, we have not yet conducted specialized training for such tasks.\n\n------\n\n**Q3: Training method.**\n\nOur training method is divided into four stages, primarily based on the following two considerations:\n\n1. Our goal is to make VITA-Audio a universal paradigm that can benefit all interleaved Speech LLMs. Specifically, the model can directly leverage the weights aligned during ASR and TTS tasks to quickly initiate the training of the second stage. To validate the generalization ability of the framework, we trained two versions, VITA-Audio and VITA-Audio-Plus, to demonstrate the effectiveness of this approach across different model architectures.\n\n2. In early experiments, we attempted to train the MCTP module directly in Stage 1, or train a version containing 10 MCTPs in Stage 2, all at once. However, when the training data was insufficient, such approaches performed worse than the staged training strategy.\n\n------\n\n[1] Cai T, Li Y, Geng Z, et al. Medusa: Simple LLM inference acceleration framework with multiple decoding heads."}, {"review_id": "XBNisxReVS", "reviewer": "Reviewer_orX3", "summary": "Problem & Motivation:\n\nTraditional cascaded speech systems (ASR → LLM → TTS) suffer from cumulative latency, loss of paralinguistic cues, and error propagation. Even recent end-to-end speech-LLM models incur high first-token delays in streaming scenarios, which hinders real-time interaction. VITA-Audio addresses this by enabling the model to output audio tokens in the very first forward pass, eliminating that bottleneck.", "questions": "Can the authors provide a translation for all non-English characters to ensure everyone can understand the examples in Figure 1?", "limitations": "Complex, Multi-Stage Training Pipeline: The four-stage curriculum (audio–text alignment → single-module training → multi-module training → fine-tuning) requires careful scheduling, hyperparameter tuning, and large amounts of pretraining data. This adds significant engineering overhead and may be hard to adapt to new domains.\nHowever, this should not be a concern for the overall ratings.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- Zero Audio Token Delay: VITA-Audio is the first end-to-end speech model that can generate audio tokens in the very first forward pass.\n\n- Interleaved Cross-Modal Token Prediction (MCTP): The lightweight MCTP modules efficiently map LLM hidden states to multiple audio tokens in one pass.\n\nWeakness:\n\nA central observation and motivation in VITA-Audio is exactly that “the hidden states of certain text tokens in the LLM backbone contain sufficient semantic information for generating the corresponding audio tokens, which means that it is unnecessary to attend to additional text tokens when generating audio.” I really like this interesting concept and the findings based on this, and also appreciate the astonishing results they achieved based on this characteristic. However, the paper doesn’t offer a full study or a preliminary experiment on an evaluation set with quantitative results to support this observation. Without this, it is not clear to the audience whether this is just a coincidence or is really a common phenomenon. Of course, I am definitely aware that the authors did mention “This finding is also reported in many literature on attention-based speech systems [7, 32]”, but the whole paper can still be improved by simply providing quantitative results based on the speech model they are using to support the hypothesis “irrelevant text tokens being masked out, the model is still able to generate the correct audio, and the pronunciation remains contextually appropriate”. Any statistical analyses or metrics would substantially strengthen the claim. \nI will uprate my score if this concern is addressed.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thanks for your positive recognition of our work. Our point-by-point responses to all your comments are provided below.\n\n\n------\n\n**Q1: Can the authors provide a translation for all non-English characters to ensure everyone can understand the examples in Figure 1?**\n\nThank you for your suggestion.\n\nIn Chinese, \"一行\" (yihang) emphasizes spatial arrangement (e.g., 一行白鹭 - a row/line of egrets), while \"一行\" (yīxíng) focuses on the concept of a group or unit (e.g., 一行人 - a group/party of people). \n\nIn the revised version, we will provide translation annotations for all non-English characters to ensure smooth understanding for the readers.\n\n------\n\n**Q2: Detailed supporting results for the hypothesis that, with irrelevant text tokens being masked out, the model is still able to generate the correct audio, and the pronunciation remains contextually appropriate.**\n\nAlthough we have not quantitatively tested this observation on a specific benchmark, we have verified it through several cases. During inference, we first locate the audio tokens that the model attends to most in the attention map for a given text token, and then perform text-token masking before generating those audio tokens. Specifically, we generate audio tokens with three different masking strategies:\n\n(1) No text token is masked at all.\n\n(2) All irrelevant text tokens except the target text tokens are masked.\n\n(3) All text tokens are being masked.\n\nWe then continue the generation after the masking. In this case, both (1) and (2) can produce normal speech.\n\nFor examples in Figure 1a), in the phrase \"there are seven days in a week,\" the attended text is on \"seven days,\" and in the phrase \"一行白鹭上青天,\" the attended text is on \"一行\" (yi hang), meaning \"One row of egrets ascends the blue sky,\" and both can produce correct pronunciation.\n\nWe are conducting quantitative experiments. Specifically, we plan to perform forced alignment on the generated speech to obtain the corresponding audio tokens corresponding to each text token. We then regenerate the speech and mask out all text tokens except for the relevant ones when reaching those audio token positions. We will assess whether the generated audio tokens are correct. Unfortunately, due to time constraints, we don't have the final result yet, but we will synchronize the results once they are available."}, {"review_id": "1hKqzqR3GU", "reviewer": "Reviewer_GWRd", "summary": "This work proposes VITA-Audio, a spoken LM that can generate text and speech token simultaneously. They introduce a lightweight multiple cross-model token prediction modules to enable fast audio token generation. With the proposed module, VITA-Audio can generate 10 audio tokens after each LLM forward pass, mitigating the latency issue in other recent speech-text joint SLMs.", "questions": "1. How does the number 10 come from for the MCTP module to generate the audio tokens in each forward?\n\n2. Figure 2 is unclear for me. Can you explain more about differences of the input of the MCTP modules and the spoken LM? Why is there only 5 audio token being fed into the decoder after each LLM’s forward? Do the MCTP modules only look at temporal hidden states instead of all historical output from the LLM? (related to weakness point 2, feel free to answer these questions and comments together)\n\n3. What is the reason of having so many variants of the VITA-Audio model? The existence of the models increase the complexity of understanding or evaluating the actual performance of the model.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n* The models exhibit reasonable latency-performance trade-off according to the result tables. \n\n**Weaknesses**\n\n* The writing style feels like a technical report rather than an academic paper. The problem is not well-formulated and the inputs / outputs of each modules are unclear. \n\n* The process of how the speech-text data is interleaved is unclear in the main context. This affects the understanding of how the SLM is trained along with the MCTP modules. I have to check the supplementary materials to find out some clues and details (Figure E1 helps a lot). My recommendation is to make a clearer illustration that combines the advantages from both the Figure 2 and Figure E1, or at least include Figure E1 and present it in a more precise way. Anyway, I would like to know how exactly the text and audio tokens are interleaved in the training and inference stages. It seems that some hyperparameters are set to ensure that the audio tokens are lagged behind the text outputs during inference. \n\n* The paper seems to be a incremental work that tries to mitigate the latency problem in GLM-4-Voice by introducing the concept in speculative decoding techniques. In this way, the novelty is considered limited for me. More importantly, the writing style should be revised towards being incremental rather than proposing an entire new type of SLM.", "was_revised": true, "final_justification": "Overall, I would like to maintain my score due to:\n\n1. The clarity issues would require a major revision.\n2. The novelty is considered limited for me—integrating MCTP into the joint SLM framework appears to be a straightforward latency enhancement at the expense of performance (Boost/Turbo vs. Vanilla), and I found no particularly intriguing or surprising empirical results.\n\nHowever, I do appreciate the fact that the authors decide to open-source the model, which should be one of the strengths that I have not mentioned in the review previously.", "rebuttal": "Sincerely thanks for your efforts in reviewing this work. We hope the detailed responses help clarify your concerns. We would greatly appreciate it if you could kindly re-evaluate our work in light of the new explanations and additional results.\n\n------\n\n**Q1: Choosing the Number of MCTP Modules:**\n\nThe number of MCTP modules is positively correlated with inference acceleration; however, it may also come with an increased risk of error rate. After a comprehensive evaluation, we selected the configuration with 10 modules to achieve the optimal balance between efficiency and accuracy.\n\n------\n\n**Q2: VITA-Audio Data Format.**\n\n\nWe will take the inference stage as an example, and denote $Y_t$ as the $t$-th token, $h_n$ and $h^i_n$ as the $t$-th hidden state from LLM and MCTP-$i$, respectively.\n\nWithout considering KV cache, LLM takes all previous $t$ tokens $( Y_0, Y_1, ..., Y_{t-1} )$ as input and predicts the next token $Y_{t}$. Note that LLM also generates hidden state $( h_0, h_1, ..., h_{t-1} )$.\n\nThen the first MCTP-1 futher takes $n$ tokens $( Y_1, Y_2, ..., Y_{t} )$ with the hidden states $( h_0, h_1, ..., h_{t-1} )$ as input and predict token $Y_{t+1}$. And MCTP-1 also generates hidden state $ h^1_{t} $ for token $Y_{t}$.\n\nThe second MCTP-2 takes the all $n$ tokens $( Y_2, Y_3, ..., Y_{t+1} )$ with the hidden states $( h_1, h_2, ..., h_{t-1}, h^1_{t} )$ as input and predict token $Y_{t+2}$.\n\nAfter one forward pass of LLM and all MCTPs, the model outputs 11 tokens $( Y_{t}, Y_{t+1}, ..., Y_{t+10} )$.\nFigure 2 may cause some misunderstandings, and we will correct this in the revised version.\n\nDuring training, the loss function is formatted as:\n\n$ \\mathcal{L}  = - \\sum_{t=1}^{T} \\log P \\( Y_t  | {Y_0, Y_1, ..., Y_{t-1} } \\)  + \\log P_1 \\( Y_{t+1}  | {Y_1, Y_2, ..., Y_{t},  h_0, h_1, ..., h_{t-1} } \\)  + \\log P_2 \\( Y_{t+2}  | {Y_2, Y_3, ..., Y_{t+1},  h_1, h_2, ..., h^1_{t} } \\)  + ... + \\log P_{10} \\( Y_{t+10}  | {Y_10, Y_11, ..., Y_{t+9},  h_9, h_{10}, ..., h^9_{t+9} } \\)  $.\n\n\nWe set the ratio of text tokens to be roughly $1:2$ following recent TTS and SLM literature (GLM4Voice [1] and CosyVoice2 [2]), which causes the audio stream to lag behind the text stream.\nDuring both inference and training, we use $n$ text tokens and $2n+2$ audio tokens as an interleaved group where the additional two tokens correspond to special audio tokens: 
 and 
.\n\n------\n\n\n**Q3: The variants of VITA-Audio.**\n\nThe variants of VITA-Audio come from two aspects:\n\n1. Two Models with Different Architectures: VITA-Audio and VITA-Audio-Plus.\n\nVITA-Audio is designed as a universal paradigm to benefit various interleaved Speech LLMs.\nTo validate the broad applicability of this framework, we conducted experiments on different architectures of interleaved speech LLMs.\nThe results show that both VITA-Audio-Plus, based on SenseVoice, and VITA-Audio, based on GLM4-Voice, significantly improve the model's accuracy and inference speed by introducing the MCTP module, demonstrating the compatibility of this approach with different model architectures.\n\n2. Four Different Inference Modes: Turbo, Boost, Balance, and Vanilla.\n\nWe have designed four different inference modes: Turbo, Boost, Balance, and Vanilla, to be applied in different scenarios.\nThe core distinction between these modes lies in the configuration of inference parameters (the number of activated MCTP modules and the ratio of interleaved text and audio token generation).\nAs a result, we do not need to train a separate model for each mode.\nOf course, targeted optimization can still be carried out for the desired inference mode if the application scenario is specified.\n\n\n------\n\n\n**Q4: Differences Between MCTP and Speculative Decoding:**\n\nExisting speculative decoding methods (such as Medusa[3]) typically require verifying all candidate tokens to ensure the correctness of the generated results.\nAnd the verification step significantly slows down the decode speed.\nIf the verification step is omitted, the accuracy rate of the generated results tends to be low.\nIn high-concurrency or resource-limited environments, the acceleration benefits of speculative decoding diminish significantly.\n\nIn contrast, the MCTP module aims to reduce the latency for generating the first audio chunk in streaming scenarios, and does not require verification.\nOur instantiated MCTP generates up to ten audio tokens at once.\nMore importantly, this module has a very low computational cost, ensuring stable and significant interaction acceleration even in high-concurrency or resource-constrained settings.\n\nIt is important to emphasize that speculative decoding methods and MCTP are not mutually exclusive.\nIn practical applications, speculative decoding can be used to accelerate text generation, while the MCTP module can be employed during speech generation, further improving overall end-to-end inference speed.\n\n------\n\n[1] Zeng A, Du Z, Liu M, et al. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot.\n\n[2] Du Z, Wang Y, Chen Q, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models\n\n[3] Cai T, Li Y, Geng Z, et al. Medusa: Simple LLM inference acceleration framework with multiple decoding heads."}, {"review_id": "MRwNzEEReg", "reviewer": "Reviewer_opy1", "summary": "As speech LLMs and voice assistant technologies continue to advance, the number of forward passes required before producing the first speech output has become an increasingly important factor. The authors introduce a novel method called Multiple Cross-modal Token Prediction (MCTP) into an interleaving-based voice interaction model, which significantly reduces latency. At the 7B scale, the method achieves a 3 to 5 times faster generation speed compared to standard autoregressive decoding.", "questions": "Questions are in Weaknesses.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. Although MCTP bears some resemblance to text-based approaches like speculative decoding and MTP (as the authors acknowledge), its adaptation to audio token prediction based on their own alignment observations during speech generation is innovative and effectively reduces latency.\n\n2. The utility of the proposed method and model is validated across a range of tasks.\n\n3. The model and code have been open-sourced, making a meaningful contribution to the broader research community.\n\n# Weaknesses and Questions\n\nWhile I do not see any major shortcomings in this work, several technical questions came to mind:\n\n1. In Table 3, it might have been helpful to include comparisons with a wider variety of voice interaction models. Currently, only parallel or interleaving-based speech LLMs are listed. Recent models like Qwen2.5-Omni, LLaMA-Omni, and MiniCPM-o incorporate pluggable decoders jointly trained with LLMs. Including results from these models would be informative. While they may show stronger performance due to better LLM preservation, I do not believe this would detract from the advantages of the proposed method, as the two approaches follow fundamentally different paradigms. I mention this only as a point of curiosity and not as criticism.\n\n2. Since the authors mention using the decoder from glm-4-voice, I assume that the token format from that model is used at the output stage. In this context, I have two specific questions:\n- (1) Does the Sensivoice encoder in the Plus model provide continuous embeddings or discretized tokens as input?\n- (2) What kind of information do glm-4-voice tokens encode? Do they contain only semantic information, or do they also represent acoustic features? Given that one of the key advantages of an end-to-end pipeline is its ability to capture paralinguistic cues, the latter would be ideal.\n\n3. I am curious whether the model was trained with multi-turn conversational data. That is, does it support multi-round interactions? From the Appendix, it appears to be single-round only, but I may have overlooked something.\n\n4. I would also like to understand how many audio tokens are typically accumulated before audio is reconstructed. Since this number directly affects user-perceived latency, it would be useful to know the design trade-offs. If the chunk size is too small, audio quality may degrade due to glitches; if too large, the latency benefits of MCTP could be reduced. I realize this might be more closely related to the glm-4-voice decoder, and I raise this purely out of interest rather than as a critique.\n\n5. Is there any evidence showing how much the original text understanding capabilities of the LLM are affected by training with interleaving or parallel generation schemes? It would be useful to see performance on benchmarks like MMLU to assess any potential trade-offs.\n\n6. Since Qwen2.5-7B was used as the backbone, I am curious whether there was a particular reason for not using an Instruct-tuned version.", "was_revised": true, "final_justification": "Since my score was already positive, I've decided to keep it unchanged.", "rebuttal": "We sincerely thanks for your positive recognition of our work. Our point-by-point responses to all your comments are provided below.\n\n------\n\n**Q1: In Table 3, it might have been helpful to include comparisons with a wider variety of voice interaction models.**\n\nThank you for your suggestion. We have consulted the results of models such as LLaMA-Omni [1] and MiniCPM-o [2] on the SQA task, and we will incorporate their results into the revised version of the paper.\n\n------\n\n**Q2.1: Does the Sensivoice encoder in the Plus model provide continuous embeddings or discretized tokens as input?**:\n\nIn the VITA-Audio-plus, the SenseVoice [3] encoder provides continuous embeddings as input. \n\n**Q2.2: What kind of information do glm-4-voice tokens encode?**:\n\nThe GLM4-Voice [4] tokenizer is capable of encoding acoustic features such as intonation, stress, and rhythm.\n\n------\n\n**Q3:  Does it support multi-round interactions?**\n\nVITA-Audio supports multi-turn conversations. In Stage 4, we trained using the VoiceAssistant-400K [5], which includes multi-turn audio dialogues. We will update the data format in the appendix in the revised version to avoid any misunderstandings.\n\n------\n\n**Q4: How many audio tokens are typically accumulated before audio is reconstructed?**\n\nWe found that GLM4-Voice struggles to encode very short audio tokens (e.g., 2 audio tokens). Based on our experience, typically 8 audio tokens are a suitable choice. This is also the reason we choose the interleaving ratio for the Boost and Balance inference modes. The Balance mode requires two forward passes to generate enough audio tokens for decoding, while the Boost mode only requires one forward pass.\n\n------\n\n**Q5: Is there any evidence showing how much the original text understanding capabilities of the LLM are affected by training with interleaving or parallel generation schemes?**\n\nWe have tested VITA-Audio after aligning with ASR and TTS tasks on several text modality benchmarks. After the ASR and TTS alignment, the model’s original text understanding capabilities were indeed affected. However, this might be due to the lack of large-scale, high-quality text data used during training. Interestingly, we observed a slight improvement in performance on GSM8K, which may be due to the fact that our training dataset included a significant amount of math-related data. We will include this result in the revised version.\n\n\n| Model | MMLU | gsm8k|\n|----------|----------|----------|\n| Qwen-7B-Instruct | 74.22 |80.06|\n| VITA-Audio-plus |   66.92 | 80.14|\n\n\n------\n\n**Q6: Was there a particular reason for not using an Instruct-tuned version?**\n\nWe used Qwen-7B-Instruct as the pre-trained text LLM. We will clarify this point in the revised version.\n\n------\n[1] Fang Q, Guo S, Zhou Y, et al. Llama-omni: Seamless speech interaction with large language models.\n\n[2] Yao Y, Yu T, Zhang A, et al. Minicpm-v: A GPT-4v level mllm on your phone.\n\n[3] An K, Chen Q, Deng C, et al. Funaudiollm: Voice understanding and generation foundation models for natural interaction between humans and LLMs.\n\n[4] Zeng A, Du Z, Liu M, et al. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot.\n\n[5] Xie Z, Wu C. Mini-omni: Language models can hear, talk while thinking in streaming."}]

Author rebuttal:

Meta-review: VITA-Audio is an end-to-end speech model designed for real-time audio generation during the first forward pass, addressing latency issues in cascaded and interleaved speech systems. 

Strengths:

•	Low-latency generation: VITA-Audio achieves 3–5× speedup over standard autoregressive models, making it suitable for real-time applications.

•	MCTP module: Efficiently maps LLM hidden states to multiple audio tokens in one pass, reducing delay.

•	End-to-end design: Avoids cascading ASR → LLM → TTS pipelines, mitigating error propagation and preserving paralinguistic cues.

•	Open-source commitment: Model, training code, and data are publicly released.

•	Comprehensive evaluation: Benchmarked on ASR, TTS, audio language modeling, and spoken question answering tasks.

Overall Reviewers’ attitudes:

Most of reviewers generally appreciate the practical impact, efficiency, and open-source nature of VITA-Audio. While some see the contribution as solid but incremental, others suggest that clarity improvements and stronger empirical validation would significantly enhance the paper. The model’s ability to generate speech in the first forward pass is seen as a notable advancement for real-time voice interaction systems.
Decision: Accept (poster)