Title: VITA-Audio: Fast Interleaved Audio-Text Token Generation for Efficient Large Speech-Language Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Models

Large Speech Language Models
Abstract: With the growing requirement for natural human-computer interaction, speech-based systems receive increasing attention as speech is one of the most common forms of daily communication. However, the existing speech models still experience high latency when generating the first audio token during streaming, which poses a significant bottleneck for deployment. To address this issue, we propose VITA-Audio, an end-to-end large speech model with fast audio-text token generation. Specifically, we introduce a lightweight Multiple Cross-modal Token Prediction (MCTP) module that efficiently generates multiple audio tokens within a single model forward pass, which not only accelerates the inference but also significantly reduces the latency for generating the first audio in streaming scenarios. In addition, a four-stage progressive training strategy is explored to achieve model acceleration with minimal loss of speech quality. To our knowledge, VITA-Audio is the first multi-modal large language model capable of generating audio output during the first forward pass, enabling real-time conversational capabilities with minimal latency. VITA-Audio is fully reproducible and is trained on open-source data only. Experimental results demonstrate that our model achieves an inference speedup of 3~5x at the 7B parameter scale, but also significantly outperforms open-source models of similar model size on multiple benchmarks for automatic speech recognition (ASR), text-to-speech (TTS), and spoken question answering (SQA) tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAs speech LLMs and voice assistant technologies continue to advance, the number of forward passes required before producing the first speech output has become an increasingly important factor. The authors introduce a novel method called Multiple Cross-modal Token Prediction (MCTP) into an interleaving-based voice interaction model, which significantly reduces latency. At the 7B scale, the method achieves a 3 to 5 times faster generation speed compared to standard autoregressive decoding.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. Although MCTP bears some resemblance to text-based approaches like speculative decoding and MTP (as the authors acknowledge), its adaptation to audio token prediction based on their own alignment observations during speech generation is innovative and effectively reduces latency.\n\n2. The utility of the proposed method and model is validated across a range of tasks.\n\n3. The model and code have been open-sourced, making a meaningful contribution to the broader research community.\n\n# Weaknesses and Questions\n\nWhile I do not see any major shortcomings in this work, several technical questions came to mind:\n\n1. In Table 3, it might have been helpful to include comparisons with a wider variety of voice interaction models. Currently, only parallel or interleaving-based speech LLMs are listed. Recent models like Qwen2.5-Omni, LLaMA-Omni, and MiniCPM-o incorporate pluggable decoders jointly trained with LLMs. Including results from these models would be informative. While they may show stronger performance due to better LLM preservation, I do not believe this would detract from the advantages of the proposed method, as the two approaches follow fundamentally different paradigms. I mention this only as a point of curiosity and not as criticism.\n\n2. Since the authors mention using the decoder from glm-4-voice, I assume that the token format from that model is used at the output stage. In this context, I have two specific questions:\n- (1) Does the Sensivoice encoder in the Plus model provide continuous embeddings or discretized tokens as input?\n- (2) What kind of information do glm-4-voice tokens encode? Do they contain only semantic information, or do they also represent acoustic features? Given that one of the key advantages of an end-to-end pipeline is its ability to capture paralinguistic cues, the latter would be ideal.\n\n3. I am curious whether the model was trained with multi-turn conversational data. That is, does it support multi-round interactions? From the Appendix, it appears to be single-round only, but I may have overlooked something.\n\n4. I would also like to understand how many audio tokens are typically accumulated before audio is reconstructed. Since this number directly affects user-perceived latency, it would be useful to know the design trade-offs. If the chunk size is too small, audio quality may degrade due to glitches; if too large, the latency benefits of MCTP could be reduced. I realize this might be more closely related to the glm-4-voice decoder, and I raise this purely out of interest rather than as a critique.\n\n5. Is there any evidence showing how much the original text understanding capabilities of the LLM are affected by training with interleaving or parallel generation schemes? It would be useful to see performance on benchmarks like MMLU to assess any potential trade-offs.\n\n6. Since Qwen2.5-7B was used as the backbone, I am curious whether there was a particular reason for not using an Instruct-tuned version.\n\nQUESTIONS:\nQuestions are in Weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes VITA-Audio, a spoken LM that can generate text and speech token simultaneously. They introduce a lightweight multiple cross-model token prediction modules to enable fast audio token generation. With the proposed module, VITA-Audio can generate 10 audio tokens after each LLM forward pass, mitigating the latency issue in other recent speech-text joint SLMs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The models exhibit reasonable latency-performance trade-off according to the result tables. \n\n**Weaknesses**\n\n* The writing style feels like a technical report rather than an academic paper. The problem is not well-formulated and the inputs / outputs of each modules are unclear. \n\n* The process of how the speech-text data is interleaved is unclear in the main context. This affects the understanding of how the SLM is trained along with the MCTP modules. I have to check the supplementary materials to find out some clues and details (Figure E1 helps a lot). My recommendation is to make a clearer illustration that combines the advantages from both the Figure 2 and Figure E1, or at least include Figure E1 and present it in a more precise way. Anyway, I would like to know how exactly the text and audio tokens are interleaved in the training and inference stages. It seems that some hyperparameters are set to ensure that the audio tokens are lagged behind the text outputs during inference. \n\n* The paper seems to be a incremental work that tries to mitigate the latency problem in GLM-4-Voice by introducing the concept in speculative decoding techniques. In this way, the novelty is considered limited for me. More importantly, the writing style should be revised towards being incremental rather than proposing an entire new type of SLM.\n\nQUESTIONS:\n1. How does the number 10 come from for the MCTP module to generate the audio tokens in each forward?\n\n2. Figure 2 is unclear for me. Can you explain more about differences of the input of the MCTP modules and the spoken LM? Why is there only 5 audio token being fed into the decoder after each LLM’s forward? Do the MCTP modules only look at temporal hidden states instead of all historical output from the LLM? (related to weakness point 2, feel free to answer these questions and comments together)\n\n3. What is the reason of having so many variants of the VITA-Audio model? The existence of the models increase the complexity of understanding or evaluating the actual performance of the model.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nProblem & Motivation:\n\nTraditional cascaded speech systems (ASR → LLM → TTS) suffer from cumulative latency, loss of paralinguistic cues, and error propagation. Even recent end-to-end speech-LLM models incur high first-token delays in streaming scenarios, which hinders real-time interaction. VITA-Audio addresses this by enabling the model to output audio tokens in the very first forward pass, eliminating that bottleneck.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Zero Audio Token Delay: VITA-Audio is the first end-to-end speech model that can generate audio tokens in the very first forward pass.\n\n- Interleaved Cross-Modal Token Prediction (MCTP): The lightweight MCTP modules efficiently map LLM hidden states to multiple audio tokens in one pass.\n\nWeakness:\n\nA central observation and motivation in VITA-Audio is exactly that “the hidden states of certain text tokens in the LLM backbone contain sufficient semantic information for generating the corresponding audio tokens, which means that it is unnecessary to attend to additional text tokens when generating audio.” I really like this interesting concept and the findings based on this, and also appreciate the astonishing results they achieved based on this characteristic. However, the paper doesn’t offer a full study or a preliminary experiment on an evaluation set with quantitative results to support this observation. Without this, it is not clear to the audience whether this is just a coincidence or is really a common phenomenon. Of course, I am definitely aware that the authors did mention “This finding is also reported in many literature on attention-based speech systems [7, 32]”, but the whole paper can still be improved by simply providing quantitative results based on the speech model they are using to support the hypothesis “irrelevant text tokens being masked out, the model is still able to generate the correct audio, and the pronunciation remains contextually appropriate”. Any statistical analyses or metrics would substantially strengthen the claim. \nI will uprate my score if this concern is addressed.\n\nQUESTIONS:\nCan the authors provide a translation for all non-English characters to ensure everyone can understand the examples in Figure 1?\n\nLIMITATIONS:\nComplex, Multi-Stage Training Pipeline: The four-stage curriculum (audio–text alignment → single-module training → multi-module training → fine-tuning) requires careful scheduling, hyperparameter tuning, and large amounts of pretraining data. This adds significant engineering overhead and may be hard to adapt to new domains.\nHowever, this should not be a concern for the overall ratings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VITA-Audio, an end-to-end speech model designed for real-time audio generation during the first forward pass, addressing latency issues in cascaded and interleaved generation pipelines. The core innovation lies in the MCTP module, which allows the model to generate decodable audio token chunks with zero audio token delay, achieving 3-5x inference speedup while maintaining high-quality output. VITA-Audio is trained solely on open-source data, fully open-sourced, and evaluated comprehensively on ASR, TTS, and speech question answering benchmarks. Results show that VITA-Audio is competitive with existing models of similar size in both efficiency and accuracy, while offering low-latency generation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper proposes a simple and practical solution to reduce token latency in interleaved speech-language models, which is important for enabling more human-like, real-time conversational systems.\n\n- The model is fully open-source and reproducible, trained entirely on public datasets.\n\n- The evaluation is comprehensive, covering multiple tasks (ASR, TTS, SQA) and speech language models.\n\n- The paper is clearly written and well-structured.\n\n\n**Weaknesses**\n\n- Methodological novelty is limited. The work builds on known ideas such as the fact that speech-text alignment is many-to-one, and adds incremental improvements (e.g., MCTP, adapter-based tuning). While useful, these do not represent a major modeling innovation.\n\n- The evaluation focuses mainly on semantic modeling. There is no acoustic modeling, which limits the system's ability to handle speaker consistency, prosody, or low-level audio fidelity. Tasks like speaker-consistent speech continuation are likely unsupported. Comparisons with acoustic-semantic models (e.g., Moshi) may be unfair, as VITA-Audio only models semantic tokens.\n\n- The paper does not compare against other methods to for parallel tokens generation (e.g., LoRA adapters, Medusa, speculative decoding), which could offer similar benefits in terms of latency compared to using MCTP modules.\n\n**Overall Assessment**\n\nVITA-Audio is a good engineering effort that advances real-time speech generation using open-source models and data. Although the methodological novelty is modest, the model is efficient, well-evaluated, and reproducible. It offers a practical approach to low-latency, semantic token-based speech generation.\n\nQUESTIONS:\n- MCTP novelty: The MCTP module is central to your approach. Could you clarify how it differs from existing chunked or multi-head generation strategies commonly used in LLMs and TTS systems? For instance, could LoRA adapters or other parameter-efficient methods be used instead of MCTP modules to achieve similar latency and throughput benefits?\n\n- Multispeaker synthesis: How is multispeaker TTS handled in VITA-Audio? Are speaker embeddings used in the decoder, and if so, how are they obtained and integrated during training?\n\n- Training stages: The current setup involves multiple stages. Are all of these steps necessary for good performance? For example, could the MCTP modules be trained directly in Stage 1 while fine-tuning the LLM, without needing the full staged pipeline?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}