VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
Authors: Chaoyou Fu
Haojia Lin
Xiong Wang
YiFan Zhang
Yunhang Shen
Xiaoyu Liu
Haoyu Cao
Zuwei Long
Heting Gao
Ke Li
Long MA
Xiawu Zheng
Rongrong Ji
Xing Sun
Caifeng Shan
Ran He
Venue: NeurIPS 2025 spotlight ()
Keywords: Multimodal Large Language Models
Large Vision Language Models
Primary area: applications

Abstract: Recent Multimodal Large Language Models (MLLMs) have typically focused on integrating visual and textual modalities, with less emphasis placed on the role of speech in enhancing interaction. However, speech plays a crucial role in multimodal dialogue systems, and implementing high-performance in both vision and speech tasks remains a challenge due to the fundamental modality differences. In this paper, we propose a carefully designed multi-stage training methodology that progressively trains LLM to understand both visual and speech information, ultimately enabling fluent vision and speech interaction. Our approach not only preserves strong vision-language capacity, but also enables efficient speech-to-speech dialogue capabilities without separate ASR and TTS modules, significantly accelerating multimodal end-to-end response speed. By comparing against state-of-the-art counterparts across benchmarks for image, video, and speech, we demonstrate that our omni model is equipped with both strong visual and speech capabilities, making omni understanding and interaction.

11TA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
Chaoyou Fu F Haojia Lin Xiong Wang". Yi-Fan Zhang Yunhang Shen Xiaoyu Liu 1.2 Haoyu Cao², Zuwei Long", Heting Gao?, Ke Li Long Ma Xiawu Zheng", Rongrong Ji Xing Sun Caifeng Shan 1.2 Ran He
1 State Key Laboratory for Novel Software Technology, Nanjing University Schooo of Intelligence Science and Technology, Nanjing University Tencent Youtu Lab, 1XMU, CCATA
Project Leader Corresponding Author
Demo Video: Click YouTube Link
Source Code: http:///hhub..oo//I//AMMMM//////
Abstract
Recent Multimodal Large Language Models (MLLMs) have typically focused on integrating visual and textual modalities, with less emphasis placed on the role of speech in enhancing interaction. However, speech plays crucial role in multimodal dialogue systems, and implementing high-performance vision and speech tasks remains challenge due to the fundamental modality differences. In this paper, we propose a carefully designed multi-stage training methodology that progressively trains LLM understand both visual and speech information, ultimately enabling fluent vision and speech interaction. Our approach not only preserves strong vision-language capacity, but also enables efficient speech-to-speech dialogue capabilities without separate ASR and TTS modules, significantly accelerating multimodal end-to-end response speed. By comparing against state-of-the-art counterparts across benchmarks for image, video, and speech, we demonstrate that our omni model equipped with both strong visual and speech capabilities, making omni understanding and interaction.
both
Introduction
Recent advancements in MILMSs [1, 2, 3, 6, 7, 9] have led to significant progress, particularly integration visual and textual modalities. The introduction visual information into LLMs has notably enhanced model capabilities across various multimodal tasks. However, with the growing appeal human-computer interaction, the role of the speech modality has become increasingly prominent, especially in multimodal dialogue systems. In such a system, speech not only serves key medium for information transmission but also greatly improves the naturalness and convenience of interactions. Consequently, integrating visual and speech modalities to achieve multimodal interactions has emerged as critical research focus.
The integration of vision and speech in MILMSs is not straightforward due to their inherently differences [10]. For example, visual data, such as images, convey spatial information, while speech data convey dynamic changes in time series. These fundamental differences pose challenges for simultaneous optimization of both modalities, often leading to conflicts during training. For
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The video is at Ix x real time.
bи 13 W A You seem to bejin a modern lounge or meeting room.
Figure 1: VITA-1.5 enables near real-time vision and speech interaction via end-to-end framework. allows you to turn on the camera and have a fluent speech conversation. Please see our demo video at this YouTube link.
instance, the inclusion of speech data may degrade performance on vision tasks, and vice versa. In addition, traditional -pepech systems rely separate modules for Automatic Speech Recognition (ASR) and Text-to-Speech, which can increase latency and reduce coherence, limiting their practicality in real-time applications [11, 12, 13, 14, 15].
In this paper, we introduce VITA-115, multimodal LLM that integrates vision, language, and speech through a carefully designed three-stage training methodology. The training strategy progressively incorporates vision and speech data, relieving modality conflicts while maintaining strong multimodal performance. In the first stage, we focus on vision-language by training visual adapters and fine- tuning the model with descriptive caption and visual QA data. This step establishes the model's foundational visual capabilities, enabling robust image and video understanding. The second stage introduces audio input processing by training an audio encoder using data, followed by fine-tuning with speech QA data. This stage equips the model with the ability to understand and respond to audio inputs effectively. Finally, in the third stage, we train an audio decoder to enable end-to-end speech output, eliminating the need for external TTS modules. This allows VITA-1.5 to generate fluent speech replies, enhancing the naturalness and interactivity of multimodal dialogue systems.
paired
We have conducted extensive evaluations on various benchmarks related to image, video, and speech understanding, comparing the results with both open-source and proprietary models. VITA-1.5 demonstrates comparable perception and reasoning capabilities comparable to leading image/video based MLLMs, and shows significant improvements in the speech capability.
2 Related Work
Recently, thanks to the rapid development of language models such as GPT's [16, 17], LLaMA [18, 19), Alpaca [20]. Vicuna [21], and Mistral [22], researchers have successfully extended text comprehension to multimodal understandiin//ccsonnng through techniques like multimodal alignment and instruction [25], MiniCPM-V 2.5 [26], DeepSeek-VLL [27], and SliME [28] have made significant advances in image perception and reasoning. while models like LAngVA [29] and Video-LLaVe VA [30] have showcased the latest progress in video understanding. These models increasingly capable of handling diverse data types, driving the continuous improvement of multimodal perception and understanding capabilities.
tuning. For example, models such as LLaVA [1], Qwen- VL [23], Cambrian- I [24].
Beyond visual modalities, recent years have also witnessed significant progress in incorporating speech capabilities into I.L.s,, driven by the increasing demand for natural human-computer interac- tion. The dominant approach has been cascade ASR, LLM, and TTS modules. This text-centric

---

approach faces fundamental limitations due to the loss of paralinguistic features like tones and emotions. While works like [31] and [32] have attempted address these issues by incorporating speech encoders and emotion vectors, they still rely on speech transcription, resulting in substantial latency issues that impact the user experience. The emergence of proprietary models like GPT-4o [33] has demonstrated the possibility of end-to-end speech interaction, inspiring a new wave of research in speech-enabled MLLMs. Following this trend, several notable works have emerged in the open- source community. Models such as Mini-Omni2 [34], LLaMA-Omni [35], and Moshi [36] have explored various strategies for aligning speech modality with LLMs and achieving duplex dialogue capabillties. While these open-source efforts have successfully enabled duplex speech interaction with LLMs, they still lack the capability to handle visual modalities as demonstrated by GPT-4o, limiting their applications in scenarios requiring both visual and speech understanding.
Despite these advances both visual and speech modalities, a significant gap remains between proprietary and open-source models. Compared to proprietary models that support multiple modalities, including audio, image, and text, e-g., GPT-4o [37] and Gemini-Pro 1.5 [38], most open-source models have primarily focused on image and text modalities [2]. Moreover, few open-source models have involved multimodal interaction capabilities, which is a relatively unexplored area. While works like VITA-1.0 [12] have made initial attempts to introduce speech for human-compute interaction, introducing additional speech data poses challenges to the model's original multimodal abilities. Furthermore, speech generation typically relies on existing TTS systems, which often results in high latency, thus impacting user experience. In this paper. we present VITA-1.5 that leverages refined training strategics, excelling in perceiving data across four modalities (video, image, text, and audio), while also realizing near real-time vision and speech interaction.
3
-1T
3.1 Model Architecture
The overall architecture of VITA-1.5 is depicted in Fig. 2. The input side is the same as that of the VITA-1.0 version [12], that is, adopting the configuration of "Multimodal Encoder-Adaptor- LLM". It combines the Vision/Audio Transformer and the Multi- Layer Connector with an LLM for joint training, aiming enhance the unified understanding of vision, language, and audio. With respect to the output side, VITA-115 has its own end-to-end speech module, instead of using the external TTS model like the original VITA-1.0 version.
Discrete Token
Text Speceh
Codec Decoder
NAR Soeech Decoder
AR Speech Decoder
VITA-1.5
Vision Adaaneer
Spesch Ailapter
Vision Frecader
Spozch
3.1.1 Visional Modality
Image
Speech
Visual Encoder. VITA-1.5 adopts Intern as the visual encoder, with an input image size of 448x448 pixels, generating 256 visual tokens per image. For high-resolution images, 11T employs a dynamic patching [39] strategy to capture local details, improving the accuracy of image understanding.
Figure 2: Overall Architec- ture of VITA-1.5. The input side consists of vision and au- dio encoders, along with their adapters. The output side has an end-to-end speech gener- ation module, rather than di- rectly using an TTS model.
Video Processing. Videos are treated as a special type of multiple- image input. If the video length is shorter than 4 seconds. frames are uniformly sampled; for videos between 4 and 16 seconds, one frame per second sampled; for videos longer than 16 seconds, 16 frames are uniformly sampled. No dynamic patching is applied to video frames to avoid excessive visual tokens that could hinder processing efficiency.
Vision Adapter. A two-layer MLP is used to map the visual features to visual tokens suitable for the subsequent understanding of LLM.
http:://wugggggggeccooo////////s 448px
3

---

3.1.2 Audio Modality
Speech Encoder. Similar [40], our audio encoding module consists of multiple downsampling convolutional layers (4x downsampling) and 24 Transformer blocks (with a hidden size of 1024). The downsampling layers help reduce the frame rate of the audio features, improving the processing speed of LLM. The audio encoder has about 350M parameters and an output frame rate of 12.5Hz. Mel-filter bank features are used as the input of the audio encoder, with window size of 25ms and shift of 10ms [40].
Speech Adapter. It consists of multiple convolutional layers with 2x downsampling.
Speech Decoder. TiCodec [41] used our codec model, customizing single codebook with size of 1024. This single-codebook design simplifies the decoding process during the inference phase. The codec model responsible for encoding continuous speech signals into discrete speech tokens with the frequency of 40Hz, and at the same time has the ability to decode them back into speech signals with the sample rate of 24.000Hz.
The current L.M can only output text tokens, and the speech generation capability requires the LLM be able to output speech tokens. To this end, we add speech decoders after the text tokens following [40]: 1) Non-Autoregressive (NAR) Speech Decoder, which processes text tokens globally and models semantic features, with the aim of generating an initial distribution of speech tokens; 2) Autoregressive (AR) Speech Decoder generates higher quality speech tokens step by step, based on the speech information produced by the NAR decoder. The final sequence of speech tokens is then decoded into continuous speech signal flow (waveform) using the speech decoder of the Codec model. We adopt LLaMA decoder layers for both NAR and AR speech decoders, where the hidden size is 896 and the parameter size is about 120M.
3.2 Training Data
As shown in Table 1, the training data of multimodal instruction tuming encompass a wide range of categories. such as caption data and QA data, both Chinese and English. During different training phases, subsets of the overall dataset are selectively sampled to serve different objectives. Specifically, the datasets are categorized as follows:
Image Captioning Data. Datasets such as ShareGPT4V [42], ALLaVA-Captio [43], CeriGGGPTo-Imaagee and synthetic data are used to train the model to generate descriptive languages for images.
Image QA Data. Datasets like LLaVA-150K3,
[1], LVIS- Instruct [44], ScienceQA [45], ChatQA [46], and subsets sampled from LLAVA-OV [47], such as general image QA and mathematical reasoning datasets, are utilized to train the model in answering image-based questions and performing visual reasoning tasks.
This category supports the model in understanding OCR and dia- gram content, using datasets such as Anyword-3M [48], ICDAR2019-LSVT UReader [49], SynDOG ICDAR2019-LSVT-QA
OCR & Diagram Data.
and corresponding data sampled from LLaVA-OV.
Video Data. Datasets like ShareGemini [50] and synthetic data are used to train the model to handle video inputs and perform tasks such as captioning and video-based QA.
Pure Text Data. This category enhances the model's capability to understand and generate languages, facilitating text-based QA tasks.
In addition to the image and video data listed in Table 1, 110,000 hours of internal apeech-transcrige paired ASR data, covering both Chinese and English, are incorporated to train the audio encoder and align the audio encoder with the LLM. Furthermore, 3,000 hours of text-speech paired data generated by TTS system are used to train the speech decoder.
hacrp:://shareggtoo.gttee 1o/ "nctp://icaa22119.org
"naver-ddoore--//ptthdo---- aactp://1cdar2019.org

---

Table 1: Training data of multimodal instruction tuning. The images of the synthetic data come from
open-source datasets like Wukong [51], LAION [52], and CC12M [53].
Data Scenario
QA Type
Dataset Name
Questions (K)
Language
SharcGPT4V
99.50
Eng
ALLaVA-
697.40
Eng
Description
ShareCTP4o-Image
55.50
Eng
Synthetic Data
593.70
CN
LLaVA-150K
218.36
General Image
L.aVA-Mixture-sampl
1872.10
Eng
LVIS-Instruct
939.36
Eng
ScienceQA
12.72
Eng
ChaiQA
7.39
Figg
LLaVA-OV General
1754.65
Eng
LLaVA-OV Math Reasoning
1140.92
Eng
Synthetic Data
212.68
CN
Anyword-3M
1709.30
ICDAR2019-LSWV
366.30
CN
Description
UReader
100.00
Fing
SynDOG-EN
100,00
Eng
OCR Diagram
SynDOG-CN
101.90
CN
ICDARRUU9-LSVT-Q
630,08
CN
QA
LLaVA-OV Doc Chart Screen
4431.50
Eng
LLaVA-OV General OCR
404.20
Eng
Description
ShareGemini
205.70
CN
General Video
Synthetic Data
569.40
CN & Eng
QA
Synthetic Data
4336.30
CN & Eng
Pure Text
QA
Synthetic Data
1574.20
CN & Eng
Total
22133.16
CN & Eng
3.3 Three Stage Training Strategies
In order to ensure that VITA-1.5 performs well in tasks involving vision, language, and audio, we have to face a key challenge, i.e., training conflicts between different modalities, For example, adding the speech data could negatively impact the understanding of the vision data, as the features of speech differ significantly from those of vision, causing interference during the learning process. To address this challenge, we devise three-stage training strategy as shown in Fig. 3. The core idea to gradually introduce different modalities into the model, allowing it to increase the power of new modality while maintaining the power of the existing modalities.
3.3.1 Stage 1: Vision-Language Training
Stage 1.1 Vision Alignment. In this stage, our goal is to bridge the gap between vision and language. The features of the former are extracted from the pre-trained vision encoder Intern ViT-300M, and the latter is introduced through the LLM. We use 20% the deseriptive caption data from Table for training, where only the visual adapter is trainable, while the other modules are frozen. This approach allows the LLM to initially align the visual modality.
Stage 1.2 Vision Understanding. In this stage, our goal is to teach the LLM to transcribe image all the descriptive caption data from Table 1. During this process, the
content. Toward this end, encoder and adapter of the visual module, as well as the I.LM, are trainable. The focus to enable the model establish strong connection between vision and language by learning from descriptive texts about images, allowing it to understand image content via generating natural language descriptions.
Stage 1.3 Vision SFT. Following Stage 1.2, the model has acquired basic understanding of images and videos. However, the instruction following ability is still limited, and it difficult to cope with the visual QA task. To achieve this, descriptive caption data to increase the diversity of the dataset and the complexity of the tasks.
all the QA data from Table while retaining 20% of the
During training, the encoder and adapter of the visual module, as well as the LLM, are trainable. The key objective of this stage to enable the model not only to understand visual content but also to answer questions following instructions.
5

---

Stage 1 Vision-Lagguage Training
Stage 1.1
Stage 1.2 Vision Understanding
Stage 1.3 Vizion SPT
Viston Alignment
Icon Description
LLM
LLM
LLM
Discrete Text Token
Discrete Speech Token
Vision Adapter
Adapter
Vision Adiepter
Continmous Task Tokon
Virual
Training
Encoder
Eccoder
Enceder
Prozen
Image
Video
Text
Image
Image
20% Caption data
100% Caption data
20% Caption 100% Q4
Stage Audio Inpot Tuning
Stage Audio Output Tuning
Stage 2.1
Stage 2,2 Audio
Stage 3.1
Stage 3.2 NAR
Audio Aligument
Codec Training
Speech Query
Decoder Training
Query
Text
dama
LLM
LLM
Codec Decoder
Speech Decuder
Specch Eneoder
Speech Encoder
Vision Adapter
Speech Adapter
Speech Decoder
=ll
Visual Encoder
Speech Eneader
Guidance Token
Codoc
LLLM Embodding
Speech
Speech
=
-Iluu
(a)
(b)
Image
Video
Text
Speech
Speech
Text
Specck-trameriipion
Sneech/Text Caption (4%) (24 (20%)
Text Speech Data
Figure 3: Training Pipeline of VITA-1.5. The training process is divided into three stages to incrementally incorporate vision and audio into the LLM while relieving modality conflicts. Stage I focuses Vision-Language Training, including vision alignment (Stage 1.1, using 20% caption data from Table 1), vision understanding (Stage 1.2, using 100% caption data), and instruction tuning for visual QA (Stage 1.3, using 20% caption data and 100% QA data). Stage 2 introduces Audio Input Tuning, with audio alignment (Stage 2.1, utilizing 1,000 hours of
pairs) and instruction tuning for speech QA (Stage 2.2, sampling 4% caption data and 20% QA data). Finally, Stage focuses on Audio Output Tuning, including the training of the codec model (Stage 3.1, using 3,000 hours of text-speech data) and speech decoder training (Stage 3.2). The percentages shown in the image correspond to the data sampling ratios specified Table 1.

---

3.3.2 Stage 2: Audio Input Tuning
Stage 2.1 Audio Alignment. After completing the training of Stage 1, the model has developed a strong foundation in image and video understanding. In this stage, our goal to reduce the discrepancy between audio and language based on Stage 1, enabling the LLM to understand audio inputs. The training data consists of 11,000 hours of speech-transcription pairs. We follow a two-step approach: (a) Speech Encoder Training: We adopt training framework used in common speech recognition systems, using a Connectionist Temporal Classification (CTC) loss function [54] to train the speech encoder. The aim is for the encoder to predict the transcription text from the speech input. This step ensures that the audio encoder can extract speech features and map them to the text representation space. (b) Speech Adapter Training: After training the speech encoder, we integrate it with the LLM, using an audio adapter to introduce audio features into the input layer of the LLM. The training objective this stage is to enable the LLM to output the transcription text of the speech data.
Besides, in step (b), we introduce special trainable input tokens to guide the speech understanding process. These tokens provide additional contextual information that guides the LLM used for the QA task to perform the ASR task.
Stage 2.2 Audio SFT. The focus of this stage is to introduce the QA functionality with speech questions and text answers. To achieve this, we sample 4% of the caption data and 20% of the QA data from Table 1. In terms of data processing, approximately half of the text-based questions are randomly replaced with their corresponding speech versions, generated using TTS system.
In this stage, both the visual encoder and adapter, the audio encoder and adapter, as well as the LLM are trainable, aiming improve the model's adaptability with multimodal inputs. In addition, we add a classification head to the LLM's output. This head is used to distinguish whether the input comes from speech text. As result, the model can more accurately interpret speech inputs and process different modalities efficiently and flexibly.
3.3.3 Stage 3: Audio Output Tuning
In the first two stages of training. the VITA-1.5 model has effectively developed its multimodal understanding capabilities. However, crucial capacity, i.e., speech output, remains absent, which is essential for its role as an interactive assistant. To introduce speech output functionality without compromising the model's fundamental abilities, we draw on the strategy [40]. using 3,000 hours of text-speech data and employing a two-step training approach (see Fig. 3).
Stage 3.1 Codec Training, The goal of this step is to train a codec model with single codebook using speech data. The encoder of the codec model has the ability to map speech discrete tokens. while the decoder can map the discrete tokens back to speech stream. During the inference phase of VITA-1.5, only the decoder is used.
Stage 3.2 NAR + AR Decoder Training. The training of this stage uses text-speech paired data, where the text fed into the tokenizer and the embedding later of the LLM obtain its embedding vectors, and the speech is fed into the encoder of the codec model to obtain its speech tokens. The text embedding vectors are sent the NAR speech decoder to get global semantic features, and then the features are sent to the AR speech decoder, which predicts the corresponding speech tokens. Note that the LLM frozen during this stage, thus the multimodal performance is not affected.
4 Evaluation
4.1 Vision-Language Evaluation
Baselines. We compare a series of open-source MLLMs, including VILA-1.5 LLaVA-Next [56], Cog VLM [57], Inteml.M-XComposee2255 [58], Cambrian-1 [24], MiniCPM-V-2.6 [26], O [59],
Intern VL-Chat- InternV--2 [60], LLa VA-OV [47], and Video-LLaVA [30], SliME [28]. and
Long VA [29], as well as 5 closed-source MLLMs, including GPT-4V², GPT-4o", GPT-40-mini, Gemini 1.5 Pro [38], and Claude 3.5 Sonnet".
card/
sonnet
7

---

Table 2: Evaluation on Image Understanding Benchmarks. VITA-1.5 shows performance compa- rable to the leading open-source models and advanced closed-source counterparts. MMB refers to MMBench, MMS to MMStar, Hal to HallusionBench, Math to Math Vista, and OCR to OCRBench. Note that after the training of Stages 2 (Audio Input Tuning) and (Audio Output Tuning), VITA-1.5 retains almost its original visual-language capabilities in Stage (Vision-Language Training).
Method
L.M
MMB
MMS
MMMU
MathV
Hal
A120)
OCR
MMVet
MME
Avg
VILA-113
Vicuna-v 1.5-188
68.5
A1.1
45.0
1718.2
32.1
LLaVA-Next
Yi-34b
77.8
51.5
48.8
40.4
34,8
78.9
574.0
50.7
2006.5
58.3
CogVLM2
70.7
50.5
42.6
38.6
41.3
73.4
757.0
57.8
1869.5
58.8
InternL M-Xcompose22
12177LL.22-88
22204
Cambrian
Now-Hermess- 2--------ii- 2-Yi-34B
77.8
54.2
50.4
50.3
41.6
79.5
591.0
53.2
2049.9
61.4
IntemVL-Char-Il
Intern. M2-20B 20B
79.7
57.1
46.8
54.7
47.4
80.6
720.0
55.4
2189.6
65.1
Ovis1.5
Gemmaz-98-44
49.7
2125.2
Intem
IntemLLM2,,570
79.4
61.5
51.2
58.3
45.0
83.6
794.0
54.3
2215.1
67.3
MiniCPM-V 2.6
Qwen2 7B
78.0
$7.5
49.8
60.6
48.1
82.1
852.0
60.0
2268.7
68.5
Proprietary
GPT-4V
65.5
50.4
59.1
48.2
39.3
71.4
678.0
49.0
1790.3
58.5
GPT-4o mini
76.0
54.8
60.0
52.4
46.1
77.8
785.0
66.9
2003.4
66.3
Gemini 1.5 Pro GPT-4o Claude3.5 5 Sonnet
59.1
60.6
57.7
45.6
79..1
754,0
64.0
2110.6
67.2
82.8
61.6
62.8
56.5
51.7
77.4
663.0
66.5
2328.7
69.3
78.5
62.2
65.9
61.6
49.9
80.2
758.0
66.0
1920.0
69.3
Open
VITA-LO VITA-1.5 (Stage I)
Mixtral-Ba 8x
71.8
46.4
47.3
44.9
39.7
73.1
678.0
41.6
2097.0
57.8
Qwen2 7B
77.1
59.1
53.1
66.2
44,1
80.3
752.0
51.1
2311.0
67.1
VIIA-115-AAdii (Stage
Qwen2-7B
2352.0
Table Evaluation on Video Understanding Benchmarks. Although VITA-1.5 still lags behind models like GPT-4o and Gemini-1.5-Prrt it performs comparably to many open-source models. Note that after the training of Stages (Audio Input Tuning) and 3 (Audio Output Tuning), VITA-1.5 retains almost its original visual-language capabilities in Stage (Vision-Language Training).
Method VA
LLM
Video-MME w/o sub
Video-MME sub
MVBeuch
TempCompass
Vicuna-vI 1,5-13
39.9
41.6
49.8
SliME
Llama3-8B-Instree
47.2
LAngVA
Qwen2-7B
52.6
54.3
$7.0
VILA-1.5
Llamu3-8B-Instrue
58.8
2.0
M2-7B
LLaVA-OneVisioo
Qwen2-7B
58.2
61.5
56.7
64.2
InterVV1.22
Internl
M2.5 M2.55 7b
66.0
MiniCPM-V-2.6
Qwen2-7B
60.9
63.7
66.3
Proprietary
GPT-4o-mini
64.8
68.9
demini-1155PPP
75.0
81.3
67.1
GPT-4o
71.9
77.2
73.8
Open Source
VITA-1.0 VITA-1.5 (Stage I) VITA-1.5 (Stage 3)
Mixtral 8x7B
55.8
59.2
62.3
Qwen2-7B
56.8
59.5
56.8
65.5
Qwen2-78
55.4
Evaluation Benchmarks. To assess the image perception and understanding capabilities of VITA-
1.5, we utilize several evaluation benchmarks, including MME [61], MMBench [62]. MMSttar [63]. MMMU [64], Math Vista [65], HallusionBench [66]. A12D [67], OCRBench [68], and MM [69]. These benchmarks cover wide range of aspects, including general multimodal capabilities (e.g.. MME, MMBench, and MMMU), mathematical reasoning (Math Vista), hallucination detection (HallusionBench), chart (A12D) and OCR (OCRBench) understanding, providing comprehensive evaluation results. For video understanding, we use representative evaluation benchmarks including Video-MME [70], MVBench [71], and TempCompass [72].
Vision-Language Capabilities. Table presents a comparison of VITA-1.5's image understanding performance. After the training of the three stages, VITA-1.5 performs comparably to the most advanced open-source models and even surpasses some closed-source models like GPT-4V and GPT-4o-mini This result highlights the robust capabilities of VITA-1.5 in image-language tasks. As shown Table 3, VITA- shows comparable performance the top open-source models in the evaluation of video understanding. The notable gap compared to proprietary models suggests that V1TA-1.5 still has significant room for improvement and potential for further enhancement in video understanding. Please note that after the training of Stages 2 (Audio Input Tuning) and (Audio Output Tuning), VITA-1.5 retains almost its original visual-language capabilities in Stage Vision-Language Training).
8

---

Table 4: Evaluation on ASR Benchmarks. VITA-1.5 has demonstrated strong performance in both Mandarin and English ASR tasks. in both languages.
outperforms specialized speech models, achieving better results
CN (CER.)
Eng (WER.)
Model
aishell-1
test net
test meeting
dev clean
dev other
test clean
test other
Wav2ve22-base
6.0
13.4
Mini-Omni2
4.8
9.8
4.7
9.4
Freeze-Ommi
2.8
12.6
14.2
4.2
10.2
4.1
10.5
VITA-1.0
12.2
16.5
7.6
16.6
8.1
18.4
VITA-1.5
2.2
8.4
10.0
3.3
7.2
3.4
7.5
4.2 Speech Evaluation
Baselines. The following three baseline models are used for comparison: Wav2vec2-base [73],
Mini-Omni2 [74], Freeze-Omni [40], and VITA-1.0 [12].
Evaluation Benchmarks. The Mandarin Evaluation Sets consists of three datasets: aishell-1 [75], test net [76], and test meeting [77]. These datasets are used to evaluate the model's performance on Mandarin speech. The evaluation metric is the Character Error Rate (CER). The English Evaluation Sets include four datasets: dev-clean, dev-other, test-clean, and test-other [78], which are used to evaluate the model's performance
English speech. The evaluation metric is Word Error Rate (WER). The evaluation results in Table indicate that VITA-1.5 achieves leading accuracy both Mandarin and English ASR tasks. This demonstrates that VITA-1.5 has successfully integrated advanced speech capability to support multimodal interaction.
Conclusion and Future Work
this paper, we has presented VITA-1.5, multimodal LLM designed to integrate vision and speech through a carefully crafted three stage training strategy. By relieving the inherent conflicts between modalities, VITA-1.5 achieves robust capabilities both vision and speech understanding, enabling efficient speech-to-specee interactions without relying separate ASR TTS modules. Extensive evaluations demonstrate that VITA-1.5 performs competitively across multimodal benchmarks. We hope that VITA-1.5 promote the progress of open-source models in the field of real-time multimodal interaction. Although VITA-1.5 has made some contributions, such as multi-modality joint training, end-to-end architecture, response latency, and basic performance, there two major areas that can be improved in our future work:
1. Personalized MLLM. Currently, VITA-1.5 is generic and do not incorporate individual preferences during interaction. For example, after learning about personal preferences content and manner of answers be adjusted accordingly.
interaction, the
Long-term memory. The process of human- computrr interaction can last 10 minutes or even several hours, in which case it is important for the human-comp interaction in real scenarios.
Impact Statement
This paper studies the technology of large models to enhance their technical level. Its influence the same that of the research other large models and will not be repeated here.
Acknowledgments
This work is funded by National Natural Science Foundation of China (Grant No. 62506158 and No. 62441234), Fundamental Research Funds for the Central Universities, Al & for Science Project of Nanjing University (No. 2024300529), and CCF-Tencent Chino-Bird Open Research Fund.

---

References
[1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. arXiv preprint arXiv:2304.0848 2023.
[2] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal Ilm with discrete sequence modeling. arXiv preprint arXiv:2402.1 222222222.2222 12226, 2024.
[3] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tunee audio- language model for video understanding. arXiv preprint arXiv:2306.00288 2023.
[4] Jinsong Zhang, Minjie Zhu, Yuxiang Zhang, Zerong Zheng. Yebin Liu, and Kun Li. Speechact: Towards generating whole-body motion from speech. IEEE TVCG, 2025.
[5] Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio andio-visual large language models. arXiv preprint arXiv:2406. 15704, 2024.
[6] Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie. Audio-visual I1m for video understanding. arXiv preprini arXiv:23/2.0677 2023.
[7] Hao Wen, Hongho Kang, Jian Ma, Jing Huang, Yuanwang Yang, Haozhe Lin, Yu-Kun Lai, and Kun Li. Dycrowd: Towards dynamic crowd reconstruction from a large-scene video. IEEE TPAMI, 2025.
(8] Yunlong Tang, Daiki Shimada, Jing Bi, Minggian Feng, Hang Hua, and Chenliang Xu. Empow- ering iims with pseudo-untrimmed videos for audio-visual temporal understanding. In AAAI, 2025.
[9] Xiaoguang Tu, Zhi He, Yi Huang, Zai-Hao Zhang, Ming Yang. and Jian Zhao. overview of large ai models and their applications. Visual Intelligence, 2024.
[10] Dan Oneată and Horia Cucu. Improving multimodal speech recognition by data augmentation and speech representations. In CVPR, 2022.
[11] V Madhusudhana Reddy, T Vaishnavi, and K Pavan Kumar. Speech-to- -oetettt--eetteeee and text-to-speech recognition using deep learning. In ICECAA. IEEE, 2023.
[12] Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, et al. Vita: Towards open-source interactive omni multimodal Ilm. arXiv preprint arXiv:2408.05211
2024.
[13] Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic tross-modal conversational abilities. arXiv preprint arXiv:2305. 10000 2023.
[14] Jingying Liu, Binyuan Hui, Kun Li, Yunke Liu, Yu-Kun Lai, Yuxiang Zhang, Yebin Liu, and Jingyu Yang. Geodetry-guided dense perspective network for speech-driven facial animation. IEEE TVCG, 2021.
[15] Jinsong Zhang, Xiongzheng Li, Hailong Jia, Jin Li, Zhuo Su, Guidong Wang, and Kun Li. Logavatar: Local gaussian splatting for human avatar modeling from monocular video. CAD, 2025.
[16] OpenAl. Gpt-4 technical report. 2023.
[17] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. NeurIPS, 2020.
[18] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
10

---

[19] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2307.09220 099888, 2023.
[20] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followin llama model, 2023.
[21] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* R chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023). 2023.
[22] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06625, 2023.
[23] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 0e0e308..2990 /2966, 2023.
[24] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang. Adithya Iyer, Xichen Pan, al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint M.XXv:2246.16860 2024.
[25] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.1888 2024.
[26] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui. Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06399 2024.
[27] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprini urXiv:2403.05525, 2024.
[28] Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang. Zhang Zhang, Liang Wang, and Rong Jin. Beyond Ilava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406. 08487, 2024.
[29] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024.
[30] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint 2r11022311.10122, 2023.
[31] Hongfei Xue, Yuhao Liang, Bingshen Mu, Shiliang Zhang, Qian Chen, and Lei Xie. E- chat: Emotion--ensitive spoken dialogue system with large language models. arXiv preprint arXiv:2401.0044 2023.
[32] Guan- Ting Lin, Prashanth Gurunath Shivakumar, Ankur Gandhe, Chao-Han Huck Yang, Yile Gu, Shalini Ghosh, Andreas Stolcke, Hung-yi Lee, and Ivan Bulyko. Paralinguistic--nhancee large language modeling of spoken dialogue. arXiv preprint 1321:22312.15316, 2023.
[33] OpenAl. htttps://optt wwwp///eee////eeo/oooo.oo htt.in/eeeeeelllog-//////// 2024.
[34] Zhifei Xie and Changqiao Wu. Mini-omni2: Towards open-source gpt-4o model with vision, speech and duplex. arXiv preprint arXiv:2410.11190) 2024.
[35] Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. arXiv preprint 24Xiv:2409.066 06666, 2024.
11

---

[36] Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer. Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dialogue. arXiv preprini arXiv:2410.00037, 2024.
[37] OpenAl. Hello gpt-4o. 2023.
[38] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv: /880, 2023.
[39] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint 2rXiv:240..18821 2024.
[40] Xiong Wang, Yangze Li, Chaoyou Fu, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: smart and low latency speech-to-speech dialogue model with frozen Ilm. arXiv preprint arXiv:2411.0077.
2024.
[41] Yong Ren. Tao Wang, Jiangyan Yi, Le Xu, Jianhua Tao, Chu Yuan Zhang, and Junzuo Zhou. Fewer-token neural speech codee with time-invariant codes. In ICASSP. IEEE, 2024.
[42] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311. 127933 2023.
[43] Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Hamesssing gpt4v- synthesized data for a lite vision-language model, 2024.
[44] Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint 1..554 2023.
[45] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 2022.
[46] Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa: Surpassing gpt-4 on conversational qa and rag. arXiv preprint arXiv:2401. 10225, 2024.
[47] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint 1rX8v:2408.03.3 2024.
[48] Yuxiang Tuo, Wangmeng Xiang, Yan He, Yifeng Geng, and Xuansong Xie. Anytext: Multilingual visual text generation and editing. arXiv preprint arXiv:2311.03054 2023.
[49] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Jun- feng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05122 2023.
[50] Share. Sharegemini: Scaling up video caption data for multimodal large language models, June 2024.
Somm/hareea 14 4/ SSarrnemmin.
[51] Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Niu Minzhe, Xiaodan Liang, Lewei Yao, Runhui Huang. Wei Zhang, Xin Jiang, et al. Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark. NewrIPS, 2022.
[52] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
12

---

[53] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
[54] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, 2006.
[55] Ji Lin, Hongxu Yin. Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023.
[56] Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal capabilities in the wild, May 2024.
[57] Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408, 16500, 2024.
(58) Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang. Haodong Duan, Hang Yan, et al. Internim sccopo--composer vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv: 12 2023.
(59) Shiyin Lu, Yang Li, Qing-Guu Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprini arXiv:2405.20797 2024.
[60] Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual- visual-linguistic tasks. arXiv preprint arXiv:2312 142.3888,, 2023.
[61] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
[62] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui Hc, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:23007. 06281/ 2023.
[63] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang. Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, al. Are the right way for evaluating large dision-language models? arXiv preprint arXiv:2403.203: 2024.
[64] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567, 2024.
[65] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255. 2023.
[66] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: advanced diagnostic suite for entangled language hallucination and visual illusion in large vasion-l models. In CVPR. 2024.
[67] Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski, Evanfiya Logacheva, Serafina Orekhova, Aino Tuomainen, Matthew Stone, and John A Bateman. Ai2d-rst: A multi- modal corpus of 1000 primary school science diagrams. Language Resources and Evaluation, 2021.
13

---

[68] Yuliang Liu, Zhang Li, Biao Yang. Chunyuan Li, Xucheng Yin, Cheng- lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of oer in large multimodal models. arXiv preprint urXiv:2305.07895, 2023.
[69] Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Me-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint 2rXiv:2308.024999 2023.
[70] Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075. 2024.
[71] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, al. Mvbench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024.
[72] Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video Ilms really understand videos? arXiv preprint arXiv:2403.004700 40300046,, 2024.
[73] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. NewrIPS, 2020.
[74] Zhifei Xie and Changqiao Wu. Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities. arXiv preprint arXiv:2410.11190, 2024.
[75] Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Aishell-1: An open- source mandarin speech corpus and speech recognition baseline. In O-COCOSDA, IEEE, 2017.
[76] Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint orXiv:2106.0699, 2021.
[77] Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang. Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, et al. Wenetspeech: 10000+ hours multi-domain mandarin corpus for speech recognition. In ICASSP, 2022.
[78] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In ICASSP, 2015.
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
No theorem and lemma.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 3.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: The code open-source, and the data processing methods have been clearly defined. The experimental settings are also included, making the experiments reproducible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. ctips.cc/public//uu guwdee/dss///// CCodeSubeesiaceecceee simiisssinn sssssnoliccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 3.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[NA)
Justification: The evaluations are performed with temperature=0, resulting in minimal variance. Due to the high computational cost of training from scratch, it is difficult to provide results from multiple reruns (5-10).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / htt///eurip..///
Answer: [Yes]
Justification:
This work meets the requirements.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section 5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work is not applicable to this.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The corresponding works are cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This work is applicable this.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work is not applicable to this.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work is not applicable to this.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This work is not applicable this.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https:/ http://eeeeepps..cooeeeeeeeeeeee / for what should or should not be described.
21


Reviews:
[{"review_id": "dPOBLXtfkR", "reviewer": "Reviewer_DC1q", "summary": "This paper proposes VITA-1.5, a multimodal large language model that supports near real-time vision-and-speech interaction. It introduces a three-stage training strategy to integrate visual understanding and speech capabilities into a base language model, thereby enabling end-to-end speech-to-speech dialogue without separate ASR or TTS components. The contributions include a significant reduction in response latency (from ~4 seconds down to ~1.5 seconds in a demo setting), enhanced multimodal performance on benchmark tests, and improved speech recognition accuracy. Extensive experiments on image, video, and speech benchmarks demonstrate that VITA-1.5 performs competitively with state-of-the-art systems.", "questions": "1. Closing the gap with proprietary models. VITA-1.5 still trails leading closed-source systems (e.g., GPT-4o, Gemini-Pro) on several benchmarks, particularly for video understanding and high-level reasoning. What potential methods do the authors envision to further narrow this performance gap?\n\n2. Discuss the difference between VITA-1.5 and other video understanding models that can also accept video and audio as input.\n\n3. This is a typo in line 285.", "limitations": "See weaknesses", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Significant Latency Reduction. Through an end-to-end codec-based speech decoder, VITA-1.5 cuts the total “speaker-to-reply” latency from ~4 s to 1.5 s in a demo setting \n\n2. Enhanced Multimodal Accuracy. Average performance on core vision–language benchmarks (MME, MMBench, MathVista, etc.) rises sharply from 59.8 to 70.8\n\n3. Substantial Speech Gains – English ASR Test-Other WER falls from 18.4 % to 7.5 %, and Mandarin CER also improves, while an integrated, end-to-end TTS module replaces separate ASR/TTS pipelines\n\nWeaknesses:\n\n1. Lack of human evaluation. Automated benchmarks, though informative, are imperfect surrogates for real-world user experience. A complementary human evaluation study would provide stronger evidence of VITA-1.5's practical effectiveness.\n\n2. Missing metric on human‐initiated interruptions. In natural conversation, a speaker can be interrupted at any time. Could the authors report whether users are able to interrupt VITA-1.5’s speech output, and, if so, what the interruption success rate is?\n\n3. Although cutting latency from 4 seconds to 1.5 seconds is a substantial improvement, 1.5 seconds is still longer than typical human reaction time. Further reductions are therefore needed before the system can truly emulate real-time human conversation.", "was_revised": true, "final_justification": "Thank you for the detailed rebuttal that addressed my concerns. I have also read the comments from other reviewers and the authors' responses. Overall, I am convinced by the clarifications and will raise my score to recommend acceptance of this paper.", "rebuttal": "We sincerely thanks for your positive recognition of our work. Our point-by-point responses to all your comments are provided below.\n\n------\n\n**Q1: Lack of human evaluation. Automated benchmarks, though informative, are imperfect surrogates for real-world user experience. A complementary human evaluation study would provide stronger evidence of VITA-1.5's practical effectiveness.**\n\n**R1**: Thanks for your constructive suggestion. To complement the quantitative results, we have conducted an initial **human evaluation study** to assess the practical effectiveness of VITA-1.5 in **real-time multimodal interactions**.\n\nSpecifically, we invited **15 participants** to interact with the deployed VITA-1.5 system through **open-ended video + speech conversations**, and rate their experience along three dimensions:\n\n1. **Answer Satisfaction (AS)** – Does the system give helpful, relevant answers?  \n2. **Interaction Fluency (IF)** – Is the response timing and speech delivery smooth and natural?  \n3. **Interruption Naturalness (IN)** – How naturally does the system handle speech interruptions?\n\nEach dimension was scored on a **0–5 scale**, with higher being better. We also included VITA-1.0 as a baseline for comparison. The results are as follows:\n\n| Model     | AS  | IF  | IN  |\n|-----------|-----|-----|-----|\n| VITA-1.0  | 3.7 | 3.5 | 4.2 |\n| VITA-1.5  | 4.1 | 4.4 | 4.4 |\n\nThese results show that **VITA-1.5 significantly improves user experience**, particularly in terms of fluency and interaction smoothness.\n\nDue to time constraints, we were not able to conduct a larger-scale or more detailed user study. However, we plan to expand this evaluation in future work, including more participants, additional metrics, and comparisons across system settings.\n\n------\n\n**Q2: Missing metric on human‐initiated interruptions. In natural conversation, a speaker can be interrupted at any time. Could the authors report whether users are able to interrupt VITA-1.5’s speech output, and, if so, what the interruption success rate is?**\n\n**R2**: Thank you for pointing this out. To evaluate this aspect, we conducted an **interruption success rate test** during the rebuttal period. Specifically, while the model was still answering a prior query, a new spoken query was issued by the user. We then measured whether the system correctly halted the ongoing speech output and responded to the new input.\n\nDue to limited time, we performed **50 such duplex interaction tests**, and VITA-1.5 achieved an **interruption success rate of 98%**, demonstrating robust capability in handling real-time speech interruptions.\n\nThis confirms the effectiveness of our duplex architecture. We plan to further expand this evaluation in future work with more diverse conversational settings and metrics such as latency to interruption and interruption naturalness.\n\n------\n\n**Q3: Although cutting latency from 4 seconds to 1.5 seconds is a substantial improvement, 1.5 seconds is still longer than typical human reaction time. Further reductions are therefore needed before the system can truly emulate real-time human conversation.**\n\n**R3**: We appreciate your insightful observation. Indeed, while 1.5 seconds is still longer than typical human reaction time, it represents a substantial improvement over the ~4 seconds latency in VITA-1.0.\n\nWe fully agree that achieving near-human latency remains an important goal. Toward this, we are actively exploring:\n- **Faster decoding strategies** (e.g., faster generation of the first voice token),\n- **Pipeline parallelism** between ASR, LLM inference, and TTS stages,\n- **Hardware-aware deployment** to minimize I/O and streaming bottlenecks.\n\nWhile VITA-1.5 marks a significant step forward, we view 1.5 seconds as a milestone—not a final destination—and we are committed to pushing latency lower in future versions.\n\n------\n\n**Q4: Closing the gap with proprietary models. VITA-1.5 still trails leading closed-source systems (e.g., GPT-4o, Gemini-Pro) on several benchmarks, particularly for video understanding and high-level reasoning. What potential methods do the authors envision to further narrow this performance gap?**\n\n**R4**: We appreciate your thoughtful question. While VITA-1.5 demonstrates strong performance among open-source models, we acknowledge that there remains a performance gap compared to proprietary systems such as GPT-4o and Gemini-Pro, especially in video understanding and high-level reasoning tasks.\n\nWe believe the gap is primarily due to the following factors, which we are actively addressing:\n\n1. **Limited high-quality video data**:  \n   VITA-1.5 currently relies on open-source video data, which are often short and lack diversity. Proprietary models benefit from access to large-scale, high-quality, curated video datasets. We plan to systematically expand our training corpus using long-form, multi-domain, and task-rich video data.\n\n2. **Restricted video input length**:  \n   VITA-1.5 currently supports only 16 video frames per input, which limits its temporal modeling capability. In future versions, we aim to extend the input window significantly (e.g., 64–512 frames) using more efficient video encoders and attention mechanisms designed for long sequences.\n\n4. **Multimodal instruction tuning and CoT-style supervision**:\n   To enhance reasoning ability, we are constructing multimodal instruction datasets that include **chain-of-thought (CoT) annotations**, where intermediate reasoning steps are explicitly guided across modalities. This helps the model develop structured thinking and better interpret complex tasks involving video, audio, and text.\n\nWhile we recognize that fully closing the gap with proprietary systems is a long-term challenge, we view VITA-1.5 as a strong open-source step toward that goal. We are committed to improving along the above directions and welcome collaboration from the community to further advance open multimodal intelligence.\n\n------\n\n**Q5: Discuss the difference between VITA-1.5 and other video understanding models that can also accept video and audio as input.**\n\n**R5**: Thank you for the question. While VITA-1.5 and prior models such as Video-LLaMA series [1] and video-SALMONN series [2] all incorporate video and audio modalities, their core objectives and system designs differ substantially. Most existing audio-visual models mainly aim to better understand video (visual frames+audio), such as designing a new Q-Former to assist in synchronizing frames and audio. \n\nIn contrast, VITA-1.5 introduces speech information to assist human-computer interaction, and thus focuses more on interference between modalities, speech interruption, recognition and generation capabilities.\n\n[1] H. Zhang, X. Li and L. Bing, “Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding,” in Proc. EMNLP, Singapore, 2023. \n[2] G. Sun, W. Yu, C. Tang and et al., “video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models.”, in Proc. ICML, Vienna, 2024.\n\n------\n\n**Q6:  A typo in line 285.**\n\n**R6**: Thank you for pointing this out. We have carefully reviewed the sentence and will correct the typo in the final version. We appreciate your attention to detail."}, {"review_id": "QHD3ouOwvk", "reviewer": "Reviewer_Zxsq", "summary": "VITA-1.5 is a Multimodal Large Language Model (MLLM) integrating vision, language, and speech for real-time interaction. It employs a three-stage training approach to mitigate modality conflicts: first building vision-language capabilities, then tuning for audio input (ASR), and finally training an end-to-end speech decoder (TTS). VITA-1.5 reduces interaction latency from ~4s to ~1.5s, improves performance on multimodal benchmarks, and offers strong speech processing, making it a competitive open-source model, validated across image, video, and speech benchmarks.", "questions": "1. Can author provide a more detailed rationale or ablation studies for the specific ordering of three-stage training pipeline?\n2. What is the impact on the language model itself after each stage of training?", "limitations": "Yes, the authors have included a section on future work that acknowledges limitations such as the lack of personalization and long-term memory.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n1. The paper demonstrates an impressive engineering effort in integrating multiple modalities (image, video, speech, text) and training on a massive and diverse collection of datasets. The resulting model, VITA-1.5, shows strong performance on various open-source benchmarks, often outperforming other open-source models in vision-language and ASR tasks.\n2. The goal of achieving low-latency, real-time multimodal interaction with end-to-end speech I/O is highly relevant and challenging. Moving away from cascaded ASR-LLM-TTS systems is a valuable research direction to reduce error propagation and latency, and the paper makes a commendable attempt in this direction.\n\n### Weaknesses\n1. The title \"Towards GPT-4o Level\" sets a very high expectation that is not substantiated by the paper's own results. \n2. A primary claim is the model's ability to enable \"fluent speech-to-speech dialogue.\" While the paper provides extensive evaluation for Automatic Speech Recognition (ASR) performance (Table 4), there is a complete absence of evaluation for the Text-to-Speech (TTS) output. For an interactive model, the quality of the generated speech is paramount. There are no objective metrics (e.g., Mean Opinion Score - MOS) or subjective user studies to assess the naturalness, prosody, intonation, or speaker similarity of the generated audio. The authors claim to overcome limitations of traditional TTS systems, such as the loss of paralinguistic features, but provide no evidence to support this.\n3. The three-stage training pipeline is presented as a fixed recipe. To strengthen the contribution, the authors should have provided ablations to justify their design choices. For instance, what happens if the training order is changed (e.g., Audio Input -> Vision -> Audio Output)? How sensitive is the model to the data mixing ratios described? A deeper analysis of the modality conflicts they aim to relieve would be more insightful than just showing the final successful outcome.\n4. VITA 1.0 proposed a multi-stage training method, and VITA 1.5 does not seem to have made any major changes, so it is difficult to be considered a core contribution.\n5. The audio input is handled by a speech encoder that processes Mel-spectrograms and maps them into the LLM's semantic space. In contrast, the audio output relies on a separate codec to generate discrete audio tokens, which the LLM is trained to predict. These input and output pathways operate in fundamentally different representation spaces. The paper does not discuss the implications of this design choice. It is unclear how this disconnect affects the model's ability to learn a unified audio representation, potentially limiting capabilities like voice cloning or maintaining prosodic consistency, and whether it introduces training inefficiencies.", "was_revised": true, "final_justification": "Although the quality of the generated speech seems a bit poor (WER seems high), considering the comprehensiveness of the model, I am willing to increase my score.", "rebuttal": "Sincerely thanks for your efforts in reviewing this work. We hope the detailed responses help clarify your concerns. We would greatly appreciate it if you could kindly re-evaluate our work in light of the new explanations and additional results.\n\n------\n\n**Q1: The title \"Towards GPT-4o Level\" sets a very high expectation that is not substantiated by the paper's own results.**\n\n**R1**: Thank you for pointing this out. We agree that the title “Towards GPT-4o Level” may set high expectations, and we would like to clarify that it is intended to convey our **directional motivation**, rather than a claim of parity with GPT-4o.\n\nSpecifically:\n- Our work was directly inspired by OpenAI’s **interactive demo of GPT-4o**, which showcased real-time multimodal interaction capabilities. This motivated us to explore similar **speech–vision–language integration** within the open-source community.\n- VITA-1.5 represents one of the **earliest open-source attempts** to build an **interactive, real-time multimodal LLM** with unified **speech input and output** capabilities.\n- While our model does not yet match GPT-4o in all dimensions, we believe our **design choices, training strategies, and system deployment** offer a solid step **towards that goal**, especially considering the challenges of **open data, limited compute, and reproducibility**.\n\nWe will revise the paper to clarify this point and avoid any unintended overstatement in the final version.\n\n------\n\n**Q2: Comment on lack of evaluation for TTS output.**\n\n**R2**: Thank you for pointing this out. We have conducted both **objective** and **subjective** evaluations to assess the performance of the speech generation module.\n\n1. **Objective Evaluation**  \nWe evaluate our speech decoder on the **Seed-TTS-eval** benchmark using **WER** (lower is better) as a proxy for intelligibility:\n- VITA-1.5 achieves **2.63% (zh)** and **8.44% (en)** WER on test sets.\n- For comparison, CosyVoice achieves **4.08% / 4.07%**, but uses over **160K hours** of training data, while VITA-1.5 uses only **3K hours**.\n- This demonstrates that VITA-1.5 achieves competitive speech output quality even with limited data.\n\n2. **Subjective User Study**  \nTo assess real-world speech quality and user experience, we conducted a **user study** involving **15 participants** who engaged in open-ended multimodal conversations (video + speech) with both VITA-1.0 and VITA-1.5. Participants rated system performance (0–5 scale) on three dimensions, including Answer Satisfaction (**AS**), Interaction Fluency (**IF**), and Interruption Naturalness (**IN**):\n\n| Model     | AS | IF | IN |\n|-----------|---------------------------|----------------------------|-------------------------------|\n| VITA-1.0  | 3.7                       | 3.5                        | 4.2                           |\n| VITA-1.5  | **4.1**                   | **4.4**                    | **4.4**                       |\n\nThese results show a significant improvement in **speech delivery fluency and natural interaction** in VITA-1.5, aligning with our goal of enabling **real-time, human-like multimodal dialogue**.\n\nDue to **time constraints** in preparing this submission, we were only able to conduct these initial evaluations. We fully agree that additional metrics such as **MOS (Mean Opinion Score)**, **prosodic quality**, and **speaker similarity** would further strengthen our claims. We plan to include these in future work, and will also clarify the current results more explicitly in the revised version.\n\n------\n\n**Q3: Comment on three-stage training order and ablation.**\n\n**R3**: Thank you for the insightful suggestion. The design of our three-stage training pipeline is based on empirical findings and practical considerations regarding **data scale**, **modality interference**, and **model stability**.\n\n- We first integrate **vision input** into the LLM because **vision-language instruction data is much more abundant** and exerts a **larger influence on LLM behavior**. Training this modality first allows the model to learn stable cross-modal alignment with language.\n\n- We then add **audio input** (ASR) through **lightweight tuning**, which we found to have **minimal impact on previously learned vision capabilities**. In contrast, we observed that reversing the order—adding audio input first and then vision—tends to **destabilize the audio pathway**, likely due to the large distributional shift introduced by vision-heavy data.\n\n- **Audio output (speech generation)** is placed last, as it involves a separate decoder trained to predict codec tokens. At this point, the LLM is **fully frozen**, and the decoder is trained independently, making it suitable for this final stage.\n\nWe acknowledge that this strategy was adopted **empirically**, and we agree that **systematic ablation studies**, such as testing different modality orders or mixing ratios, would provide valuable insight into **modality interference and optimization dynamics**. Due to time constraints, we were unable to include these experiments in the current submission, but we plan to conduct and release such studies in the final version of the paper.\n\nWe appreciate the reviewer’s suggestion and will clarify the rationale behind our training pipeline in the revised version.\n\n------\n\n**Q4: VITA 1.0 proposed a multi-stage training method, and VITA 1.5 does not seem to have made any major changes, so it is difficult to be considered a core contribution.**\n\n**R4**: Thank you for raising this point. Compared to VITA-1.0, VITA-1.5 introduces **substantial and meaningful improvements** in terms of **training methodology, model capability, and system performance**. Below, we clarify the major differences:\n\n1. **Refined and Progressive Multi-Stage Training**  \n   - VITA-1.0 adopts a coarse three-stage training: (i) align vision, (ii) align audio (both with LLM frozen), and (iii) joint SFT with all modalities (LLM unfrozen). This causes **visual and audio capabilities to be fully formed only in the final joint stage**, which increases the risk of **modality interference**.\n   - In contrast, **VITA-1.5 introduces a novel six-phase progressive curriculum**, where:\n     - The model is first fully trained on vision-language tasks,\n     - Then gradually and separately exposed to audio input,\n     - Finally, joint tuning is performed only after each modality is already stabilized.\n   - This **progressive modality integration** significantly alleviates training conflicts and improves robustness. We believe this design is of broad relevance for the community working on MLLMs.\n\n2. **End-to-End Speech Generation Capability**  \n   - VITA-1.0 depends on an **external TTS module**, which leads to pipeline latency and fragmented control.\n   - VITA-1.5 integrates an **in-model speech decoder**, enabling **low-latency speech output (reduced by ~1.4s)** and real-time interaction, with support for **streaming and speech interruption**.\n\n3. **Significant Performance Gains**  \n   - On multimodal benchmarks, VITA-1.5 improves performance from **59.8 to 70.8**, and reduces ASR WER from **18.4% to 7.5%**.\n   - Additionally, VITA-1.5 demonstrates **near real-time human-computer interaction**, as validated by both latency measurements and user studies.\n\nIn summary, VITA-1.5 is **not a minor extension**, but a significant advance over VITA-1.0 across **training strategy, model architecture, and system behavior**. We will make these distinctions clearer in the revised version of the paper.\n\n------\n\n**Q5: Comment on audio input/output representation discrepancy.**\n\n**R5**: Thank you for raising this important point. This design choice reflects a **practical trade-off**:\n- Mel-spectrogram-based input encoding provides **dense, high-resolution prosodic features** for robust ASR performance.\n- Codec-based output enables **lightweight, low-latency generation**, with high-fidelity audio reconstruction and compatibility with streaming decoders.\n\nAlthough the input/output use different spaces, we have found in practice that this **does not prevent the model from learning effective cross-modal alignment**, thanks to the **central shared LLM** which serves as a semantic bottleneck. Our user study and benchmark results show that VITA-1.5 maintains good **speech understanding and generation** capabilities, with good **prosody and interaction fluency**.\n\nWe acknowledge that this architecture may currently **limit capabilities such as speaker identity retention and unified audio representation learning**. In future work, we plan to explore:\n- **Shared latent audio embeddings** (e.g., semantic speech tokens),\n- **Self-supervised audio encoders-decoders trained jointly**,\n- And **speaker/personality control** in end-to-end training.\n\nWe will add a discussion of these trade-offs and future directions in the final version.\n\n------\n\n**Q6: What is the impact on the language model itself after each stage of training?**\n\n**R6**: Thank you for the thoughtful question. Due to time constraints, we conducted experiments on three language evaluation benchmarks throughout the multi-stage training process. The results show that our progressive training strategy **preserves the core language capabilities** of the LLM with **only minimal changes**:\n\n- On **MMLU**, the score consistently stays around **74**, with **only fractional-point differences** across stages.\n- On **GPQA-Diamond**, the score remains stable at approximately **36**, with **only fractional-point differences**.\n- On **LCB**, we observed a slightly larger fluctuation (47 to 43), possibly due to its sensitivity to fine-grained commonsense and factual reasoning during modality tuning.\n\nThese results suggest that our multi-stage training procedure is effective at **minimizing disruption to the LLM’s linguistic competence**, while enabling successful integration of vision and audio modalities.\n\nWe will include this analysis in the revised version of the paper."}, {"review_id": "gGUJ4CRHFL", "reviewer": "Reviewer_RHor", "summary": "This paper introduces an omni-modal model, VITA-1.5. The training of VITA consists of three stages: the first stage focuses on acquiring visual understanding, the second stage develops speech understanding, and the third stage enables speech generation. VITA-1.5 achieves strong performance across image, video, and speech benchmarks. Demonstrations show that VITA-1.5 is capable of real-time interaction and supports features such as speech interruption.", "questions": "1. **Speech Interruption.** The video demonstration showcases the model’s ability to handle speech interruption. The authors are encouraged to include a description in the paper detailing how this feature is implemented, including what modules were introduced and which data were used.\n\n2. **Speech decoder.** The speech decoder in this paper consists of 4 layers of AR decoder and 4 layers of NAR decoder. The authors should clarify the motivation for this design choice, rather than using only AR or only NAR decoders. It would be helpful if the authors could provide an explanation or include ablation studies to justify this architectural decision.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. **Paper writing.** This paper is well written and easy to follow; the design of the 3-stage training pipeline is reasonable.\n\n2. **Interesting demo.** This paper provides a demo video that supports speech interruption and works in real-time. I like this demo video.\n\n### Weakness\n\n1. **Streaming speech output.** The speech decoder consists of 4 layers of NAR decoder and 4 layers of AR decoder. Does the NAR mean that full attention, rather than causal attention, is conducted across all audio tokens? If yes, then I think VITA-1.5 is hard to support streaming speech output. Users have to wait until all the text is generated before they can hear the speech output.\n\n2. **Lack of comparison.** VITA-1.5 achieves good performance on various image, video and speech benchmarks. The author should add comparisons with some new open-source MLLMs, such as LLaVA-Onevision[1], and Qwen2-VL[2] (It is okay if the performance is lower than Qwen2-VL series, since they didn't release training data).\n\n3. **Technical novelty.** This paper introduces a three-stage pipeline for training an Omni Model. However, the training pipeline and model architecture described are quite similar to those of other models in the field, such as Mini-Omni2[3], Llama-Omni[4], and Lyra[5]. As a research paper, I think it is important for the authors to clearly highlight the methodological novelty of VITA-1.5, compared to other related works.\n\n[1] Li, Bo, et al. \"Llava-onevision: Easy visual task transfer.\" arXiv preprint arXiv:2408.03326 (2024).\n\n[2] Wang, Peng, et al. \"Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution.\" arXiv preprint arXiv:2409.12191 (2024).\n\n[3] Xie, Zhifei, and Changqiao Wu. \"Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities.\" arXiv preprint arXiv:2410.11190 (2024).\n\n[4] Fang, Qingkai, et al. \"Llama-omni: Seamless speech interaction with large language models.\" arXiv preprint arXiv:2409.06666 (2024).\n\n[5] Zhong, Zhisheng, et al. \"Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition.\" arXiv preprint arXiv:2412.09501 (2024).", "was_revised": true, "final_justification": "The author's detailed rebuttal has basically addressed my concerns. After reading the author's responses to the other reviewers' comments, I have decided to raise my score to 5.", "rebuttal": "We sincerely thanks for your positive recognition of our work. Our point-by-point responses to all your comments are provided below.\n\n------\n\n**Q1: Streaming speech output. The speech decoder consists of 4 layers of NAR decoder and 4 layers of AR decoder. Does the NAR mean that full attention, rather than causal attention, is conducted across all audio tokens? If yes, then I think VITA-1.5 is hard to support streaming speech output. Users have to wait until all the text is generated before they can hear the speech output.**\n\n**R1**: Thank you for your insightful question. Indeed, the NAR decoder in VITA-1.5 applies full attention across input tokens. However, VITA-1.5 is specifically designed to support **low-latency, streaming speech output** through a hybrid decoding strategy:\n\n- The NAR decoder operates in a **sentence-by-sentence manner**, where sentences are segmented by punctuation (e.g., commas). These segments are typically short, allowing efficient chunk-wise speech token generation without requiring the full text beforehand.\n- The AR decoder follows and performs causal decoding over the generated tokens, enabling **real-time, progressive speech synthesis**.\n- Most importantly, the **generation rate of text tokens is significantly faster than that of speech tokens**. On NVIDIA H20 GPUs, generating speech tokens corresponding to 1 second of text typically takes 3.2 seconds. Additionally, waveform decoding and audio playback introduce further delay.\n\nThis natural gap means that **text generation typically stays ahead of speech synthesis**. Thus, the chunk-wise NAR decoding **does not become a bottleneck** for streaming, as long as the first sentence is short, allowing the system to start generating and playing speech almost immediately. **Users do not have to wait until the full text is generated** to begin hearing the response.\n\nIn summary, while the NAR decoder is non-causal, VITA-1.5 effectively supports streaming speech output in practice through architectural design and scheduling strategies. We will make this explanation more explicit in the revised version.\n\n------\n\n**Q2: Lack of comparison. VITA-1.5 achieves good performance on various image, video and speech benchmarks. The author should add comparisons with some new open-source MLLMs, such as LLaVA-Onevision[1], and Qwen2-VL[2] (It is okay if the performance is lower than Qwen2-VL series, since they didn't release training data).**\n\n**R2**: Thanks for your valuable suggestion. We have conducted additional comparisons between VITA-1.5 and recent open-source multimodal models, specifically **Qwen2-VL-7B** and **LLaVA-OneVision-7B**, on widely adopted benchmarks including **MME**, **MMMU**, **MMBench**, **MathVista**, and **MMStar**. The results are summarized below:\n\n| Model                | MME  | MMMU | MMBench | MathVista | MMStar |\n|----------------------|------|------|---------|-----------|--------|\n| **Qwen2-VL-7B**        | 2276 | 53.7 | 81.0    | 61.6      | 60.7   |\n| **LLaVA-OneVision-7B** | 1998 | 48.8 | 80.8    | 63.2      | 61.9   |\n| **VITA-1.5 (ours)**    | 2311 | 52.1 | 76.6    | 66.2  | 60.2   |\n\nAs shown:  \n- VITA-1.5 achieves the **highest score on MME and MathVista**, demonstrating strong **perception and reasoning** capabilities, especially in math-related tasks;  \n- VITA-1.5 performs competitively on other benchmarks, despite its additional support for **audio and speech interaction**, which are not covered by the above baselines;  \n- We emphasize that VITA-1.5 is designed as an **omni** model with **real-time speech interaction**, whereas Qwen2-VL and LLaVA-OneVision focus solely on **vision-language** tasks.\n\nWe will include these comparison results in the revised version.\n\n------\n\n**Q3: Technical novelty. This paper introduces a three-stage pipeline for training an Omni Model. However, the training pipeline and model architecture described are quite similar to those of other models in the field, such as Mini-Omni2[3], Llama-Omni[4], and Lyra[5]. As a research paper, I think it is important for the authors to clearly highlight the methodological novelty of VITA-1.5, compared to other related works.**\n\n**R3**: Thanks for the thoughtful comment on the novelty of our approach. We would like to emphasize that while VITA-1.5 shares high-level goals with recent works such as Mini-Omni2, LLaMA-Omni, and Lyra (**will all be cited in our revised version**), our contributions lie in the **systematic, scalable, and practical realization** of real-time omni-modal interaction. Specifically, VITA-1.5 distinguishes itself from these works in the following key aspects:\n\n1. **Systematic Three-Stage Progressive Training Pipeline**  \nWhile these works often adopt a one-shot or loosely staged training approach, VITA-1.5 introduces a **carefully crafted three-stage curriculum** that incrementally incorporates vision and speech modalities while alleviating cross-modal conflicts. This includes:\n- Stage I: Vision-language alignment and instruction tuning.\n- Stage II: Audio input integration with fine-grained CTC alignment and modality-type classification.\n- Stage III: End-to-end speech generation via codec + NAR+AR decoding.  \nThis structure ensures **stable convergence and modular controllability**, which is not present in Mini-Omni2 or LLaMA-Omni.\n\n2. **Sentence-wise NAR + AR Speech Decoder Design for Streaming Output**  \nVITA-1.5 proposes a **sentence-segmented NAR** followed by an **AR decoder for causal streaming**, enabling both fast generation and real-time playback. Unlike Lyra, which focuses on efficiency, our method **balances latency and fidelity** with a decoding design tailored for **interactive speech responses**.\n\n3. **Unified and Practical Real-Time System Deployment**  \nVITA-1.5 has been **deployed in a real-world demo**, achieving <700ms model latency (on A800), and ~1.5s total latency (including preprocessing and audio playback). To our knowledge, it is the **first open omni-modal system** that demonstrates this level of **real-time vision and speech interaction** in a single end-to-end pipeline.\n\n4. **Robust Benchmark Results Across Modalities**  \nOur model achieves **competitive results** on vision-language tasks (e.g., MathVista, MME), while also delivering strong **speech recognition performance**. This demonstrates that **adding speech capability has little impact on vision performance**, which is a known challenge in multimodal joint training.\n\nIn summary, the **novelty of VITA-1.5 lies not in isolated architectural tweaks**, but in its **coherent, practical, and extensible framework** that effectively brings together high-quality perception, reasoning, and real-time interaction across modalities. We will clarify and highlight these novel aspects more explicitly in the revised version.\n\n------\n\n**Q4: Speech Interruption. The video demonstration showcases the model’s ability to handle speech interruption. The authors are encouraged to include a description in the paper detailing how this feature is implemented, including what modules were introduced and which data were used.**\n\n**R4**: Thank you for your insightful question. In VITA-1.5, **speech interruption** is supported through a **duplex inference design**, which deploys **two identical models** in parallel and coordinates their responses via a lightweight voice activity detection (VAD) mechanism.\n\nSpecifically:\n- During duplex interaction, both **Model-A** and **Model-B** are initialized identically.\n- When the user issues **Query-1**, Model-A starts generating the spoken response.\n- If the user begins **Query-2** while Model-A is still speaking, the **VAD detects user speech onset**, and the system **immediately stops Model-A’s decoding**.\n- At the same time, **Model-B takes over** and starts generating a response to Query-2, ensuring smooth and low-latency interaction.\n\nThis design offers several advantages:\n- It allows **true real-time bidirectional control**, as models can be switched instantly without reloading or context corruption.\n- The interruption logic is implemented **at the orchestration level**, without modifying the model architecture.\n- The use of **autoregressive speech decoding** enables token-level control and precise halting of generation.\n\nWhile we did not use explicitly labeled \"interruption\" data during training, VITA-1.5 was trained on **dialog-style multimodal instruction tuning data**, which implicitly encourages responsiveness and turn-taking behavior.\n\nWe will update the paper to explain more clearly.\n\n------\n\n**Q5: Speech decoder. The speech decoder in this paper consists of 4 layers of AR decoder and 4 layers of NAR decoder. The authors should clarify the motivation for this design choice, rather than using only AR or only NAR decoders.**\n\n**R5**: Thank you for raising this important point. The design of the VITA-1.5 speech decoder is motivated by the need to **balance latency, fluency, and controllability** in speech generation.\n\n- **NAR decoder (fast and global)**: The NAR decoder operates in parallel and produces a **coarse global layout** of speech tokens conditioned on the semantic content of the text. This enables **low-latency initial speech token generation** and facilitates **sentence-level semantic alignment**.\n- **AR decoder (fine and causal)**: The AR decoder then performs **causal refinement** of the NAR output, generating high-quality speech tokens **token-by-token**. This step ensures **natural prosody, smoothness**, and compatibility with **streaming playback**.\n\nOtherwise: \n- Using **only AR decoding** would result in better quality but **significantly increases latency**, especially for long responses, which hurts real-time user experience.\n- Using **only NAR decoding** would reduce latency but often leads to **unnatural prosody and token artifacts**, particularly in expressive or long-form speech.\n\nWe will include the explanation in the revised version."}, {"review_id": "66zNxbxXS1", "reviewer": "Reviewer_RiXE", "summary": "- The paper presents VITA-1.5, an end-to-end multmodal model capable of tackling video/audio/image understanding tasks with speech and text outputs.\n- The authors introduce a comprehensive set of datasets with a multi-stage training technique to develop the proposed model.", "questions": "- The video shared in the appendix is in chinese, how well does it perform with english audio/speech?\n- How much compute was used to train the model?", "limitations": "Yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 1, "strengths_and_weaknesses": "### Strength\n\n- The model is able to output speech and is end-to-end.\n\n### Weaknesses\n\n- I have trouble finding a technical takeaways from this paper that could help research move forward. There are no details shared about data curation, the model architecture follows VITA-1.0 to most extent, no real ablations on different training stages or dataset choices.\n- The authors say in the checklist that they do not provide access to data stating that its widely available with preprocessing details shared but i do not find the mention of that for any kind of data. They are very vague about the audio and speech data used in sec 3.3.2 and 3.3.3, so that justification is wrong.\n- Do the authors plan to release details or checkpoints?\n- The results lack behind most methods in Tab. 2", "was_revised": true, "final_justification": "During the rebuttal and discussion phase, the authors shared a few more details about their work, however, all the useful details are missing from the current article and i feel it is not right to accept it in the current state. It requires many more ablations and dataset curation details/experiments to have publishable quality.", "rebuttal": "Sincerely thanks for your efforts in reviewing this work. We hope the detailed responses help clarify your concerns. We would greatly appreciate it if you could kindly re-evaluate our work in light of the new explanations and additional results.\n\n------\n\n**Q1: I have trouble finding a technical takeaways from this paper that could help research move forward. There are no details shared about data curation, the model architecture follows VITA-1.0 to most extent, no real ablations on different training stages or dataset choices.**\n\n**R1**: We respectfully clarify that our paper presents several important technical takeaways that we believe are valuable for advancing multimodal interaction systems:\n\n1. **Progressive Multi-Stage Curriculum to Alleviate Modality Interference**  \n   While VITA-1.0 proposed an initial coarse-grained multi-stage pipeline, VITA-1.5 introduces a significantly refined **six-phase training strategy**. This curriculum gradually incorporates different modalities in a stabilized order—first vision, then audio input, and finally audio output.  \n   This progressive approach **mitigates modality interference** and maintains the core linguistic capabilities of the base LLM (e.g., <1% variation on MMLU and GPQA-Diamond), which we believe is a **methodologically valuable insight** for future MLLM training paradigms.\n\n2. **End-to-End Real-Time Speech Interaction**  \n   VITA-1.5 upgrades from the cascaded TTS pipeline in VITA-1.0 to an **end-to-end speech generation system** using codec tokens.  \n   This architectural shift enables **low-latency speech output** (↓1.4s) for nearly real-time interaction—an important system-level contribution to interactive AI agents.\n   \n\n3. **Data Handling and Reproducibility**  \n   The training data used in this work are from **publicly available datasets**, including audio (internally organized open-source audio data), vision, and multimodal instruction corpora. Due to space constraints, we briefly described audio data use in Sec 3.3.2 and 3.3.3, but we acknowledge the descriptions were insufficiently detailed.  \n   In the revised version, we will:\n   - Provide a **complete list of datasets**, \n   - Clarify **data preprocessing and filtering procedures**, and\n   - **Release training&inference codes and model checkpoints** to ensure **reproducibility**.\n\n4. **Ablation Studies**  \n   We agree that ablations would strengthen the contribution. Due to time limitations, we prioritized delivering an integrated, end-to-end functioning system. We are currently supplementing ablations on training order and data mixing ratios, and plan to include them in final version.\n\nWe will make these contributions and distinctions more explicit in the revised manuscript.\n\n------\n\n**Q2: The authors say in the checklist that they do not provide access to data stating that its widely available with preprocessing details shared but i do not find the mention of that for any kind of data. They are very vague about the audio and speech data used in sec 3.3.2 and 3.3.3, so that justification is wrong.**\n\n**R2**: We acknowledge that Sections 3.3.2 and 3.3.3 lack sufficient detail regarding the audio and speech datasets used, and we appreciate the opportunity to clarify and improve this aspect.\n\nThe data used in this work are sourced from **publicly available datasets**. Specifically:\n\n- For **audio input (ASR)**, we use well-established datasets such as **AISHELL-1**, **Common Voice**, and **LibriSpeech**.\n- For **speech output (TTS)**, open-source text corpora are translated to audio by a TTS system.\n- For **instruction tuning**, we synthesize multimodal instruction-style annotations based on these datasets to align with the model’s multi-turn conversational capabilities.\n\nTo ensure **reproducibility**, we will:\n- Provide a **list of used datasets** in the final version, including download links;\n- Describe our **data preprocessing pipeline** in detail (e.g., how to use TTS systems to synthesize audios);\n- **Release our training&inference codes and model checkpoints**.\n\nWe are fully committed to open research and making VITA-1.5 reproducible, including all necessary resources for the community to replicate our results and build upon our work.\n\n------\n\n**Q3: Do the authors plan to release details or checkpoints?**\n\n**R3**: Yes — we will **definitely release** the training&inference codes and model checkpoints. Ensuring **reproducibility and openness** is a core goal of our work. We will also provide clear documentation and usage instructions to support the community.\n\nWe are fully committed to supporting open research and lowering the barrier for future work in multimodal interaction.\n\n------\n\n**Q4: The results lack behind most methods in Tab. 2.**\n\n**R4**: We appreciate the reviewer’s concern regarding performance in Table 2. While it is true that VITA-1.5 slightly lags behind certain **closed-source models** such as GPT-4o (66.8 vs. 69.3), we would like to emphasize several important clarifications:\n\n1. **Strong Performance Compared to Open Models**  \n   VITA-1.5 achieves highly competitive performance compared to advanced **open-source models**. For instance, it is only **0.5 points behind InternVL2** (66.8 vs. 67.3) in average performance, despite InternVL2 using many private training data and focusing solely on image-text tasks without supporting audio or speech interaction.\n\n2. **Multimodal Interaction Capability**  \n   Unlike many vision-language models such as InternVL2, VITA-1.5 supports **end-to-end multimodal interaction**, including nearly real-time **speech input/output** and **speech interruption handling**, making it more broadly applicable to real-world interactive scenarios.\n\n3. **Competitive Results vs. Fully Open Models**  \n   Compared to **fully open-source baselines**, such as LLaVA-OneVision, which also use only publicly available training data, VITA-1.5 demonstrates **consistently stronger performance** across benchmarks:\n\n   | Model                 | MME  | MMMU | MMBench | MathVista | MMStar |\n   |----------------------|------|------|---------|-----------|--------|\n   | LLaVA-OneVision-7B   | 1998 | 48.8 | 80.8    | 63.2      | 61.9   |\n   | **VITA-1.5-7B**       | **2311** | **52.1** | 76.6    | **66.2**  | 60.2   |\n\n   These results highlight the **solid multimodal foundation** of VITA-1.5, achieved while simultaneously supporting speech modalities — which LLaVA-OneVision does not.\n\nIn summary, VITA-1.5 strikes a meaningful balance between **broad modality coverage** and **strong open-source benchmark performance**, and we believe this makes it a valuable step forward in the direction of real-time, speech-enabled MLLMs.\n\n------\n\n**Q5: The video shared in the appendix is in chinese, how well does it perform with english audio/speech?**\n\n**R5**: While the demonstration video in the appendix is in Chinese, we confirm that **VITA-1.5 fully supports both Chinese and English** for speech input and output. In fact, we evaluate English speech performance using standard benchmarks. For example, in Table 4, VITA-1.5 achieves better than Wav2vec2-base, Mini-Omni2, and Freeze-Omni on the English ASR tasks. The English WER on the Seed-TTS-eval benchmark is 8.44% (use only 3K hours training data), which has demonstrated the TTS performance.\n\nWe will provide an English version of the demo video in the final submission to better showcase these capabilities.\n\n------\n\n**Q6: How much compute was used to train the model?**\n\n**R6**: The training took 5,300+ H20 GPU hours."}]

Author rebuttal:

Meta-review: This submission presents VITA-1.5, a multimodal large language model enabling near real-time speech–vision interaction through a progressive multi-stage training pipeline and an integrated speech decoder. The reviewers generally agree that the paper represents a solid engineering advance and provides a valuable open-source step toward speech-capable MLLMs. However, opinions diverge sharply, with three reviewers recommending accept (ranging from borderline to strong accept) and one reviewer recommending reject.

Strengths identified across reviews:
+ Significant latency reduction (~4s → ~1.5s) in interactive speech-to-speech dialogue (DC1q, Zxsq, RHor).
+ Strong empirical performance across image, video, and speech benchmarks, competitive with leading open-source baselines (DC1q, Zxsq).
+ Well-structured training methodology that progressively integrates modalities while mitigating interference (DC1q, RHor).
+ Convincing demo showcasing real-time duplex speech interaction and interruption handling (RHor).
+ Comprehensive system-level contribution: end-to-end speech generation without external ASR/TTS modules (all positive reviewers).

Weaknesses raised:
- Limited novelty beyond VITA-1.0; clearer differentiation from related omni-models (Mini-Omni2, LLaMA-Omni, Lyra) would strengthen the case (Zxsq, RHor).
- Insufficient evaluation of speech generation quality (e.g., TTS prosody, MOS scores) and lack of ablations on training order or data mixing (Zxsq).
- Latency of 1.5s, though improved, still exceeds human conversational speed (DC1q).
- Concerns over missing details in dataset description, limited ablation, and reproducibility under realistic compute budgets (RiXE). Reviewer RiXE also dismissed widely adopted benchmarks (MME, MMBench, MathVista, MMMU), asserting MMStar is the “only good” one, leading to strong reviewer–author disagreement.

Rebuttal and Discussion:
The authors addressed nearly all substantive concerns: they provided additional comparisons with recent baselines (Qwen2-VL, LLaVA-OneVision), clarified their hybrid NAR+AR decoder design for streaming speech, reported human evaluation and interruption success rate (98%), and conducted initial TTS quality studies. They also committed to releasing code, checkpoints, and detailed dataset descriptions. DC1q, Zxsq, and RHor all explicitly raised their scores after rebuttal, acknowledging that concerns had been satisfactorily addressed. Reviewer RiXE maintained a reject recommendation, emphasizing compute cost, lack of ablations, and skepticism of benchmarks, despite clarifications and community-standard practices provided by the authors.

Three reviewers are aligned that the paper is technically solid and merits acceptance, while the single negative review appears disproportionately harsh. The rebuttal convincingly strengthened the submission and demonstrated the authors’ commitment to reproducibility. Given the significance of providing an open-source foundation for real-time multimodal interaction, I recommend accept.
Decision: Accept (spotlight)