OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM
Authors: Hanrong Ye
Chao-Han Huck Yang
Arushi Goel
Wei Huang
Zhen Wan
Jinchuan Tian
An-Chieh Cheng
Ligeng Zhu
Yuanhang Su
Yuming Lou
Yong-Xiang Lin
Dong Yang
Sreyan Ghosh
Zhijian Liu
Yukang Chen
Ehsan Jahangiri
Ambrish Dantrey
Daguang Xu
Ehsan Hosseini-Asl
Seyed Danial Mohseni Taheri
Vidya Nariyambut Murali
Sifei Liu
Yao Lu
Oluwatobi Olabiyi
Yu-Chiang Frank Wang
Rafael Valle
Bryan Catanzaro
Andrew Tao
Song Han
Jan Kautz
Hongxu Yin
Pavlo Molchanov
Venue: ICLR 2026 Poster (2026)
Keywords: Omni-modal models
Multimodal LLMs
Large Language Models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Advancing machine intelligence requires developing the ability to perceive across multiple modalities, much as humans sense the world. We introduce OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We carefully study the design choices across model architecture and data curation. For model architecture, we present three key innovations: (i) OmniAlignNet for strengthening alignment between vision and audio embeddings in a shared omni-modal latent space; (ii) Temporal Embedding Grouping for capturing relative temporal alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding for encoding absolute temporal information in omni-modal embeddings.
We introduce a curation and synthesis pipeline that generates 24M single-modal and omni-modal conversations. We find that modalities reinforce one another in both perception and reasoning. Our model, OmniVinci, improves over Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while using just 0.2T training tokens — a 6× reduction compared to Qwen2.5-Omni’s 1.2T. We finally demonstrate omni-modal advantages in downstream applications spanning robotics, medical AI, and smart factory.

Published as a conference paper at ICLR 2026
OMNIVINCI: ENHANCING ARCHITECTURE AND DATA FOR OMNI-MODAL UNDERSTANDING LLM
Hanrong Ye
Chao-Han Huck Yang"
Arushi Goel"
Wei Huang"
Ligeng Zhu"
Yuanhang Su
Sean Lin'
An-Chieh Cheng"
Zhen Wan"
Jinchuan Tian"
Yuming Lou"
Dong Yang"
Zhijian Liu
Yukang Chen
Ambrish Dantrey
Ehsan Jahangiri
Sreyan Ghosh
Daguang Xu
Ehsan Hosseini-Asl Danial Mohseni Taheri
Vidya Murali Sifei Liu
Yao Lu
Oluwatobi Olabiyi
Yu-Chiang Frank Wang Rafael Valle
Bryan Catanzaro Andrew Tao
Song Han
Jan Kautz
Hongxu Yin
Pavio Molchanov
NVIDIA
"Core Contribution
CCorresponding Authors
Equal Advisory
Code
Model
Webpage
ABSTRACT
Advancing machine intelligence requires developing the ability to perceive across multiple modalities, much as humans sense the world. We introduce OmniVinci, an initiative build strong, soene,,uree, omni-modal L.M. We carefully study the design choices across model architecture and data curation. For model architecture, we present three key innovations: (i) OmniAllignNt for strengthening alignment between vision and audio embeddings in shared omni-modas latent space; (ii) Temporal Embedding Grouping for capturing relative temporal alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding for encod- ing absolute temporal information in omni-modal embeddings. We introduce a curation and synthesis pipeline that generates 24M single-moda and omni-modal conversations. We find that modalities reinforce one another in both perception and reasoning. Our model, OmniVinci, improves over Owen2.5-Omní with +19.05 on DailyOmni (cross-modal understanding), +1.7 on MMAR (audio). and +3.9 on Video-MME (vision), while using just 0.2T training tokens 6x reduction com- pared to Owen2.5-Ommi 1.2T. We finally demonstrate omni-modal advantages in downstream applications spanning robotics, medical AI, and smart factory.
Omni Dalycmmi
Omni Worldserse
Audio MIMAU
Audio MMARR
Vision MVBench
Vision Video-MMEE
70
66.5
71.6
70.6
88.2
58.4
55
48.23
40
25
Internt2 Qwenz-ia Vt QwanZ 5-0mi
Ours
LLaVA-Video 4-
Owen2 5-Omri Curs
Kim- Audo
Gemin-2 5-Pro
Gwen2 5mm
SALAMONN-1 38
Ours
GPT-do-min miny
Qwanz s-Omn
Ours
Video CCAM
Gwen2 5amm
Ours
NVILA
Qwen2 S-V/
Owen2 S-mmm
Ours
Intemiide22
Figure 1: Omni Vinci demonstrates strong performance across widely used omni-modal (+19.05 on Dailyomni), audio (+1.7 on MMAR), and vision (+3.9 on Vidco-MME) understanding benchmarks.
INTRODUCTION
The progress of multimodal LLMs has demonstrated appealing applications when LLMs learn to see with vision (Lin et al./ 2024b Liu et al. 2023| Alayrac et al. (2022) listen with audio (Goel et al.) 2025) Chu et al. 2202) Tang et al. 2023a). Recent work has enabled joint video-audio alignment.
further unifying their strengths towards general intelligence (OpenAl 2024 Wu al. 2024b Tang

---

Published as a conference paper at ICLR 2026
Image
Video
(Souns/Speech)
Text/Speech Prompts
Encoder
Audio Encoder
Encoder
Omni-Madal Input Fiibeedingg
Downsample Proj.
Downsample
Comprr Embeoss
Prompt
Prompe
OrniAlignNet
Tamporal E mbed Grouping
Algnment
Contrentive
Constrained Rotary Time Embed
Text
LLM
Berrlesery tettiiig win "Dood tay""
TTS
Dutput
Crose-Entropy
Figure 2: We introduce a foundation model for omni-modal understanding. Our model blends information from vision, audio, and text modalities into unified omni-modal token sequence via the proposed omni-modal alignment mechanism.
al. 2023b Ye et al. 2024 Abouelenin al. 2025 Xu et al. 2025). However, training such an omni-modal system can be expensive and challenging across many dimensions, as it relies on proper choices of network architecture and data recipe.
This work presents a systematic exploration of developing omni-modal LLMs aiming to enable simultancous understanding of vision, audio (encompassing both natural sounds and human speech), and language. We ablate and validate the design choices oversecing model architecture design, data curation, and training strategy. For model architecture, introduce harmonize vision and audio embeddings in a unified omni-modal embedding space, featuring three new techniques: (i) OmniAlignNer that learns to construct modality-shared space to align vision and audio embeddings from the same video; (ii) Temporal Embedding Grouping that divides the time dimension into multiple chunks and reorganizes the vision and audio embeddings according their timestamps to align with the corresponding chunks; (iii) Constrained Rotary Time Embedding to directly insert periodic temporal information into vision-audio embeddings. We observe noticeable performance improvements with these techniques, as shown later in our experiments. On the data front, we curate 24 million high-quality multimodal conversation samples that span a diverse set of tasks across audio, video, and image domains, including both modal-specific conversations and omni-modal conversations. We tackle the scarcity of omni-modal data by exploiting existing video- with-audio QA data, which implicitly encodes amni-modal signals (implicit learning). To further facilitate omni-modal learning, we generate synthetic conversations with explicit umni-modal labels (explicit learning).
framework to
Our findings enable frontier omni-modal model, named Omni See a quick performance comparison in Figure and more in experimental section. Compared prior such as Qwen2.5-Omni and Gemini-2.5-Pro, Omni further pushes the boundary of various multimodal understanding tasks, with gains of +2.83% on WorldSense and +19.05% on Dailyomni for joint vision-audio understanding. +1 +1.7% on MMAR for audio understanding, and +3.9% on Video-MME for vision understanding. OmniVinct also pushes on efficiency fronts using only 0.2T training tokens, around 6x fewer than Qwen2.5-Omm Ommniss 1 1.2T tokens. More encouragingly, we observe the synergy between audio and video, not only for perception, but also for reasoning. Finally, we demonstrate that Omni Vinci has enabled or improved a wide range of important downstream applications, including robotics, video broadcasting, medical, and smart factory use cases.
MODEL ARCHITECTURE
The key objective of model architecture design is to support composable cross-modal understanding through integrating heterogeneous input from images, videos, audio, and text, into shared omni- modal latent space. As shown in Figure 2 we adopt an auto-regressive regime to encode visual and audio signals, and then align them input of LLM backbone.
Omni-Modal Input Embedding, To simplify the network design, we (i) decompose video into sequence of temporally correlated images and audio, and (ii) employ unified audio encoder to
2

---

Published as a conference paper at ICLR 2026
handle both acoustic and speech information in context and prompt. We present the encoder sharing paths in Figure 2 and describe the details of encoding streams in Appendix D.I
2.1 OMNI-MODAL ALIGNMENT MECHANISM
We next integrate embeddings from all modalities into a unified latent space as input for LLM.
Omni AlnnAAlggNeet module. For a given input video, the audio and vision streams have an inherent semantic connection, providing complementary information for each other. Such correlation provides a natural way to more effectively learn and align vision and audio embeddings in the unified latent space. To this end, we propose OmniAlignNet,, which strengthens the learning of vision and audio embeddings via exploiting their complementary information. As illustrated in Figure 3 the Omni OlgiAlign module first maps visual and audio embedding sequences (outputs of modality- specific projectors) into shared latent embedding space and then aligns them via contrastive learning, inspired by ImageBind (Girdhar et al. 2023).
Given an input video with an accompanying audio stream, we denote the sequence of visual embed- dings produced by the visual projection layer as E,, E RBN and the sequence of audio embeddings produced by the audio projection layer E, RN,xC. with N, and .N, represent the number of visual and audio embeddings, respectively, while denotes the latent dimensionality. To align representations,
initialize a vision query embedding Q2 E IR 1xC and an audio query embedding Qu E R LxC These queries are used to project E, and E,, into fixed-size embeddings of shape (1 x C). Suppose each batch has K videos, the projected features layers of self-attention modules and L2 normalized, yielding the vision-omni embedding V E R x and the audio-omni embedding A E R respectively, modality-shared latent space.
then processed through three
With embeddings V and A in the shared la- tent space, we now apply CLIP-style contrastive loss (Radford et al., 202I) on the output em- beddings to minimize intra- saralsample cross-modal distance, while maximizing inter-sample cross- modal distance. Let (V,, A be the set of L2- 1.2-normalized visual and audio embeddings for a batch of K video clips. The similarity be- tween the i-th visual embedding and the j-th audio embedding is computed as their dot prod- uct, 8i, V A,. The contrastive loss is then formulated as symmetric cross-entropy loss over the similarity score. The loss for aligning vision audio (Lw-ra) and audio vision (La-vo)
Query Embh
Vision Takerts
Cross Alti Layat
SA
Audia Takerne
+
Cross Attn Layer
Query
Sel Atranttiiion Lapar
Pravediy
Figure 3: Illustration of the proposed OmniAlign- Net module.
is:
N
exp(s;i
N
exp(su)
L Lo
log
La-ve
log
(1)
j= exp(s1))
N
1=1 j==1 exp(s;))
The final objective for the OmniAlignNet module, Looalign is the average of these two directional losses, encouraging bidirectional alignment between the modalities: Lo-align = 1/4 (Lo_a + La-+o).
While OmniAlignNet effectively aligns the high-level semantics of visual and audio embeddings, falls short in modeling their temporal relationships. To overcome this limitation, we introduce two techniques: Temporal Embedding Grouping and Constrained Rotary Time Embedding, which are described in the following sections.
Temporal Embedding Grouping (TEG). We first impose temporal order to visual-audio embeddings by organizing them into groups based on their timestamps. The relative temporal order information is then encoded the position of visual and audio embeddings in the input sequence.
Let the duration of each temporal group be To, which controls the granularity of the grouping. For simplicity, consider a case where we only sample four visual frames at timestamps {12 /2 13 ( and four audio samples timestamps (12, 12, 1a (1). These timestamps satisfy f2 < 1 To
t < t < 2IG and t1 < th < To < t² < Eg 2Tc. The corresponding set of visual embeddings 11 is E,, (el e e."," e }. where each embedding e, E ((I))xC, Here, H and W represent the height and width of the visual feature map, and C' is the latent dimension. Similarly, the set of audio
3

---

Published as a conference paper at ICLR 2026
embeddings is E, = (ee en ed" eaa) with each e, E RIXC, Based on their timestamps relative the duration To. the embeddings for each modality partitioned into two temporal groups:
2 1.
G. = (e: e. G. (e. J. (e G. (e e
Then we combine the visual and audio groups based temporal order, and obtain the omni-modal embedding sequence:
e dl [G Ga. Go, C.. = ee ed ea e
E Egroup
ea
This temporal organization of the embedding sequence allows the subsequent LLM backbone to better capture the temporal relationships among embeddings from different modalities. Our experiments show that this time-based grouping improves the model's ability to comprehend information from multiple modalities.
Constrained Rotary Time Embedding (CRTE). TEG incorporates relative temporal order into em- beddings but still lacks the ability to encode absolute timestamp information. Prior work, RoTE (Goel et al. 2024), explored embedding rotations to inject absolute timestamps, but this method remains sensitive to minor timestamp fluctuations and struggles to capture larger temporal shifts effectively. To overcome these limitations, introduce constrained timestamp embedding strategy that de- fines a maximum time horizon, Tomx. enabling a more balanced temporal sensitivity. Our approach comprises three stages: base frequency construction, frequency modulation, and element-wise rotary embedding, as described next.
Base Frequency Generation. We first define base frequencies as:
2m
Wi
Toax Of/C
for i = 1..
(4)
where w; the base frequency for dimension i, C is the embedding dimension, 0 2 1 controls frequency scaling, and max defines the coarsest temporal resolution. A smaller T max increases frequency and sensitivity to fine-grained differences, while a larger one captures broader trends but may blur close timestamps, and is thus critical for balancing local and global temporal encoding.
Frequency Modulation. To adapt frequencies to actual timestamps, we scale them as: S4. = wi tt. where 24, is the modulated frequency at dimension and time t, for sample j. This step ensures that temporal differences are reflected in the rotation applied to embeddings.
Rotary Embedding Application. Similar to RoPE (Su et al. 2024). given an embedding vector x E RC of sample (a sampled frame for video or sampling point for audio), we apply rotation as:
CRTE(x, Q.5)) = x x cos( RotateHal((x) @ sin.)..)),
(5)
where @ denotes clement-wise multiplication, and RotateHalf rotates each pair of dimensions by 90"; RotateHalī(x)) = [-32, X1, -I,, - IC-]]. The RotateHall function effectively groups the entire C-dimensional embedding vector into C/2 independent 2D planes. Each of these 2D planes gets its own rotation, and the angle of rotation can be different for each pair. We apply rotations at varying frequencies across different pairs of dimensions for two primary reasons: enables rich, multi-scale representation of temporal information, and it preserves the semantic integrity of the original embedding vectors.
Final Embedding Sequence. After CRTE, the temporally-aligned omni-modal embedding sequence is passed into the LLM backbone, allowing it to integrate both fine- and coarse-grained timing cues during downstream processing.
Input-Output Configuration. The final architecture perceives flexible input modality combinations with a subset or union of all modalities, e.g., video with or without audio, with speech or text prompts. On the output end, the text-output based system can be connected with off- thethhh-seeff Text-to- Speech (TTS) modules we analyze their tradeoffs in Section[E.4] Without bells and whistles, users can generate spoken descriptions for videos, answer spoken questions, verbally instruct robots.
4

---

Published as a conference paper at ICLR 2026
"che video beqina with cleaemus blue undernataz zobot noving throuch a dazk bluc ccean. it bevins with hich-tash undersaler suplotatioon then Eransit Lons to sore naturel aet Lingn, hight iigttin the contrant betrees hunan technology and natural world."
Vision Captioning
"Break vidco time order, descriiig each scene
detail
Understanding
Villo
wadia
clip of susic playing the background.
Audio Captioning
there's also male voice apeaking knglish. be saya,
you trave to the deepect pointe of Farth ... te seere like vidco detail. Transcribe speech you hear?"
"Describe this austo from
hoto talking about the nystorics of the doep oscan oz nayas woun the Eazth's a interiog. Wrong Understanding
203
[Vision Captions]
(adiaa Captions]
[Vision Captions)
[Aodio Captions]
Omni-Caption:
"che video coptiivating journey
through noves away fron viewer... The Correct Omni-Modal Understanding
blge waters, its Lights ousting through the gloon 05
Darcation explaine the Journay from Japan to the Challanger beep ....
Figure 4: Omni- modal captions generation pipeline. Video is segmented into 20-second clips. Visual and audio captions are generated independently for each segment, but lack cross-modal context and contain wrong understanding mnodality-speciffe hallucination). separate LLM performs cross- modal correction and summarization to create accurate omni-modal captions.
3 TRAINING STRATEGY
To gradually enable comprehensive omni-modal understanding of a pretrained LLM, we use two- stage approach: we first conduct modality-peccific training to develop individual capabilities for each modality, followed by omni-modal joint training to integrate these capabilities.
3.1 MODALITY-SPECIFT TRAINING
We use a two-stage approach for training: Starting from pretrained LLM, we first conduct modality- specific training to develop individual capabilities for each modality, followed by omni-modal joint training to integrate these capabilities. Due to space limitations, we present comprehensive details of this phase in Appendix D.3 and proceed directly to describe the subsequent omni-modal joint training phase in the next section.
3.2 OMNI-MODAL JOINT TRAINING
We employ two types of data in the omni-modal joint training phase: (i) modality-speciff data, randomly sampled from the datasets used in the earlier vision-only and audio-only training, and (ii) omni-modal data, which contains both vision and audio inputs. For the amni-modal data, which contains both visual and audio inputs, can be further divided into two categories, i.e., implicit omni- modal learning data and explicit omni-modal learning data, depending how the omni-modal understanding ability supervised in training.
(i) Implicit Learning Data. Videos naturally omni-modal when visual and audio streams present simultaneously but remains under explored. We first take advantage of the existing video QA datasets supervise the visual-audio joint understanding ability implicitly, which is underutilized in most previous video LLMs. This practice, we refer as implicit omni-modal learning, leads to notably improved performance in video understanding that remains under utilized by prior work.
(ii) Explicit Learning Data. To obtain more direct and accurate supervision for joint visual-audio understanding ability, we further propose an omni-modal data engine synthesize omni-modal labeling for videos with audio tracks, enabling us to conduct explicit omni-modal learning.
Omni-Modal Data Engine. The whole data engine is visualized in Figure We start with synthetic audio and video captions using pretrained vision captioning model (Zhu et al.) (025) and audio captioning model (Xu et al., (025). We immediately observed that captions generated from either modality alone can lead to wrong understanding due to the inherent modality-specific limitations. As illustrated in Figure 4 the video is centered around deep-sea exploration. However, the vision- captioning model incorrectly interpreted it as being only about human technology, relying solely on visual cues without access to the speech in video. Conversely, the audio-captioning model wrongly labeled it as related to "Earth's 's interior". since it could only draw meaning from the audio track. We refer to this limitation as condality-speciffe hallucination" To address this issue, we employ a I.LM (Yang et al. 2025a) to correct and summarize the visual and audio captions based on information
5

---

Published as a conference paper at ICLR 2026
Q Where does Kostas Glincs work? simagn A: Kostaz Elines world ot Furopeen Commission
Q. What the name Use paintin?? 2rg? -imager & Alha 2011
Document 7%
Knowledge 8%
in the accomponyyng diagram of mombus
Genreral 19%
the mesqure
Qu "Boosed the austa, which type music likeyy being prayee?"
measure of anolo CDA 1407
[mage 36%
(8)
Audio QA 20%
<images
Sound 21%
A: The anzmer is
-10
Video 11%
What first this sudto the drum treak the vocale? <Sounds
Speech 17%
Video QA 11
Omni 15%
Actn this audio cllp, the ciamn break comes first The drumg ace the thythm and groave before the vocals enter the mix.
Q What in the video? <Videcn At Acpocent to the toilet is and whrts
Omnii Caption 3is
several
Nittonn,
together...
Audi G4 4%
Omni QA 12%
Q Describe or coption the video details. -Video ustio A: The vidoo immeraes the viewer in deep exploration. The audio men spewking wbout esploring
sprech spoken contont the nwyy.
Overall distribur iistribution of
meters
face...
training
Figure 5: Pie chart of overall distribution of training data across modalities, showing proportions for image (36%), non-speech sound (21 %), speech (17%), omni (15%), and video (11%).
from both sides, producing a comprehensive joint caption for each 2-minute segment. From our observation, this method can help achieve correct omni-modal understanding, as shown in the example enhance the diversity and quality of the omni-modh data by synthesizing QA pairs with reasoning trace from the omni-modal captions using a reasoning LLM (Guo et al. 2025a). The resulting dataset greatly assists with learning,
in Figure Furthermore,
show in experiments.
Key Insight Captioning based solely audio visual is often inaccurate because of the inherent limitations of each modality. Hence, a joint captioning approach is preferred to integrate both modalities and produce comprehensive summaries across clips.
Joint Training Data Distribution. As shown in Figure 5 the overall training dataset contains 24 million modality-specific conversations from 150+ datases across image, video, and audio understanding tasks. Ondil data contributes 15%, image data constitutes the largest share at 36%, speech data represents 17% of the total, and video data forms the remaining %. For more details, please refer to Appendix D.5) To enable audio- pootmippppppee ability, we convert text prompts in multimodal tasks into speech using Magpie TTS model (Hussain et al. 2025 Neekhara et al.) 2024 Casanova et al. 2025), generating omni-modal speech-visual input pairs. The questions are generated from a comprehensive collection of multimodal datasets, including general multimodal understanding, image captioning, spatial relationship reasoning and referring, chart and table inter- pretation, scientific figure analysis, document understanding, and multi-hop reasoning. This diverse range enables comprehensive evaluation across core vision-language capabilities such as factual grounding, reasoning over structured data, and complex multi-step inference in both scientific and general domains. See detailed distribution of speech-prompted omni QA datasets in Figure [4
4 EXPERIMENTS
We start with ablations validate design options Section 4.1, before large-scale training towards frontier performances in Section 4.2
Table 1: Ablation study for omni-modal alignment. The pro- posed Temporal Embedding Grouping (TEG), Constrained Rotary Time Embedding (CRTE), and OmniAlignNet consis- tently achieve better average performance across modalities.
Omni
Method
4.1 DESIGN CHOICE ABLATION
Worldsense Dailyomni Omnibennch
Average
Token Concatenation Baseline
54.55
45.51
4.1.1 VISUAL-AUDIO ALIGNMENT SCHEME
TEG (ours)
44.51.2.0
00.99.u..4
37.65...19
47.72
Leamed Time Embedding
2..5
60.40....ao
36.91-p.4
47.30,1.7
RoTF
44.42 24.4
60. 90.7446.30
38.24.1.m
47.80.
Baseline Setup. To investigate the behavior of omni-modal models under
++ CRTE (ours)
45. 4.55
65. .1.1
39.64-3
40.25..474
#+ OlniAlignNer (ours)
46.21 1.00
68.88,
12.05
48.74....x
$2.59.7.0

---

Published as a conference paper at ICLR 2026
various experimental conditions
gradually introduce new techniques onto a baseline model trained with 10B tokens randomly sampled subset of the full data mixture (the sampling process is weighted according to the original dataset sizes). We evaluate model performance on Worldsense (Benchekroun et al. 2023), Dailyomni (Zhou al. 2025), and Omnibench (Li et al. 2002c).
Temporal Embedding Grouping. We observe immediate performance improvements with TEG applied to the baseline and present the results in Table I thanks to the enhanced temporal alignment of modality tokens.
Constrained Rotary Time Embedding. We next compare CRTE with other design choices: (i) "Learned Time Embedding" that defines a trainable embedding matrix, where each discrete timestamp in the range [0, Tmaz] mapped to a unique vector via MLP. (ii) "RoTE" (Goel et al. 2024).
recent embedding method introduced in Section 2.1, As summarized in Table ! the "Learned Time Embeddding method slightly degrades performance (47.30), indicating it is unsuitable for absolute timestamps. RoTE offers only marginal gains, while the proposed Constrained Rotary Time Embedding achieves the best score (50.25), clearly improving over the baseline.
OmniAlign Finally, we impose the proposed Omni OmniAlignNet on top of TEG and CRTE. As shown the bottom section of [ OmniAlignNer delivers significant performance boosts across all benchmarks. The average score improves from 50.25 to 52.59 (+2.34), and the model achieves considerable gains on Omnibench (+6.1), Worldsense (+0.75), and Dailyommi (+1.17).
4.1.2 IMPLICIT AND EXPLICIT LEARNING
We next validate implicit and explicit omni-modal learning as detailed in Section 3.2 For implicit learning, we continue to finetune the above model on 270K video conversations with au- dio stream. Results in Table 2 show clear gains on VideoMME (uu et al.] 2024a), even when subtitles are pro- vided, highlighting the value of learn-
Table 2: Ablation study on joint visual-audio learning meth- ods. "Visual+Audio" uses audio in video for implicit learning (IL), while "data engine" generates omni-modal data for ex- plicit learning (EL).
Method
Video
VideoMME w/o
w/ subities who subriles Short
Medium Long
Aloone Visual Audio (lL.)
66.96.a5
63.76,220
21.312m1
64. 16 58 82.271
Visual Audio Data Engine (BL) 68.63
67.37.3m
76.78.23
07.56.,39 57.78.0m
ing directly from audio. Further adding explicit learning data from our omni- modal data engine yields stronger improvements across benchmarks, showing the effectiveness of our data pipeline.
4.2 SCALING AND EVALUATION
With validated design choices, we now scale up the experiments using the full post-training omni- modal dataset introduced in Section 3.2 Training details are in Appendix D.4
4.2.1 OMNI-MODAL BENCHMARK
We first evaluate on omni-modal understand- ing benchmarks and show results in Table 2T Omni Vinci sets new state-of-the-art average score of 53.73, and marks a notable improve- ment of +4.07 compared to the next best model, Qwen2.5-0mni. On the Worldsense benchmark, our model achieves the highest score of 48.23, surpassing Qwen2.5 5-mni by +2.83. The ad- vantage even more significant on the Daily- omni dataset, where our model attains score of 66.50, leading by +19.05 over Qwen2.5-Omni and by +5.18 over Gemini-2.0 2.0--200-10 O-Flash-Lite. In the Omnibench benchmark, our model shows a solid score of 46.47, higher than Gemini 1.5 Pro.
Table 3: Omni benchmarks, including video-audio datasets Worldsense and Dailyommi, as well as the image-aude audio dataset Omnibench.
Onnni
Minell
Worldsense
Dailyumni (Vioo-Aadio
Ommibench
(Mides-Audio
Umaige-Audoo
Geemini
61.32 (2.0 Mash Lac)
42.91 (1.5 Pro)
GPT--4
42,60
IntemV1.2
39.10
(v.55
Qwen2
32.40
48.60
Vwen2.5:Oma Omna
45.40
47.45
$6.15
49.66
OmniVinci
48.23
66.50
46.47
53.73
4.2.2 AUDIO BENCHMARK
Audio QA. We assess our model on audio understanding benchmarks, MMAR (Ma et al.
2025 and
7

---

Published as a conference paper at ICLR 2026
Table 5: Video benchmarks. Omni Vinci Table 6: Image benchmarks. Vinci maintains compa-
outperforms NVILA baseline.
rable image understanding performance with NVILA.
Medd
Vidkolecck
Vidoo-MM
AID Clurga DecYQA
IEEVIAA
Haslha
K3NNL
Work 0/
Fmage
S0DD TOuVIAA
54.2
55.3
12.8
20.2
05.6
01.
86.
512
13.4
10.2
77.4
10.3
GPT-4a
71.
Ocean the
56.4
$7.2
13.3
61.
03.9
622
97.
42.5
70.4
75.7
10.2
18
54.5
65.8
$5.3
TEANETS
015
18.1
197
117
51.
16.3
13.3
15.5
18 56.5
36.3
58.2
-310
616
4011
131
Long
67.3
6011
560
01:3
3000
335
08.5
66.2
88.8
42.6
26.1
15.5
15.3
3811
Qneak-V VL
18
18 18.3
154
68.1
65.3
714
02.5
10.3
10.6
36.5
31.2
425 2011
19.2
77.4
18.3
Qwee2.5
70.3
64.3
153
96.6
037
71.6
533
654
72.5
762
603
NVILA
18
57.
35.7
(8.1
64.2
UnmVad
10
61.3
02.0
7066
05.2
03.5
497
945
364
075
753
153
854
Table 4: Audio QA bench- mark.
MMAU (Sakshi et al. 2024), with results reported in Tables and 16 On MMAR, Omni Vinci surpasses Qwen2.5-Omm by +1.7, and on MMAU by +0.6, highlighting significant improvement in general audio understanding.
Model
MMAR (1)
LTU
19.20
Audio Flamingo
Qwen-22Audio
30,40
Speech Recognition. To assess the automatic speech recogni- tion (ASR) capabilities of OmniVinci, we evaluate on four widely
SALAMONN
33.20
Baichuan Omni-1.5
used benchmarks: LibriSpeech (Panayotov al. 2015), AMI (Kraai] et al. 2005), Tedlium (Rousseau et al.] (012)). and VoxPopuli Wang
Qwen2.5-Omml
$6.70
OmniVVinei
58.40
al. 2021), comparing against leading multi-modal models. As shown in Table our model achieves
competitive word error rates (WER) of 1.7 on LibriSpeech-clean and 3.7 on LibriSpeech-other, closely matching or surpassing the latest works.
We further investigate OmniVinci's performance under two aggntic-cascadee setups: (i) incorporating ASR text history (Huang et al. (025) and (ii) leveraging retriever- based training as shown in Figure [6] These techniques help boost Omni Vinci's capacity, yielding average WERs of 5.7 and 5.0, respectively. These test-time scaling studies
Table 7: Multi-domain speech recogni- tion benchmarks. *Results taken from related papers; details in Appendix
WER (.)
Model
LS
LSwr
AMI
Ted.
Vox.
Avg.
are provided in Appendix
E.3 (Table 19)
Whisper-large-33
1.8
3.6
16.1
3,9
10.1
7.1
Qwen2-Andio
1.7
4.1
15.2
3,1
7.1
6.4
GPT-4o-real-time
2.5
5.0
19.3
4.1
12.1
8.6
4.2.3 VIDEO BENCHMARK
Gemini-2.0-Flade
2.5
5.9
21.5
3.0
7.9
8.2
Phi-4-MM
1.7
3.8
11.5
2.9
5.9
5.2
We compare with other open-source video-language mod-
Qwen2.
1.8*
3.4
17.9
5.2
5.8'
6.8
els in Table On the Long VideoBench (Wu et al., (024a) val set, Omni achieves score of 61.3, outperforming NVILA by a margin of +3.6. Similarly, our model improves on MVBench (Li et al. score of 70.6, outperforming also the recently released Qwen2.5-Omin (70.3).
OmniVinei
1.7
6.8
2024b) with a
Furthermore, on the Video-MME (Fu et al., 2024a) benchmark (without subtitles hints), Vinci again sets high score at 68.2, surpassing Qwen2.5-VL-77 by +3.1, positioning it as a leading open-source model for video comprehension tasks.
Audio understanding capacity enables consistent metric improvements across video benchmarks, akin to human perception.
Key Insight 2.
4.2.4 IMAGE BENCHMARK
We evaluate OmniVince ten image benchmarks to test its versatility. These tasks range from
understanding diagrams and charts (AI2D (Kembhavi et al. 2016)), ChartQA (Masry et al. (022)) to
document analysis VQo (Mathew et al. 2021)), I), mathematics (Math Vista (Lu et al. (024bb)
and general visual question answering (VQAv2 2-tesdeev (Goyal et al. 2017 ). As shown in Table 6
Omni Vinci consistently achieves competitive scores across the board.
4.3 OMNI-MODAL REASONING
Building on advances in the Group Relative Policy Optimization (GRPO) (Sho et al. 2024 algorithm
and prior work on multi-modal reasoning training (Chen et al.) 2025 Feng et al. 2025). ). we next tackle omni-modal reasoning through accommodating audio tokens in addition to visual ones.
Specifically, for each given question and omni-modal input 9 = {g., qo. 9a (q: is textual input, qu is visual input, and go is audio input, respectively), the sampling number is G. the policy model, under
8

---

Published as a conference paper at ICLR 2026
Accurncy
Acourcty Frl
251
Rowardd
077
2.7h; Flater Format
Reward Convergence
+0.1 Better
Convergence with
Audio RIL
+7.8% Acc.
0.0
0.5
0.5
0.3
0.4
0.2
0.2
0.1
11
21
Stapa
11
21
31
Staps
Steps
Ours
Owen2 5-Omni
Ours
Owen25-0mni
sudio woo audio
Figure 6: Left: Accuracy reward and format reward curves of OmniViner and Qwen2.5-Omm in RL training. Right: Accuracy reward curve of Vnnci with and without audio.
along with corresponding rewards (ri, (2,, TC), where the rewards are computed by rule-based function that evaluates format and accuracy (Shao et al. 2024). The model Ty is then optimized by maximizing the following objective:
the old policy Touts generates a set of candidate answers {0r, ,2 02: 9G}
G
To Gos qa) A,, clip( To (oilge qu Qa))
[(0)- Eor G
(min(
1 c,1 c)A,)
Tone (or qts qui qa)
Tone (0g 9t qus qa)
DD (Tu]|[re))]] (6)
where and [3 are hyper-panne parameters of each loss part, the sampling number G' is set as 8. The rewards {ri, r2, rG) are normalized to get the advantages (A.) for updating the model:
r;s mcan( fri, (2........)). ra))
A
std r,......)))
We apply GRPO post-training to the fi- nal Vinci checkpoint to enhance
Table 8: Ablation study of GRPO post-training
Omni
its performance omni-modal un-
Model
Worldsense (?) Dailyommi (?)
Omnibench (f) Avg. (1)
derstanding benchmarks. For training data, we curated a 18K omni-modal MCQ dataset using the omni-modal data engine, as detailed in the meth-
OmniVinci
48.23
66.50
46.47
53.73
OmniVinci + RL 48.70
67.08.
47.79...3
54.52.c79
ods section. During GRPO training, we utilize the Long-RL. (Chen et al. 2025) as the training
framework, configure the model process up 64 video frames. with maximum prompt length of 1024 tokens and a maximum response length of 2048 tokens. The update batch size is set to 64, with the rollout number of 8 for each sample, ensuring robust gradient estimation. We employ a temperature of 1.0 and a top-p value of 0.99 for sampling, facilitating diverse exploration during training. These training configurations are carefully designed to optimize the model's ability to handle complex omni-modal reasoning tasks effectively and efficiently.
As shown in Table we observe consistent performance gains across all benchmarks after applying RL training. Comparing convergence with Qwen2.5-Omi under the same recipe (Figure 6). both models benefit from our multi-modal RL framework, but Omni Vinci leverages stronger base per- formance and instruction-follovin to surpass Qwen2.5-Omni the GRPO accuracy curve within 15 steps, while also converging faster on formatting tasks. Ablation experiments further show that including audio input boosts RL effectiveness: with audio, accuracy reward converges +0.1 higher than video-only training (Figure 6, right), highlighting the importance of audio for video learning.
Joint audio-visual input surpasses the visual-alone input for GRPO training, offering faster and better convergence.
Key Insight 3.
4.4 DOWNSTREAM TASKS
VmniVinci also improves downstream tasks that benefit from video-audio perception, including speech prompted robot navigation (Appendix Sec. C.I). sports video understanding (Appendix
Sec. C.2 cross-lingual speech translation (Appendix Sec. C.3), medical analysis considering physi-
C.3.
cian verbal explanations (Appendix Sec. C.4). and semiconductor factory monitoring (Appendix
Sec. C.5.1
Omni Vinci enables new frontier performances in these domains.
9

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
We present OmniVinci, a systematic effort to build an omni-modal LLM that allows joint perception of images, videos, audio, and text. We discuss architectural innovations including OmniAlignNett Temporal Embedding Grouping, and Constrained Rotary Time Embedding, joint with an enhanced data and training recipe. OmniVinet showeases frontier omni-modal performances, down on training costs, and improves downstream agentic applications.
ETHICS STATEMENT
This proposed approach exclusively utilizes existing open-souree datasets that have been established in previous academic research, without generating or incorporating any novel image, video, or audio materials. The data employed in this study is designated solely for research purposes.
REPRODUCIBILITY STATEMENT
To facilitate replication by the research community, this project will be publicly available as open- source software. Our model architecture is thoroughly detailed in Section 2 while Section 4 provides comprehensive training procedures and implementation specifics, including hyperparameter configurations.
LLM USAGE
The authors used LLMs for minor refinement of the wording during writing.
REFERENCES
Marah Abdin, Sam Ade Jyothi, al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv, 2024.
AbdeIrahman Abouelenin, Atabak Ashfaq. Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras, arXiv, 2025.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In NeurIPS, 2022.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-Languuae Navigation: Interpreting visually-grounder navigation instructions in real environments. In CVPR, 2018.
Anthropic. Claude 2024.
R. Ardila, M. Branson, K. Davis, Henretty, M. Kohler, J. Meyer, Morais, L. Saunders, F. M. Tyers, and G. Weber. Common voice: A masivvly-multiinguall speech corpus. In LREC 2020. 2020.
Shuai Bai et al. Qwen2.5-vl technical report. arXiv, 2025.
Youssef Benchekroun, Megi Dervishi, Mark Ibrahim, Jean-Baptist Gaya, Xavier Martinet, Grégoire Mialon, Thomas Scialom, Emmanuel Dupoux, Dieuwke Hupkes, and Pascal Vincent. Worldsense: A synthetic benchmark for grounded reasoning in large language models. arXiv, 2023.
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El- Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin. Yian Zhang. Tugrul Konuk, Gerald Shen, Ameya Sunil Mahahaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen,
10

---

Published as a conference paper at ICLR 2026
Zhilin Wang. David Mosallanezhad, Adi Renduchintala, Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong, Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabeeh Karimi, Ying Lin, Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha, Brandon Norick, Joseph Jennings, Shrimai Prabhumoye. Syeda Nahida Akter, Mostofa Patwary Abhinav Khattar, Deepak Narayanan, Roger Waleffe, Jimmy Zhang, Bor- Yoing Su, Guyue Huang. Terry Kong. Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky, Robert McQueen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen, Manoj Kilaru, Andrew Wang. Anna Warno, Abhilash Somasamudramath, Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Oleksandr Romanenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edelman, Anahita Bhiwandiwalla, Muthu Subramaniam, Smita Ithape. Karthik Ramamoorthy, Yuting Wu, Suguna Varshini Velury, Omri Almog, Joyjit Daw, Denys Fridman, Erick Galinkin, Michael Evans, Katherine Luna, Leon Derczynski, Nikki Pope, Eileen Long, Seth Schneider, Guillermo Siman, Tomasz Grzegorzek, Pablo Ribalta, Monika Katariya, Joey Conway, Trisha Saar, Ann Guan, Krzysztof Pawelec, Shyamala Prayaga, Oleksii Kuchaiev, Boris Ginsburg, Oluwatobi Olabiyi, Kari Briski, Jonathan Cohen, Bryan Catanzaro, Jonah Alben, Yonatan Geifman, Eric Chung, and Chris Alexiuk. Llama-nemotron: Efficient reasoning models. arXiv, 2025.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335-359 2008
H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma. Crema-d: Crowd- sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4): 377-390, 2014.
Jean Carletta. Unleashing the killer corpus: experiences in creating the multi-everything ami meeting corpus. Language Resources and Evaluation, 41(2):181-190, 2007.
Edresson Casanova, Ryan Langman, Paarth Neekhara, Shehzeen Hussain, Jason Li, Subhankar Ghosh, Ante Jukić, and Sang-gil Lee. Low frame-rate speech codec: codec designed for fast high-quality speech Ilm training and inference. In ICASSP. IEEE, 2025.
Peihao Chen, Xinyu Sun, Hongyan Zhi, Runhao Zeng. Thomas H. Li, Gaowen Liu, Mingkui Tan, and Chuang Gan. A nav: Action-aware zero-shot robot navigation by exploiting vision-and-langee and-language ability of foundation models. arXiv, 2023a.
Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor: Vision lisionaudio-llangage omni-perception pretraining model and dataset. arXiv, 2023b.
Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A wtt---adiioouuiiilleeeeeee
omni-modality foundation model and dataset. NeurIPS, 2023c.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang. Zhijian Liu, Ethan He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. Longvila: Scaling long-context visual language models for long videos. In ICLR, 2024a.
Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojuan Qi, et al. Scaling rl to long videos. NeurIPS, 2025.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv, 2024b. InternLL
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo. Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang. Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu,
11

---

Published as a conference paper at ICLR 2026
Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How Far Are We to GPT-V? Closing the Gap to Commercial Multimodal Models with Open-Soure Suites. arXiv:2404. 16821,, 2024c.
An-Chich Cheng, Yandong Ji, Zhaojing Yang, Xueyan Zou, Jan Kautz, Erdem Biyik, Hongxu Yin, Sifei Liu, and Xiaolong Wang. NaVILA: Legged Robot Vision-Language-Actioo Model for Navigation. arXiv:2412.0445. 2024a.
Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mitsufuji. Mmaudio: Taming multimodal joint training for high-quality video-to-aude tuuiioo synthesis. In CVPR, 2025.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Lim, Luowei Yang, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv, 2024b.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv, 2023.
Yunfei Chu. Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-pus purepoe- vision-language models with instruction tuning. arXiv, 2023.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn J Keogh. The ucr time series archive. IEEE/CAA Journal of Automatica Sinica, 2:1293133 2018.
Google DeepMind. Gemini 2.0 technical report. Technical Report, 2025.
Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), NeurlPS, Curran Associates, Inc., 2023.
Soham Deshmukh, Shuo Han, Hazim Bukhari, Benjamin Elizalde, Hannes Gamper, Rita Singh, and Bhiksha Raj. Audio entailment: Assessing deductive reasoning for audio understanding. In AAAI, 2025.
Mattia A Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. Must-c: multilingual speech translation corpus. In NAACL. Association for Computational Linguisties, 2019.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv, 2020.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff. Andy Zeng. Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. arviv, 2023. URL htt.tppp:///xxi.ooo////aabaab /ttp:////aaa///33333333
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP. IEEE, 2020.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, and Xiaoli Li. Tslanet: Rethinking transformers for time series representation learning. arXiv, 2024.
Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jan Kautz, Jang Hyun Cho, Marco Pavone, Song Han, and Hongxu Yin. VILA?: VILA Augmented VILA. arXivv2407.1745. 2024.
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milms. CoRR, ahs/2503.21776 2025.
12

---

Published as a conference paper at ICLR 2026
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: open dataset tuman-llbbled sound events. IEEE/ACM Transactions on Audio, Speech, and Language 2021.
Processing,
Chaoyou Fu, Yuhan Dai. Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-Modal LLMs in Video Anal- ysis. arXiv:2405.2107, 2024a.
Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao. Yifan Zhang, Xiong Wang, Di Yin, Long Ma. Xiawu Zheng. Ran He, Rongrong Ji, Yunsheng Wu, Caifeng Shan, and Xing Sun. Vita: Towards open-source interactive omni multimodal 1lm. arXiv, 2024b.
Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Reddy Evuru, S Ra- maneswaran, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Compa: Address- ing the gap in compositional reasoning in audio-language models. In ICLR, 2024.
Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro. Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities. arXiv, 2025.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023.
Arushi Goel, Karan Sapra, Matthieu Le, Rafael Valle, Andrew Tao, and Bryan Catanzaro. Omcat: Omni context aware transformer. arXiv, 2024.
Arushi Goel, Sreyan Ghosh, Jachyeon Kim, Sonal Kumar, Zhifeng Kong, Sang- Lee, Chao- Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. Audio flamingo Advancing audio intelligence with fully open large audio language models. arXiv, 2025.
Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass, Listen, think, and understand. arXiv, 2023.
Google. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11880 2023.
Google. Gemini 1.5: Unlocking Multimodal Understanding Across Millions of Tokens of Context. arXiv:2403.05530, 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Star Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In CVPR, 2017.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv, 2025a.
Yuxin Guo, Shuailei Ma, Shijie Ma, Xiaoyi Bao, Chen-Wei Xie, Kecheng Zheng, Tingyu Weng, Siyang Sun, Yun Zheng, and Wei Zou. Aligned better, listen better for audio-visual large language models. arXiv, 2025b.
Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang. Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In SLT. IEEE, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. ICLR, 2021.
Rui Hu, Delai Qiu, Shuyu Wei, Jiaming Zhang, Yining Wang, Shengping Liu, and Jitao Sang. Investigating and enhancing vision aision-audio capability in omnimodal large language models. arXiv, 2025.
13

---

Published as a conference paper at ICLR 2026
Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, et al. Step-audio: Unified understanding and generation in intelligent speech interaction. arXiv, 2025.
Rongjie Huang, Mingze Li, Dongchao Yang. Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In AAAI, 2024.
Shehzeen Hussain, Paarth Neekhara, Xuesong Yang, Edresson Casanova, Subhankar Ghosh, Mikyas T Desta, Roy Fejgin, Rafael Valle, and Jason Li. Koel-tts: Enhancing IIm based speech generation with preference alignment and classifier free guidance. arXiv, 2025.
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. NeurlPS, 2024.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A Diagram is Worth a Dozen Images. In ECCV, 2016.
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In NAACL-HLT, 2019.
Minsu Kim, Chae Won Kim, and Yong Man Ro. Deep visual forced alignment: learning align transcription with talking face video. In AAAI, 2023.
KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng. Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, Jun Chen, Yanru Chen, Yulun Du Weiran He. Zhenxing Hu, Guokun Lai, Qingcheng Li, Yangyang Liu, Weidong Sun, Jianzhou Wang, Yuzhi Wang, Yuefeng Wu, Yuxin Wu, Dongchao Yang, Hao Yang. Ying Yang. Zhilin Yang, Aoxiong Yin, Ruibin Yuan, Yutong Zhang, and Zaida Zhou. Kimi-audio technical report, 2025, URL httttt
//arxivvorr/abs/2554....8
Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In MT Summit, 2005.
Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. In ICML, 2024.
Wessel Kraaij, Thomas Hain, Mike Lincoln, and Wilfried Post. The ami meeting corpus. In Behavioral Research, 2005.
Jacob Krantz, Erik Wijmans, Arjun Majundar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision and language navigation in continuous environments. In ECCV, 2020.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision: Easy Visual Task Transfer. arXiv:2408.00 033366, 2024a.
Caorui Li, Yu Chen, Yiyan Ji, Jin Xu, Zhenyu Cui, Shihao Li, Yuanxing Zhang, Jiafu Tang, Zhenghao Song, Dingling Zhang, al. Omnivideobench: Towards audio visual understanding evaluation for omni milms. arxiv, 2025a.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv, 2023.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024b.
Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, al. Baichuan-omni-Is. technical report. arXiv, 2025b.
14

---

Published as a conference paper at ICLR 2026
Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang. Jian Yang, et al. Omnibench: Towards the future of universal omni- amni-language models. arXiv, 2024c.
Paul Pu Liang, Yun Cheng. Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, et al. Quantifying & modeling multimodal interactions: An information decomposition framework. NeurIPS, 2023.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv, 2023.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-Aware Weight Quantization for On-Device LLM Compression and Acceleration. In Conference on Machine Learning and Systems (MLSys), 2024a.
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, 2024b.
Ting- Ting-Hung Lin, Hung-Jen Chen, Yuan-Chung Wei, Sung-Po Yang, Yung-Lun Lin, Parthasarathy Sriram, Guan-Hong Liou, Yu-Chieh Huang, Yi-Hsuan Chiu, Po-Chun Lai, Yiyi Wang, and Mark Peng. Genai applications of vision-language models for semiconductor defect classification. In SPIE. SPIE, 2025a.
Yen-Ting Lin, Zhehuai Chen, Piotr Želasko, Zhen Wan, Xuesong Yang, Zih-Ching Chen, Krishna Puvvada, Ke Hu, Szu- Wei Fu, Jun Wei Chiu, et al. Neko: Cross- coos--modllity post-recognition error correction with tasks-guided mixture-of-experts language model. In ACL, pp. 222-236, 2025b.
Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho- aqa: A crowdsourced dataset for audio question answering. EUSIPCO. IEEE, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao. Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. In CVPR, 2025a.
Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment. arXiv, 2025b.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In CVPR, 2024a.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Math Vista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. In ICLR, 2024b.
Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, et al. Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix. arXiv, 2025.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In ACL, 2024.
Divyam Madaan, Taro Makino, Sumit Chopra, and Kyunghyun Cho. Jointly modeling inter-& modra- dependencies for multi-modal learning. NeurlPS, 2024.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. In ACI. 2022.
15

---

Published as a conference paper at ICLR 2026
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. DocVQA: Dataset for VQA on Document Images. In WACV, 2021.
Brandon McKinzie, Zhe Gan, Jean-Phillppe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng. Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training, arXiv, 2024.
Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen, A multi-device dataset for urban acoustic scene classification. arXiv, 2018.
Paarth Neekhara, Shehzeen Hussain, Subhankar Ghosh, Jason Li, Rafael Valle, Rohan Badlani, and Boris Ginsburg. Improving robustness of Ilm-based speech synthesis by learning monotonic alignment. arXiv, 2024.
OpenAL. GPT-4o, 2024.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In ICASSP. IEEE, 2015.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi palti-party dataset for emotion recognition in conversations, arXiv, 2018.
Vinay Prithyani, Mohsin Mohammed, Richa Gadgil, Ricardo Buitrago, Vinija Jain, and Aman Chadha. On the feasibility of vision-language models for time-series classification. arXiv, 2025.
Qwen.
Qwen2.5: A party of foundation models, September 2024.
URL https://
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML, 2023.
Srijith Radhakrishnan, Chao-Han Huck Yang, Sumeer Ahmad Khan, Rohit Kumar, Narsis A Kiani, David Gomez-Cabres and Jesper N Tegner. Whispering Ilama: A cross-modal generative error correction framework for speech recognition. EMNLP, 2023.
Anthony Rousseau, Paul Deléglise, and Yannick Esteve. Ted-lium: an automatic speech recognition dedicated corpus. In LREC, 2012.
Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nicto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv, 2024.
Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. Tomato: Assessing visual temporal reasoning capabilities multimodal foundation models. arXiv,, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300,, 2024.
Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? ECCV, 2024.
Baifeng Shi, Boyi Li, Han Cai, Yao Lu, Sifei Liu, Marco Pavone, Jan Kautz, Song Han, Trevor Darrell, Pavlo Molchanov, al. Scaling vision pre-training to 4k resolution. CVPR, 2025.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Nearocomputing, 568: 127063, 2024.
16

---

Published as a conference paper at ICLR 2026
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models, 2023a.
Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. In NeurIPS, 2023b.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv, 2024.
HumanOmni Team. Humanomni: A large vision-speech language model for human-centric video understanding. arXiv, 2025.
Changhan Wang, Anne Wu, and Juan Pino. Covost 2 and massively multilingual speech-to-text translation. arXiv, 2020.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv, 2021.
Chenyu Wang, Sharut Gupta, Xinyi Zhang, Sana Tonekaboni, Stefanie Jegelka, Tommi Jaakkola, and Caroline Uhler. An information criterion for controlled disentanglement of multimodal data. arXiv, 2024a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VE VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. arXiv: /2191, 2024b.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, al. Internvide02: Scaling video foundation models for multimodal video understanding. arXiv, 2024c.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang. Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In NeurIPS, 2024d.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. NeurlPS, 2024a.
Ming-Ju Wu, Jyh-Shing Jang, and Jui- Jui-Long Chen. Wafer map failure pattern recognition and similarity ranking for large-scale data sets. IEEE Transactions on Semiconductor Manufacturing, 28(1):1-12, 2015.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 11m. In 2024b.
Haotian Xia, Zhengbang Yang, Junbo Zou, Rhys Tracy, Yuqing Wang, Chi Lu, Christopher Lai, Yanjun He, Xun Shao, Zhuoqing Xie, et al. Sportu: A comprehensive sports understanding benchmark for multimodal large language models. In ICLR, 2025.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In ICML, 2023.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv, 2025.
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv, 2025a.
Chao-Han Huck Yang, Yile Gu, Yi-Chieh Liu, Shalini Ghosh, Ivan Bulyko, and Andreas Stolcke. Generative speech recognition error correction with large language models and task- activviii prompting. In ASRU. IEEF, 2023.
17

---

Published as a conference paper at ICLR 2026
Chao-Han Huck Yang. Sreyan Ghosh, Qing Wang. Jaeyeon Kim, Hengyi Hong, Sonal Kumar, Guirui Zhong, Zhifeng Kong, S Sakshi, Vaibhavi Lokegaonkar, Oriol Nieto, Ramani Duraiswami, Dinesh Manocha, Gunhee Kim, Jun Du, Rafacl Valle, and Bryan Catanzaro. Multi-domain audio question answering toward acoustic content reasoning in the dease 2025 challenge, 2025b. URL
Yuan Yao, Tianyu Yu. Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. MiniCPM-V: A GPT-4V Level MILMM on Your Phone. arXiv: 0/800, 2024.
Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, et al. X-vila: mross-todality alignment for large language model. arXiv, 2024.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv, 2023.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning, In SIGKDD, 2021.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
Bo Zhang al. Llava-next: Stronger Ilms supercharge multimodal capabilities in the wild. LLavA Blog, 2024a. URL llava-next/
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuneel audio-visuu language model for video understanding. arXiv, 2023. /tLiittss///rrvw..rrr///// abs http:///aabb33300583333
Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and Wang He. NaVid: Video-based VLM Plans the Next Step for Vision-and- Language Navigation. In Robotics: Science and Systems (RSS), 2024b.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024c. URL
https: //1p::://1avae waavavvl..cmm www.ggit..u...... wtw.. /
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video Instruction Tuning With Synthetic Data. arXiv:2410.02713, 2024d.
Ziwei Zhou, Rui Wang, and Zuxuan Wu. Daily-omni: Towards audio-visual reasoning with temporal alignment across modalities. arXiv, 2025.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv, 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijic Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv, 2025.
18

---

Published as a conference paper at ICLR 2026
APPENDIX
APPENDIX TABLE OF CONTENTS
Related Works
20
Real-World Qualitative Study
21
Downstream Agents
21
Robotics: Gheech-Driven Vision Language Navigation
21
Sport Video Understanding
22
Speech Agent: Speech Translation
24
................................
Medical Al
24
C.5
Smart Factory Agents
................................
25
C.5.1
Semiconductor Manufacturing
25
C.5.2
Factory and Industrial Time Series Understanding
26
D
Method Details
27
D.1
Omni-Modal Input Embedding
27
D.2
More Discussion Constrained Rotary Time Embedding (CRTE)
28
D.3
Modality-Specific Training
28
D.3.1 Vision Training
28
D.3.2
Audio Training
29
D.4
Omni-Modal Joint Training Details
29
D.5
Extra Details of Training Data
29
More Experiments and Discussion
30
Audio Encoding
30
2
Model Quantization and Efficient Deployment
................................
31
Omni Vinci with ASR Test- Time Scaling Methods
32
E.3.1
Instructional Formatting for Cross-Modal Decision Making
33
E.3.2
Supervision for Decision-Aware Fine- Tuning
34
Inference- Time Control Flow
E.4
Speech Output
34
E.5
Analysis: Why Do We Need Omni-Modal LLMs?
36
E,6
Evaluation on More Video Benchmarks
36
Text Understanding Evaluation
37
Loss Summanization
37
E.
Ablation Study of TEG Group Duration
Ta
37
19

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
A significant body of work has focused on augmenting LLMs with individual sensory capabilities, primarily vision and audio, often following a similar architectural blueprint. In the visual domain, the dominant paradigm involves using vision encoder (e.g.. ViT (Dosovitskiy 20200) to extract features which are then aligned with the LLM's input space via a bridging module. Pioneering models
like Flamingo (Alayrac
al. 2022) introduced sophisticated cross-attention mechanisms, while
subsequent works (Li et al. 2023: et al. 2023 Ye et al., [023 Driess et al. [023] Liu et al. 2023]
Lin et al., 2024b [iu et al. 2025a| McKinzie et al. 2024] Dai et al. 2023] Zhang et al. 2023; Wang
et al. c024c Maaz et al., 2024; Fang et al., 2024 Shi et al. 2025). demonstrated the remarkable
effectiveness of a simple projection layer combined with visual instruction tuning. A parallel line
of research has applied this pattern to the auditory domain, where Audio-Language Models like
LTU Gong et al. 2023). Whispering-LLant LLaMA (Radhakrishnan et al.
2023). Audio-Flamingo (Goel
et al.
2025 Qwen-Audio (Chu al. 2023), and others (TTng al.
2023a Deshmukh et al. 2023
Kong et al. 2024 Ghosh et al. 2025; Huang et al., 2024; Chu et al.
2024) use audio encoders to
process speech, music, and ambient sounds. These specialized models represent crucial stepping stones toward the more holistic goal of unified, omni-modal understanding. While specialized models
for vision and audio have become increasingly capable, the development of foundational, omni-moda LLMs remains relatively nascent. For example, such a single omni model that can natively process and reason across text, vision, audio, and potentially other data types.
The endeavor presents various challenges in terms of model architecture, data curation, and the immense computational resources required for training. Recent pioneering efforts have addressed the challenges of multimodal understanding and reasoning. Google's 's Gemini (Google) (2023) represents significant advancement as a natively multimodal model designed to seamlessly integrate and reason across interlcaved text, images, audio, and video inputs. However, it remains proprictary and is not available to the open- open-source community. Within the open-source community, several noteworthy
efforts anni- LLMs have been introduced (Li et al. 2025b Lu ct al.
2024a Wu et al.)
2023b Liu et al. 2025b Fu 2024b Ye et al. 2024| Chen et al. 2023c; Hu et al., 2025: Chen et al.
et al. 2024b), demonstrating strong capabilities in joint vision-audio understanding tasks. Among these, Phi-4-MM (Abouelenin et al. 2025) and Qwen2.5-0mni (Xu et al. 2025) achieve the strongest results to date; however, their accompanying technical reports reveal relatively simple architectural choices and a lack of thorough ablation studies to systematically examine critical design decisions. In contrast, our work not only proposes several novel techniques for omni-moda understanding but also adopts more rigorous experimental approach by conducting comprehensive ablation studies before scaling to large-scale datasets. We systematically evaluate various architectural choices and design decisions, providing detailed experimental analyses that we make publicly available. Through this methodical investigation, we aim to contribute valuable insights that can inform and inspire future research directions in omni-modal large language models.
Some research work study multimodal alignment in other domains. For instance, Cheng et al. (0025) investigates vision-audio alignment for audio generation rather than multimodal understanding as we do. Their method applics RoPE within transformer blocks to align visual and audio tokens, but RoPE encodes only relative temporal information. In contrast, OmniVinci incorporates both relative and absolute temporal cues directly into the input sequence before tokens enter the L.M backbone. Kim et al. (2023) target vision-text alignment and do not address the vision-audio setting. It trains a neural network to predict a text transcription for each frame. By comparison, our TEG and CRTE methods require no training or supervision; instead, they inject temporal information directly using predefined formula. Guo et al. (2025b) fuse visual and audio tokens at the same timestamp via cross cooss-attention and then forward them into the LLM. However, it does not embed absolute temporal information into the tokens. Our method differs by encoding absolute temporal cues through embedding rotation in CRTE. Finally, there are also research works on time-series series alignment (Zerveas et al. 1 Eldele et al. 2024h that do not involve vision or audio modalities.
Compared Models in Experiments. In the experimental section, we compare with prior works on vision LLMs, audio LLMs, and omni-modal LLMs on various multimodal benchmarks. Specifically, we list the reference here due to the space limit in main text. Compared models include Gemini (Google 2023 2024), GPT-4o (OpenAl) 2024), Claude-3.0 (Anthropic. 2024), Intern (Chen et al.) 2024c), Qwen2-Ve VL (Wang et al. (2024b). Awen2.5-0mmn (Xu et al., 2025). 2,2), Phi-4-MM (Abouelenin Team et al. 2025), Audio Flamingo (Goel et al. 2025). Qwen2
et al.] 2025), Kimi-Audio (KimiTeam
20

---

Published as a conference paper at ICLR 2026
User: Describe video detoil.
User: Assess the video, followed by detailed descritiio of video and audio conteents
Answer: video begins Hoang a modern, woll-it room with pocket
Video -15
large and greenery
Answer: The video features pedrast disnassion belween two ind viduuls in light blue shart and the other in black blazer, set in professional studio envirnment with betge, conversttion revalves specificall focusing the imitations and challenges of achieving human- ievel Al through noural networks (NN)). The individual in the ight blue shirt. who is the pa primaty speaker, expresses skeplition about the feasibiity of wass reaching human-level inteligence, emphastong the need for different approach, He highights the Imitations of Nhis in handlngg complex tasks and the pcsential for diminishing returns on investment in this area. The duviduall in the black blzzer who lssens attentaclly and occssianally inter sheres wis perspective the patentisl of Al the im importncee understanding the physical wordd for Al systems to reacon and plan effectively also discusses the challenges of achteving human- the need community address these challanges. pranticaa
golrf by open large black with gold nebonn person's hand webble focus then shifts
wnting message Trjoy
pert The cmessage
the and mcropphne artificial
The
the transttonn
box the tabie,
white
the NIDDAA
wee peeksshp industria setting The rabut hollts the same back gift box with the gold cublen and how, and if opeen the bux reveal the black card with mecsage. The rabot esomines the card closely. The namatie contrrues with the robot, ctill in the warishop setting, notding the back grft oox the rabot opens the box, revealing sloek, white device with a black screon, cestled in crsmpled blank paper. The rabot examones the nevice closely, then places it bank into the box and cinses it The arane transation to diffeweed sething where the rabot is now in modeer office envrooment with grean wall and multale computer monitors, The robot stands behind the aoooed gift bax, gesiuuing with Its hands as if axplaiiing or presenting sometiingg The video wraps up with the robot to the moder office envronment gesturing with its hands The scene fransitions detavled foatures and expressive eyos.
abor ative aporoach among research discussion is analytical providing insights and perspectives on topic.
eval
a clece-up of the rebor's fane, showing its
Joint Vision-Audio Perception
Joint Vision-Audio Perception
User: [What is speaker doing in video?! 04044
Transer be Sowech, O +
Thee first part lat's just ge what the foundation We
that mentioned, fundamentaly ntally
brcardng problem few cifferent and seecht the best ansowr Maybe solves the same problem veriety ways and
oaches problom
Answer: 0.00 04
Video +15 minutes
has the seme answer consistancy checking. Dr maybe, after it's daww deriing the answer, plugs it back into the equation -n maybe quadratic equation to confirm that, fact, that's the right answor Insteed of just one-shot blurting out.
man is speaking to crewd in large industrial space, ng microphone and gesturing as he speake.]
User: O ++
[What the misiion of his company?!
User: Transer be to Arabic
+000 64
Answer" aes wotn مارر je wes des (لldea aca you lad d80 ..."S0 upd اني jهبر ja Lday di co0l yoll
[The mission of his company establish a zelf sustaining civllzation on Mars]
Joint Vision Parception and Speech Generation
Speech Transcription Translation
Figure 7: OmniVinci demonstrates strong vision and audio perception capabilities to handle single or joint modality scenarios. The model also supports audio prompts and outputs.
Audio (Chu al. 2023), Gemma TTam et al.
2024), LTU (Gong al.
2023) SALAMONN Tang
et al.
2023a).
Baichuan-Omni-ls. (Li et al.
(025b). Whisper-large-2 (Radford et al. 2023). LLaVA
NeX1
Video
(Zhang et al., 2024c). Intern VL2 (Chen al. 2024c). LLaVA-One Vision (Li al.
2024a).
Long VILA (Chen et al. 2024a), Qwen2.5-VL VL (Wang et al. 2024b), NVILA (Liu et al.
2025a).
Video-ChatGPT (Maaz et al. 2024). VideoChat2 (Li et al. 2024b).
B REAL-WORLD QUALITATIVE STUDY
To evaluate the performance of the model real-world omni-modal signals, we test it using recently released online vidcos, as shown in 7 Our results demonstrate that the model can thoroughly comprehend both visual and audio inputs from previously unseen videos and generate responses based on this information, highlighting its strong generalization capabilities. The model successfully integrates speech cues with visual data, allowing for more effective interaction with the environment. These qualitative observations demonstrate the effectiveness of the proposed explicit and implicit training strategy.
C DOWNSTREAM AGENTS
Next, we demonstrate the applicability of VmniVinci in a wide range of downstream agentic tasks that yield consistent improvements across benchmarks while enabling new capabilities.
C.I ROBOTICS: SPEECH-DRIVER VISION LANGUAGE NAVIGATION
Prior work (Cheng et al., 2024a:
Zhang et al. 2024b Chen et al. 2023a in Vision-Language
Navigation (Anderson et al. not always practical for real world scenarios where the most convenient and natural way to command robot is through human speech. As first step toward this goal, we introduce a speech-driven vision
20188 has predominantly relied based prompts. However, this is
21

---

Published as a conference paper at ICLR 2026
Instruction: Enter the bedroom and stand at the foot of the bed. Walk right and bedroom from the door with letter "B" the wall to your left. Take right turn out of the bedroom and walk in through another bedraom's door where you a light colored shaggy rug. Take another right and stand jusi through the door after the white little dog on floor to your left. cm.
Agent:
De
Instruction: Step through the bathoom dooors, make a left turn and take step, then sharp left lurn continue walking until you've made through the bedroom door. Stop.
Agent: Turn left 15 degrees.
Orade
Prediction
Figure 8: An illustration of our speech-driven navigation agent based on Omni Left: Agent's current visual observation. Middle: Top-down map indicating the goal position and the agent's past trajectory. Right: the input speech instruction and the agent's predicted action given the current observation.
language navigation task. This task is inherently more challenging than its text-based counterpart, as interpreting the nuances of speech is more complex than processing clean text.
Table 9: Vision Language navigation results on R2R-CE. Our speech-driven model, Omni Vinci, achieves comparable performance to the text-driven NVILA, with a lower navigation error.
R2R Val-Unseen
Model
Size
Obs.
Instruction
NE 4
OS
SR 1
SPL
Seg2Sey
RGB
Text
10.10
8.0
0.0
0.0
CMA
RGB
Text
9.55
10.0
4.0
NaVid
7B
RGB
Text
5.47
49.0
37.0
35.0
NVILA
SB
RGB
Text
5.43
60.4
53.3
48.8
OrniVVnni
RGB
Audio andior Text
5.67
60.8
50.6
45.1
Specifically, we fine-tune Omni on the training split of R2R-CE (Krantz et al. 2020
benchmark for Vision-and- and-Laguuage Navigation in continuous environments, with speech prompts, using 8 history frames for context in line with NVILA (Liu et al. (025a). As shown in the results Table 9. Omni surpasses many text-based models and achieves performance comparable to NVILA. We present qualitative examples in Figure & that illustrate how our speech-driven vision- language-action (VLA) navigation agent functions in practice. The agent is deployed in the Habitat simulator under the continuous environment setting. The demo provides three synchronized views: (1) the agent's 's current observation RGB (left), (2) a top-down map indicating the goal location and the trajectory taken so far (middle), and (3) the spoken instruction together with the agent's 's predicted action, such as moving forward a certain distance or turning left or right by specified angle (right).
C.2 SPORT VIDEO UNDERSTANDING
Understanding videos of complex sports scenarios requires models to capture both visual dynamics and contextual cues. To evaluate the sports understanding capability of our proposed OmniVinci Vinci, we conduct experiments fine-grae -raingriind sports comprehension. As shown in Table Omni Vinci-98 9B delivers strong performance despite its compact scale of billion parameters. These results confirm the effectiveness of model
the SPORTU-video dataset (Xia et al. 2025), large-scale benchmark for
22

---

Published as a conference paper at ICLR 2026
Serve
Rally
Rally
End point
Frame
Federer Roger Federer is player the white polo T-shirt and matching shorts. has white bandanna fied around his forehcad and a white wristband on his right wrist. He is also wearing pair of white shoes with white socks. Benneicau Julien Bennetcau is player white polo
System Prompt:
matching shorts. He with black markings them. He has black ankle brace on his right foot well.
black
wristbund right white pair of shoes
Instruction: What happened in this point? Provide a detailed caption.
Agent: Benneteau's
lands outside service court, but where Benneteau hits the ball out of bounds with forehand shot.
atternpt successful.
point a lengthy rally
Instruction: How this point ended hetween Federer and Benneteau.
Agent: The point when Bennetenu
out of bounds.
Figure 9: Example of tennis broadcast commentary generation. For better visualization, we added red circle highlights to the tennis ball.
Table 10: Overall performance of MI.MMs on SPORTU-video for multiple-choice questions. The best results within each category bolded. Notably, OmniVinci model achieves highly competitive accuracy compared with both flosedsooure and ooen-source models.
(Ace., 1)
Close Source Model
Claude 3.D Haiku
47.95
Gemini 1.5 Pro
64.93
Gemini 1.5 Flash
62.52
GPT-tomini
58.19
GPT-4o
68.79
Open Source Model
ChatUniVi
41.89
LLaVA NeXT
63.72
mPLL'G-OWB
60,80
ST-LLM
46.39
Tarsier
60.99
Video-ChaGGPP
34,05
VideuChal2
61.53
Qwen2. 5-m22-5-mmni-7
60.49
Omni Vinci-9B (ours)
67.30
design and motivate its extension to more demanding, real-world applications such as live sports broadcasting, where both accuracy and efficiency are essential.
To further assess performance in realistic broadcasting settings, we curate a tennis-specific dataset collected from 14 full matches. The dataset contains 24,078 multiple-choice questions and 20,214 open-ended questions derived from pre-clipped videos, each spanning 3-120 seconds with precisely annotated start and end points. Since sports broadcasting requires synchronizing visual actions with speech cues (e.g., live commentators' narration umpire calls) to enable professional-style commentary, tennis provides an ideal domain for multimodal evaluation.
In our tennis experiments, we evaluate tasks such as identifying the server from player characteristics, determining the point winner, and classifying the outcome type (e.g., ace, forced error, unforced error). The benchmark Vinii processes clips at their native resolution (primarily FHD 1920 1080), using 128-frame segments per point. As shown Table I1 Omni Vinci substantially outperforms Owen2.5-Omm in predicting point outcomes and rally length, demonstrating the advantages of high-resolution spatiotemporal modeling. Figure Iillustrates sample videos with action explanations, along with generated open-ended commentary styled after professional broadeasters.
23

---

Published as a conference paper at ICLR 2026
Table 11: Comparison of video understanding accuracy (%) for tennis broadcasting. Results are evaluated with multiple-choice questions (MCQ). Inference time is measured on an NVIDIA A100, with input clips averaging around 20 seconds in duration. AWQ indicates model quantization performed with the AWQ technique (Lin et al. (024a).
Inference Time (Seconds L)
Server & Winner
Receiver Winner
Point Ending
Shots Exchanged
Model
Owen2.5-Omni
3.34
96.2
90.7
48.6
38.3
OmniVinei
3.29
85.7
Vini w/ AWQ
1.85
100.0
100.0
85.7
85.1
Table 12: Performance comparison of different models on Covost2 speech translation tasks measured by BLEU scores. EN + X denotes translation from English to the target language, and X EN denotes translation from the target language to English. Languages: zh Chinese, ja = Japanese, ar Arabic, de German.
EN
X (Acc., f)
EN (Asc., f)
Model
zb
ja
ar
de
avg.
2h
ja
ar
de
ayg.
Qwen2-andio
45.2
24.8
20.1
29.9
30.0
24.4
18.7
19.5
35.2
24.5
Qwen2..5-omni
41.4
26.0
19.7
30.2
29.3
29.4
12.1
19.3
37.7
24.6
Phi-4-mm
38.0
31,9
35.3
28.9
24.9
33.3
5.5
37,9
25.7
OmniVVinei
39.7
32.6
23.3
38.5
32.8
29.9
33.7
20.1
32.6
29.1
For efficient deployment, we adopt the LLM-AWQ implementation of Activation-aware Weight Quantization (Lin al. (004a),, which enables 4-bit quantization while preserving accuracy. Inference executed using the TinyChat engine on NVIDIA hardware, supporting multimodal video-audio inputs. On single NVIDIA A100, ViniVinci achieves average latency of under seconds per pre-quantized clip, delivering a 45% boost inference speed and making it well-suited for live broadcasting scenarios. We further validate deployment NVIDIA L40s GPUs, demonstrating the practicality of our approach in resource-constrainee environments.
C.3 SPEECH AGENT: SPEECH TRANSLATION
We benchmark OmniVinci the CoVoST2 (Wang et al. BLEU scores across multiple target languages in both tuning on related data, and show the results in Table 12 Our model delivers competitive translation quality across most directions, with particularly strong performance in - HN for Japanese (23.2 BLEU) and Arabic (23.0 BLEU). This balance of accuracy across languages highlights the benefit of integrating speech translation corpora within our omni-modal training pipeline, enabling to perform both recognition and translation a unified framework. The ability to handle multilingual speech understanding and cross cross-lingual transfer further broadens the applicability of our model in real-world communication, international dialogue systems, and cross- cross-border information access.
2020 speech translation task, measuring and X-EN directions, after fine-
C.4 MEDICAL AI
We evaluate OmniVinci's zero-shot generalization to the medical domain using 49 privacy- deidentified, radiologist-curated video clips of whole-body CT interpretations. As illustrated in Figure 10 each 2-minute recording captures a radiologist interpreting real-world clinical images with 2D allal- viewer, including scrolling through slices, placing measurements and annotations, zooming, adjusting window/level, and, when relevant, comparing the same image under different window settings.
From these video-audis pairs and their transcripts, we construct 588 multiple-choice questions spanning four categories-(i) long-horizon temporal reasoning and localization, (ii) audio-visual synchronization and understanding, (iii) anti-shortcutiing (resisting language priors without visual evi- dence), and (iv) temporal reasoning- approximately balanced across categories with three options per item. The dataset curated with assistance from the L. --Nemotrooo------ Ultra-25BB (Bereovich) jet al. 2025), leveraging both the visual content and transcripts. We report comparative performance for Vinivinci and Qwen2.5-O0l mmmi in Table 13
24

---

Published as a conference paper at ICLR 2026
Whole-Body CT Interpretation Videos
Speech Audio
starting to reviow chest, abdomen, lunas and the liver. We're going to begin our on using the luna wincow And first I'm checking the right lung, So here at the apex you are seeing some locsened cyctic areas. These are probabbly some air cysts and related fibrotic changes at the right lung apex. And then like like similarfy on the left side, you seeing those air filled cavities and cysts. So, the first thing is to look some nodules, lunge, lobe here you are seeing some bronchectasee
Radiologist:
we be focusing on
nodules, infirrates and potential masses, So, I'm
the right mid lobe
anything. Now again sir-filled cavities and cysts, they also very prominently visible, this like basically located at right upper lbbo. So, middle of be oetween the two fissures, And again, we are seeing an cyst .."
for these
right
Figure 10: Sample frames and transcript trunks from one of the curated radiolegist-narrated CT interpretation video. For annotation, the radiologist maintains 2D axial view while progressively adjusting visualization (e.g., window/level, zoom) and annotating across slices.
Table 13: Performance comparison between Omni Vinci and Qwen2.5-Omm on
multiple- choice QA datasets across four categories. Abbreviations: LH long-horizon temporal reasoning & localization; AVS = audio-visual synchronization & understanding: AS = anti-shortcutiing (resisting language priors without video evidence); TR temporal reasoning.
Method
Acc. (LH) 1
Acc. (AVS) 1
Acc. (AS)
Ace. (IR)
Average 1
Qwen2.5-Omm
0.83
0.75
0.91
0.70
0.79
OmniVVinci
0.84
0.76
0.92
0.76
0.82
OmniVinci consistently outperformed Qwen2.5-Ommi across all four categories, yielding an overall gain of about +2.0 percentage points. Its largest margin was in temporal reasoning (TR; +6.1), highlighting stronger capabilities event sequencing, change detection, and temporal cue mod- eling-often the most demanding aspects of video comprehension in clinical workflows. Stable improvements were observed in long-horizon reasoning (LH) and audio-visual synchronization (AVS) (+0.7 each), reflecting better preservation long-range context and closer alignment between narra- tion and visual content. The anti-shortcutting (AS) category also showed a gain of +0.7, suggesting that OmniVinct is more robust against linguistic shortcuts and leans more heavily on visual evidence. Some qualitative comparisons of test samples are presented in Figure 11
C.5 SMART FACTORY AGENTS
C.5.1
SEMICONDUCTOR MANUFACTURING
Wafer maps are essential in semiconductor manufacturing for visualizing defect distributions, enabling yield monitoring, process drift detection, and preliminary root cause identification. It is a domain with a significant gap from multimodal L.M.. To study whether we can leverage our omni-modal ViniVinci on this task, we fine-tune OmniVinci on wafer map data, aligning visual and textual features for robust defect analysis, as illustrated in Figure| 12 On the WM-81 IK dataset (Wu et al. 2015), Omni achieves superior performance over VILA (Lin et al.) 2024b) and NVILA (Lin et al. 2025a Liu et al. 2025a) (which has been trained for wafer defect classification), and our model demonstrates further improvements, summarized in Table [4] Beyond classification, this framework can be extended to support interactive querying and automated reasoning for Root Cause Analysis, systematically linking defect clusters to process tools, wafer locations, or temporal drifts.
25

---

Published as a conference paper at ICLR 2026
hong-horizon temporal reasoning localization
Audio -visual synchronization understanding
You will be asked multi-choice sionsions. Your replies must contain single lettor (olther B, C, If subtie intensity change the king fields represents 5% adjustment diagnostic confidence for pneumonia, how many such changes occur from to 120 seconds, requining tracking across the entire video duration?
You will be asked multi-choice questions. Your replies must contain single letter (either DI, What structure highlighted by the green circular marker added near the lung area 20-30
Spine
4 adjustments (20% to 10 adjustments (50% total) adjustments (30% total)
C.
Bronchus
Lung nodule
8 ad ustmmntttss (40% total))
Ground
Owen2.5-0mni Ours: B
Owen2 6-Omni:
Durs: D
Anti- shatssttottttingggg
Temporal reasoning
You asked multi-choice questions. replies must contain only a single lettor (either A, R, C, Dj How many bone lesions were identified the thorax that would support diagnosis metastasis?
questions. Your replies contain only a singla letter (either A. B, DJ How do the lung textures in the scan change time, based visual tues?
A. They transition uniform density
A. No lesions
B. They became more homageneous
lesions (+3)
They increasing bright
One icsion
D. They display consistent heteroganaous potterns
D. Two lesions
Ground truth: C Owen2.5-0mni:
Owen2 Owen2.5-0mm
Durs: D
Ouns
Figure 11: Qualitative comparison between Omni'Vinci and Qwen2.5- 5m0 on an omni-modal medical QA task based on radiologist-narrated CT interpretation videos. We organize the evaluation into four categories of questions: long-horizon temporal reasoning and localization, audio-visua synchronization and understanding, shortuutoouu.,,g and temporal reasoning.
User: This image of wafer map, the yellow pattern in the circle refers to the defect pattern. There are possible types defect wafer map (1) loc. (2) edge-loc. (3) center. edge (5) scratch. (6) near-full, donut. random. What type of anomaly does the provided image present?
edge-loc
edge-ring
Figure 12: Illustration of wafer robust defect analysis task for smart factory agent.
C.5.2 FACTORY AND INDUSTRIAL TIME SERIES UNDERSTANDING
We apply Omni Vinci to Statistical Process Control (SPC) chart recognition, a representative task in industrial quality monitoring and root cause analysis. Our model recognizes a wide range of fault categories, including out-of-control points such as spikes or drops, persistent runs and monotonic trends such as level shifts up or down, cyclic oscillations, mixture random fluctuations, as well as missing values or short outages, as illustrated in Figure 13 On a held-out test set, our model achieves 87% accuracy, showing that by transforming time- series signals into visual representations, we can effectively leverage large-scale vision-language pretraining for sensor monitoring and industrial diagnostics. This demonstrates the feasibility of deploying our framework in real manufacturing pipelines, where timely detection of process abnormalities is crucial for preventing defects and reducing downtime.
We assess our framework time series classification tasks using datasets from the UCR archive (Dau et al. 2018), where time series are transformed into line plots to exploit large-scale vision- pretraining. Our first comparison is against VLM-TSC (Prithyani et al., (2255), LLaVa-based VLM that adopts a similar conversion strategy. As shown Table 15] our approach achieves superior performance the PenDigits and Iaayno datasets.
26

---

Published as a conference paper at ICLR 2026
Table 14: Comparison of VILA, NVILA, and OmniVinci on wafer defect classification.
VILA
Lin
2024b)
NVILA Liu al 2025a
OmniVinni (ours)
Parameters
40B
9B
Resolution
336x
x48
448 x-44
Model size
75 GB
16 GB
18 GB
Accuracy
90.8%
97.6%
98.1 %
User: these images belong to? The possible classes cluster, constant, eycling, missing. period_ tr trending, petoiic_atttrmss, shift, trending, uneven.
por
Answer: period_trendingg
Answer: missing
Figure 13: Illustration of SPC chart recognition for industrial fault detection.
D METHOD DETAILS
D.1 OMNI-MODAL INPUT EMBEDDING
Image. Similar to NVILA (Liu et al. 2025a). we start with pretrained the SigLip (Zhai al. 2023 vision encoder and augment it with x "Spatial Scale-Thent "Thhe--Copress" Dynamic S2 (Liu et al. 2025a Shi al. 2024) to accomondate for multi-scale and high resolution images. Given input image of varying dimensions, the overall encoding module adapts the largest scale to the nearest tile-aligned size divisible by 448 and maintains the aspect ratio. Feature maps from all scales are aligned to this largest scale and concatenated, processed by a 2-layer MLP for projection into a latent space shared by embeddings of different modalities.
Audio. We adopt a single audio encoding pipeline for both speech and non-speech audio. Raw audio waveforms sampled 16 kHz and converted into audio frames using the Short-Time Fourier Transform (STFT). These frames are then processed by the Audio Flamingo (Goel et al.) 2025) audio encoder to extract acoustic features in both speech and natural sound. The encoder consists of convolutional layers followed by transformers, enabling it to capture both local and global audio patterns. The extracted features are subsequently projected into the modality-share latent space using a 2-layer MLP.
Video. Videos contain two modalities introduced above. namely vision and audio. For the vision stream, the video frames are temporally sampled uniformly to reduce redundancy and computational load. Each frame processed individually through the lboove-mentioee image input pipeline, and the resulting features are aggregated temporally. We then utlize temporal pooling the feature sequence to further compress visual information. For audio stream, we extract features with the same Audio modality pipeline mentioned above. Meanwhile, we extract the timestamps for each visual and audio embeddings to act as temporal guidance on interleaved token arrangement as explained later.
Prompt. For text prompts, employ standard text encoder (Qwen 2024), which first tokenizes the input into discrete tokens and then maps them into a continuous semantic embedding space via an embedding layer. This space is shared with embeddings from other modalities. For speech prompts, we use the previously described audio encoder to generate the corresponding continuous semantic embeddings. Finally, the resulting prompt embeddings are concatenated with the visual and audio embeddings introduced earlier.
Model version "pa" geema-sessss sigli 1--s0400--patch
27

---

Published as a conference paper at ICLR 2026
Table 15: Performance comparison of test accuracy (%) on selected UCR datasets (Dau al. 2018
Dataset Info
Acc.
Dataset
Type
Length
Train
Text
Class
VLM-IIC Prithyynn et
2025
Ours
PenDigits
MOTION SENSOR
7494
3498
10
85.08
96.88
Poeeyon
1029
2
95.00
D.2 MORE DISCUSSION ON CONSTRAINED ROTARY TIME EMBEDDING (CRTE)
The base frequency in CRTE, wi is designed to have a geometric progression frequencies. For small values of (e.g., the first pairs of dimensions), the denominator is smaller, resulting in higher frequencies (w; is large). These dimensions undergo rapid rotation with respect to time. Consequently, they are highly sensitive fine-grained temporal differences and are effective distinguishing between timestamps that are close to one another. For large values of (e.g.. the last pairs of dimensions). the term gi/d becomes significantly larger. resulting in lower frequencies (w, is small). These dimensions rotate slowly, making them suitable for encoding coarse, long-range temporal relationships. They provide a stable signal for large time intervals without the issue aliasing or "wrapping around" that would occur with high-frequency signals. By partitioning the embedding space into spectrum of frequencies, the model can concurrently attend to both local and global temporal contexts. This multi-scale approach provides robust and comprehensive representation of absolute time.
D.3
MODALITY-SPECIHC TRAINING
D.3.1 VISION TRAINING
The modality-specifet vision training aims to train the model with visual understanding ability. We follow NVILA (Liu et al. 2025a) training recipe including five stages;
Stage 1 Vision Projector Alignment. This stage learns to project visual information through a visual projector. This stage ensures that the visual embeddings compatible with the language model's 's token embeddings, which is essential for smooth downstream integration. The model is trained image-text pairs with simple captioning-style supervision, setting a baseline understanding of visual semantics. Only the vision projector is tuned during this process.
Stage 2 Vision Encoder Alignment. With the projector aligned, the model now focuses enhancing the vision encoder's capacity to process diverse visual content. In this stage we train only the vision encoder and visual projector.
Stage Vision Pre-Traie Fre-Training. During this core stage, the model is trained on large-scale multimodal data to learn how interpret and generate image descriptions. The vision encoder is kept frozen, while the vision projector and the LLM are fine-tuned.
Stage 4 Image Instruction Tuning, In this stage the model is fine-tuned with vision instruction- following capabilities. It is trained to answer multimodal questions, generate captions, reason over scenes, interpret documents, and more. Training data covers a broad range of multimodal capabilities. It includes high-quality instructional examples to align the model with human preferences, datasets for generating rich image captions, and tasks that develop logical and visual reasoning skills. The model is also trained to interpret documents and embedded text, answer general and knowledge- visual questions, and handle diagrams, visual dialogues, and multimodal instructions. In this stage, all model parameters are fine-tuned.
Stage 5 Video Instruction Tuning, In the final vision alignment stage, the model is adapted to video understanding. The goal here is to enable temporal reasoning and visual understanding over sequences of frames. This includes tasks such as activity recognition, multi-frame object tracking, and answering time-sensitive questions. The whole model is fine-tuned.
Through this vision alignment process, we obtain the "vision preliminary checkpoint" with well- trained vision encoder, projector, and language model.
28

---

Published as a conference paper at ICLR 2026
D.3.2 AUDIO TRAINING
Starting from the language model in the above vision preliminary checkpoint we next train the audio understanding ability of our model, which involves (i) audio projector and encoder alignment step followed by (ii) audio instruction tuning.
Stage 1 Audio Projector& & Encoder Alignment. This phase focuses on aligning audio encoder and its associated compression layer. We keep the parameters of the language model and vision side fixed. Training consumes 50K audio-language pairs curated from public datasets spanning across audio-based (music, non-speech sound, and speech) question answering, speech-to-text captioning. and automatic speech recognition. By training on this heterogeneous dataset, we encourage the audio projection module to learn a unified representation that aligns well with the language model's semantic space.
Stage Audio Instruction Tuning. During the second stage of training, the audio encoder, audio projection module, and language model are fine-tuned in a unified, end-to-end manner. This joint optimization allows the system to develop a comprehensive and deeply integrated understanding of audio. This stage consumes comprehensive audio-SFT SFT dataset overseeing 9.6 million samples, including but not limited to audio- -baedd question answering (AudioEntailmentAA (Deshmukh et al.) 2025). Clotho-AQA (Lipping et al. 2022). DCASE-2025-train (Yang et al.) (2025b), etc.), audio captioning (AudioCaps (Kim et al. 2019), Clotho-v2 (Drossos et al. (020), Miradata (Ju et al. 2024)- recaptioned, etc.). speech emotion recognition (CREMA-D (Cao et al. 2014). IEMOCAP (Busso et al. 2008), MELD (Poria et al., 20188. etc.), automatic speech recognition (CV-ASR (Ardila et al. 2020 Europarl-ASR (Koehn 2005). LibriSpeech-ASE (Panayotov et al. 2015). etc.), and speech translation (MuST-C (Di Gangi al., 2019), Emilia (He et al. 2024), etc.). This allows the model to learn both low-level acoustic features and high-level semantic representations, enabling robust generalization across multiple audio understanding tasks and versatile capabilities in interpreting complex auditory inputs. At this point, we find that the model's ability to perform visual understanding tasks is worse, which motivates us to pursue the subsequent omni-moda joint training.
D.4 OMNI-MODAL JOINT TRAINING DETAILS
We adopt a cosine learning rate schedule, preceded by linear warm-up up phase over the first 3% of the training data. The base learning rate is set to 2 x 10-5, During training, the vision and audio encoders are kept frozen. The total token count is approximately 200 billion.
D.5 EXTRA DETAILS OF TRAINING DATA
This section describes the comprehensive multi- modaiitt training data used for developing the pro- posed omni-modal LLM, which are designed to handle diverse types of audio, visual, and textual information. Our training corpus encompasses a wide range of modalities including speech recog- nition, audio question answering, audio captioning, audio classification, video question answering, and image understanding tasks. The dataset is carefully curated to provide robust coverage across multiple domains, enabling the model to develop strong cross-modal understanding and reasoning capabilities.
There are 3.6 million omni-modal conversations, 8 million image-text conversations, 2.7 million video- text conversations, 5.3M speech-text conversations, and 4.3 million speech-text conversations. Omni- modal data contributes 15%, consisting of omni question answering (12%) and omni captioning (3%). Image data constitutes the largest share at 36%, with notable subcategories including general image tasks (19%), ), knowledge-based tasks (8%). and document processing (7%). Sound (non-speech) data accounts for 21%, predominantly driven by audio question answering (20%). Speech data represents 17% of the total, primarily comprising automatic speech recognition (10%), audio question answering (4%), and speech translation (2%). Video data forms the remaining 11% entirely attributed to video question answering. The training data consists of approximately 24 million samples distributed across three main categories: Speech, Sound, and Image/Video. The Speech category includes datasets for automatic speech recognition (ASR). speech translation, and emotion classification. featuring well-established corpora such AMI (Carletta, 2007). Common Voice (Ardila al.) 2020), and LibriSpeech (Panayotov et al. 2013). The Sound category encompasses audio question answering datasets like MMAUQA (Goel et al. 2025) and CompA-R-AQA (Ghosh al. 2024).
29

---

Published as a conference paper at ICLR 2026
Table 16: MMAU audio benchmark.
Music
Sound
Speech
Avg
Model
Test
Test-mini
Test
Test-mini
Test
Test-mini
Test
Test-miol
Gemini 2.5 Pro
68.26
64.77
70.63
75.08
72.67
71.47
71.60
69.36
Gemini 2.5 Flush
76.58
69.40
65.57
69.30
71.80
68.27
69.57
67.39
Kimi-Aodio
63.93
66.77
70.70
$6.57
64.40
68.
Phi-4-multimdaal
Audio Flamingo
62.40
GPT-4o Audio
$6.29
63.20
64.56
66.67
60.82
62.50
Owen2-Audio-nnstru.
55.26
$5.67
56.29
61.17
55.37
57.40
59.60
Gemma 3n 4B
61.26
53.20
56.89
50.27
58,00
62.13
58.00
55,20
Qwen2.5-Co Ommii
67.33
65,90
76.77
78.10
68.90
70.60
71.00
71.50
OmniVinnl
73.07
73.65
73.57
78.68
68.17
66.97
71.60
73.10
Damo ca
CLEVR
3%
CLEVI Mart
Cauldron 7%
LNQA 11%
Knowledge 31%
9%
LVIS-Inst4V 8%
General 28%
aava-Instructe 150k 5%
Captions 6%
ShurughrA 2%
Spatal 6%
Document 26%
TextCaps %
GQA 5%
MMC-inst 13%
Hae took
PlotQA 6%
Figure 14: Data distribution of our synthetic
multimodal conversation.
audio captioning datasets such as Clotho-v2 (Drossos et al. 2020). and various audio classification datasets including UrbanSound8K (Mesaros al. 2018) and FSD50k (Fonseca et al., 202T). The image/Video category includes datasets for visual question answering, document understanding, and general image comprehension tasks.
E
MORE EXPERIMENTS AND DISCUSSION
E.
AUDIO ENCODING
Audio Encoder Backbone. To investigate the choice of audio representations for the omni-modal
model, we evaluate two state-of-the-art audio encoders: Qwen2-Audio (Chu et al. 2023) used by Owen2.5-Omni (Xu et al. 2025), and the AF-Whisper backbone (Goel et al. (0255) from Audio Flamingo GGol et al. 2025). This comparative analysis enables us to identify the backbone that provides the most effective encoding for downstream multimodal tasks. Specifically,
ablate these
key components by aligning them with the LLM backbone model we used in audio-only training.
We 10% of the audio/speech training data fairly evaluate the effectiveness of the two encoders
30

---

Published as a conference paper at ICLR 2026
under the same data budget. As shown in Table 17, AF-Whisper consistently outperforms the Qwen-2 Audio encoder backbone on audio and speech understanding tasks. Therefore, our final model architecture adopts the AF- AF-Whisper backbone to extract informative audio features.
Table 17: Ablation study on different Audio Encoder backbones.
Andio Encoder
LS-elean
LS-other
MMAU-mini
MMAU
Qwen2-Audio
3.5
61.3
59.0
AF-Whisper chosen
70.5
63.3
Audio Token Compression. For the AF- AF-Whisper encoder, similar Whisper-large-v3 (Radford et al. (2023), the process begins by resampling the audio to 16 kHz sampling rate, followed by transforming the raw waveform into a 128-channel mel-spectrogram using a 25 analysis window and 10 ms hop interval (i.e., a hop length of 160). This yields 3,000 audio frames for 30-second audio, which are then processed through convolutional layers and a transformer model to extract audio features, resulting in 750 sequential audio feature vectors. Therefore, each second of audio is roughly represented by 25 tokens. While this may not seem like for a 30-second audio, encoding one hour of audio would require about 90,000 tokens, which could overwhelm the context length of multimodal models.
We next explore several audio information compression strategies to improve efficiency in representing audio information. In our ablation study, we fine-tune the preliminary checkpoint before large-scale training 2.6M audio-only dataset, referring this configuration the Baseline. We then evaluate two audio feature compression methods: (i) Applying 1-D convolution with kernel size and stride before audio projector, or (ii) Applying average max pooling with kernel size 2 before audio projector. We assess performance on audio understanding benchmarks, including Librispeech, Gigaspeech, VoxPopuli, and Long Audio Bench (Goel al. 2025) and present results in Table T8] We also report the embedding per minute of input audio and the average end-to-end latency of the LLM forward pass on Long Audio Bench for each variant in the table.
Table 18: Downsampling method comparison for audio token compression in OmniVinci. For Librispeech, Gigaspeech, and VoxPopuli we report WER (lower is better). For Long Audio Bench report accuracy (higher is better) and latency (lower is better). Gains are computed relative the baseline (All audio tokens).
Emb/min (1)
Librispeech-dl.
Librispeech-oth. WER (L)
Gigaspeech
VoxPopulI-ASR WER (4)
Long Audio
Model
WER ())
WER (
Acc. (f)
Lat. (1)
Baseline- - audio tokens
750
4.49
10.77
5.89
1.78
Audio Compression
Conv stride 2
375
2,10
5.22
11.01
6.25
41.79kosi
1.4,am
Avg. pooling
375
1996
4.75
10.85
6.24
42.16
1.41aam
Max pooling- chosen
375
1.93.m
4.99
10.78.g.
6.17
43.15
1.40
We observe several advantages via compression. Halving audio tokens leads to significantly shorter latency, from 1.78 sec/sample to 1.40 sec/sample (+17.7% improvement). For the long audio under- standing task, applying audio token downsampling improves the accuracy by 2% as it compresses information into more condense representative embeddings, alleviates the burden on L.MMs when handling large volumes of audio embeddings, For short-form benchmarks, we study varying down- sampling options, where we observe max pooling maintains performance across benchmarks without minimal accuracy degradations.
E.2 MODEL QUANTIZATION AND EFFICIENT DEPLOYMENT
Although OmniVinci demonstrates strong omni-modal performance, real-world deployment quickly encounters multiple constraints. Large models or long video sequences often exceed device memory capacity, while interactive applications demand extremely low latency. To meet these challenges, we compress the model via quantization and optimize the system for speedup. A detailed analysis of the inference pipeline reveals distinct bottlenecks: the vision and audio towers are dominated by dense matrix multiplications, processing large batches of tokens in parallel and thus primarily computation-bon bound:; in contrast, the LLM decoding stage-where where the model consumes and generates one token at time- is memory-bandwidee limited and becomes the key latency bottleneck in
31

---

Published as a conference paper at ICLR 2026
20
2.72x
Time-to-FirsttTTakee (ims)
600
Decade Latency (ms)
1.78x
400
10
200
GOM
16 Qwen2. n-omni
32
64
Frames
8
16
32
64
Fromes
Ours
w/ Audio
Owen2.5-omn
Ours
w/ Audio
Figure Latency comparison between Qwen2.5-Omni and our OmniVinci model on GeForce RTX 4090 GPU. Our model achieves 1.7 x faster time-to- to-tirtttttoenn latency and 2.72x faster decoding latency.
long-context scenarios. To address this, we adopt component-aware quantization strategy. For the vision and audio towers, we apply W8A8 quantization, reducing arithmetic cost while preserving representational quality. For the LLM, we employ W4A16 quantization, compressing weights into bits while retaining 16-bit computation, which accelerates muniiidth-lmmitdd decoding. Finally, to
recover accuracy, we integrate Activation-Awarr Weight Quantization (AWQ) (Lin al. 2024a and SmoothQuant (Xiao al. 2023).
We measure the time-to to-first-token latency and decoding latency on a single GeForce RTX 4090 GPU using video clips ranging from 2 to 32 seconds (at 2 frames per second), and compare the performance against Qwen2.5-Omme
in Figure T5 Overall, these quantization methods allow a 8B model to handle videos of up to 64 frames 24GB RTX 4090 GPU, while achieving 1.7x lower time-to- toee-to-irstttoeen latency and 2.72 x faster decoding latency, For 16-frame video with audio stream, our model needs only around 160ms produce the first token.
E.3 OMNIVINCI WITH ASR TEST-TIME SCALING METHODS
To push the limit of transcription accuracy, investigate our model's ability to leverage pre- trained ASR models in downstream speech understanding tasks. In cascaded post-ASR processing setup (Yang al., 2023) as shown in Figure [6 (a), speech inputs are first transcribed by the model's ASR module and then processed by LLM based generative ASR error correction. We use a popular from SimulStreaming as the cascaded ASR module.
800M streaming variant Whisper-v3-
The results are also shown in Table 19 The cascaded pipeline yields additional improvements on ASR tasks, making particularly beneficial in offline transcription scenarios. We use Phi-4-mm-instruet 5-shot (Abouclenin et al. 2025) speech modeling setup one test-time baseline. For Qwen2.5-0mmi experiment, we follow the official inference scrip for the evaluation reported in the fourth row of Table 19 with the original results shown in the third row. From the extended test-time scaling results, Omni Vinci-cascaded improves average WER from 6.3 to 5.7. The Omni -AAG setup yields further improvement, reducing average WER from 6.3 to 5.0 with the same model size of ASR parallel cascading by using ASR text as index for Omni Vinci on mutlimodal ASR correction (Lin
al. 2025b). We introduce the retriever training details of this setup in the following section.
Vinci with ASR based Retriever-Augmente Training.
As shown in Figure 16 (b), given primary acoustic input, A, our objective is generate final, high-fidelity textual output Trnal (either a transcription for ASR or translation for ST). The model has access to two streams textual information:
1. Internal Hypothesis (Tinternat:: first-pass generation produced by the omni-modal model itself, conditioned solely on the acoustic input A. This represents the model's direct, audio-grounded interpretation.
We follow the official ASR cookbook in ma htt.bboookkamiieessssaaaaaaiaiie /
/bl ob
ng. ipynb and related discussion in httpa on the Omni settings used in evaluation.
32

---

Published as a conference paper at ICLR 2026
Table 19: Speech Recognition WER (%) comparison of different models on speech recognition datasets.
(
Model
LSccoo
LSutee
AMI
Tedlium
Voxpopuli
Avg.
Phi-4-MM Phi-4-Me 4-MM shols)
1.7
11.5
2.9
5.9
5.2
Qwen2.5-ommi: reported Xu et al.
2025
1.8
3,4
5.8
Qwen2.5. oommi:: repraduced
2.1
3.8
17.8
5.2
6.1
7.0
OmniVVnni
1.7
3.7
16.1
3.4
6.8
6.3
OmmiVinci-cascala
1.6
3.0
14.1
3.3
6,5
5.7
OmaiVinci
Vinci-RAG RAG
1.5
3.0
11.6
3.0
5.7
5.0
Audio
Streaming ASR Model
text
OmniVinci
transcription
Audio (a) OmniVinci-Cascaded: serial connection from ASR to Omni model
Audio
Streaming
transcription
ASR Model
Special token <need_ref> <no_ref>
Audio
OmniVinci
transcription
(b) OmniVinci-RAGe Parallel connection between ASR and Omni models using retrieval token
Figure 16: We illustrate two test-time scaling methods using an extra ASR model: (a) Omni Vinci- Cascaded, using ASR history as an additional input to the Omni model with the audio inputs, and (b) Omni Vinci-RGG, using the retrieval token for prediction. The related results reported in Table 19]
2. External References (H)): A set of candidate transcriptions, H. f2,,12.... fe................... hN generated by one or more external systems. This set represents external, text-only evidence that may contain valuable corrections introduce noise.
The task is formulated as conditional generation problem that jointly models the final output and decision variable, d. The model learns to generate control token indicating its strategy, followed by the refined text. This decision hinges the model's ability to align A, Timernat. and 76 to determine the most reliable path to the ground truth, Tg--
E.3.1 INSTRUCTIONAL FORMATTING FOR
DECISION MAKING
To facilitate this crrision-mainng process, we structure the input as comprehensive instruction that forces the model to weigh different sources of evidence. The model presented with modalities and explicitly prompted to declare its generation strategy.
Task: Ferform reference- veference-augmentee augmented correction for given speech input.
Objective: Evaluate the quality of an internally generated hypothesis against external candidates. First, select generation strategy by producing control taken. Then, generate the final, corrected text. the internal hypothesis well-aligned with the audio, select <accept_internal If the external candidates provide necessary corrections, select sirterrate_referrnnee
deemed superior and
33

---

Published as a conference paper at ICLR 2026
Acoustic Evidence: [AUDIO]
External Candidate Transcriptions:
1. (h_1}
2. (h_2)
3. (h_3)
(h_4)
5. (b_5)
Internal Hypothesis:
(T [[iintrnaa)
Output:
The model is then trained to generate the complete target string, beginning with either daccept_interpe or centegrate__eferenneee followed by the corrected and final- ized text. We expand the model's vocabulary with these two special tokens serve as explicit control signals.
E.3.2 SUPERVISION FOR DECISION-AWARF PINE-TUNING
Supervision for this decision-aware fine-tume tne-tunnng is derived by comparing the internal hypothesis (Timerma)) against the ground truth (Tgi) and the external references (H). The decision label is deter- mined as follows:
<accept internal>: This label is assigned when the word error rate (WER) of Tineraal is below predefined threshold or when the external references in H offer no improvement or introduce hallucinations. This teaches the model to trust its own cross-modal alignment between audio and text when its confidence is high.
reference>: Letterrate_refeeencee:: This label is assigned when Tinermal contains correctable errors and at least one hypothesis in H. provides information that reduces the WER relative to Tg.. This trains the model to identify valuable external information and integrate it, effectively re-aligning its understanding based on supplementary textual evidence.
The final training target is the concatenation of the assigned decision token and the ground-truth transcript Tg--
E.3.3 INFERENCE-TIME CONTROL FL.OW
At inference, the hypothesis, and the external references. The first token generated by the model dictates the subsequent control flow:
model processes the aulti-source input containing the audio, its internal
If the model generates pecccept_interpee 1 it signals high confidence in its audio- to-text mapping. For the final output, we can simply use its pre-computed internal hypothesis, Tintemal- allow the model regenerate it.
If the model generates <integrate wwweraatteeeeeeeeeeeeeeeeeeeeeee it indicates that the external textual evidence is necessary for achieving a better output. The full sequence generated by the model following this token is taken the final, corrected transcript.
This mechanism provides an interpretable and controllable framework for test-time adaptation, allowing the model to dynamically adjust its reliance on external knowledge based on the specific challenges of each input. This is critical for robust performance in both ASR, where the focus is on transcription fidelity, and ST, where correct semantic understanding grounded in both audio and reference text is paramount for accurate translation. Table presents the performance of this method, denoted as OmniVinei-RAa. Vinei-RAG. substantially improves the model's results across all speech recognition benchmarks.
E.4 SPEECH OUTPUT
Rather than training a speech generation model from the ground up. we leverage saitee-f-thheeee pre-trained text-to-speech (TTS) systems produce speech in relevant scenarios, and adapt
34

---

Published as a conference paper at ICLR 2026
approach using speech codec when needed. Our evaluation focuses on English omni- moni-modal-in and voice-out, using two complementary metrics: mean opinion score (MOS; higher indicates greater naturalness) and TTS word enror rate (WER; lower indicates higher intelligibilityy,, the latter measured through an external ASR system. As reported in Table 20 existing off-the-shelf models already yield high-quality, neutral speech suitable for assistant-style applications. Among the back ends tested, Omni Vinci-Magpie achieves the best overall balance (MOS 4.63, WER 2.7%). followed closely by gpt-4o-mini-tts (MOS 4.59, WER 3.1%) and Qwen-omni (MOS 4.53, WER 3.2%). Omni Sinci--taabeeodee delivers a competitive WER (2.9%) but with slightly reduced naturalness (MOS 4. 12), highlighting that intelligibility and perceived naturalness are not always aligned. In contrast, Bark underperforms on both measures (MOS 3.32, WER 8.2%), consistent with its more stochastic synthesis approach.
Setup. We evaluate prompt following on VoiceBench style/control splits and conversational control tasks. We compare three prompting strategies over interleaved audio-vision contexts: (i) Tran- script prompting (ASR-Hext) [aud, vis|x3 text-prompt, (ii) Native audio prompting (encoder fea- tures): [aud, vis] x3 + aud-prompt, (iii) 7TS-injected prompting (render text to speech, then encode); [aud, vis] x3 + (Txx(te--pprmmp) mid [aud, vis]. aud-prompp],, [aud, vis] x2, and suffi [aud, vis]*3 (aud-prompt].
We also ablate prompt position: prefix [aud-prompt] + [aud, vis]
Metrics. We report (a) Prompt Adherence Rate (PAR; judged by paired preference and rubric scoring), (b) slot accuracy for constrained commands (names, numerals, entities), and (c) latency/efficiency (no additional ASR pass). For speech rendering quality, MOS/WER results summarized in Table 20
Key Insight (1) Native audio prompting is the most robust accents, background noise, and overlapped speech; it preserves prosodic cues (rate, emphasis) that pure transcripts dis- card, leading to higher PAR and slot accuracy noisy and accented conditions. (2) Transcript prompting is competitive on clean speech but degrades when ASR struggles named entities or code-switched fragments. (3) TTS-injected prompting reduces acoustic mismatch in far-field sce- narios and is effective when a consistent house voice is desired, but it transfers less speaker/style information than using the raw prompt audio. (4) Prompt suffix placement-immediaeee immediately before the model's response consistently outperforms prefix and mid insertion, likely due reduced long-range interference in the attention context.
Encoding the audio prompt directly (no external ASR) yields the best prompt following under realistic noise/accents while lowering latency and memory by avoiding an extra ASR pass. Suffix-position audio prompts provide the strongest control.
Table 20: English naturalness MOS (higher is better) and TTS word error rate (WER; lower is better). Best per column in bold.
Setup
Regime
MOS 1
WER (5E)
Qwen-Ornni
suto-regressive
4.53
GPT-4o
4.59
3.1
VnniVincii Cozy Voice
agentic cascuded
4.54
3.0
OmniVinci Bark
agentic cascaded
3.32
8.2
Vinnc--SabloCooeee
auto-regrossive
4.12
2.9
Vinci-Magpie (chosen)
agentic cascaded
Beyond raw scores, we observe consistent performance across synthesis regimes. Agentic cascaded setups that decouple text planning from acoustic rendering tend to produce strong MOS and low WER in our pipeline, while rego-segres models are competitive but show greater variance. Importantly, swapping the TTS back end does not alter OmniVinee language understanding or response planning; it only affects the surface realization of speech, simplifying deployment-time time customization (e.g.. voice, rate).
For interactive agents, streaming synthesis and low perceived latency are crucial. Our chosen back ends support incremental generation, enabling prompt first-audio while the remainder of the utterance is synthesized. In production, we prioritize (i) stability numerals, abbreviations, and named
35

---

Published as a conference paper at ICLR 2026
entities, (ii) speaker consistency across turns, and (iii) graceful handling of punctuation and prosody cues from text.
E.5 ANALYSIS: WHY DO WE NEED OMNI-MODAL LLMs?
Humans perceive the world through vision and audio simultaneously and rely on both modalities to perform many tasks. While certain academic benchmarks for visual audio understanding may require only single modality, we believe that developing omni-modal models is critically important and represents the right long-term direction. As noted by Liang et al. (2023), different modalities can exhibit synergy, effectively supporting one another. |Mdaan et al. (2024) show that multimodal learners often outperform their unimodal counterparts. Similarly, |Wang et al. (2024a) argue that modalities benefit each other by helping the model triangulate shared underlying reality.
Regarding whether performance improves or degrades benchmarks after our omni-modal training, we observe substantial gains in video understanding (as shown in Table 2 VideoMME w/o subtitles 61.67->63.70 +2.09) and comparable performance on most image (Table and audio tasks (Table relative to single-modality baselines, without increasing model size to accommodate these additional modalities. We expect that scaling the model to three times its current size would yield further performance improvements various benchmarks of different modalities.
Furthermore, we evaluate the performance of the model checkpoint following modality-specific training and compare it with OmniVind after omni-modal joint-tuning, As shown in Table 21 omni-modality joint tuning enhances performance across most image and audio tasks.
Table 21: Comparison of model performance between Podality-Speciffe Training and Omni Vinci (After Omni-Modality Tuning) across various benchmarks.
Benchmark
VQ4r2
RWQ4
SRED Imuge
MMMU
ChariQ4
A120
Math
BarVQ4
VidroMME.
Omni
Daily
sense
Modility Specide Trinise
66.71
73.97
77.02
$2.6
88.56 92.9
36.64 66.50
44.20 48.23
After Chen- Olmn-Medalryy Modaley Tunning
67.5
77.1
49A7
84.6
63.
68.18
46.47
3.7
E.6 EVALUATION ON MORE VIDEO BENCHMARKS
Table 22: TOMATO Benchmark Results (Shangguan et al. 2024
Category Scores
Model
All
Model Rolation
Direction
Velocity & Frequency
Shape & Trend
Visual
Count
Cues
VideoL.LaMA 2 70
Cheng et al.
2024h
10.10
22.
15.70
18,80
31.40
19.50
18.50
Phi 3.5 Vision
Abdin et al
2024
20.30
16.60
14.30
23.30
40.00
24.70
20.70
InterVLL 8B
Chon
2024b
17.10
25.10
09.00
28.70
31.40
22.90
21.70
LLaVA-NeXI=
Vid
Zhang et
2024a
20.60
26.30
12.40
24.20
30.00
24.30
22.70
InterVLL 2 26B
Chen
2024h
18.50
29.30
10.50
31.40
11.40
25.70
23.30
VideoLLaMA
728 Chong
2024b
14.30
24.60
22.40
26.50
27.10
28.80
23.50
17.90
34.30
20.90
Video
Video-Only Only
//ang
38.60
Vinci (ours)
21.30
20.80
09.50
21.10
42.90
40.80
24.30
We further evaluate Omni Vinci on two additional benchmarks: TOMATO (Shangguan et al. 2024).
which emphasizes video temporal reasoning, and Omni VideoBench (Li et al. (002a)), which focuses on visual-audio synergistic understanding, including long-term temporal reasoning. The quantitative results for both benchmarks are presented T2bles[2 and 23 On TOMATO, Omni achieves competitive performance across diverse temporal reasoning categories, notably excelling in Visual Cues and Count. On Omni VideoBench, Omni Vinci also establishes new state-of-the-art overall score among 7B-scale models, demonstrating robust capability across different video duration ranges. These results collectively highlight Omni OmniVinci's strong generalization ability in complex temporal and multimodal video understanding tasks.
36

---

Published as a conference paper at ICLR 2026
Table 23: Beni Benchmark Results
(Li et
2025a
Model
LLM Params
[0.1] min
(1,5)
(5.10) min
(10,30) min
Overall
Lad2 Cheng ci al.
2024b
7B
32.0
28.23
29.6
28.29
29.2
Qwen2.5-Omni (78) (Xu
2025
7B
41.57
27.41
25.33
26.72
29.3
Qwea225-VL (72B) Bai a
2025
72B
33.13
30.03
31.88
24.43
29.5
MiniCPM-o Yao 2024
31.43
28.49
34.53
26.15
29.7
Qwen2.5-VI. (7B) (Hai
201.5)
25.93
30.03
31.88
30.15
29.8
HumanOmmiiV2 V2
36.57
29.36
29.62
2.0-Flash DeepMind
35.86
32.75
Qwen2.5-VL. (32B) Bai
2025
38.55
31.22
29.26
30.53
VmniVinnci (surs)
38.55
34.11
30.13
27.16
32.1
E.7 TEXT UNDERSTANDING EVALUATION
The primary goal of this work is omni-modal understanding across video, audio, and images, with text-only tasks falling outside its scope. For reference, we evaluate our model on two widely adopted text understanding benchmarks: MMLU (Hendrycks et al., 2021) and MMLU-Pro (Wang et al. 2024dh, Table 24 These tasks assess a model's ability to reason over a broad range of academic and professional subjects. We compare system against the closely related Qwen2.5-0mmi (Xu et al. 2025). Our model outperforms Qwen2.5-Omi on both benchmarks, with particularly notable improvement on the challenging MMLU-Pro dataset, highlighting its enhanced text understanding capabilities.
Table 24: Comparison of model performance on MMLU and MMLU-Pro benchmarks.
Model
MMLU
MMLU-Pro
Qwen2.5
OnniVVnnii
E.8 LOss SUMMARIZATION
We use a weighted sum of the OmniAlignNet loss and cross-entropy loss, defined by the equation:
Lfinal = Lee + aLoo-align-
We determined the optimal value of a 0.01 via an ablation study, the results of which are presented
in Table 25
Table 25: Ablation study results for different values of a.
Omnibbnnh
Dailyomni
Worldseuse
0.1
44.26
65.75
45.48
0.01
45.74
65.83
46.21
0.001
40.12
65.67
45.90
F.9 ABLATION STUDY OF TEG GROUP DURATION To
The group duration TG of Temporal Embeding Grouping (TEG) controls the granularity of the grouping. To examine its impact on model performance, we conduct a series experiments with varying group durations, shown in Figure 17 Empirically, find that duration of 30 seconds yields the best overall performance. We hypothesize that small To may break the semantic coherence between tokens.
37

---

Published as a conference paper at ICLR 2026
Worksddderss
Dailyornii
Omnibench
64.-
33.8
00
Score
11.2
25.8
62.6
364
Boudine
Bonsine
TEG dis
Te0 Mi
mo0
Bocdine
necu
Methoy
Method
Methoo
Figure 17: Performance comparison using different TEG group duration TG. We find that duration of 30 seconds yields the best overall performance.
38


Reviews:
[{"review_id": "lJPB4hWkn4", "reviewer": "Reviewer_hewe", "summary": "The paper introduces OmniVinci with three techniques including i) OmniAlignNet to align vision and audio in a video ii) Temporal embedding grouping for capturing the temporal alignment between vision and audio and iii) Constrained rotary time embedding for adding temporal information into vision-audio embeddings. The experiments show improvement on various multimodal understanding, audio understanding and vision understanding tasks over prior work.", "questions": "Please refer to my comments above.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper tackles an important goal of building open-source, omni-modal LLM by incorporating OmniAlignNet, temporal embedding grouping and constrained rotary time embedding. The paper curated a large-scale dataset spanning audio, video and image domains and shows improvements to multimodal understanding, audio understanding and vision understanding tasks over prior work, which would be of interest to the community. Overall, the proposed method is simple and paper is easy to read and well-written.", "weaknesses": "* The distinction of OmniAlignNet module, use of position encoding from the current video-audio alignment common in existing work [1,2,3] is unclear. The paper lacks a discussion with these works making its positioning among them unclear.\n* Similarly there exists many studies [4,5] that have incorporated the temporal sequence in multiple ways, which the paper lacks a comparison or distinction with.\n* The paper argues the need of Omnimodal data engine and gives an example of where both audio and video are required. But as shown in many prior multimodal studies [6,7,8], there exists many datasets where one modality suffices and thus explicitly enforcing interactions is suboptimal and often leads to unnecessary correlations. The paper lacks any discussion in this aspect as well. \n* The paper highlights modality-specific training in section 3.1 by using data for each modality but it is unclear how this is incorporated in the omni-modal joint training and more details need to be provided on the separation of the modality-specific and omni-modal training. \n* The claims of the paper are not well supported by the empirical results. For example, i) While OmniVinci only improves the performance on Dailyomni in Table 3, its worse than almost all models on Omnibench with upto 10% worse than Qwen. The performance on Worldsense is also not convincing without confidence intervals. Similar conclusions hold for Image benchmarks in Table 7 and speech recognition benchmarks in Table 5, where OmniVinci obtains worse performance across baselines. \n\nOverall, in the current state I recommend recommend rejection due to the lack of discussion with prior work in multiple aspects and claims not being supported by empirical results.\n\nThe following can improve the paper further:\n* A common trend for multimodal models is the lack of temporal reasoning. It would be useful to see the performance of the proposed method on cases [9,10] which are explicitly designed to evaluate the same. \n* The font size and presentation for most results is extremely small, which makes it challenging to interpret the results meaningfully.\n* The position of Table 5 and Table 6 can be switched.\n\nReferences:   \n[1] Cheng et al. MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis.     \n[2] Kim et al. Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Video.    \n[3] Guo et al. Aligned Better, Listen Better for Audio-Visual Large Language Models.  \n[4] Zerveas et al.  A transformer-based framework for multivariate time series representation learning.    \n[5] Eldele et al. TSLANet: Rethinking Transformers for Time Series Representation Learning.  \n[6] Liang et al. Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework.  \n[7] Madaan et al. Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning.   \n[8] Wang et al. An Information Criterion for Controlled Disentanglement of Multimodal Data.  \n[9] Shangguan et al. TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models.   \n[10] Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QedgswHAuc", "reviewer": "Reviewer_Tky5", "summary": "This paper introduces a suite of modules to enhance video–audio multimodal understanding, including OmniAlignNet, Temporal Embedding Grouping (TEG), and Constrained Rotary Time Embedding (CRTE).\nOmniAlignNet is proposed to align video and audio latent representations. TEG and CRTE are designed to improve the temporal alignment of video and audio features, thereby facilitating model learning.\nAblation studies validate the effectiveness of each component. The proposed method outperforms Qwen2.5-Omni on several video and audio understanding benchmarks.", "questions": "Since the 24M multimodal model is part of the contributions, is there a plan to open-source it?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Experiments on dataset engines highlight the importance of fully exploiting cross-modal information in sounding videos, which benefits both clean data construction and model performance. These findings offer meaningful insights for future research.\n- The proposed modules improve model performance from two key perspectives—video–audio semantic alignment and temporal alignment—both of which are well-motivated and demonstrated to be effective for video and audio understanding tasks.", "weaknesses": "- Although the authors claim the model is omni-modal, the work primarily focuses on video and audio, leading to degraded performance on the image modality. Furthermore, results for text-to-text tasks are not reported. Combined with the claim that the model uses far more tokens than Qwen2.5-Omni, it raises concerns that the proposed approach may neglect text and image modalities.\n- TEG involves interleaving video and audio tokens when feeding the LLM, which is similar to the approach used in Qwen2.5-Omni. Likewise, contrastive learning is a common practice for aligning video and audio semantic features.\n- Comparisons with Qwen2.5-Omni in terms of audio generation latency, training cost, and total parameter count are not reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dt9UBn5vJq", "reviewer": "Reviewer_xmSv", "summary": "This paper introduces OmniVinci, an omni-modal Large Language Model (LLM) designed for comprehensive cross-modal understanding by jointly processing vision, audio, and language. The authors present notable architectural contributions, including OmniAlignNet for vision-audio alignment, Temporal Embedding Grouping (TEG) for structured temporal representation of modality tokens, and Constrained Rotary Time Embedding (CRTE) for encoding absolute temporal information. They also curated and synthesized a diverse, large-scale dataset comprising 24 million conversations spanning both single- and multi-modal scenarios. Commendably, the work includes demonstrations of practical applications in real-world settings.", "questions": "same as weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The work proposes the innovative multi-modal integration architecture, OmniAlignNet, which aligns image and video dimensions within a unified feature space. The introduction of TEG and CRTE further enhances modality feature learning, substantially boosting the model's overall omni-modal understanding performance.\n\n- The work construct a substantial dataset of 24 million samples and implemented multi-modal reasoning augmentation. The workflow illustrat in Figure 4 provides a clear mechanism for handling modality-specific hallucinations and generating high-quality cross-modal supervision.\n\n- The paper includes a relatively comprehensive set of evaluations and ablation studies, effectively demonstrating the model's capabilities. Furthermore, the work showcases initial deployment and application potential in real-world scenarios.", "weaknesses": "- The study employs a paradigm where single modalities (image, audio) are trained separately before a unified modal alignment is performed. The paper lacks a performance comparison of modality-specific tasks before and after the cross-modal unified alignment. Reporting the respective performances on pure Image and Audio tasks before and after this alignment stage would significantly help validate the effectiveness of the proposed method.\n\n- The image data significantly outweighs the audio data during pre-training. I wonder what the ratio of image-to-audio modality tokens is during the cross-modal alignment phase. Moreover, if the compression ratio for both the vision and audio encoders is identical during alignment,  how to eliminate potential bias resulting from the inherent token quantity imbalance between these modalities.\n\n- Some reported model results are not reflective of the current state-of-the-art. It would be beneficial to use more recent, cutting-edge model results for comparison (e.g., updating InternVL-2 to InternVL-3 and Qwen2-vl to Qwen3-vl) to ensure the novelty claims are appropriately contextualized against the strongest contemporaries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b79ODxwqcU", "reviewer": "Reviewer_uWWz", "summary": "This paper introduces OmniVinci, an open-source omni-modal LLM. The core contributions are twofold: (1) A new model architecture featuring three specific mechanisms (OmniAlignNet, TEG, and CRTE) to improve semantic and temporal alignment between vision and audio. (2) A novel data curation pipeline, the \"Omni-Modal Data Engine,\" designed to mitigate \"modality-specific hallucination\" by using an LLM to correct and fuse single-modality captions. The authors demonstrate that OmniVinci achieves SOTA performance, notably outperforming Qwen2.5-Omni with 6x fewer training tokens.", "questions": "- On the Combination of Loss Functions: The OmniAlignNet module introduces a contrastive loss, $L_{o-align}$ (Eq 1)20, while the LLM backbone uses a standard generative cross-entropy loss (implied in Figure 2). How are these (and potentially other) losses combined during the omni-modal joint training phase? Are they simply summed, or are they weighted (e.g., $L_{total} = \\alpha L_{LLM} + \\beta L_{o-align}$)? If weighted, how were these hyperparameters determined?\n- Regarding Modality Conflict at Inference: The data engine is designed to resolve modality-specific hallucination during training . How does the final OmniVinci model handle new, explicit modality conflicts at inference time? For example, if the model is fed a video showing a dog but the audio narration says, \"this is a cat,\" how does the model prioritize or fuse these contradictory signals?\n- Clarification on 6x Training Efficiency: The 6x reduction in training tokens (0.2T vs. 1.2T)  is a very impressive efficiency claim. To fully contextualize this: (a) Does the 0.2T token count include the computational cost of running the \"Omni-Modal Data Engine\" to generate the 24M samples? (b) What was the total computational cost (e.g., total GPU-hours) and wall-clock time required for the omni-modal joint training phase, compared to the baseline?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Novel Alignment System: The paper introduces a novel system for aligning information across text, audio, and video. The proposed architectural modules (OmniAlignNet, TEG, CRTE) are well-motivated and supported by extensive ablation studies (e.g., Table 1).\n- Innovative Data Curation: The authors build an \"Omni-Modal Data Engine\"  to address the common and difficult problem of \"modality-specific hallucination.\" They provide valuable insights (in Figure 4) into the limitations of single-modality captioning .\n- State-of-the-Art Performance: The model delivers SOTA results on numerous industry benchmarks for audio, video, and omni-modal tasks among models of similar scale (including Qwen2.5-Omni). Significant improvements are shown on key tests like DailyOmni (+19.05) and Video-MME (+3.9).\n- High Training Efficiency: The model achieves this strong performance while being highly efficient, using only 0.2T training tokens compared to Qwen2.5-Omni's 1.2T", "weaknesses": "- Limited Gains from RL Post-Training: The performance improvement from the GRPO reinforcement learning (RL) post-training appears relatively modest. As shown in Table 8, the score improvements on Worldsense, Dailyomni, and Omnibench are all less than 1 percentage point. Given the complexity and cost of RL training, does this minor gain suggest a bottleneck in the current method or data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)