OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding
  LLM
Authors: name: Hanrong Ye; name: Chao-Han Huck Yang; name: Arushi Goel; name: Wei Huang; name: Ligeng Zhu; name: Yuanhang Su; name: Sean Lin; name: An-Chieh Cheng; name: Zhen Wan; name: Jinchuan Tian; name: Yuming Lou; name: Dong Yang; name: Zhijian Liu; name: Yukang Chen; name: Ambrish Dantrey; name: Ehsan Jahangiri; name: Sreyan Ghosh; name: Daguang Xu; name: Ehsan Hosseini-Asl; name: Danial Mohseni Taheri; name: Vidya Murali; name: Sifei Liu; name: Jason Lu; name: Oluwatobi Olabiyi; name: Frank Wang; name: Rafael Valle; name: Bryan Catanzaro; name: Andrew Tao; name: Song Han; name: Jan Kautz; name: Hongxu Yin; name: Pavlo Molchanov
Published: 2025-10-17 17:59:59

Abstract: Advancing machine intelligence requires developing the ability to perceive
across multiple modalities, much as humans sense the world. We introduce
OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We
carefully study the design choices across model architecture and data curation.
For model architecture, we present three key innovations: (i) OmniAlignNet for
strengthening alignment between vision and audio embeddings in a shared
omni-modal latent space; (ii) Temporal Embedding Grouping for capturing
relative temporal alignment between vision and audio signals; and (iii)
Constrained Rotary Time Embedding for encoding absolute temporal information in
omni-modal embeddings. We introduce a curation and synthesis pipeline that
generates 24M single-modal and omni-modal conversations. We find that
modalities reinforce one another in both perception and reasoning. Our model,
OmniVinci, outperforms Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal
understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while
using just 0.2T training tokens - a 6 times reduction compared to
Qwen2.5-Omni's 1.2T. We finally demonstrate omni-modal advantages in downstream
applications spanning robotics, medical AI, and smart factory.