OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attention
Authors: name: Zhangquan Chen, name: Jiale Tao, name: Ruihuang Li, name: Yihao Hu, name: Ruitao Chen, name: Zhantao Yang, name: Xinlei Yu, name: Haodong Jing, name: Manyuan Zhang, name: Shuai Shao, name: Biao Wang, name: Qinglin Lu, name: Ruqi Huang
Organization: Tencent

While humans perceive the world through diverse modalities that operate synergistically to support a holistic understanding of their surroundings, existing omnivideo models still face substantial challenges on audio-visual understanding tasks. In this paper, we propose OmniVideo-R1, a novel reinforced framework that improves mixed-modality reasoning. OmniVideo-R1 empowers models to "think with omnimodal cues" by two key strategies: (1) query-intensive grounding based on self-supervised learning paradigms; and (2) modality-attentive fusion built upon contrastive learning paradigms. Extensive experiments on multiple benchmarks demonstrate that OmniVideo-R1 consistently outperforms strong baselines, highlighting its effectiveness and robust generalization capabilities.