LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
Authors: name: Yifan Dai, name: Zhenhua Wu, name: Bohan Zeng, name: Daili Hua, name: Jialing Liu, name: Bozhou Li, name: Yuran Wang, name: Chengzhuo Tong, name: Hao Liang, name: Xiaochen Ma, name: Junbo Niu, name: Tianyu Guo, name: Yang Shi, name: Yue Ding, name: Yiyan Ji, name: Bingyin Mei, name: Yushuo Guan, name: Yuanxing Zhang, name: Pengfei Wan, name: Fangcheng Fu, name: Wentao Zhang
Organization: Kling Team

Joint audio-visual reasoning is essential for omnimodal understanding, yet current multimodal large language models (MLLMs) still struggle when reasoning requires fine-grained evidence from both modalities. A central limitation is that explicit text-based chain-of-thought (CoT) compresses continuous audio-visual signals into discrete tokens, weakening temporal grounding and shifting intermediate reasoning toward language priors. We argue that a unified latent space is a better medium for such reasoning because it preserves dense sensory information while remaining compatible with autoregressive generation. Based on this insight, we propose LatentOmni, a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states. LatentOmni introduces feature-level supervision to align latent reasoning states with task-relevant sensory features and uses Omni-Sync Position Embedding (OSPE) to maintain temporal consistency between latent audio and visual states. We further construct LatentOmni-Instruct-35K, a dataset of audio-visual interleaved reasoning trajectories for supervising latent-space reasoning. Comprehensive evaluation across multiple audio-visual reasoning benchmarks demonstrates that LatentOmni achieves the best performance among the evaluated open-source models and consistently outperforms the Explicit Text CoT baseline, supporting latent-space joint reasoning as a promising path toward stronger omnimodal understanding.