InternVL3: Exploring Advanced Training and Test-Time Recipes for
  Open-Source Multimodal Models
Authors: name: Jinguo Zhu; name: Weiyun Wang; name: Zhe Chen; name: Zhaoyang Liu; name: Shenglong Ye; name: Lixin Gu; name: Yuchen Duan; name: Hao Tian; name: Weijie Su; name: Jie Shao; name: Zhangwei Gao; name: Erfei Cui; name: Yue Cao; name: Yangzhou Liu; name: Weiye Xu; name: Hao Li; name: Jiahao Wang; name: Han Lv; name: Dengnian Chen; name: Songze Li; name: Yinan He; name: Tan Jiang; name: Jiapeng Luo; name: Yi Wang; name: Conghui He; name: Botian Shi; name: Xingcheng Zhang; name: Wenqi Shao; name: Junjun He; name: Yingtong Xiong; name: Wenwen Qu; name: Peng Sun; name: Penglong Jiao; name: Lijun Wu; name: Kaipeng Zhang; name: Huipeng Deng; name: Jiaye Ge; name: Kai Chen; name: Limin Wang; name: Min Dou; name: Lewei Lu; name: Xizhou Zhu; name: Tong Lu; name: Dahua Lin; name: Yu Qiao; name: Jifeng Dai; name: Wenhai Wang
Published: 2025-04-14 17:59:25

Abstract: We introduce InternVL3, a significant advancement in the InternVL series
featuring a native multimodal pre-training paradigm. Rather than adapting a
text-only large language model (LLM) into a multimodal large language model
(MLLM) that supports visual inputs, InternVL3 jointly acquires multimodal and
linguistic capabilities from both diverse multimodal data and pure-text corpora
during a single pre-training stage. This unified training paradigm effectively
addresses the complexities and alignment challenges commonly encountered in
conventional post-hoc training pipelines for MLLMs. To further improve
performance and scalability, InternVL3 incorporates variable visual position
encoding (V2PE) to support extended multimodal contexts, employs advanced
post-training techniques such as supervised fine-tuning (SFT) and mixed
preference optimization (MPO), and adopts test-time scaling strategies
alongside an optimized training infrastructure. Extensive empirical evaluations
demonstrate that InternVL3 delivers superior performance across a wide range of
multi-modal tasks. In particular, InternVL3-78B achieves a score of 72.2 on the
MMMU benchmark, setting a new state-of-the-art among open-source MLLMs. Its
capabilities remain highly competitive with leading proprietary models,
including ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro, while also
maintaining strong pure-language proficiency. In pursuit of open-science
principles, we will publicly release both the training data and model weights
to foster further research and development in next-generation MLLMs.