InternVL3: Exploring Advanced Training and Test-Time Recipes for
  Open-Source Multimodal Models
Authors: name: Jinguo Zhu, name: Weiyun Wang, name: Zhe Chen, name: Zhaoyang Liu, name: Shenglong Ye, name: Lixin Gu, name: Yuchen Duan, name: Hao Tian, name: Weijie Su, name: Jie Shao, name: Zhangwei Gao, name: Erfei Cui, name: Yue Cao, name: Yangzhou Liu, name: Weiye Xu, name: Hao Li, name: Jiahao Wang, name: Han Lv, name: Dengnian Chen, name: Songze Li, name: Yinan He, name: Tan Jiang, name: Jiapeng Luo, name: Yi Wang, name: Conghui He, name: Botian Shi, name: Xingcheng Zhang, name: Wenqi Shao, name: Junjun He, name: Yingtong Xiong, name: Wenwen Qu, name: Peng Sun, name: Penglong Jiao, name: Lijun Wu, name: Kaipeng Zhang, name: Huipeng Deng, name: Jiaye Ge, name: Kai Chen, name: Limin Wang, name: Min Dou, name: Lewei Lu, name: Xizhou Zhu, name: Tong Lu, name: Dahua Lin, name: Yu Qiao, name: Jifeng Dai, name: Wenhai Wang

We introduce InternVL3, a significant advancement in the InternVL series
featuring a native multimodal pre-training paradigm. Rather than adapting a
text-only large language model (LLM) into a multimodal large language model
(MLLM) that supports visual inputs, InternVL3 jointly acquires multimodal and
linguistic capabilities from both diverse multimodal data and pure-text corpora
during a single pre-training stage. This unified training paradigm effectively
addresses the complexities and alignment challenges commonly encountered in
conventional post-hoc training pipelines for MLLMs. To further improve
performance and scalability, InternVL3 incorporates variable visual position
encoding (V2PE) to support extended multimodal contexts, employs advanced
post-training techniques such as supervised fine-tuning (SFT) and mixed
preference optimization (MPO), and adopts test-time scaling strategies
alongside an optimized training infrastructure. Extensive empirical evaluations
demonstrate that InternVL3 delivers superior performance across a wide range of
multi-modal tasks. In particular, InternVL3-78B achieves a score of 72.2 on the
MMMU benchmark, setting a new state-of-the-art among open-source MLLMs. Its
capabilities remain highly competitive with leading proprietary models,
including ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro, while also
maintaining strong pure-language proficiency. In pursuit of open-science
principles, we will publicly release both the training data and model weights
to foster further research and development in next-generation MLLMs.