MiniCPM-V: A GPT-4V Level MLLM on Your Phone
Authors: name: Yuan Yao, name: Tianyu Yu, name: Ao Zhang, name: Chongyi Wang, name: Junbo Cui, name: Hongji Zhu, name: Tianchi Cai, name: Haoyu Li, name: Weilin Zhao, name: Zhihui He, name: Qianyu Chen, name: Huarong Zhou, name: Zhensheng Zou, name: Haoye Zhang, name: Shengding Hu, name: Zhi Zheng, name: Jie Zhou, name: Jie Cai, name: Xu Han, name: Guoyang Zeng, name: Dahai Li, name: Zhiyuan Liu, name: Maosong Sun

The recent surge of Multimodal Large Language Models (MLLMs) has
fundamentally reshaped the landscape of AI research and industry, shedding
light on a promising path toward the next AI milestone. However, significant
challenges remain preventing MLLMs from being practical in real-world
applications. The most notable challenge comes from the huge cost of running an
MLLM with a massive number of parameters and extensive computation. As a
result, most MLLMs need to be deployed on high-performing cloud servers, which
greatly limits their application scopes such as mobile, offline,
energy-sensitive, and privacy-protective scenarios. In this work, we present
MiniCPM-V, a series of efficient MLLMs deployable on end-side devices. By
integrating the latest MLLM techniques in architecture, pretraining and
alignment, the latest MiniCPM-Llama3-V 2.5 has several notable features: (1)
Strong performance, outperforming GPT-4V-1106, Gemini Pro and Claude 3 on
OpenCompass, a comprehensive evaluation over 11 popular benchmarks, (2) strong
OCR capability and 1.8M pixel high-resolution image perception at any aspect
ratio, (3) trustworthy behavior with low hallucination rates, (4) multilingual
support for 30+ languages, and (5) efficient deployment on mobile phones. More
importantly, MiniCPM-V can be viewed as a representative example of a promising
trend: The model sizes for achieving usable (e.g., GPT-4V) level performance
are rapidly decreasing, along with the fast growth of end-side computation
capacity. This jointly shows that GPT-4V level MLLMs deployed on end devices
are becoming increasingly possible, unlocking a wider spectrum of real-world AI
applications in the near future.