MiniCPM-V: A GPT-4V Level MLLM on Your Phone
Authors: name: Yuan Yao; name: Tianyu Yu; name: Ao Zhang; name: Chongyi Wang; name: Junbo Cui; name: Hongji Zhu; name: Tianchi Cai; name: Haoyu Li; name: Weilin Zhao; name: Zhihui He; name: Qianyu Chen; name: Huarong Zhou; name: Zhensheng Zou; name: Haoye Zhang; name: Shengding Hu; name: Zhi Zheng; name: Jie Zhou; name: Jie Cai; name: Xu Han; name: Guoyang Zeng; name: Dahai Li; name: Zhiyuan Liu; name: Maosong Sun
Published: 2024-08-03 15:02:21

Abstract: The recent surge of Multimodal Large Language Models (MLLMs) has
fundamentally reshaped the landscape of AI research and industry, shedding
light on a promising path toward the next AI milestone. However, significant
challenges remain preventing MLLMs from being practical in real-world
applications. The most notable challenge comes from the huge cost of running an
MLLM with a massive number of parameters and extensive computation. As a
result, most MLLMs need to be deployed on high-performing cloud servers, which
greatly limits their application scopes such as mobile, offline,
energy-sensitive, and privacy-protective scenarios. In this work, we present
MiniCPM-V, a series of efficient MLLMs deployable on end-side devices. By
integrating the latest MLLM techniques in architecture, pretraining and
alignment, the latest MiniCPM-Llama3-V 2.5 has several notable features: (1)
Strong performance, outperforming GPT-4V-1106, Gemini Pro and Claude 3 on
OpenCompass, a comprehensive evaluation over 11 popular benchmarks, (2) strong
OCR capability and 1.8M pixel high-resolution image perception at any aspect
ratio, (3) trustworthy behavior with low hallucination rates, (4) multilingual
support for 30+ languages, and (5) efficient deployment on mobile phones. More
importantly, MiniCPM-V can be viewed as a representative example of a promising
trend: The model sizes for achieving usable (e.g., GPT-4V) level performance
are rapidly decreasing, along with the fast growth of end-side computation
capacity. This jointly shows that GPT-4V level MLLMs deployed on end devices
are becoming increasingly possible, unlocking a wider spectrum of real-world AI
applications in the near future.