How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal
  Models with Open-Source Suites
Authors: name: Zhe Chen; name: Weiyun Wang; name: Hao Tian; name: Shenglong Ye; name: Zhangwei Gao; name: Erfei Cui; name: Wenwen Tong; name: Kongzhi Hu; name: Jiapeng Luo; name: Zheng Ma; name: Ji Ma; name: Jiaqi Wang; name: Xiaoyi Dong; name: Hang Yan; name: Hewei Guo; name: Conghui He; name: Zhenjiang Jin; name: Chao Xu; name: Bin Wang; name: Xingjian Wei; name: Wei Li; name: Wenjian Zhang; name: Lewei Lu; name: Xizhou Zhu; name: Tong Lu; name: Dahua Lin; name: Yu Qiao
Published: 2024-04-25 17:59:19

Abstract: In this report, we introduce InternVL 1.5, an open-source multimodal large
language model (MLLM) to bridge the capability gap between open-source and
proprietary commercial models in multimodal understanding. We introduce three
simple improvements: (1) Strong Vision Encoder: we explored a continuous
learning strategy for the large-scale vision foundation model -- InternViT-6B,
boosting its visual understanding capabilities, and making it can be
transferred and reused in different LLMs. (2) Dynamic High-Resolution: we
divide images into tiles ranging from 1 to 40 of 448times448 pixels
according to the aspect ratio and resolution of the input images, which
supports up to 4K resolution input. (3) High-Quality Bilingual Dataset: we
carefully collected a high-quality bilingual dataset that covers common scenes,
document images, and annotated them with English and Chinese question-answer
pairs, significantly enhancing performance in OCR- and Chinese-related tasks.
We evaluate InternVL 1.5 through a series of benchmarks and comparative
studies. Compared to both open-source and proprietary models, InternVL 1.5
shows competitive performance, achieving state-of-the-art results in 8 of 18
benchmarks. Code has been released at https://github.com/OpenGVLab/InternVL.