Seeing and Understanding: Bridging Vision with Chemical Knowledge Via
  ChemVLM
Authors: name: Junxian Li; name: Di Zhang; name: Xunzhi Wang; name: Zeying Hao; name: Jingdi Lei; name: Qian Tan; name: Cai Zhou; name: Wei Liu; name: Weiyun Wang; name: Zhe Chen; name: Wenhai Wang; name: Wei Li; name: Shufei Zhang; name: Mao Su; name: Wanli Ouyang; name: Yuqiang Li; name: Dongzhan Zhou
Published: 2024-08-14 01:16:40

Abstract: In this technical report, we propose ChemVLM, the first open-source
multimodal large language model dedicated to the fields of chemistry, designed
to address the incompatibility between chemical image understanding and text
analysis. Built upon the VIT-MLP-LLM architecture, we leverage ChemLLM-20B as
the foundational large model, endowing our model with robust capabilities in
understanding and utilizing chemical text knowledge. Additionally, we employ
InternVIT-6B as a powerful image encoder. We have curated high-quality data
from the chemical domain, including molecules, reaction formulas, and chemistry
examination data, and compiled these into a bilingual multimodal
question-answering dataset. We test the performance of our model on multiple
open-source benchmarks and three custom evaluation sets. Experimental results
demonstrate that our model achieves excellent performance, securing
state-of-the-art results in five out of six involved tasks. Our model can be
found at https://huggingface.co/AI4Chem/ChemVLM-26B.