Seeing and Understanding: Bridging Vision with Chemical Knowledge Via
  ChemVLM
Authors: name: Junxian Li, name: Di Zhang, name: Xunzhi Wang, name: Zeying Hao, name: Jingdi Lei, name: Qian Tan, name: Cai Zhou, name: Wei Liu, name: Weiyun Wang, name: Zhe Chen, name: Wenhai Wang, name: Wei Li, name: Shufei Zhang, name: Mao Su, name: Wanli Ouyang, name: Yuqiang Li, name: Dongzhan Zhou

In this technical report, we propose ChemVLM, the first open-source
multimodal large language model dedicated to the fields of chemistry, designed
to address the incompatibility between chemical image understanding and text
analysis. Built upon the VIT-MLP-LLM architecture, we leverage ChemLLM-20B as
the foundational large model, endowing our model with robust capabilities in
understanding and utilizing chemical text knowledge. Additionally, we employ
InternVIT-6B as a powerful image encoder. We have curated high-quality data
from the chemical domain, including molecules, reaction formulas, and chemistry
examination data, and compiled these into a bilingual multimodal
question-answering dataset. We test the performance of our model on multiple
open-source benchmarks and three custom evaluation sets. Experimental results
demonstrate that our model achieves excellent performance, securing
state-of-the-art results in five out of six involved tasks. Our model can be
found at https://huggingface.co/AI4Chem/ChemVLM-26B.