Towards 3D Molecule-Text Interpretation in Language Models
Authors: name: Sihang Li, name: Zhiyuan Liu, name: Yanchen Luo, name: Xiang Wang, name: Xiangnan He, name: Kenji Kawaguchi, name: Tat-Seng Chua, name: Qi Tian

Language Models (LMs) have greatly influenced diverse domains. However, their
inherent limitation in comprehending 3D molecular structures has considerably
constrained their potential in the biomolecular domain. To bridge this gap, we
focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular
Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze
3D molecules by equipping the LM with a 3D molecular encoder. This integration
is achieved by a 3D molecule-text projector, bridging the 3D molecular
encoder's representation space and the LM's input space. Moreover, to enhance
3D-MoLM's ability of cross-modal molecular understanding and instruction
following, we meticulously curated a 3D molecule-centric instruction tuning
dataset -- 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric
instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder
and LM. It significantly surpasses existing baselines on downstream tasks,
including molecule-text retrieval, molecule captioning, and more challenging
open-text molecular QA tasks, especially focusing on 3D-dependent properties.