Towards 3D Molecule-Text Interpretation in Language Models
Authors: Sihang Li
Zhiyuan Liu
Yanchen Luo
Xiang Wang
Xiangnan He
Kenji Kawaguchi
Tat-Seng Chua
Qi Tian
Venue: ICLR 2024 poster (2024)
Keywords: 3D molecules
Large Language Model
3D-text interpretation
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Language Models (LMs) have greatly influenced diverse domains. However, their inherent limitation in comprehending 3D molecular structures has considerably constrained their potential in the biomolecular domain. To bridge this gap, we focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze 3D molecules by equipping the LM with a 3D molecular encoder. This integration is achieved by a 3D molecule-text projector, bridging the 3D molecular encoder’s representation space and the LM’s input space. Moreover, to enhance 3DMoLM’s ability of cross-modal molecular understanding and instruction following, we meticulously curated a 3D molecule-centric instruction tuning dataset – 3D-MoIT. Through 3D molecule-text alignment and 3D molecule-centric instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder and LM. It significantly surpasses existing baselines on downstream tasks, including moleculetext retrieval, molecule captioning, and more challenging open-text molecular QA tasks, especially focusing on 3D-dependent properties. We will release our codes and datasets at https://github.com/lsh0520/3D-MoLM.

Published as a conference paper at ICLR 2024
3D-MOLM: TOWARDS 3D -OLECULE-TEXT INTER- PRETATION IN LANGUAGE MODELS
Sihang Li
Zhiyuan L
Yanchen Luo
Xiang Wang!
Xiangnan Hel3t
Kenji Kawaguchi²
Tat-Seng Chua?
Qi Tian
University of Science and Technology China 2 National University Singapore
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, USTC "Inttitute of Dataspace, Hefei Comprchensive National Science Center
Huawei Cloud
ABSTRACT
Language Models (LMs) have greatly influenced diverse domains. However, their inherent limitation in comprehending 3D molecular structures has considerably constrained their potential in the biomolecular domain. To bridge this gap, we focus 3D molecule-text interpretation. and propose 3D-MoLM: 3D- Moleccull Language Modeling. Specifically, 3D-MoL. enables an LM to interpret and analyze 3D molecules by equipping the LM with 3D molecular encoder. This integration is achieved by a 3D molecule-text projector, bridging the 3D molecular encoder's representation space and the LM's input space. Moreover, enhance 3D- MoLM's ability of cross cross-modal molecular understanding and instruction following, we meticulously curated a 3D molecule-centee instruction tuning dataset 3D- MoIT. Through 3D molecule-tex. alignment and 3D molecule-centri instruction tuning, 3D-MoLMM establishes an integration of 3D molecular encoder and I.M. It significantly surpasses existing baselines on downstream tasks, including molecule- text retrieval, molecule captioning, and more challenging open-text molecular QA tasks, especially focusing 3D-dependent properties. We release our codes and datasets at https: / / thub. com https/shh/00///// MMMMM...
INTRODUCTION
The advancement of Language Models (LMs) (Devlin et al., 2019; OpenAI, 2023b; Touvron et al., 2023a) has triggered a series remarkable innovations across multiple disciplines (Zhao al., 2023). Notably, LMs excel text-based molecule understanding tasks, such as question-answering (QA) in the chemical and medical domains (Taylor et al., 2022), by pretraining on extensive biochemical literature. Recognizing the potential of LMs in harnessing extensive biochemical knowledge for molecule-relevant tasks, molecule-tes text modeling emerges as a new research direction (Edwards et al., 2021; 2022). Previous works have been dedicated harmonizing texts with 1D molecular sequences (Zeng et al., 2022; Taylor et al., 2022) and 2D molecular graphs (Liu et al., 2023b; Su et al., 2022: Liu et al., 2022a), aiding in tasks like molecule-text retrieval and molecule captioning. However, they mostly leave 3D molecular structures untouched, which are crucial to understanding molecular dynamics, protein-ligand interactions, enzymatic functions, and range of other biomolccular phenomena (Karplus & McCammon, 2002; Jorgensen, 2004).
To bridge this gap. we focus on molecule-tex interpretation, with the goal enabling an LM to interpret and analyze 3D molecular structures through text generation. Given the recent successes of 3D molecular encoders in tasks like molecule property prediction, docking, and conformation prediction (Zhou et al., 2023; Lu et al., 2023; Fang et al., 2022), it is promising to incorporate one as an LM's perception module for 3D molecules. Upon examination of existing literature (Dai et al., 2023; Hong et al., 2023; Chung et al., 2022), identify two key challenges to scamlessly integrate 3D molecular encoder into an LM for 3D molecule-text interpretation:
3D Molecule- Text Alignment maps 3D molecular representations into the input textual space where the LM can understand.
"qqual contribution.
"oorrsponndece Xiang Wang and Xiangnan He. {xiangwang|233 1223, xiangnanhe) @ gmail.com

---

Published as a conference paper at ICLR 2024
Molecule-Text Retrieval
3D-MoLM: 3D-Molecular Language Modeling
Molecule Question Answering
Query: This modequle is hydrii- yolnnoni is oiniami....
instuction
Molecular * Encoder (Uni-Mol)
describos lask, palioo with an ippl that provides father condear. Wille a response that apprspnontyy completes the request
Rank:
Instruction: What is the
crscorsoiceor G-00)-010j010
3D Moloucle- Text Projector (Q- (Q-former)
molenule? CCYC/G2H0JC1 -BICmo C
+30 molecuoor embeds>
Molecule Captioning
Response:
This molequle's 's's sructune contains coro,
Digycli
this (C)e)) DC/+CCHG0HG J0 <30 motecular embeds>
Instraction:
Language Model (Liama 2)
*
LoRA
Instruction: Please provide the SCP
wallecale GC1-DCX-- Co---CC---/CC-----0 (00--0------- DC-D ZCCANIC x30 molesster anobedie
Cofteit nott hydrosya-
Respanse: Acalytoaleyyii aoti Apploars
cystel
noamir ardd
inpit moleruie -11152 flew.
Response: energy
Figure I: Demonstration of 3D-MoLM, 3D-MoLM is general purpose molecular LM that can be applied for molecule-text retrieval, molecule captioning, and molecular QA tasks. Flame denotes
general-purpose
tunable modules, while snowflake * indicates frozen modules.
3D Molecule-centric Instruction Tuning fine-tunes the model to follow human instructions 3D molecule relevant tasks.
To address these challenges, we propose 3D-MoLM: 3D-Molecular Language Modeling, depicted in Figure 1. Specifically, it consists of two key components: 1) a 3D molecule-text projector for 3D) molecule-text alignment, which aligns the latent representation spaces between the 3D molccular encoder and the LM, and 2) dataset for 3D Molecule-centric Instruction Tuning, 3D-MolT, as shown in Figure 3. 3D-MolT enhances the model's ability to follow human instructions and discern 3D-dependent properties of molecules.
For 3D molecule-text alignment, we employ Q-Former (Li et al., 2023) as the 3D molecule-text projector, drawing inspiration from leading vision-language modeling methods (Zhu et al., 2023; Dai et al., 2023). Given a molecule's 3D structure, Q-Former converts it into tokens, which serve as 1D) soft prompts (Li & Liang, 2021), harmonizing scamlessly with the language space of the LM. This translation facilitates the LM's interpretation of 3D molecular structures. To cultivate the Q-Former's alignment capability, two training stages are conducted - the first stage focuses on 3D molecule-text representation learning, while the second stage optimizes for 3D molecule-text alignment. As depicted in Figure 3, these two training stages facilitated by collected 316K molecule-text pairs from PubChem (Kim et al., 2021). To promote the 3D molecule-text alignment process, we manipulate the dataset by generating the 3D conformations based on SMILES using RDKit (Landrum et al., 2013) and enriching the molecular descriptions with GPT-3.5 (OpenAI, 2023a). We will detail the collection and enrichment of PubChem Dataset in Section 2.2.1 and Appendix B.
Upon aligning 3D molecules with texts, we conduct instruction tuning using our curated dataset 3D-MoIT. It is designed cultivate 3D-MoLM's ability to follow instructions, and enhance its perception of 3D-dependent molecule properties. Specifically, 3D-MolT is sourced from two databases: 1) PubChem, which offers wide range of molecular properties, origins, and applications. and 2) PubChemQC (Nakata, 2015), which specializes in 3DD-ependent molecular properties. As shown Figure 3, for the PubChem portion, we leverage GPT-3.5 generate QA pairs based on their descriptions. Yet, molecular properties collected from PubChem (e.g., molecular weight and LogP) can be largely inferred from ID or 2D molecular data. To enhance 3D-MolT's perception of 3D molecular structures, we further incorporate data from PubChemQC, which includes 3D-dependent molecule properties (e.g.. HOMO and LUMO; McQuarrie Simon (1997)). We fill these properties into set of text templates, transforming them into instruction tuning formats, as Figure 1 illustrates.
Our contributions can be summarized as follows:
We propose MD-MoLM, a new framework for 3D molecule-text interpretation. 3D-MoLM employs a 3D molecule-text projector to bridge the modality gap between a 3D molecular encoder and an LM, enabling the LM to perceive 3DD molecular structures.
We curate 3D-MolT, 3D molecule-centric instruction tuning dataset. We extract and transform data from PubChem and PubChemQC to an instruction following format, to cultivate 3D-MoL abillty in instruction following and 3D molecule-text interpretation.
3D-MoLM achieves state-of- tfate-of-the-art performances in extensive downstream tasks. Notably, on the PubChem Dataset, for molecule-text retrieval and molecule captioning, it outperforms baselines by
2

---

Published as a conference paper at ICLR 2024
Projector: O-Former
3D Molecule-Tast
Text Matahing
Contrasting
Captioning
3D malecule branch
branch
Text
Next layer
Next
Malecule- Matching
Molecule tranefommr
Nx
Test transformer
Feed-forward
Feed-forward
Self-att
Self-att
3D Molecular Encoder
Cross-att
X
layer
layer
Malecule Text
Contrasting
Sharod Saf-eilantio tioe
Self-att
Self-att
Self-att
Next layer
Next layer
Molscule Captioning
Loamable
Caffeic ackd la
tolkans
hydroay
Self-att
Self-att
L.Q.ommer's archilect.ue.
1.. Self-atiention masking sirwdagy difiewnt tooks.
(a) Stage 1. The 3D molecule-text projector (i.e. Q-Former) with the attached frozen 3D molecule encoder is optimized for 3D molecule text representation learning. Stage involves three training objectives: molecule-tee matching, molecule-text contrasting, and molecule captioning.
Molecular * Encoder (Uni-Mol)
C1=cGi=O/C=C1 C=CC--Ojojaj0
Moleucle-Text 19 Projector (Q-Former)
physical pr- opertiss caffelc acid?
Text Input
Language Model (Llama
LoRA
3D Molecule Input
Text Output
yollow sightl....
(b) Stage & 3. 3D-MoLM is trained to perform 3D molecule-to-terr text generations given 3D molecular tokens (extracted by the Q-foreer) and ID textual prompt tokens.
Figure 2: Illustration of 3D-MoLM's architectures at different stages.
20% accuracy and 6.47 ROUGE-L, respectively. Further, it surpasses the baselines with ID 2D molecular perceptions open-text QA tasks, especially on 3D-dependent properties, verifying the capability of 3D molecule-text interpretation.
2 3D-MOLM: 3D MOLECULAR LANGUAGE MODELING
3D-Mol M incorporates a 3D molecular encoder into an LM, aiming to align 3D molecular geometries with textual concepts and facilitate a comprehensive cross-modal understanding of molecules. Conse- quently, 3D-MoLM is able to read 3D molecular structures, amplifying its molecular understanding and facilitating 3D-text interpretation. Our idea draws from related works in molecule-text modeling, multi-modal instruction tuning, and multi-modal LMs. See Appendix A for comprehensive literature review. Here we delve into 3D-MoLM's architecture and its training pipeline.
2.1 MODEL ARCHITECTURE
3D-MoLM's architecture consists of three key components: 1) 3D molecular encoder, focusing on encoding 3D molecular structures; 2) 3D molecule-text projector, aiming to map the 3D molecular encoder's representations to the input space of the LM; and 3) an LM, which specializes in text generation and is later adapted for understanding 3D molecular structures.
3D Molecular Encoder. We adopt Uni-Mol (Zhou et al., 2023) as our 3D molecular encoder fmol- Specifically, Uni-Mol is pretrained conformations. Formally, let m = (V. h, C) denote a molecule, where V and h separately represent atomic nodes and their features, and C E R collects the 3D coordinates of nodes. Uni-Mol, the representation for each pair of atoms is initialized using invariant spatial positional encoding derived from 3D coordinates C. This encoding, grounded in the pair-wise Euclidean distances between atoms, ensures that the representation remains consistent regardless of global rotations or translations. Subsequently, representations of atoms and atom pairs engage in self-attention mechanism, generating the molecular representation with 3D spatial information. Overall, the 3D molecular encoder fmol performs molecule encoding procedure to obtain the atomic representations:
large molecule dataset comprising 209M 3D molecular
x B2, T = Smol (m).
(1)
where æ; corresponds the representation of the i-th atom.

---

Published as a conference paper at ICLR 2024
Malacuor Eacader
Molecabor Eeceder
Nobeester
Excoder
Projecer
Projecer
Model
LoRA
Proecter
Neck!
Stiee: 1:30 Midscude Toxt Reprovetation Learning
Stage 2: Meloculo Text Generatie Learning
Stage 3:30 Molee cemble
Teak Decoreatr fnoria Dorrcctio parec
ogaan countar balcga qul lolocc
at sau
PeoOheeOG Instrcction Farmet
de rusbll segeeeet la motz The inchary fat
conturt insanine sbuct is placit eructur retulleu natirr/ pruc tnghteed yrum and long olycica vapa net caroog grus aoce
S OFFA4
Rid CONC wOw.OOOOOCCCCOOOO0000 ADD vosesar bean
karacee::
Figure 3: Illustration of the model architectures (upper part) and the dataset usage (bottom part) for the three training stages. PubChem is used for the stage 1 (i.e., 3D molecule-text representation learning) and stage 2 (i.e., 3D molecule-text alignment via generative learning). 3D-MolT is used for 3D molecule-centric instruction tuning. Texts in the same color indicate the same information source.
3D Molecule-Texx Projector. Taking inspiration from the leading vision-language models (Li et al., 2023; Dai et al., 2023), we architect the 3D molecule-text projector fps as a Querying Transformer (i.e.. Q-Former) and initialize it from the Sci-BERT's checkpoint (Beltagy et al., 2019). As illustrated in Figure 2a, Q-Former has two transformers with shared self-attention layers: one molecule transformer for processing 3D molecule features, and one text transformer for processing texts. The text transformer follows the same architecture of BERT (Devlin et al., 2019), while the molecule transformer adds cross-altention modules between the modules of self-attention and feed-forward to extract molecule features. Specifically, the molecule transformer maintains K learnable query tokens. Given 3D molecule input, the query tokens can interact with the 3D molecular encoder's is representations through the cross attention modules. Therefore, the K query tokens' output representations contain molecule information, represented as M [m,, m2, ..., mx). The 3D molecule-text projector's 's forward function can be written as:
M [m1, m2, ..., mx] = fpoo ()..
Language Model (LM). We employ Llama2 (Touvron et al., 2023b) as our base LM fi to leverage its powerful text generation capability and internal chemistry knowledge. Although pretrained for general-purpose usage, the extensive biomedical literature in Llama 2's pretraining corpus enables efficiently interpret ID molecular sequences (e.g., SMILES) and proficiently address essential QA tasks that are relevant to molecular understanding. In this work, we let Llama2 process mixed token sequences that includes both textual tokens and 3D molecular tokens, which is detailed in Section 2.2.1. Formally, we denote a mixed token sequence that include I textual and molecular tokens as Z = [2], Z2. .... z1). Further, the LM adopts a causal mask to generate textual response Ž with length n, where the prediction for the i-th token, 2, is dependent on its previous tokens:
Ż = [ 21, 2t4n), 2 = fi (Zes),
Z [z], z2, .... 21, 21+1: 1/1 (3)
where each 2; is later transformed by a linear layer Swcah accompanied by softmax function, converting into probabilistic distribution over the vocabulary. The final prediction ž; for the i-th token is the word in the vocabulary with the highest probability, defined as:
2 = arg max fvocab ( (&)) [w].
ai Erevabb
2.2 MODEL TRAINING
To tackle the identified two challenges of 3D molecule-text alignment and 3D molecule-centric instruction tuning, we delineate three-stage training pipeline (cf. Figure 3) for 3D-MoLM, including 3D molecule-text representation learning, 2) 3D molecule-texe alignment via gerative learning, and 3) 3D molecule-centric instruction tuning.
2.2.1
3D MOLECULE-TEN. ALIGNMENT
Data Preparation - PubChem. A substantial collection of 3D molecule-text pairs is essential to bridge the gap between the 3D molecular representation space and the LM's input space. We begin
4

---

Published as a conference paper at ICLR 2024
Table 1: Statistics of 3D-MoIT.
property; Desc. ddeseriie property.
PubChhmQCC
PuhChem
Subset
#Mol
#Comp. QA
#Mol
#Comp. QA
#Desc QA
Pretrain
3,119,717
12,478,868
301,658
1,199,066
1,508,290
Train
623,944
2,495,776
12,000
46,680
60,000
Valid
77,993
311,972
1,000
3,898
5,000
Test
77,993
311,972
2,000
7,785
10,000
by collecting molecular SMILES text pairs from PubChem and further employ GPT-3.5 to enrich the less annotated molecular descriptions. Molecular 3D conformations are obtained by running the MMFF algorithm in RDKit (Landrum et al., 2013). As a result, we obtain 316K 3D molecule-text pairs for the alignment pretraining and downstream tasks of molecule-text retrieval and molecule captioning. The details of processing, subset split, and quality evaluation are in Appendix B.
Stage 1: 3D Molecule- Text Representation Learning. In the first stage, we jointly pretrain the Q-Former together with the frozen 3D molecular encoder on the collected 3D molecule-text pairs. Following BLIP-2 (Li et al., 2023), we perform multi-objective training, including molecule- text matching, molecule-text contrasting and molecule captioning (cf. Figure 2a). These training objectives are designed to cultivate Q-Former's ability to extract molecular features that resonate profoundly with the corresponding text. Specifically, the molecule-text matching task mandates the model differentiate between matched and unmatched pairs, enhancing the fine-grained alignment of cross- moosll representations. In molecule-text contrasting, the similarity between molecule and its corresponding text (i.e., positive pair) is contrasted against those of negative pairs, aiming to maximize the mutual information between the molecular and textual representations, In molecule captioning, the Q-Former is trained to generate text descriptions, based on given molecular inputs.
Stage 2: 3D Molecule-Texs Alignment via Generative Learning. In this stage, connect the 3D molecular encoder and the Q-Former with the LM for 3D molecule-to-tee toeeet generation, as illustrated in Figure 2b. The training objective is conditional language modeling: 3D-MoL.M is trained generate textual responses given 3D molecular tokens and 1D) textual prompt tokens. Specifically, the 3D molecular tokens are extracted by the Q-Former, while the ID textual prompt tokens include the molecule's ' SMILES sequence and a textual task description. See Appendix E for ablation studies on prompt templates. This training objective encourages the LM to discern the contextual interplay between textual and molecular tokens, thus aiding in 3D molecule-tex. interpretation. For efficiency consideration, we freeze the 3D molecular encoder and employ LoRA tuning for the LM (Hu et al., 2021). Overall, this stage aligns 3D molecular representations with the LM's textual input space, aiding effective molecule-to-text generation.
2.2.2 3D MOLECULE-CENTRIC INSTRUCTION TUNING
Stage 3: 3D Molecule-centrie Instruction Tuning. In the final stage, freeze the 3D molecular encoder and conduct instruction fine-tun socitunnng to jointly optimize the 3D molecule-text projector and the L.M This fine-tuning has two purposes: 1) to enhance our model's ability to follow various instructions; and 2) to improve our model's understanding of 3D molecular structures, especially in recognizing 3D-dependent properties. It is framed as conditional text generation optimization based on the input prompt and the standard language modeling loss is adopted. Now, we elaborate on the details of our instruction tuning dataset 3D-MoIT.
Data Preparation - 3D-MolT. 3D-MoIT sources data from the PubChem (Kim et al., 2021) and PubChemQC (Nakata, 2015) databases. Specifically, the instruction tuming data from PubChem can be divided into two types: computed molecular properties and descriptive texts, where computed molecular properties are numerical values and descriptive texts characterize molecule properties in free texts. To effectively activate the 3D perception, we further include computed molecular properties from PubChemQC. PubChemQC contains 3D-dependent molecule attributes, which mostly cannot be inferred from ID or 2D molecular representations. PubChemQC also includes DFT-determined 3D conformations, commonly regarded as the ground truth 3D molecular modeling. The statistics of 3D-MolT are shown in Table 1. We detail the dataset construction as follows:
PubChem: Instruction Format. We select the following computed properties: molecular weight, LogP, TPSA, and complexity. They are transformed into instruction tuning format using a pre- defined set of text templates. For descriptive text, adopt GPT-3.5 to read molecular descriptions
5

---

Published as a conference paper at ICLR 2024
Table 2: Molecule-Texx retrieval results on the PubChem Dataset. f denotes pretraining on the original PubChem texts without GPT-3.5 enrichment. We report performances of both using molecule to retrieve text (M2T) and using text to retrieve molecule (T2M).
Retrieval batch
Retrieval
M2T (%)
T2M (%)
M2T (%)
T2M (%)
Model
Ace
R/@20
Ace
R@
Acc
R0
Acc
R.820
ID SMILES
BEiB
85.32
98.74
84,20
98.43
41.67
87.31
40.18
86.77
KV-PLM
86.05
98.63
85.21
98.47
42.80
88,46
41.67
87.80
Graph
MoMu-S
87.58
99.2 24
86.44
99.38
47.29
90.77
48. 13
89.92
MoMu-K
88.23
99.41
87.29
99.42
48.47
91.64
49.46
90.73
MoMu-St
90.43
99.53
99.60
60.51
58.36
91.35
MoMu-KI
90.89
99.67
90.16
99.44
62.07
93.06
59.11
92.01
Conformation
3D-McLMM
99.74
99.34
71.30
95.96
3D MoLM
93.50
100.00
92.89
99.59
69.05
95.91
70.13
94.88
and generate five QA pairs for each molecule, as illustrated in Figure 3. PubChem includes diverse range molecule properties, enhancing the comprchensiveness 3D-MoIT.
Instruction Format. We select the following computed molecular properties: HOMO, LUMO, HOMO-LUMO Gap, and SCF-energy. These properties are transformed into instruction tuning format by filling the values into pre-defined text templates, as illustrated in Figure 3. We use the processed dataset released by (Xu et al., 2021) and follow the scaffold split.
PubChemQC:
EXPERIMENT
In this section, we conduct extensive experiments, including molecule-text retrieval, molccule captioning, and open-text molecular QA tasks, demonstrate the effectiveness of 3D-MoLMM for 3D molecule-text interpretation. See Appendix C' for experimental details of each downstream task.
3.1 MOLECULE-TEX TEXT RETRIEVAL
We assess the Stage-1 I checkpoint of 3D-MoLM on the downstream subsets of PubChem Dataset for molecule-text retrieval. These subsets contain real-world molecules paired with textual descriptions longer than 20 words. We opt not to evaluate existing molecule-text datasets of PCDes (Zeng et al., 2022), because of data leakage. PCDes and curated PubChem dataset stem from the same source (i.e., some PCDes test data present in our pretraining set). We employ Sci-BERT (Beltagy et al., 2019), KV-PLM (Zeng al., 2022), and MoMu (Su al., 2022) as baselines and evaluate the performance by Accuracy and Recall @ 20, both within a batch of 64 samples and across the entire test set. Baselines are initiated from their official checkpoints and finetuned using the downstream partition the PubChem Dataset, except MoMu', which is our re-implementation with the original PubChem texts without GPT-3.5 enrichment. From Table 2, we have the following observations:
3D-MoLM surpasses existing baselines, including both molecular ID-language models (i.e., Sci- BERT, KV-PLM) and 2D-language models (i.e., MoMu-S/K), by a large margin. The improvement can be attributed to two pivotal factors. Firstly, the performance gain 3D-MoLM' over MoMui, which are both pretrained on the same PubChem molecule-text pairs, demonstrates that Q-Former benefits from multi-objective pretraining across diverse granularities, distinguishing it from other models that predominantly rely on the molecule-text contrastive learning objective. Secondly, the scale of the PubChem Dataset, which we curated, offers a considerable advantage, which is verified by the performance gain MoMu over the original MoMu. With the collection of 301K molecule-text pairs for pretraining, it markedly surpasses the 15K pairs in the MoMu w.r.t. scale. This increased scale provides model with richer and more diverse set of training examples, allowing it to better capture the molecular-ter interactions. Such superior performance underscores the effectiveness of 3D molecular representation learning in Stage 1, demonstrating the capability of Q-Former to extract molecular features that are strongly related to the textual deseriptions.
2) The retrieval performance the PubChem test set appears be negatively impacted by GPT-3.5 enrichment. We infer that this decline caused by the enrichment process enlarging the distribution gap between the pretraining and downstream subsets. While the original texts might be succinct and

---

Published as a conference paper at ICLR 2024
Table 3: Molecule captioning results on PubChem Dataset. f denotes pretraining on the original PubChem texts without GPT-3.5 enrichment, Llama2-7B, without molecule-text projector, goes through Stage training with the prompt of 1D SMILES. 2D-MoLM, replaces the 3D molecular encoder with 2D molecular encoder and goes through the same training process as 3D-MoLM.
(a) Molecule captioning results.
Type
Model
BLEU-2
BLEU-
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
ID SMILES
MolT5-Small
15.23
13.45
23.98
MolTS-Base
24.51
16.61
32.19
14.04
21.35
26.10
MollS-Large
25.87
17.28
34.07
16.42
23.41
28.04
Specialist
ID SMILES 2D Graph
MoMu-Small
22.86
16.01
24.35
MoMu Mosu-Base
24.74
16.77
32.45
14.62
22.09
27.16
MoMu-l Large
18.01
16.86
28.73
ID SMILES 3D Conformation
3D-MoLMT
29.82
22.39
37,23
22.49
31.07
32.69
3D-MoLM
30.32
22.52
36.84
22.32
31.23
33.06
ID SMILES
Llama2-7B
27.01
20.94
20.68
28.88
32.11
Generalist
1D SMILES
2D Graph
2D-MoLM
27.15
21.19
36.02
20.76
29.12
32.28
SMILES
3D Conformation
3D MD-MMLMi
29.25
22.07
36.48
21.80
30.95
33.12
MD.MoLM
21.26
33.55
(b) sample of molecule captioning.
Molecule
Ground Trush
3D-MoLM
Globostellatic C is a tricyclic triterpenoid of the isenalabaricane group. It has role anti- neoplastic agent and a metabolite. It is a tricyclic triterpencid. acctate ester, an enone, an ether and an oao munocarboxylic acid. It is conjugate acid globostellatate C(I-).
Globostellatic is trieyelic triterpenoid with formula C34H4806, originally isolated from Glo- bostellata. It has role as metaholite. tricyclic triterpenodd, monocarboxylic an acetate ester. Globostellatic acid B is a natural product Globostellata data available.
and
hence less informative, they exhibit word patterns more congruent with those in the test set, in stark contrast to the distinctive GPT-style exhibited by the enriched texts. Nonetheless, we argue that this enrichment process benefits forging more comprehensive connection between molecular structures and their inherent properties. This assertion is further substantiated by subsequent experiments of textual generation tasks, where GPT-3.5 enrichment boosts the performance of 3D-MoL.M.
3.2 MOLECULE CAPTIONING
Generalist and Specialist Models. For generation tasks of molecule captioning and open-text molecule QA, we present 3D-MoL MoMMM'ss performances of two different variants: the specialist, which fine-tuned for each generation task separately, and the generalist, which is unified model trained on all the generation tasks. Both models are obtained by fine-tuning the Stage-2 checkpoint.
Here we evaluate 3D-MoLM for molecule captioning on the PubChem Dataset. We opt not to use CheBI-20 (Edwards et al., 2022) for two reasons: firstly, its origin from PubChem poses potential data leakage issues, akin to the concerns with PCDes; secondly, CheBI-20 is curated in a way that molecular names replaced with "the molecule". driving the model to emphasize properties instead of names. However, the molecular nomenclature system is intrinsically related to the identification of distinct molecular structures, encompassing features such as hydrocarbon chains and benzene rings. Consequently, model's ability to accurately predict these names serves as testament to its adeptness comprehending the underlying molecular structures. Thus, we elevate the task's complexity by retaining molecular names in the texts, positioning this task as combination of molecule captioning without molecular names (Edwards et al., 2022), and name prediction (Favre Powell, 2013). To demonstrate the effectiveness of 3D molecular perception, we include 3D-MoLM's variants of 1D (i.e., Llama2-7B) and 2D (i.e., 2D-MoLM) perception as baselines. Specifically, Llama2-7B. without molecule-ter projector, goes through Stage 2 & 3 training using ID SMILES as molecule representations. 2D-MoLM replaces the 3D molecular encoder with a 2D molecular encoder (Liu al., 2022b), and undergoes the same training process M..MoL Specialist models
7

---

Published as a conference paper at ICLR 2024
fine-tuned using the training set from the PubChem Dataset. Table 3 presents the performances with metrics of BLEU, ROUGE, and METEOR, accompanied by a concrete sample. We observe that:
1) 3D-MoLM demonstrates superior performance across the board, with the highest scores all evaluation metrics. While slightly erring in identifying the molecule as "Globostellatic acid B" rather than "Globostellatic acid C", it pinpoints roles and primary structures such as the tricyelic triterpenoid, acetate ester, and monocarboxylic acid. This demonstrates the effectiveness of 3D molecule-ter alignment training to bridge the gap between 3D molecular representations and LM's input space. We also provide detailed analysis and discussion on failure cases in Appendix D.
2) The enrichment process via GPT bolsters the text generation capability based on 3D) molecular structures. This underscores the hypothesis that the enrichment strengthens the connection between molecular structures and their associated properties, enhancing cross-modal molecular understanding. Similar gain can be observed in the following open-text QA tasks as well.
3) MDMMss''s heightened performance, when juxtaposed with finetuned Llama2-7B and 2D-MoLM subjected to similar training process but modeling molecules as ID SMILES and 2D graphs, highlights the pivotal role of 3D structure perception in bolstering molecular understanding.
3.3 OPEN-TEXT MOLECULAR QUESTION-ANSWERIN (QA)
We evaluate 3D-MoLLM for open-text molecular QA on the 3D-MolT dataset. Considering that open-text molecular QA mostly unexplored in existing works, we mainly compare MD-MoL.M with its variants of ID or 2D molecular perceptions. Notably, we report performances of specialists, which are trained for each task separately, and generalists, which are unified models trained on all the generation tasks. Table 4 presents the quantitative evaluations and QA samples. We observe that:
Observations for Descriptive Property QA: 1) Superiority of 3D-MoLM over baselines. It exhibits commendable performance in 3D molecule-text interpretation, clearly surpassing the baselines. Specifically, It correctly identifies beryllium acetate as a solid that is soluble in water and goes beyond the ground truth by providing supplementary details, such pinpointing the elevated melting and boiling points and attributing them to the potent intermolecular forces between berylfium and acetate, which underlines its depth of molecular understanding.
2) Benefit of descriptive instructions. Incorporating them amplifies the model's molecular compre- hension. This can be verified by that through instruction tuning, Llama2-7B (generalist) advances over its initialization point, manifesting improvements of 2.46 in BLEU-2 and 3.22 in METEOR.
Observations for Computed Property QA: 1) 3D-MoLM achieves superior performances on computed property QA task, consistently achieving the lowest MAE, especially on those properties intrinsically determined by 3D conformations (i.e., highlighted properties Table 4c). A remarkable performance lead of 0.77 eV among generalists on HOMO-LUMO Gap accentuates 3D-MoLM's adeptness at 3D molecular understanding, which we attribute to its 3D perception. However, for properties that mainly stem from atomic compositions and interatomic connections (i.e., molecule weight, LogP, TPSA, and complexity), the advantage, while persistent, more subdued. This aligns with Uni-Mol, which displays larger advantages over 2D molecular models for predicting quantum mechanical properties grounded in 3D structures.
2) Efficacy of instruction tuning. Instruction tuning amplifies the model's capability to adhere to and act upon instructions. This is illuminated by Llama2-7B's ' official checkpoint's 's occasional ineptitude in generating valid numerical responses, particularly when tasked with approximating 3D 3D-dependent computed properties that aren't 't straightforwardly derivable from SMILES.
3) Comparison with Uni-Mol. 3D-MoL.M enhance the accuracy of molecular property prediction by leveraging both the rich contextual knowledge found in chemistry literature and 31) molecular conformations, For instance, the pretraining corpus of 3D-MoLM contains descriptions of hydropho- bicity (LogP) and solubility (TPSA). While Uni-Mol excels at predicting molecular properties by interpreting 3D conformations, cannot utilize textual descriptions of chemical properties. This dual-source knowledge utilization can potentially enhance the prediction of molecular properties.
Observations for generalist and specialist: While the generalist model slightly underperforms in comparison to the specialist models, it still exhibits a performance gain over other baselines. This underscores MD' versatility and capability to effectively handle multiple tasks.
8

---

Published as a conference paper at ICLR 2024
Table 4: Open-text QA results on 3D-MoIT. * denotes the official checkpoint without any finetuning. i denotes molecule-text alignment on the original PubChem texts without GPT-3.5 enrichment. Llama2-7B, without a molecule-text projector, goes through Stage 3 instruction tuning by modeling molecules as ID SMILES. 2D-MoLM replaces the 3D molecular encoder with 2D molecular encoder and goes through three-stage training as 3D-MoLM.
(a) Descriptive property QA results.
Type
Model
BLEUU-
BLEU-4
ROUGE.1 ROUGE-2
ROUGE-L
METEOR
Llama2-7B
35.14
30.41
Specialist
2D MoLM
30.84
25.09
38,46
24.22
33.04
50.92
3D-MoLMf
30.33
38,
23.93
32.98
51.33
3D-MoLM
32,00
26.13
40.13
25.55
34,64
52.15
Llama2-78*
25.22
21.16
31.48
19:21
25.22
43.17
Llama2-7B
27.68
22.81
34.73
21.55
29.91
46.39
Generalist
2D-MuLM
30.23
3D MoLMi
29.92
24.44
38.62
22.83
32,30
50.81
3D-MoLM
31.81
26.08
40.13
25.87
34,99
51.93
(b) concrete sample of descriptive property QA.
Molocule
Question
Ground truth
3D-MoLM
What are the physical
Beryllium acctate is typically white or colortess solid and soluble in water.
On the physical properties of herylllium acctate, it white crystalline solid that soluble water. has relatively high melting point and boiling point due to the strong inter- molecular forces between berylllum acetatle.
properties of beryllium acetate??
(c) Computed property QA results. We report the MAE with valid answer rate (%) LMs sometimes fail follow instructions to generate numerical responses. highlight properties largely dependent on 3D structures.
Type
Madel
Weight (g/mol)
Logg
TPSA (A2)
Complexity HOMO (eV) LUMO (eV) H.I. Gap (eV) SCF (10*eV)
Non-LM Uni-Mol
20.35
0.59
13.48
57.24
0.32
0.35
0.21
0.45
Llama2-77 7B
22.10
1.45 (95%) 15.87 (92%)
69.74 (93%)
1.24 (96%)
1.04 (95%)
0.88 (92%)
0.70 (99%)
Specialist
MD-MoLL
21.48 (94%)
0.88 (96%)
13.52 (92%)
74 (94%)
0.92 (98%)
0.80 (96%)
0.67 (93%)
0.71 (99%)
3D-MoLMf
(96%)
0.95 (96%)
10.26 (94%)
(95%)
0.45 (98%)
0.36 (96%)
0.41 (94%)
0.39 (99%)
3D-MoLM
14.79 (95%)
0.66 (975)
9.71 (93%))
44.85 (94%)
0.26 (97%)
0.25 (94%)
0.28 (94%))
0.35 (99%)
Llama2-7B*
42.18 (82%)
2.10 (85%) 27.11 (84%) 121.87 (76%)
2.87 (70%)
1.89 (71%)
1,86 (70%)
3.84 (23%)
Llama2-7B
27.42 (92%))
1.78 (93%) 17.07 (90')
78.16 (92%)
1.89 (90%)
1.26 (90%)
1.25 (91%)
0.87 (991S)
Generalist 2D MoL.M
20.80 (92%)
1.36 (94%)
12.47 (89%) 52.70 (91%)
1.52 (93%)
1,13 (92%)
1.09 (88%)
0.96 (99%)
3D-Mol Mf
19.54 (93%)
0.92 (92%) 14.14 (92%) 54.68 (90%)
0.65 (94%)
0.41 (92%)
0.55 (89%)
0.49 (99%)
MoLM
(92%)
(95%) 10.90 (90%)
(89%)
0.35 (95%)
0.36 (93%)
0.32 (909))
0.38 (98%))
(d) Concrete samples of computed property QA.
Molecule
Question
Ground truth
3D-MoLM
Please provide Molecular Weight value for molecule. If uncestain, provide an estimate. Re- spond with the numerical value only.
The Molccular Weight for the input molocule is 286.28 g/mol.
Input molocule
Molecular Weight of 288.30 30 g/mol.
Could you give me HOMO LUMO Gap value of this molecule? If uncertain, provide an estimate. Respond with numerical value only.
LUMO Gap for the input molecule is 5.325 eV.
H10MO LUMO Gap for the input molecule is 5.762 eV.
CONCLUSION
In this work, we introduce 3D-MoLM, a new approach tailored for 3D-molecular language modeling. 3D-MoLM equips an LM with 3D molecular encoder for 3D molecule-ter interpretation. This is achieved by a 3D molecule-text projector that aims to map 3D molecular representations into the LM's textual space. Furthermore, 3D-MoL.M incorporates 3D molecule-centric instruction tuning, enhancing both its adherence to human instructions and 3D molecular understanding. Extensive experiments demonstrate that 3D-MoLM excels in various tasks, including molecule-text retrieval, molecule captioning, and open-text molecular QA.
Despite the promising results, our work has few limitations. Compared with vision-language modeling methods (Dai et al., 2023; Hong et al., 2023), the scale of our 3D molecule-text dataset is notably constrained, inherently limiting the performance. This motivates texts closely related to 3D molecular structures. Furthermore, this study does not explore other intriguing capabilities of large LMs, such in-context learning and cf reasoning.
search for high-quality

---

Published as a conference paper at ICLR 2024
5 ACKNOWLEDGEMENTS
This research is supported by the National Natural Science Foundation of China (922701 14) and par- tially supported by the National Research Foundation Singapore under the Al Singapore Programme (AISG Award No:
the Singapore Ministry of Education Academic Research Fund Tier (Award No: TI 251RES2207) and the Google Cloud Research Credits program This research is also supported by the CCCD Key Lab of Ministry of Culture and Tourism and NExT Research Center.
000--TCC-2223000--33
(Award No:
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In NeurlPS, 2022.
Iz Beltagy, Kyle Lo,, and Arman Cohan. Scibert: A pretrained language model for scientific text. In EMNLP/JJCNLP (1), pp.
Association for Computational Linguistics, 2019.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling language models. CoRR, abs/2210.11416, 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards ceneral-purposs. vision-language models with instruction tuning. In NeurIPS, 2023.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-ILLT (1), pp. 4171-4186. Association for Computational Linguistics, 2019.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In ICMI,, 2023.
Carl Edwards, ChengXiang Zhai, and Heng Ji. Text2mol: Cross-modal molecule retrieval with natural language queries. In EMNLP (1), pp. 595-607, Association for Computational Linguistics, 2021.
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation between molecules and natural language. In EMNLP, pp. 373-413. Association for Computational Linguistics, 2022.
Junfeng Fang, Wei Liu, Yuan Gao, Zemin Liu, An Zhang, Xiang Wang, and Xiangnan He. Evalu- ating post-hoc explanations for graph neural networks via robustness analysis. seinth- Conference on Neural Information Processing Systems, 2023a.
Junfeng Fang, Xinglin Li, Yongduo Sui, Yuan Gao, Guibin Zhang, Kun Wang, Xiang Wang, and Xiangnan He. Exge: Bridging efficiency and explainability in graph condensation. WWW. ACM, 2024a.
10

---

Published as a conference paper at ICLR 2024
Junfeng Fang. Shuai Zhang, Chang Wu, Zhiyuan Liu, Sihang Li, Kun Wang, Wenjie Du, Xiang Wang, and Xiangnan He. Moltc: Towards molecular relational modeling in language models, 2024b.
Xiaomin Fang, Lihang Liu, Jicqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Genneety molecular representation learning for property prediction. Nat. Mach. Intell., 4(2):127-134, 2022.
Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions: A large-scale biomolecular instruction dataset for large language models. arXiv preprint :tt.::3000000// 2023b.
Henri A Favre and Warren H Powell. Nomenclature of organic chemistry: IUPAC recommendations and preferred names 2013. Royal Society of Chemistry, 2013.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the world into large language models. arXiv preprint arXiv:2307. 12981, 2023.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models. CoRR, abs/2106.09685, 2021.
Robert Jones. Density functional theory: Its origins, rise to prominence, and future. Reviews of modern physics, 87(3):897, 2015.
William L Jorgensen. The many roles of computation in drug discovery. Science, 303(5665): 1818-1818, 2004.
Martin Karplus and Andrew McCammon. Molecular dynamics simulations of biomolecules. Nature structural biology, 9()):646-65) 662, 2002.
Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A. Shoemaker, Paul A. Thiessen, Bo Yu, Leonid Zaslavsky, Jian Zhang, and Evan Bolton. Pubchem in 2021: new data content and improved web interfaces. Nucleic Acids Res., 49Daataasse-Issue) D1388-D1395 2021.
Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alán Aspuru-Guzik. Self- referencing embedded strings (SELFIES): A 100% robust molecular string representation. Mach. Learn. Sei. Technol., 1 (4)):40024, 2020.
Greg Landrum et al. Rdkit: software suite for cheminformatics, computational chemistry, and predictive modeling. Greg Landrum, 8:31, 2013.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping lanuage-immae pre-training with frozen image encoders and large language models. CoRR, abs/2301. 12597, 2023.
Sihang Li, Xiang Wang, An Zhang, Yingxin Wu, Xiangnan Hc, and Tat-Seng Chua. Let invariant rationale discovery inspire graph contrastive learning. In International conference machine learning, pp. 13052-13065. PMLR, 2022a.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACI/JJCNPP (1), pp. 4582-4597 Association for Computational Linguistics, 2021.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang. Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive language-iman image pre-training paradigm. In /CLR. OpenReview.ner 2022b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.0844 08485, 2023a.
Shengehao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Anima Anandkumar. Multi-modal molecule structure-text model for text-based retrieval and editing. CoRR, abs/2212. 10789, 2022a.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. In ICLR. OpenReview.net, 2022b.
11

---

Published as a conference paper at ICLR 2024
Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat- Seng Chua. Molca: Molecular adapter. In EMNLP, 2023b. URL https: /wwp//peeeeeeeeeeeeeeeeeeeeeeeee wtt.:/eee///teee wwwhh-1
modeling with cross-modal projector and uni-modal
Zhiyuan Liu, Yaorui Shi, An Zhang, Enzhi Zhang. Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Rethinking tokenizer and decoder in masked graph modeling for molecules. In NeurIPS, 2023c. URL https: openrevieeeeeeeeeeeeeeeeee
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Tay, Denny Zhou, Quoe Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 1.3300.13660 13688, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018.
Shuqi Lu, Zhifeng Gao, Di He, Linfeng Zhang, and Guolin Ke. Highly accurate quantum chemical property prediction with uni-mol+, CoRR, abs/2303. 16982, 2023.
Chenyang Lyu, Minghao Wu, Longyue Wang. Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-Ilm: Multi-modal language modeling with image, audio, video, and text integration. arXiv, 2023.
Donald Allan McQuarrie and John Douglas Simon. Physical chemistry: molecular approuch, volume University science books Sausalito, CA, 1997.
Maho Nakata. The pubchemqe project: a large chemical database from the first principle calculations, 2015.
OpenAI. Chatgpt, 2023a. URL https:/ /ttpt:::::///paaa.l..e http:///pppaa..ccooooooaoooo//// chatgpt.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08777 2023b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. $748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text 2020.
transformer. Mach. Learn. Res., 21:1
Yaorui Shi, An Zhang, Enzhi Zhang, Zhiyuan Liu, and Xiang Wang. ReLM: Leveraging language models for enhanced chemical reaction prediction. In Findings of the Association for Computational December 2023.
Linguistics: EMNLP 2023, pp. 5506-5522
Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji-Rong Wen. A molecular multimodal foundation model associating molecule graphs with natural language. CoRR, (bbs/200.00488.1 2022.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. CoRR, abs/2211 1. .00085, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CORR, hbs/23002.13711. 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In NewrlPS, pp. 200-212, 2021.
12

---

Published as a conference paper at ICLR 2024
Alain C Vaucher, Philippe Schwaller, Joppe Geluykens, Vishnu H Nair, Anna Iuliano, and Teodoro Laino. Inferring experimental procedures from text-based representations of chemical reactions. Nature communications, 12(1):2573, 2021.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. arXiv preprini arXiv:2308. 16911, 2023.
Zhao Xu, Youzhi Luo, Xuan Zhang, Xinyi Xu, Yaochen Xie, Meng Liu, Kaleb Dickerson, Cheng Deng, Maho Nakata, and Shuiwang Ji. Molecule3d: A benchmark for predicting 3d geometries from molecular graphs, 2021.
Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature communications, 13(862), 2022.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv, 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
Zipeng Zhong, Jie Song, Zunlei Feng, Tiantao Liu, Lingxiang Jia, Shaolun Yao, Min Wu, Tingjun Hou, and Mingli Song. Root-aligned smiles: tight representation for chemical reaction prediction. Chemical Science, 13(31):9023-9034 2022.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: universal 3d molecular representation learning framework, In ICLR. OpenReview.net, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing languae understanding with advanced large language models. arXiv preprint ar0iv:2304. 10592, 2023.
13

---

Published as a conference paper at ICLR 2024
A RELATED WORKS
In this section, we provide a review of literature related to molecule-text modeling, multi-modal instruction tuning, and multi-modal language models.
Molecule-Tese Modeling. Early research approaches molecular understanding by representing molecules as ID sequences. Specifically, KV-PLM (Zeng et al., 2022) represents molecules using ID SMILES and employs masked- wwwpledd--angggg--moooee modeling loss for pretraining on biomedical texts. MolT5 (Edwards et al., 2022) is T5-based (Raffel et al., 2020) model tailored for molecular tasks (iee. SMIL.ES-t--tett and Eox-to-SMILLES translations). Mol-Instruct (Fang et al., 2023b) utilizes the ID) SELFIES (Krenn ct al., 2020) molecular descriptor and offers an instruction dataset for biomolecular research. Subsequently. 2D molecular graph encoders (Liu et al., 2023c) are introduced to explicitly capture topological information. For example, MoMu (Su et al., 2022) and MoleculeSTM (Liu et al., 2022a) use cross- moosl contrastive learning to bridge the representation spaces of molecular graphs and texts. MolCA (Liu et al., 2023b) combines ID SMILES and 2D graph representations for molecule-to-text generation. ReLM (Shi et al., 2023) applies ChatGPT to rerank reaction prediction results proposed by GNN model. However, current molecule-text modeling approaches mostly rely on ID sequences or 2D graphs for molecular representation, overlooking the vital spatial information in 3D molecular conformations.
Multi-Modal Instruction Tuning, Instruction tuning (Chung et al., 2022; Longpre et al., 2023) trains LMs adhere to natural language instructions, enhancing their ability to generalize to unseen tasks and instructions. It has been extended in the multi-modal domain to augment the LM's cross-modal comprehension. For instance, InstructBLIP (Dai et al., 2023) adapts 26 vision-language datasets to the instruction tuning format and introduces instruction -wwaee visual feature extraction, allowing adaptive and informative feature extraction based on provided instructions. L.LaVA (Liu et al., 2023a) and MiniGPT-4 (Zhu et al., 2023) leverage GPT-3.5/GPT-4 (OpenAI, 2023a:b) convert polish eestruction-followiiii format, facilitating integrated visual and
image-text pairs into high-quality language understanding. In the 3D visual domain, 3D-LLM (Hong et al., 2023) curates 3D-language instruction tuning dataset to instill a 3D perspective into LMs. They have showcased the potential of instruction tuning to enhance the cross-modal understanding of LMs. However, its application in the 3D molecule-text domain remains largely unexplored.
Multi-Modal Language Models. Multi-modal LMs aim to merge the realms of text with other modalities. Early vision-language models (Radford et al., 2021; Li et al., 2022b) employ contrastive learning fuse the visual and textual representations. Recent studies (Li et al., 2023; Alayrac et al., 2022; Tsimpoukelli et al., 2021; Driess et al., 2023) suggest that visual feature spaces can be seamlessly aligned with the input spaces of LMs, enabling them to undertake language generation tasks based on visual inputs. Aside from images, researchers have integrated preceptors of diverse modalities, including video (Zhang et al., 2023), audio (Lyu et al., 2023) and 3D point clouds (Xu et al., 2023), into LMS, giving rise to series of multi-modal LMs. We draw inspiration from these revelations to devise 3D-MoLM.
B PUBCHEM DATASET AND ENRICHMENT PROCESS
GPT-3.5 Enrichment. We begin our data preparation by collecting 324,689 molecular SMILES-text pairs from PubChem (Kim et al., 2021). However, they include large number of uninformative texts the shortest description has only 1 word and 82.29% of the collected molecules have descriptions less than 20 words. To ensure the effective evaluation for alignment, we first extract a high- subset of 15K pairs encompassing descriptions with more than 19 words. This curated subset is subsequently partitioned into train validation / test sets containing 12K / 1K / 2K pairs, respectively. The less informative remainder is reserved for alignment purpose (i.e., stage 1&2 pretraining). Yet, we argue that those less informative texts potentially offers an insufficient perspective on molecular properties and structures, which could compromise the 3D molecule-text alignment. To address this shortfall, as illustrated in Figure 4, we employ GPT-3.5 to enhance these molecule descriptions, thus bolstering the less annotated molecules. We note that the enrichment process using GPT is exclusively applied to the pretraining set, ensuring that evaluations in downstream tasks (i.e., train / validation/ test sets) conducted on original real-world data.
14

---

Published as a conference paper at ICLR 2024
000-000-C0-CC
D33HOCO
RDKI
RDK
Pub Sigle snc Deni
Pub hee
soaul produc fand Sclatur Siserssa
diiul C- C-llt1o0 in cingins
briwn aa pelicen piteer meol
uan facie becama Inlentuaa the aememnne en liple dodd teme
fanil
hha sor ti (P0) porr and
S
cali wilh voggle 222
whe
lards
sepeoment Ho'' in porn
Evired
unelle hight deles ciedere
Cociee
propuces One leportent poopery to Rich oi reciiiis frans.
irjuncc
Figure 4: Utilizing GPT-3.5 to enrich the descriptions in the pretraining subset of PubChem Dataset.
Table 5: Number of molecule-text pairs in subsets of PubChem Dataset.
Subset
#Molocale-Test Pair
#Min Word
#Avg Word
Pretrain
301.658
17.84
Train
12,000
57.24
Valid
1.000
20
58.31
Test
2,000
20
55,21
Correctness and Comprehensiveness Evaluation. To evaluate the correctness and comprehensive- ness of the original texts in PubChem and those enriched by GPT-3.5, we solicited evaluations from three human experts in chemistry and also from GPT-4 with the same score standards in Figure 5. These evaluations, conducted on subsets of 100 (for human experts) and 1,000 (for GPT-4) samples respectively, are presented in Figure 6. The results indicate that the enhanced descriptions have superior comprehensiveness, while maintaining high correctness. Adopting the GPT-3.5 generation and subsequent GPT-4 evaluation approach, rather than directly using GPT-4 for text enrichment, was primarily driven by cost considerations; utilizing GPT-4 for the enrichment would have incurred expenses exceeding $5000.
Raw 3D conformation Generation. Subsequently, we leverage RDKit to transform the ID SMILES into raw 3D conformations. While the raw conformations derived from ID SMILES might not match the accuracy of costly DFT (Jones, 2015) equilibrium conformations, which necessitate several hours for single molecule property calculations, they suffice for preliminary cross-modal alignment. Meanwhile, RDKit occasionally fails to generate 3D conformations. As such, pairs failing this generation process are excluded. As shown in Table 5, the resultant dataset split for peerranntrain//aliiatiooo/ comprises 301,658/ / 12,000 / 1,000 / 2,000 3D molecule-text pairs, respectively. The pretraining set with GPT-enriched descriptions is utilized for 3D molecule-text alignment, detailed in Section 1. Mcanwhile, the train, validation, and test sets used for cross-modal retrieval and molecule captioning tasks, detailed in Section 3.1 and 3.2, serving as a measure of alignment quality.
C EXPERIMENT SETUPS
In this section, we present the experimental setups for three pretraining stages and downstream tasks.
Stage 1 Pretrain. The Q-former attached with frozen 3D molecular encoder is pertrained for 50 epochs and the number of query tokens in it is set to 8. AdamW (Loshchilov & Hutter, 2018) optimizer is adopted with weight decay of 0.05 and a learning rate scheduler of a combination of linear warmup with 1000 steps and cosine decay, in which the peak and minimal learning rates are le-4 and 5e-6, respectively. And the batch size and maximal text length 64 and 256, respectively. The computation overhead is 40 GPU hours on NVIDIA A A100 with 16 Mixed precision.
Molecule-text retrieval. The Stage-1 1 checkpoint is finetuned for 10 epochs and evaluated on the downstream subsets (i.e., train/validation/tess)) of PubChem with the same optimizer and learning rate scheduler configurations, with the exception of the warmup steps, which set to 200, as stage 1
15

---

Published as a conference paper at ICLR 2024
Description enrichment prompt template
Instrction: You urean sapert chaniia Givsn the moleenie SMI ES and mes desctiption, your talk prisa the now description providt the comeet aná decalled descrptton of the malesuirs and plasical prgertis using your chemocal molectar knokdedige
Input: Killegale SMILLES: (S) Row Benergton:: FRow tecil
Ourgut:
Carrectness standard molecular description
Instrurtion: You opet chenzis and will gave you the SKILES of moberule and panapupl of batual desctigion focit. our takks evaleate the oometaess of this tuctal deciption you chanical meterslar knorslidge ledge and scoe from I to acconting to following staraled.
Maxccue Descrition Conectasss Scoce Staddand (3-3)
1. Inaocurate
Contains zultple factual exurs lncocstpaacies
Milks non-slancard recemwt nomreclattrr.
Procides insomet connot reganding molacciles propertao or sigri
Partsaly
Contalas nsse
and miizakes inconsitaaciio.
romendater.
limited
might partly
Contaies onisiions giveing facttel
Mostly stmdarii nomendittin
devisbion..
Grves beat, overveww meleziie's spnihonce propettees
4. Montly Correc:
Provides riverigtion with mos dstails being worrewx
Contaits mizuto inacurcce but no lon that auntioanty detracts tteeraal commtaass.
Consoteetly employs sandant aomenslatur with mino scceptons.
5. Dighly Conrect
Offere heliste and deceled deseiption of the molende withost any Dacemmile
Perfectly udhares to stasdard acqueniittrr and conventions.
Pariites a tharogg undestarding of the molecuee' siguitcuace, bocked by accurato contodt and information.
Input: SMILLS; (SMILDS)
Deserittonn (Moleculartest
Ourput:
Compr eluntsvveees eases standard for molecular description.
Instruttion: tu ume ergent therist and will give you the SMILES of roolesale and penggoph of untual description forit. taokk is anlleste the compnhersiverene of this sonul dewrrgtir por using your chemaod mo'waiiee knawleiig and some from to acconting to following sondant
Moxecuee Descripton Comperensweess Soor: Staziard 5)
1. Ineufficree
Desmpton prnvices very minimal orno informtion about molectle
Omnis key deseiss meking chillreging to furn dier undenstnningg molscie
proettes about the
superficial eventeee, perhaps
grimary
Mix atob deseeption may on maltple of
siguiflcance
Medeente:
Gives deseription the molecuees Seauues, but there night laak depth.
Oovers
detall Providle contact obout the mallesalis propetion sigilficanc may not delve into leiser-kacors attiiiutes.
extauttive breath.
4. Dasalled:
Offes deep dive lato the moleoata, dutxiing is prinary featans and mone.
Costains micute onitaions but paviides will ronded codenizadaa moleccll.
Explome veriers regects the molsenn from strectim to sguificance in spesfic fulis.
Prisenss a bmader contect prssilly mesboring iss inteaattoos,, ons, nále nesorc,, other
pertirent oeether
facts.
5. Highly Congriihees eisie
Delivoss a holisii undentaadiag, lawing so stone untumed
Deactbos not only the kos featums but als dives las cussces, history, rdeeaaos, and loper kcaws atttibutes
Mekon stancand commndittoo, and poorilly includes stemative namnng or misred melecales
Paintsa pictamoof melecale's sigiiiicanee interations, raes, and any other deteil.
Input:
Descrittion (Molecalartet)
Oargoti
Figure 5: The prompt template for textual description enrichment and quality evaluation.
pretraining. Note that given the relatively small scale of the Q-former and the associated Uni-Mol model, stage does not involve LoRA tuning.
Stage 2 Pretrain. Initiating from the Stage-1 checkpoint, we proceed to the pretraining of Stage 2 for text generation training of 10 epochs, maintaining the optimizer and scheduler configurations from the prior stage. To reduce CUDA memory usage, we integrate LoRA with parameters set to r= 8, 0 32, dropout 0.1. This integration applied the modules of [k-proj, v-proj. q-proj, o-proj, gate.proj, up-proi, down-proi]. Utilizing this configuration, the resultant LoRA adapter comprises 19M trainable parameters, representing a mere 0.29% of the total parameter count in Llama2-7B. With batch size of 16, maximal token length of 320, and accumulated gradients from 2 batches, the computation overhead 60 GPU hours on NVIDIA A100 with BFloat16 Mixed precision.
Stage 3: Generalist Model. To obtain one checkpoint for all tasks encompassing molecule captioning. descriptive, cocenticc property, and 3D-centric property open-text QA, mix the training
16

---

Published as a conference paper at ICLR 2024
dncrpiire 1a0aam
wus aoel lptens Yoa Pacchem
Comptren
Eipen2
Oxper
0914
Eepot:1
Spprt
Opert3
(a) Correctness.
(h) Comprehensiveness.
Figure 6: Correctness and comprehensiveness evaluation for subsets of original PubChem texts and GPT-3.5 enriched ones. Figure shows score standards.
sets of these tasks. Given the substantial difference in the size of each dataset, we adopt sampling datasets with probabilities that are proportional to the fourth root their sizes as in (Dai al., 2023). Subsequently, the model undergoes training for 400,000 steps with validation interval of 50,000 steps, utilizing identical optimizer, learning rate scheduler, and LoRA configurations as in stage pretraining. Following this, the checkpoint with the lowest validation loss is evaluated the test sets of all tasks. The performance this model is presented in Section as a generalist.
Stage 3: Specialist for Molecule Captioning. The Stage-2 checkpoint is finetuned for 10 epochs and evaluated on the downstream subsets (i.e., (rain/validation/tess) of PubChem with the same optimizer, learning rate scheduler, and LoRA configurations stage 2 pretraining, with the exception of the warmup steps, which are set to 200,.
Stage 3: Specialist for Open-text QA. We possess three specialists for descriptive, non-3D 1aa-3--centric property, and 3DDcentric property Open-text-QA, respectively. Initially, they universally undergo a comprehensive epoch of all instruction tuning data in 3D>-MolT, employing the same configurations as stage 2. but limited to a single epoch. The computational overhead amounts to 200 GPU hours on NVIDIA A100, utilizing B6 Mixed precision. Subsequently, the specialists for descriptive, non-3D-eentric property and 3D-centric property Open-text-QA are fine-tuned on the sub-dataset of the corresponding task for 5, 5, and epoch, respectively.
D FAILURE CASE STUDY
Table 6a shows the mistakes 3D-MoLM made in the molecule captioning task, and we offer the subsequent analysis:
3D-MoL.M incorrectly identifies Globostellatic acid C as Globostellatic acid B, which are struc- turally distinct solely in the placement of methoxy branch. As illustrated in Table 6(b), they have remarkably similar records on PubChem. This suggests that despite accurately identifying primary functions and structures, the discernment of 3D-MoLM at more refined granularity requires enhancement. We will persist in investigating this in future studies.
3D-MoLM inaccurately enumerates the number of atoms in Globostellatic acid C as C34 Hs whereas the actual composition is Ca 07. Although this may seem trivial even non-expert humans, counting atom numbers and providing chemical formulas pose common challenge for language models. As demonstrated in Figure 7, even the powerful GPT-4 model lacks the ability to directly interpret SMILES strings and provide correct chemical formulas.
To rectify these deficiencies, we will persist in refining 3D-MoLM from various perspectives, in- cluding 1) integrating a more powerful 3D molecular encoder to distinguish molecules with similar structures, investigating more effective projector structures bridge the gap between 3D molecular representation and LM space, and 3) curating more fine- fine-grained datasets for molecule-tex alignment and molecule-centric instruction tuning.
17

---

Published as a conference paper at ICLR 2024
Table 6: Failure case study.
(a) Case of molecule captioning. Mistakes are highlighted.
Ground
3D-MoLM
Globostellatic
trieyclic triterpenoid
Globosiellatic acid is a trieyclic triterpenoid the isomulabaricane group. has mle an antincoplastic agent and a metabolite, It is ester, enone.
with formula C34H14806, originally isolated from Globostellata.
as a metabolite. triccclic triterpenoid, monocarboxylic acid and an acetate ester. Globostellatic acid is natural product found in Globostellata with data available.
tricyclic triterpenoid, an ether and an oxo monocarboxylee lic acid. It is conjugate globostellatate C(1-).
(b) PubChem records of Globostellatic acid C and Globostellatic acid Their differences circled in the 2D molecular graphs below, we visualize 2D molecular graphs instead 3D structures for visual clarity.
Molecule
Synonym
Molocular Formula
Textual description on PubChem
Globostellatic acid trieyclic triterpenoid of the isomal- aburicane group. It has role as an untineuplastic agent and metabolite. It is trieyclic triterpenoid, enone, monocarboxylic acid, an acctate ester and an ether. It conjugate acid globostellatate B(1-).
Globostellatic
Globostellatic acid is trieyclic triterpenoid of the isomal- abaricane group. It has role as an antincoplastic agent and metabolite. triecclic tritepenoodd, an enune, an ether and un oso monocarboxy lic acid. It is conjugate globostellatate C(I-).
Globostellatic acid
ester,
What's the chemical formula of
The SMILES string you provided, htc.re/ccccccccccceeooeeeeeeeoee (C30)0C, represents specific organic compound. number of each atom in the molecule.
However, manually counting atoms a SMILES string proo--pooe consuming. A more efficient
cheminformatic software online tools that interpret SMILES strings and provide the chemical formula.
of knowledge cutoff in September and provide chemical formulas. recommend using a tool like RDKit, Open Babel, or an online SMILES get
have capability directly interpret SMILES
formula compound.
Figure 7: Existing LLMs (e.g., GPT-4) are incapable of accurately counting numbers of atoms and generating chemical formulas of given SMILES. We use Globostellatic acid B in this example.
E ABLATION STUDY ON INPUT PROMPT
To demonstrate that integrating 3D molecular tokens with ID SMILES in the prompt enhances molecular understanding, we perform ablation study and report the results in Table We observe that using combination of both 3D molecular tokens and 1D SMILES as prompts consistently yields superior performance compared to using only one type. This can attributed to that in- cluding SMILES ensures the LM taps into its inherent understanding of ID SMILES. Thus, when integrating the 3D molecular encoder and molecule-tex projector, the distilled 3D molecular tokens concentrate discerning 3D-dependent properties. This is akin to enhancing model with additional features while retaining its existing representation ability, ensuring a more comprehensive understanding.
18

---

Published as a conference paper at ICLR 2024
Table Ablation study. The instruction prompt of 3D-MoLM contains both 3D molecular tokens and ID SMILES (cf. Figure 1). Llama2-7B, going through stage instruction tuning, is prompted with ID SMILES, while $ denotes prompting with only 3D molecular tokens.
(a) Molecule captioning on the PubChem Dataset.
Model
BLEU:2
BLEU-
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Llama2-7B
27.01
20.94
35.76
20.68
29.88
32.11
3D-MoLME
27.12
21.10
35.63
20.47
28.84
32.41
3D-MoLM
29.25
22.07
36.48
21.80
30.95
33.12
(b) Descriptive Property QA results on 3D-MoLM.
Model
BLEU:2
BLEU-4
ROUGE-1
ROUGE.2
ROUGE-L
Llama2-7B
27.68
22.81
34.73
21.55
29.91
46.39
3D Mo
28.05
23.38
35.47
21.98
30.29
47.20
3D-MoLM
31.61
25.88
39.93
25.68
34.78
51.74
(c) Computed property QA results on 3D-MoLM.
Model
Weight (g/mol)
LogP
TPSA
Complexity
HOMO (eV)
LUMO (eV)
H.L. Gap (eV) SCF (10%eV)
Llama2-7B
27.42 (92%)
1.78 (93%)
17.07 (90%)
78. 16 (92%)
1.89 (90%)
1,26 (90%)
1,25 (91%)
0.87 (99%)
3D-MoLMt
24.15 (93%)
142 (93%)
15.02 (92%)
(91%)
0.68 (94%)
(93%)
(92%)
0.51 (99%)
3D-MoLM
16.58 (92%)
0.78 (95%)
10.90 (90%)
43.49 (891)
0.35 (95%)
0.36 (93%)
0,32 (90%)
0,38 (98%)
Table 8: Ablation study of 3D perception on MolT5.
Perception
BLFU--
BLFUU-
ROUGE.
ROUGE-2
ROUGE-
METEOR
MolTS-smal small
22.53
15.23
30.44
13.45
20.30
23,98
23.73
16.57
31.57
14.34
21.63
24.81
16.61
MoIT5-bar base
V 25.55
17.31
33.36
15.49
22.62
27.54
MolTS-large
X
25.87
17.28
34,07
16.42
23.41
28.04
26.91
18.50
35.25
17.37
25.50
29.45
F THE EFFECTIVENESS OF 3D PERCEPTION
To demonstrate the effectiveness of 3D perception and eliminate the advantage of model scale (i.e., Llama2 7B versus T5 780M), we integrate the 3D molecule encoder and the projector into MolT5. This facilitates a fair comparison, maintaining the same model size.
As shown in Table 8, MolT5s equipped with 3D perception consistently surpass those lacking 3D perception across all scales. This result substantiates our claim that the incorporation of 3D geometry perception enhances the molecular understanding of language models.
G LIMITATIONS AND FUTURE DIRECTIONS
In this paper, we delve into the domain of 3D molecule-text interpretation, aiming bridge the understanding between 3D molecular representations and textual descriptions. Although the proposed approach demonstrates promising results on various tasks, we recognize the following limitations:
Expanding to More Molecule- Textt Modeling Tasks. Looking forward, tasks like experimental procedure prediction (Vaucher al.. 2021), retrosynthesis (Zhong et al., 2022), and drug- interaction (Fang et al., 2024b) can potentially benefit from 3D molecular-text modeling, but are beyond the scope of this work. This limitation stems partly from the lack of openly available datasets for certain tasks and our aim to maintain focused scope in this study. Additionally, existing tasks like molecular captioning and molecule-text retrieval can be further developed by introducing new datasets that focus
complex and novel chemical properties (e.g., molecular spectra). Using texts for graph explanation (Li et al., 2022a; Fang et al., 2023a; 2024a) and property prediction (Liu et al., 2023c) can also be explored. These opportunities for expansion are left for future research.
Fine-grained 3D Molecule-Texi Alignment. We include case studies in Appendix D showing that M is limited in discerning fine-grained small molecular structures. Overcoming this issue is
19

---

Published as a conference paper at ICLR 2024
crucial for enhancing performance in tasks that demand precise understanding of 3D molecular structures. One potential direction to curate a new 3D molecule-tex dataset with explicit 3D coordinate references within textual descriptions. Another direction is to explore more powerful 3D molecular encoders and 3D molecule-text projectors.
Dataset Scale. Other fields of multi-modal modeling have access datasets comprising more than 10 million paired samples (Changpinyo et al., 2021). In contrast, the datasets in this work (ie., PubChem 300K and PubChemQC 3M), are relatively limited in scale. This limitation of dataset scales can impede models from reaching its full potential, and underscores the need for larger and better datasets, especially those with high-quality textual descriptions closely related 3D molecular structures.
Unexplored Capabilities of LMs. Large LMs often exhibit intriguing abilities like in-context learning and thain-of-thoughe reasoning. However, this research does not venture into leveraging these capabilities, primarily due to the lack of specialized datasets detailing step-by-step inferenced.
Potential Data Leakage. The evaluation molecular LMs presents another challenge, particularly with large LMs. The vast corpora on which these models are pretrained might inadvertently encompass test datasets. This overlap can introduce biases, thereby inflating performance metrics.
Future work might address these constraints, ensuring a more holistic and accurate understanding of 3D molecule-text interpretation.
20


Reviews:
[{"review_id": "kuNMOhO24U", "reviewer": "Reviewer_HgTH", "summary": "This paper focus on 3D molecule-text interpretation, and propose 3D-MoLM: 3D-Molecular Language Modeling. Specifically, 3D-MoLM enables an LM to interpret and analyze 3D molecules by equipping the LM with a 3D molecular encoder. This integration is achieved by a 3D molecule-text projector, bridging the 3D molecular encoder’s representation space and the LM’s input space. Overall, this is an interesting work.", "questions": "Refer to Weakness.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-organized and written. The proposed 3D-MoLM enables an LM to interpret and analyze 3D molecules by equipping the LM with a 3D molecular encoder which is interesting and motivating.", "weaknesses": "The analysis of the proposed 3D-MoLM is not enough. The proposed method is interesting and straight-forward, any insight or analysis that could be offered to read to have a good understanding?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wb0c3rNtPe", "reviewer": "Reviewer_mT4V", "summary": "This paper presented the 3D-MoLM, a novel molecule-text multimodal language model. This model is a combination of two pretrained foundation models: Uni-Mol and Llama2 for molecular and language understanding. The key contribution of this paper lies in its three-staged training process that integrates these two pretrained models.\n\nIn the first stage of training, the authors adopted the idea of Q-former and multi-task training approach, a method that has been previously proposed in vision-language multimodal model, to align the representation spaces of molecules and languages. Afterwards, the Q-former and Llama2 models are finetuned by solving the molecular captioning task. In the final stage, instruction tuning was performed to the model. To this end, the authors constructed 3D-MoIT, a 3D molecule-centric instruction tuning dataset.\n\nThe proposed method was tested on several molecule-language multimodal tasks, including molecule-text retrieval, molecule captioning, and molecular question answering. The experimental results demonstrated that the superiority of the proposed models by comparing them with several baseline models.", "questions": "First of all, as mentioned in the weakness section, I would like to ask the authors about the motivation and need for molecule-language multimodal learning. What are the specific use cases of the proposed model molecule-language model? How would scientists or practitioners (i.e., domain experts) be able to harness the proposed method in their research or business?\n\nRegarding the question-answering task, the authors performed molecular properties prediction. Basically, properties including HOMO/LUMO are typically derived from the density functional theory calculations or molecular representation models such as Uni-Mol. Is there any specific rationale or chance that the language model enhances the prediction accuracy of these properties? Additionally, this work only compared the prediction accuracy with language models, and the ability to predict the properties comes from the Uni-Mol as aforementioned; therefore, I think the Uni-Mol should also be the baseline of the experiment, which will demonstrate the effect of the multimodal learning.\n\nFinally, PubChem database provides the 3D conformation of each molecule. While I am not sure what method were employed to obtain the 3D structure in PubChem, it might be worthwhile to consider leveraging these data instead of MMFF-relaxed structures.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "As a machine learning researcher, multimodal learning between molecule and language domain was expected to appear sooner or later. In this light, I think this study was well presented at the right time. The authors effectively adopted several techniques of existing visual-language multimodal learning to combine a molecular domain model and large language model. In addition, initial results of instruction tuning via 3D-MoIT dataset demonstrated the potential usage of large language model in molecular domain.", "weaknesses": "The main criticism I have is that the motivation and needs for molecule-language multimodal model remains unclear. Of course, the multimodality between molecule and language domain is a promising research direction for machine learning researchers and practitioners. However, let us consider the situation where scientists actually conduct research using the proposed model. In what scenarios can the proposed model be utilized and how can it lead to the synthesis or retargeting of novel/existing compounds? The molecular domain is the task for domain experts (i.e., chemists and biologists), not the general users. Thus, discussion on the use case of the molecule-language multimodality for domain experts is crucial for setting the ultimate goal for this line of research. Unfortunately, however, this paper and other work in this field, seems to avoid answering this pivotal question. I hope that the authors address this question in the rebuttal or the revised manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cgiIrK9Un3", "reviewer": "Reviewer_Zm2K", "summary": "This paper proposes a framework to interpret and analyze 3D molecules by equipping the LM with a 3D molecular encoder. The framework integrates a 3D unimol and a LLama language model, the framework is the same as BLIP2. Through a pre-train and fine-tune strategy, the authors demonstrate the effectiveness of the molecular caption, retrieval, and open QA tasks.", "questions": "1. The first question is about the Q-former (this may be the same as the Q-former in BLIP2, but the process is not clear here).  From Figure 2, the caption task and the contrastive task will use the text branch for text processing. These tasks are only used in pre-training, hence the question is that in the inference stage, the text branch in Q-former will be ignored (if I understand correctly). Therefore, the pre-training and the fine-tuning are mismatched, since we will only use llama2 for text generation, instead of the text branch in Q-former. What's the performance if we remove these two pertaining tasks and also the text branch in Q-former?\n2. In Figure 3, stage 1 is molecule-text retrieval, and stage 2 is the molecular caption, but in section 2.2.1, stage 1 describes multitask training with the whole three tasks. See \"Following BLIP,m we perform multi-task training, including....\". Please clarify the correct one. \n3. For the downstream tasks, the model is fine-tuned after different stages, which causes multiple different models, each task for a model. I am wondering about the multitasking fine-tuning and the performance of one model. Or at least, after three/two stage trainings, then finetune on the downstream tasks. \n4. At the beginning of section 3.1, the authors mentioned \"we assess the stage-1 checkpoint of ...\", but the last sentence on page 6 is \"Q-former benefits from the multi-task pre-training\", if the stage only contains retrieval task, the claim is wrong.\n5. From Table 4, it seems the performance gain is not as large as expected on open-text QA, do the authors have some analysis? 2D is similar to 3D.\n6. In Appendix C, did the retrieval task use LORA for fine-tuning?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes a new framework for 3D molecule-text pertaining and fine-tuning to conduct multiple 3D molecular tasks.\n2. The framework is clear and reasonable.\n3. The experiments are conducted different tasks and the effectiveness is demonstrated.", "weaknesses": "1. The novelty of this work is actually hard to say enough. It is clear that the framework is almost the same as BLIP2, the pre-training tasks and the fine-tuning stages are also the same. \n2. Besides, there are multiple different small places where I would like to ask questions. Most of them are unclear descriptions. Please look in the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cYMdK2OGfT", "reviewer": "Reviewer_3SWU", "summary": "This paper proposes to learn 3D molecule representations that can be used with pretrained large language models. Using pretrained 3D molecule encoder and large language models, the proposed method fine-tunes Q-Former to get molecule representation in language model space, and fne-tunes large language model to predict text (i.e., description) given at molecule embeddings and SMILES representations. The proposed method shows improvements compared with using large language model (e.g., LLaMA) naively.", "questions": "- Have the authors tried using larger model (e.g., LLaMA-13B)? If the authors tried, does it show the better performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- I'm not an expert in this area, but it seems the proposed approach is quite novel to leverage pretrained language model for downsteam tasks related to 3D molecules.\n- The paper is generally well-written and easy to follow.\n- The paper is well-motivated. \n- The paper includes code for reproducibility.", "weaknesses": "- The paper misses to include some failure cases (e.g., hallucination) of the proposed method. For instance, in Table 3(b), it seems the proposed 3D-MoLM wrongly interprets Globostellatic acid B (C34H48O6 while the ground truth is C33H48O7); the paper needs to discuss when the model tends to predict the wrong results. \n- I think the comparison in Table 3 might be somewhat unfair due to the different model sizes used for evaluation. Specifically, MolT5-Large has 800M parameters, while 3D-MoLM mainly uses LLaMA-7B models. In this respect, I suspect the improvements might simply come from the simple usage of language models with a large number of parameters and large training data rather than considering the 3D geometry of molecules. A good supporting experiment can be conducted by using a smaller LM for experiments (e.g., FLAN-T5-large).\n- I have a similar concern in Table 2 as well since the model used in the proposed method is much larger and uses much more data for pretraining compared with other baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)