LLaMo: Large Language Model-based Molecular Graph Assistant
Authors: Jinyoung Park
Minseong Bae
Dohwan Ko
Hyunwoo J. Kim
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Molecular Graph-Language Model
Molecular Graph-Language Instruction tuning
Graph Neural Networks
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Large Language Models (LLMs) have demonstrated remarkable generalization and instruction-following capabilities with instruction tuning. The advancements in LLMs and instruction tuning have led to the development of Large Vision-Language Models (LVLMs). However, the competency of the LLMs and instruction tuning have been less explored in the molecular domain. Thus, we propose LLaMo: Large Language Model-based Molecular graph assistant, which is an end-to- end trained large molecular graph-language model. To bridge the discrepancy between the language and graph modalities, we present the multi-level graph projector that transforms graph representations into graph tokens by abstracting the output representations of each GNN layer and motif representations with the cross-attention mechanism. We also introduce machine-generated molecular graph instruction data to instruction-tune the large molecular graph-language model for general-purpose molecule and language understanding. Our extensive experiments demonstrate that LLaMo shows the best performance on diverse tasks, such as molecular description generation, property prediction, and IUPAC name prediction. The code of LLaMo is available at https://github.com/mlvlab/LLaMo.

LLaMo: Large Language Model-based Molecular Graph Assistant
Jinyoung Park Minseong Bae Dohwan Ko
Hyunwoo J. Kim Department of Computer Science and Engineering, Korea University (1pmn678, bms2002, ikodoh, hyunwoo Okorea. kr
Abstract
Large Language Models (LLMs) have demonstrated remarkable generalization and instruction- folliooin capabilities with instruction tuning. The advancements LLMs and instruction tuning have led to the development of Large Vision-Language Models (LVLMs). However, the competency of the LLMs and instruction tuning have been less explored in the molecular domain. Thus, propose LLaMo: Large Language Model-based Molecular graph assistant, which is an end-to- end trained large molecular graph language model. To bridge the discrepancy between the language and graph modalities, we present the multi-level graph projector that transforms graph representations into graph tokens by abstracting the output representations of each GNN layer and motif representations with the cross-attention mechanism. We also introduce machine-generalle instruction data to instruction-tus general- purppss molecule and language understanding. Our extensive experiments demonstrate that LLaMo shows the best performance on diverse tasks, such as molecular description generation, property prediction, and IUPAC name prediction. The code of LLaMo is available at hhttt::://// / thub. www.hu....oomoaaoaaaaaaaeeee.e.e www.///vaab/LLaa...
molecular graph the large molecular graph-language model for
Introduction
In recent years, molecular machine learning 2 has received significant attention, addressing diverse tasks in the chemical domain. The predominant approach for molecular tasks is graph machine learning [5] 6 7 that leverages the molecular graph structure, which is a natural and expressive representation molecules. Although basph methods have successfully represented molecules, they have limited interpretability and incompatibility to solve multi-modie molecular tasks dealing with pairs of texts and molecules. address these issues, recent works 4 train both a language model and graph encoder with cross modal contrastive learning. However, the models trained with cross-modal contrastive learning are insufficient to perform open-ended molecule-to-text generation tasks 3 which are more applicable to practical use.
Large Language Models (LLMs) 2
have shown impressive progress and accomplished human-like open-ended text generation with the power of billions of parameters. To leverage the instruction-following capability of LLMs, many works employ instruction-tuning approaches [2] 13
14 for general-purpose language models. Motivated by the development of LLMs and instruction tuning, Large Languuge Models (LVLMs) have recently been explored and achieved success on image comprehension and image-to-ter generation tasks [10] [I] 15] [6] I7 [8). Despite the success of LLM-bbsed approaches on natural language processing and machine vision domains,
the research on the integration of language models and molecular graphs has been less studied due the lack of consideration of the architecture design of Large Molecular Graph-Language
Models (LMGLMM)) and the molecular graph instruction data.
"corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this paper, we propose LLaMo: Large Language Model-based Molecular graph assistant, which seamlessly integrates molecular graph encoder and large language model to enable the instruction- following response generation in molecular domain. Specifically, LLaMM consists of the molecular graph encoder, large language model, and multi-level graph projector that bridges the graph encoder and large language model. The multi-level graph projector abstracts the representation of each GNN layer and motif representation using a cross-attention mechanism, ensuring a thorough understanding of molecular structures. Furthermore, we introduce machine-generatee molecular graph instruction data through the pipeline to convert molecular descriptions and IUPAC names into a multi-turn conversation format. The generated instruction-follont following data enhances the model's ability to perform general-purposs purpose molecule and language understanding, bridging the gap between molecular graph analysis and language- based tasks. Our proposed LLaMo outperforms the LLM-based works such as GPT-4 across diverse tasks, including molecular description generation, property prediction, and IUPAC name prediction.
Our contributions are summarized as follows:
We propose LLaMo: Large Language Model-based Molecular graph assistant consisting of graph encoder, language model, and multi-level graph projector equipped with a multi-level graph projector that captures rich information of the graph structure at multiple levels.
We introduce GPT-4 generated molecular graph-text multi-turn conversation data to address the data scareity problem of molecule-text datasets and improve the cestuction-folloviii capabilities of large molecular graph-language model.
Our experiments demonstrate that LLaMo achieves the best performance on various tasks such as molecular description generation, property prediction, and IUPAC name prediction.
2
Related works
Molecular graph modeling. Molccular graphs serve as natural and expressive representation of molecules, effectively capturing the structural information. Graph neural networks [19 20, 21 22 are commonly utilized architectures for molecular graph representations. To learn graph neural networks with the limited molecular graph data [23). sulf-supervised learning has been explored. For example, various approaches [24] 251 have been developed to capture multi-level features of molecular graphs, such node-level masked atom modeling 24], motif- based self-supervii learning [25] 26], and
graph-level contrastive learning 27 28 With the advance of multi-modal large language models,
molecule-language tasks such as molecule-text retrieval g or molecule captioning 20) have recently drawn significant attention. Recent works 3
4 30 have attempted to enable language models to
understand molecular graphs. [30] treated nodes of molecular graphs tokens of language models. Some works have adopted GNN-based encoders, either by propagating their outputs to language models through MLP (4) or employing cross-modal projectors (3). However, these methods fail to consider molecular graphs at multiple levels and are hindered by inherent limitations of graph encoders, such as the over-smoothing problem [3T]. To address these challenges, we propose a novel architecture, LLaMo, which effectively propagates multi-level information of molecular graphs to language models.
Instruction tuning. Recent advancements of LLMs lead to extensive research on instruction funing, aimed improving the model's capability to follow human instructions 12
35 To has adopted datasets and integrated them with a new structure and template. On the other hand, recent studies [12 36| 37 on instruction tuning have collected data samples from strong LLMs like GPT-4 [TO]. These works first manually construct annotated seed instruction samples and expand them by prompting LLMs. As a result, several instruction-tun been proposed from the open- source LLMs, e.g., LLLMA (9] and shown generalizability across a wide range of instructions. More recently, those studies on instruction tuning have been expanded
32
33
construct high-quality instruction tuning data, a line of previous approaches 34
35
existing numan-annotale
LLMs [14 [6] 37 have
to visual instruction tuning in image [5 17 38 and video 39 40 domain to enable the model to understand the visual contents. Inspired by the instruction tuning for multi-modal LLMs in other domains, in this work, we study instruction tuning specifically for molecule graphs, which has been underexplored in the literature.

---

Input 2D
Learnable Query Tokens
Molecular Graph
pl
pili
pimoti
Cross Attention
Multi-Level
Graph Projector
Cross Attenition
ZEC
GNN Loyer
p
100
GNN Loyer
Instruction
Cross
(UICAE)
Could you provide description of this molecule?
pt
Input SMILES
C=-CCf-CC=CI (c|-D|jo-jojo
MLP
Large Language Model
Generate
The malecuie 2-hyoraxyearbaxyllte that
Figure I: Overall framework LLaMo. LLaMo consists of a graph neural network, multi-level graph projector, and large language model. It first encodes an input 2D molecular graph with the graph neural network and then converts the encoded graph into molecular graph tokens with the multi-level graph projector. Finally, the large language model generates the instruction-following response given the input SMILES, graph tokens, and the instruction.
LLaMo: Large Language Model-based Molecular Graph Assistant
The primary goal is to seamlessly integrate a molecular graph encoder and a Large Language Model (LLM) to generate instructionnfollooiing responses to the input texts and molecules. To achieve it, we propose LLaMo: Large Language Model-based Molecular graph assistant, a general- purpose Large Molecular Graph-Language Model (LMGLM) equipped with multi-level graph projector. Specifically, the proposed framework utilizes three input modalities: ID SMILES [41]. 2D molecular graph, and text (instruction). SMILES [41] is a 1D representation of a molecule, and a 2D molccular graph is processed by a GNN. The three input modalities are fed as a sequence of tokens and our LLaMo autoregressively generates text responses. Formally, given SMILES S, molecular graph tokens G, and text (instruction) the proposed method renders the response Y {y:l as:
p PYSS.G. = II p (y. G. T.yc)),
(1)
where Y<v indicates generated token sequences until i-th token.
3.1 Model Architecture
The overall architecture LLaMo illustrated Figure LLaMo consists of graph encoder, multi-level graph projector, and backbone large language model. The graph encoder g(-) takes 2D molecule graph an input and outputs their node representations multi-level graph projector ProjMc(-) (-) transforms the sequence of node representations into molecular tokens to align them with the LLM. Then, the LLM f(-) processes molecular and text tokens and provides a response in an autoregressive manner.
sequence of tokens. The
Graph encoder. We adopt Graph Neural Networks (GNNs) molecular graph encoder. Given the graph G. graph neural networks g (.) iteratively update node representation z( ( via the pessagge-asssigg framework. With the message- message-passing, L-layer GNN provides node representations (P) that express an I.-hop ego-graph given the node v as a center node. More details about graph neural networks are in the Appendix C
3

---

(n) I-layer
2-Layer
4-yyer
(d) 5-Layer
Figure 2: Node representations of graph encoder with 1,2,4,5 layers. As the number of layers increases, node representations collapse.
Multi-level graph projector. The goal of multi-level graph projector is align the graph encoder with the LLM by transforming set node representations z (1)
graph into a sequence of molecular graph tokens Hgraph- enables the language model to utilize graph information. In the literature, projectors
have been proposed mainly for Large Vision-Language Models (LVLMs) 17
T8 38
43 They
are usually implemented using a linear projection [38] or an abstraction of visual features [7 42)
which outputs of the final layer of visual encoder given input image. Analogously, the projector for large molecular high-level node representations from the pre-trained graph encoder, which is formulated
design models with linear projection or an abstraction of
H graph Proj srapp)
where ((L) g (9),
where graph
(L) 20
R Vixx(() the concatenation of node representation (L)
R.((4) from L-th layer GNN and Proj (-) is the projector.
However, we observe that the high-level representation is not effective in capturing the local infor- mation due to the over-smoothing problem (31] which means that the node representations become indistinguishable, as the number of layers in the GNN increases. Figure depicts node represen- tations (yellow dots) of graph encoder with 1 1.2.4.5 layers on one molecular graph sample. (More samples are in Appendix ! As mentioned above, node representations become over-smoothed as the number of layers increases, leading to nearly identical node representations in the final layer. Consequently, conventional projectors relying on high-level node representations have limited capability to preserve the detailed or local information of molecular graphs. Moreover, many tasks require multi-scale information, including atom, atomic group, and molecule levels. Hence, the projector that solely utilizes features from the top layer is suboptimal for the tasks.
Motivated by the observations, we propose a novel multi-level graph projector generate graph tokens that contain richer information reflecting the graph structure at multiple levels. The multi-level graph projector ProjNn (-) is formulated as
H graph ProjMG ((
(1 (0) )
) where
{
Z graph
i-
The method captures multi- graph information by leveraging node representations from all layers of GNN. To handle an arbitrary number of nodes, yielding a variable length |V] L features, we adopt the cross-attention with learnable tokens ((f) pi .p......pp. p!") Rexd for 0,... L,
where i is the number of learnable prompts. Here, [:: indicates the concatenation operation. The learnable tokens aggregate 1-th layer GNN representations into fixed number of tokens as:
p
Atm (1)
(p())
bxd
graph
graph
where Attn (Q, (, K,K) 1) is the attention operation with query Q, key K, and value V.
For more detailed representations of the input molecule, LLaMo also has learnable tokens P (moxif) use the functional groups as motifs, which are the statistically
for motif-level representations. important subgraphs in the molecular graphs. To construct functional group representations Zra. we initially identify functional groups, following 23). Then, we vectorize the main characteristics of each functional group, which is represented ZFG. Finally, the functional group representations ZPg

---

Trainable
Frozen
Zunt
Zec
Graph Encoder
Multi-level Graph Projector
Graph Encoder
Multi-level
Graph Projector
Could you provide desesiption
C1=C21-CC-C1C [eteoJ(o-Jooo
What the IUPAC
nome of
Large Language Model
Large Language Model
LORA
The motecale a hydracyeoraauukke that is..
hyydrr 2-4 hytrowyhhenslloeetr
(a) Stage graph-language alignment
(b) Stage 2: instruction-tunine tunnng
Figure 3: Two-stage training pipeline. Stage involves training the graph encoder, and stage 2 entails fine-tuning the L.M using LoRA. In both stages, the multi-level graph projector continuously trained. All training processes are performed by generating the instruction-following response.
are constructed by concatenating all individual functional group representations Zga,i, which is formulated ZFG |MFG.0..... ZPG, ZPG.M], M.. where M indicates the number of the functional groups in the given molecular graph. Given the functional group representations ZRG of the input molecule, we obtain P (moxif] = Attn (moiff) (P(matif), ZFG, ZFG) with the cross-attention.
Then, obtain the graph-level representations by applying MLP to the multi-hop and motif-level representations, ie., [Po. ((L), (mmoiff) It formulated
Hgraph MLP
p())
((moir) 1) (1+22x
(5)
where H graghh is sequence of graph tokens to be fed into the L.M.
Large language models. After constructing tokens for encoding molecular graphs, LLaMo fuses SMILES representation, graph, and text tokens and puts them into the language model to generate an instruction-following response.
3.2 Training LLaMo
Similar to most LVLMs 18
we train LLaMMo in the two-stage pipeline: (1) pre-training for molecular graph-language alignment, and (2) instruction-tuning end-to-end in Figure 3
38 42
Stage 1. Pre-training for molecular graph-language alignment. The first stage focuses on the alignment between the graph encoder and large language model by learning our multi-level graph projector. In this stage, with the LLM frozen, we train the multi-level graph projector and the graph encoder by generating molecule descriptions. For training, we molecule-description pair dataset (e.g., PubChem (44) consisting of ID SMILES representation of molecule and molecular graph and corresponding description.
Stage 2. Instruction-tuning end-to-end. In the second stage, we train the LLM to enhance the ciiruction-follwinng capabilities and enable deeper understanding of molecular graphs. In this stage, we freeze the graph encoder and train both the multi-level graph projector and the LLM. Since it is too expensive train the full LLM, employ LoRA 455 adapt LLM to the data. For instruction-folloviii
we use the GPT-generated instruction followinn multi-turn conversation dataset, which will be introduced in Section In addition to our generated cnsruuctionfoolowiing dataset, use diverse set of datasets with various instructions: molecule description generation, molecular property prediction, IUPAC name generation, forward reaction prediction, and retrosynthesis datasets.
4 GPT-assisted Molecular Graph Instruction Data Generation
Instruction data are essential for improving the instruction-follooinn capabilities -.ased models. Despite active research on instruction-tuning, the instruction-followingg data for molecular
5

---

Table I: One example to illustrate the instruction-folloiing data. The top block shows the contexts such as SMILES, description, and IUPAC name used to prompt GPT, and the bottom block shows the response of GPT-4.
Context type 1: SMILES representation
=C(CCC=-0)0)
Context type 2: Description
The molecule is polyene antibiotic that is TMC-IA in which the 2,4- 4--dimthyloott---onee group has been replaced by
1155-4,6-dimettyldeeca d.4-oic group. TMC-IC antitumour antibiotic isolated from Streptomyces sp. <-230..<omiited>
Context type 3: IUPAC name
Response: Multi-turn conversation
Question:
What is the IUPAC name of the molecule you are analyzing?
Answer:
The IUPAC name of molecule is (LADDA/SSSASA-444ydyyy(000000:2: 2amethyyllea-2244ddieaaaaa
Question:
Can you identify the type of acid or base this molecule can act in reaction?
Answer:
This molecule can act as weak acid due to the presence the carboxylic acid group.
graphs have been less explored in the literature since annotations require expertise. To alleviate the need for expertise and minimize the manual efforts, we utilize GPT-4 [T0] to generate molccular graph-text cestruction-folloiing data using graph-text pair datasets.
construct multi-tur conversation datasets, which more diverse and effective for instruction tuning compared to simple pairs of questions and answers. We leverage GPT-4 to generate multi-turn conversations with tailored contexts/prompts that consist of two representations for molecular graphs and description: (i) SMILES representation that describes the chemical structures with special strings, (ii) captions that explain the molecule, and (iii) IUPAC name that describes the molecule based on its chemical composition and structure. These representations enable the GPT-4, which inherently lacks in-depth molecular knowledge, understand and generate a diverse and high-quality set of examples. One example of the input representations is shown in the top block of Table
Inspired by previous results |46
Specifically, we generate the multi-turn conversation data in three steps: 1) select exemplar con- versations among machine-generatte instruction-tuning data, generate multi-turn conversations via in-context learning with the exemplar conversations as prompts, and 3) filter out incomplete conversations and those with many turns. In the first step, we generate exemplars with brief human-written instruction as shown in Appendix H However, we found that GPT-4 frequently fails to generate complete multi-turn conversations without the exemplars. To address this issue, we generate the instruction data with in-context learning. We sample exemplars from small set of complete conversations generated by GPT-4 in the first step. Then, GPT-4 generates the complete multi-turn conversation data for the instruction tuning guided by the prompts wrapped with the generated exemplars. To validate the quality of the generated conversation, we sample 500 subsets generated via in-context learning. We find that some conversations consisting of large number of turns are prone to generating incomplete and inaccurate outputs. So, we filter out incomplete conversations and those with many turns. The example of the generated multi-turn conversation is in the bottom block of T total, we generate 12K unique molecular graph-language cetrucctionfollowwiin
samples using PubChem324k dataset 3 44.

---

Table 2: Performance (%) of generalist models three tasks: molecule description generation, IUPAC prediction, and property prediction. Mol. Inst. tuned denotes the molecular instruction-tun sennnid model. The result is not available since LLaMA2 fails generating numerical outputs. denotes the experimental results drawn from Mol-Instruction (88.
Mol. Inst. tuned
Molecule Description BLEU (1)
IUPAC Prediction
Property pred.
Model
LLM
METEOR (1)
BLEU (f)
METEOR (1)
MAE (1)
GPT-3.5
GPT-3.5
2.2
19.7
33.4
52.6
0.075
GPT:3.5 (ICL)
GPT-3.5
28.4
56.1
50.3
62.0
0.028
GPT-4
GPT-4
0.8
16.7
29.0
48.1
0.098
GPT-4 (ICL)
GPT-4
27.0
52.2
51.8
0.019
Galacticat
Galactica
0.8
6.5
0.568
Text+Chem 15)
T5-Base
3.6
13,9
ILaMA2
LLaMA2-7B
0.0
14,1
0.0
0.4
N/A"
Mol-Instructionst
LLaMA2-7B
14.3
25.4
0.013
LLaMo (Ours)
LLaMA2-7B
38,9
67.1
56.0
73.2
0.006
Experiments
5.1 Experimental Settings
Benchmarks. To evaluate the efficacy of the proposed method, we evaluate the model for three tasks such 1) molecule description generation, 2) IUPAC name prediction, 3) property prediction (re- gression). We conducted experiments under two major settings: generalist and specialist models. In the generalist setting, one model handles all three tasks, whereas in the specialist setting, train model for each downstream task. More details about benchmarks are in Appendix G
Implementation details. For the generalist models, Llama 1ama-2-750 2--b-cchtt [9] for fair comparison with Mol-Instructions [48]. For the specialist models, train our I.LaMo with Galactica 1.3B for fair comparison with MolCA [3]. To train the generalist variant of LLaMo, we use training split of molecular description generation dataset of Mol-Instruction 48 in stage 1. In stage 2, the model is instruction-tunet with training split of description generation, property prediction, forward reaction, and retrosynthesis instruction dataset of Mol-Instruction [48). IUPAC name prediction from R and our GPT-generated instruction-followin data. To train the specialist variant of LLaMo, we follow MolCA B3 to train the model with pretraining split of PubChem324kV2 in the stage 1 phase and fine-tune the model for each specific downstream task in the stage 2. We adopt long training schedule (epoch pre-training, epoch 3 instruction tuning) for the final models. For analysis, we use short training schedule (epoch pre-training, epoch instruction tuning). For further implementation details, refer to Appendix E.
train
LLaMo based
Baselines. For the generalist models, we compare our LLaMo with (1) LLM-based generalist models including Galactica (2). LLaMA2-7B 9 GPT-3.5, and GPT-4, (2) Molecule-specializee LLM such as Text+Chem T5 [49], and (3) Molecule instruction -etuuettio-tueedd generalist model such as Mol- Instructions [48]. Since GPT-3.5 and GPT-4 have difficulty in solving the tasks without in-context learning, we additionally measure the performance of GPT-3.5 and GPT-4 with 4-shot in-context learning, which GPT-3.55 (ICL) and GPT-4 (ICL). For the specialist models,
single-task specialist molecule-language models as baselines, including MolT5 (29]. MoMu 4 and MolCA B
5.2 Experimental Results
Generalist models. We provide the experimental results of generalist models in molecular description generation, IUPAC name generation, and property prediction tasks. Our LLaMo is built on LLaMA- 7B and it is fine-tuned by our instruction-tuma certiiontttinngg method. Table 2 shows that our LLaMo achieves the best performance all three tasks. In comparison GPT-4 (ICL), which is GPT-4 with in-context- learning, LLaMo shows a performance improvement of 11.9 in BLEU-4 and 14.9 in METEOR for molecular description generation. Furthermore, LLaMMo outperforms Mol-Innstuutions, an instruction- tuned model with molecular data, by substantial performance gain of 41.7 in METEOR for molecular description generation and a 0.007 performance gain in MAE on the property prediction task. More experimental results on forward reaction prediction and retrosynthesis are in Appendix D
7

---

Table 3: Performance (%) of specialist models molecule captioning with the PubChem324k and ChEBI-20 datasets and IUPAC name prediction. Full denotes full parameter fine-tuning.
PubChem324RV2
ChEBI-20
IUPAC METEOR
Model
LLM
Training type
BLEU METEOR
BLEU
METEOR
MofTS-Small
T5-Small
full ft
8.5
18.5
43.6
55.1
42.5
MofTS-Base
T5-Base
full ft
20.9
35.6
45.7
56.9
53.2
MofgS-Large
T5- T5-Large
full ft
22.2
36.6
50.8
61.4
58.5
MoMu-Smin
TS-Small
ft
21.8
44.5
57.6
MoMu-Base
T5-Base
ft
34.2
46.2
57.6
Moru-Large
T5-Large
ft
36.2
51.5
59.7
MolCA, GalaCyzsM
Galactica- 125M
ft 30.3
41.6
52.6
63.6
71.8
MoICA, Galac
Galactica-
LoRA
45.6
53.1
72.1
LLaMo (Ours)
Galactica-
LoRA
34.4
48.0
54.8
66.6
73.4
Table 4: Performance comparison according to the projector type.
Molecule description
IUPAC prediction BLEU (1)
Property QA MAE (J)
Projector
BLEU (1)
METEOR (t)
METEOR (1)
w/o Graph
26.1
56.6
36.3
62.2
0.013
MLP (w/ low-level)
32.4
62.
42.2
68.4
0.009
MLP (w/ high-level)
33.8
63.4
45.5
67.4
0.008
MLP (w/ concat)
34.8
64.1
47.1
70.2
0.007
Resampler
34.4
62.8
43.4
65.2
0.009
MGProj (w/o motif)
36.1
65.3
48.8
69.8
0.008
MGProj (Ours)
37.8
66.1
49.6
70.9
0.007
Specialist models. We also evaluate the performance of specialist models to validate the effectiveness of our LLaMo, which is individually fine-tuned for each dataset. Table 3 demonstrates that our I.LaMo consistently achieves the best performance across all tasks and datasets. Specifically, L.LaMo outperforms the second-best model MolCA with Galactica 1.3B, by 4. in BLEU-score and 2.4 in METEOR the PubChem324KV dataset. For IUPAC name prediction, LLaMo also shows superior performance, achieving a METEOR score of 73.4, which surpasses MolCA with Galactica 1.3B by margin of 1.3 points. This experimental result indicates that our LLaMo is consistently effective in comprehending molecular graphs based on diverse large language models.
5.3 Analysis
Impact of multi-level graph projector. validate the effectiveness our multi-level graph projector, compare the performance of the multi-level graph projectors (denoted by MGProj) with other projectors in Table including two widely-used projectors such as MLPs and resamplers. Additionally, measure the performance of the base model without graph (and a projector) denoted as w/o Graph for the ablation study. MLP (w/ low-level) and MLP (w/ high-level) denote the MLP
projectors where the input is low-level representation Z graph and high-level representation z!!) respectively. MIP (w/ concat) indicates the MLP projector with the concatenated representations of all GNN layers as an input. Resampler denotes the cross-attention based resampler projector designed in Qwen- VL [50]. MGProj (w/o motif) and MGProj without and with motif tokens ((moiit)
multi-level graph projector
Table shows that our multi-level graph projector (MGProj) achieves the best performance across all three tasks. Specifically, the lulti-level graph projector achieves 49,6 BLEU and 70.9 METEOR scores with a significant improvement compared MLP projectors the IUPAC prediction task. These experimental results demonstrate that our multi-level graph projector is more effective than conventional projectors by capturing multi-scale information, including atom, atomie group, and molecule- information.

---

Table 5: Ablation studies training stage and GPT-gennrated instruction tuning data.
Molecule deseription
IUPAC prediction
Property QA MAE (J)
Stage
Stage
GPI-generated data
BLEU (t) METEOR (t)
BLEU (t)
METEOR (t)
0.0
14.1
0.0
0.4
N/A
35.5
64.8
7.3
16.9
N/A
37.2
65.1
47.5
70.2
0.007
37.8
66.1
49.6
70.9
0.007
Table 6: Performance comparison according to the training type.
Molecule description
IUPAC prediction
Property QA MAE (U)
Training type
BLEU (t)
METEOR (1)
BLEU (t)
METEOR (1)
w/o inst. tuning (Stage
35.5
64.8
7.3
16.9
N/A
Multi-task
36.9
49.4
70.5
0.218
Instruction- instruction-tuning (Ours)
37.8
66.1
49.6
70.9
0.007
Coarse-grained (high-level)
Fine-grained (low-level)
Lager
Layer
The molecule is prostaglandin carboxylic acid anion that the conjugate base 13, an-oillyyroonyrrttaaalannn FZalpha, obtained by deprotonation of the carbowy is conjugate base of 13,14- ditydrroxyrrottalandin F2alpha.
member of morpholines.
Figure 4: Visualization of attention maps for samples with coarse-grained caption (left) and fine- grained caption (right). The attention scores of high-level features are relatively high when generating coare-graiined captions, whereas those of low-level features are high for fine-grained captions.
Impact of GPT-generated instruction-tuning data. In Table 5 provide the ablation studies of each training stage and our GPT-generated instruction dataset. The experimental results reveal that the instruction tuning with generated multi-turn conversation data enhances the performance of LLaMo compared to the models trained via one or two-stage training without our GPT GPT-generated instruction data. This indicates that instruction tuning with GPT-generated multi-turn conversation data provides the model with more detailed and instruction-followii guidance.
Instruction tuning multi-task learning. Table 6 shows the advantages of instruction-tuning based on task instructions compared to multi-task learning using the simple task identifier. We use the task name simple task identifier for multi-task learning. From the table, the model without instruction tuning (Stage 1) achieves BLUE score of 35.5 and 7.3 on molecule description and IUPAC prediction tasks, respectively. The multi-task learning approach improves the scores to 36.9 for molecule description and 49.4 for IUPAC prediction. However, the intructiiontuuingg method demonstrated the most significant enhancement, achieving the highest scores of 37.8 for molecule description and 49.6 for IUPAC prediction. These results indicate that instruction tuning outperforms both the baseline and multi-task learning methods, suggesting its effectiveness in improving model performance on general-purpose training.
Visualization of attention maps. We visualize the attention map explore the effect of the multi- level graph projector in Figure 4 The figure illustrates the attention maps of graph tokens for generating coarse-grained (left) and fine-grained (right) descriptions. Interestingly, the attention scores the low-level are relatively higher than the high-level when generating fine-grained captions, whereas the attention value of the high levels is high when generating coarse- coarse-grained captions. This indicates that both low and high-level graph structural information is crucial in expressing the molecules, and the attention matrix is adaptive the caption types.
Qualitative analysis. Figure shows GT description and the molecular descriptions generated by the model with and without the molecular graph (SMILES representation only). As shown in the

---

GT: hy-hyyoooyiinoleiccc acid
LLaMo w/o graph: 1-hydroxy-2-oxo-+
LlaMo w/ graph: 13- 13-hydroxyoctadec O-enoic
cxocyclohexane-1. 1,22-
kmmamm
Invalid
H
GT: The molecule is an omega: hydroxy fatty acid anoin that the conjugate base of 18-hydroxylinoleic acid, obtained by deprotonation the carboxy group; major species at polyunsaturated fatty acid anion and omega hydroxy: long-chain fatty acid anion. It is conjugate base of 18- hydroxylinoleic acid.
LLaMo w/ graph: The molecule is hydroxy-long-cainn fatty acid anion that is the conjugate base of 13- (edrroooctaeec---nnaee acid, obtained by deprotonation of the carboxy group; major species at pH 7.3. is vydroxy-long-chee fatty acid anion and 13-hydroxy-omega long-chain fatty.
w/o graph: molecule 1-hydroxx-2- 2-ooy----xo------- oxxcyclohhxxaee 2-2-iol that is the 1-hydroxy-2 2- 2-oxo- which
exwcyclohexane-1.222 the hydroxy group position is 1-hydroxy-2-owe 2-oxx-4-oxxxr 4-oxccy.
Figure 5: An example of molecular description generation results of LLaMo w/o graph and LLaMo w/ graph given the molecule ("C(CCC/C=C) CCCCC///CCCCCCCCCCCCCCCCCCCCCCCC (CCCCCCCCCCCCCCCCCOCC0CCCCC.. In the top box, the molecular graphs of IUPAC and functional groups in the descriptions are depicted.
pyrazine, secondary carboxamide, tertiary carboxamide
LLaMo MGProj: pyridine, carboxamide
LLaMo MGProj: pyrazine, secondary carboxamide, tertiary carboxamide
LLaMo w/o MGProj:
The molecule is
LLaMo w/ MGProj: The molecule is member of pyrazines, a secondary carboxamide and tertiary carboxamide.
The molecule member of pyrazines, secondary carboxamide and tertiary carboxamide.
pyridinecarboxamide. It is member of pyridines and carboxamide.
Figure 6: An example of molecular description generation results of LLaMo w/o MGProj and LLaMo w/ MGProj given the molecule ("CIC@ @ H1 CNCC-OCCCCCCCCCCCCCNCCC--C C3) @ @HH]ICCC][CC (1 In the top box, the molecular graphs of IUPAC and func- tional groups in the descriptions depicted.
figure, LLaMo with graph denoted as LLaMo w/ graph generates better molecular description compared to LLaMo without a graph (LLaMo w/o graph). The GT description explains the molecule with 'omega-hydroxy-bon hyyrooo--oongchaaii fatty acid anion'. Since LLaMo w/o graph does not have any graph structural information, it fails generate a description with invalid IUPAC name ('1-hydroxy-2- oxo-4-ox 4x0-4-xxocycloexxaneee dio'",,,"), while LLaMo w/ graph generates a more related description with hyyrrox-long-chale fatty acid anion'. addition, know that LLaMo w/ graph accurately predicts the long-chain structure of the molecule.
We also perform another qualitative analysis by comparing molecular descriptions generated from the model with and without our Multi-level Graph Projector (MGProj) denoted by LLaMo w/ MG Proj and LLaMo w/o MGProj in Figure The figure shows that the multi-level graph projector plays a crucial role in capturing the details of the molecule. Compared to LLaMo w/o MGProj generating 'pyridine", the model with MGProj generates accurate molecular description including "pyrazine' same as GT description. This demonstrates that the multi-level graph projector is effective molecule understanding and generation by preserving the molecular graph structural information.
Conclusion
We propose LLaMo:
Large Language Model-based Molecular graph assitant, an end-to-end trained Large Molecular Graph Language Model, to perform various molecule-related tasks with a single model. For the projector, we newly introduce a multi-level graph projector, which addresses the over-smoothing problem of the graph encoder and captures multi-hop graph information. We also present machine- -eeeratte instucction-oollwinn data in the form of multi-turn conversations to improve the instruction- instruction-following following capabilities of the large language model.
10

---

Acknowledgement
This work was partly supported by ICT Creative Consilience Program through the Institute of Information & Communications Technology Planning & Evaluation (ITTP) (ITP-2002-RSS20220 11201819, 10%) and the National Research Foundation of Korea (NRF) (NRF-2033RA22220053) 45%) grant funded by the Korea government (MSIT), and the Virtual Engineering Platform Project (Grant No. P0022336, 45%), funded by the Ministry of Trade, Industry & Energy (MoTIE, South Korea), We appreciate Dr. Jaesung Kwak for valuable comments and discussions.
References
[1] Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nat. Commun., 13(1):862, 2022.
[2] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshom, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. 0tt::2211.9000. 2022.
[3] Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Can, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. In EMNLP, 2023.
[4] Bing Su, Dazhao Du, Zhao Yang. Yujic Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji- Ri-Rong Wen. A molecular multimodal foundation model associating molecule graphs with natural language. mmXiv:220,00588/ 2022.
[5] David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Suzuk, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In NeurlPS, 2015.
[6] Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. In ICML workshop. 2018
Kenneth Atz, Francesca Grisoni, and Gisbert Schneider. Geometric deep learning on molecular representa- tions. Nat. Mach. Intell., 3(12): (012))102311000 023--102,, 2021.
[8] Carl Edwards, ChangXiang Zhai, and Heng Ji. Text2mol: Cross-modal molecule retrieval with natural language queries. In EMNLP, 2021.
[9] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosalc, et al. Llama 2: Open foundation and fine-tuned chat models. arXin:2307.09288, 2023.
[10] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Nlee Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. 3kXi:2300300874,, 2023.
[11] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wa, Jean- Cean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv:231.11180,, 2023.
[12] Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In ACL, 2023.
[13] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Scaling instruction-finttueed language models. JMLR, 25(70):1-3,, 2024.
[14] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang. and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-foolloiing llama model, 2023.
[15] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter- visual instruction model. arXiv:230..15110 2023.

---

[16] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. In ICLR, 2024.
[17] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurlPS, 2023.
[18] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024.
[19] Thomas N Kipf and Max Welling. supervise classification with graph convolutional networks. In ICLR, 2017.
[20] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et al. Graph attention networks. In ICLR, 2018.
[21] Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR,, 2019.
[22] Jinyoung Park. Sungdong Yoo, Jihwan Park, and Hyunwoo J Kim. Deformable graph convolutional networks. In AAAI, 2022.
[23] Zewei Ji, Runhan Shi, Jiarui Lu, Fang Li, and Yang Yang. Relmole: molccular representation learning based on two-level graph similarities. J. Chem. Inf. Model., 62(22):5311-5372, 2022.
[24] Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z Li. Mole-bert: Rethinking pre-traiming graph neural networks for molecules. ICLR, 2023.
[25] Zaixi Zhang, Qi Liu, Hao Wang. Chengqiang Lu, and Chee- Kong Lee. Motif-based graph supe--sue learning for molecular property prediction. In NewrIPS, 2021.
[26] Xuan Zang, Xianbing Zhao, and Buzhou Tang. Hicrarchical molecular graph self-supervised learning for property prediction. Commun. Chem., 6( 1):34, 2023.
[27] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurIPS, 2020.
[28] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nat. Mach. Intell., 4(3):279-287, 2022.
[29] Carl Edwards. Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation between molecules and natural language. In EMNLP, 2022.
[30] Haiteng Zhao, Shengchao Liu, Ma Chang. Hamnan Xu, Jie Fu, Zhihong Deng, Lingpeng Kong, and Qi Liu. Gimlet: A unified graph-text model instruction-basse molecule zero-shot learning. In NeurIPS, 2023.
[31] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In AAAI, 2018.
[32] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In ICLR, 2022.
[33] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
[34] Shayne Longpre, Le Hou, Tu Vu, Alhert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In JCML, 2023.
[35] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Amaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In ICLR, 2021.
[36] Xue Fuzhao, Jain Kabir, Hitesh Shah Mahir, Zheng Zangwei, and You Yang. Instruction in the wild: A user-based instruction dataset. http://hhhbbccomo/uuuuuu/u////tt 2023.
12

---

[37] Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023.
[38] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[39] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed vidco understanding via large vision and language models. In ACI, 2024.
[40] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv:2305.06335 2023.
[41] David Weininger. Smiles, chemical language and information system. 1. introduction methodology and encoding rules, Chem. Inf. Comput. 28(1):31-36, 1988.
[42] Qinghao Yc, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv:2304. 14178, 2023.
[43] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: enhance projector for multimodal Ilm. In CVPR, 2024.
[44] Sunghwan Kim, Jie Chen. Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He. Qingliang Li, Benjamin Shoemaker, Paul A Thiessen, Bo Yu, et al. Pubchem in 2021: new data content and improved web interfaces. Nucleic Acids Res., 49(DI):D1388 1838818,,,,, 2021.
[45] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[46] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Qiao, and Ping Luo. Lylm-chub: A comprehensive evaluation benchmark for large rision-languuge models. arXiv:2306. 09265, 2023.
[47] Huayang Li, Siheng Li, Deng Cai, Longyue Wang. Lemao Liu, Taro Watanabe, Yujiu Yang, and Shuming 2024.
Shi. Textbind: Multi-turn interleaved multimodal instruction instruction-following following. In
[48] Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions: A large-scale biomolecular instruction dataset for large language models. In ICLR, 2024.
[49] Dimitrios Christofidellis, Giorgio Giannone, Jannis Born, Ole Winther, Teodoro Laino, and Matteo Manica. Unifying molecular and textual representations via multi-task language modelling. In ICML, 2023.
[50] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilitics. arXiv:2308. 12966, 2023.
[51] Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data, In EMNLP, 2023.
[52] Team GLM. Aohan Zeng, Bin Xu, Bowen Wang. Chenhui Zhang, Da Yin. Diego Rojas. Guanyu Feng. Hanlin Zhao, Hanyu Lai, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv:2406. /2793, 2024.
[53] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. 2rXiv:2302.
2023.
[54] He Cao, Zijing Liu, Xingyu Lu, Yuan Yao, and Yu Li. Instructmol: Multi-modal integration for building versatile and reliable molecular assistant in drug discovery. arXiv:2311.16208, 2023.
[55] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- perfoorrfomaccc deep learning library. In NewrIPS, 2019.
[56] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. In ICLR workshop, 2019.
13

---

[57] Thomas Wolf, Lysandre Debut, Victor Sanh. Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-ar natural language processing. In EMNLP demo, 2020.
[58] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-aa parameter-efficient fine-tuning methods, 2022.
[59] Shengding Hu, Ning Ding, Weilin Zhao, Xingtai Lv, Zhen Zhang, Zhiyuan Liu, and Maosong Sun. Opendelta: A plug and-play library parameter-efficiin adaptation of pre-trained models. In ACL, 2023.
[60] Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang. Chaowei Xiao, and Animashree Anandkumar. Multi-modal molecule structure-text model for text-based retrieval and editing. Nat. Mach. Intell., 5(12):1447-1455 2023.
[61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[62]
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In NeurlPS, 2020.
[63] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[64] Zhenqin Wu, Bharath Ramsundar, Evan N Feinherg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu. Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chem. Sci., 9(2):513-553 5300, 2018.
[65] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
[66] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In workshop. 2005.
[67] Taffee Tanimoto. Elementary mathematical theory of classification and prediction. International Business Machines Corp., 1958.
[68] Harry L Morgan. The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service. J. Chem. Doc., 5(2):107-113, 1965.
[69] Zequn Liu, Wei Zhang, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Ming Zhang, and Tie-Yan Liu. Molxpt: Wrapping molecules with text for generative pre-training. In ACI, 2023.
[70] Yizhen Luo, Kai Yang, Massimo Hong, Xingyi Liu, and Zaiqing Nic. Molfm: A multimodal molecular foundation model. arXiv: 2023.
[71] Pengfei Liu, Yiming Ren, Jun Tao, and Zhixiang Ren. Git-mol: A multi-modal large language model for molecular science with graph, image, and text. Comput. Biol. Med., 171:108073, 2024.
[72] Janna Hastings, Gareth Owen, Adriano Dekker, Marcus Ennis, Namrata Kale, Venkatesh Muthukrishnan, Steve Turner, Neil Swainston, Pedro Mendes, and Christoph Steinbeck. Chebi in 2016: Improved services and an expanding collection of metabolites. Nucleic Acids Res., 41D1))212144-1211, 2016.
[73] Henri A Favre and Warren H Powell. Nomenclature of organic chemistry: IUPAC recommendations and preferred names 2013, Royal Society of Chemistry, 2013.
[74] Jin-Mao Wei, Jie Yuan, Qing-Hua Hu, and Shu-Qin Wang. A novel measure for evaluating classifiers. Expert Syst. Appl., 37(5):799-33899 2010.
[75] Jicyu Lu and Yingkai Zhang. Unified deep learning model for multitask reaction predictions with explanation. J. Chem. Inf. Model, 62(6):137--1387, 2022. 2022.
14

---

The appendix is organized into the following sections.
Appendix
Limitations
Appendix
Broader Impacts
Appendix
Explanation on Graph Neural Networks
Appendix
Additional Experimental Results
Appendix
Detailed Experimental Settings
Appeedix
E.1
Implementation Details
Appendix
Metries
Appendix
Baselines
Appendix
Benchmarks
Appendix
Prompts
Appendix
Over-smoothing Problems
Appendix
More Qualitative Samples
A Limitations
Our LLaMo is built upon an LLM, e.g. LLaMA and Galactica, and is fine-tuned on molecule benchmark datasets by leveraging its pretrained knowledge. Given that LLMs pretrained using extensive web-crawled corpora, it is uncertain whether the data used for LLMs' pretraining and the test samples molecule benchmark datasets are mutually exclusive. This results implicit data leakage when fine-tuning and evaluating LLaMo on molecule benchmark datasets. Furthermore, LLMs inherently require large memory and computational costs and cause hallucination problems where the model generates incorrect but plausible text. Our LLaMo may inherit these LLMs' problems due to LLMs' powerful pre-traine pre-trained knowledge.
B Broader Impacts
We proposed the first molecular graph-based general-purposs model, LLaMo, which is widely applicable to various molecule tasks such molecule captioning, property prediction, and IUPAC naming. Our LLaMo itself does not have negative societal impacts. However, discussed above, since our model built upon an LLM, the model sometimes generates biased output concerning race, religion, culture, and gender, resulting in the misusage of our model. Also, training LLMs requires massive amounts CO2 emission promoting global warming.
C Explanation on Graph Neural Networks
Let G = (V,E, E, X) denote the input graph, where V, E are set of nodes and edges, respectively, and X indicates a set of input node features. The input feature of node v E is defined as x, and the edge between node and is represented with (u, v) E V x V. The neighbor set of node the input graph is denoted by N₁ = {ul (u, v) E &}. Given the graph, graph neural networks iteratively update node representation (t) IR ( via the following peessae--passing framework:
UPDATE (20 (1-1,, AGGREGATE" :.c=)))uENN)))
1= 1. (6)
(0)
where 2, X., in that the node representation of (-thh layer is input node features. AGGREGATE function aggregates the representations of the neighbor set with a particular function. UPDATE
function is designed to update node reprsentation 2!! (f-1) with the aggregated information produced by AGGREGATE (.). With the pessage-ppassigg I-llayer GNN provides node representations 2!
that express an L-hop egograph given the node center node. In this paper, a pre-trained
15

---

Table 7: Performance chemical reaction tasks, including forward reaction prediction and retrosyn- thesis. denotes the model fine-tuned with task-specific instruction data.
Model
Exactt
BLEUT
evvenshtiin
RDK FTS?
MACCS FIST
Morgan FTS*
Validiyt
Forward Reuction Prediction
Alpaca
[0]
0.138
Baize! 50
0,900
0.044
41.500
0.904
0.025
0.009
0.097
CharGI.M
40.008
0.044
LLaMA
53
0.000
0.020
42,002
0.001
0,0012
0.001
0.039
Vicuna
0.016
0.006
0.059
[LaMA"
53
0.012
0.804
29.947
0.499
0.649
0.407
1.000
InstrccMollGi
0.153
0.906
20.155
0.519
0.717
0.457
1.000
InstrcctMol-GS
S
0,536
0.967
10.851
0,776
0,878
0.741
1,000
LLaMM (Ours)
0.584
0.894
6.162
0.857
0.918
0.841
0.938
Retrasyuthesis
Alpaca! 14
0.000
0.063
46.915
0.005
0.023
0.007
0.160
Baizel
0.095
44,714
0.025
0.050
0.023
0.112
ChatGLM
52
0.000
0.117
48.365
0,056
0.075
0.043
0.046
LLaMA!
53
Vicunal
37
0,000
0.057
46.877
0,025
0.030
0.021
0.017
LLaMA" 33
0.294
0.106
Mol-Insiruction
48
0.000
0.705
31,227
0.283
0.487
0.230
1.000
InstructMol-Q G
54
0.114
0.586
21.271
0.422
0.523
0.285
1.000
InstructMol-C
0.852
LLaMo (Ours)
0.341
0.830
12.263
0.793
0.868
0.750
0.954
GIN 21
with 5 layers for the graph encoder, which has widely been applied to molecule graph understanding tasks 44. Formally, GIN updates node representations z!! with the following equation:
2 = MLP ((++ 2
111)
Σ NEN.
2 1
where (1++) is a learnable parameter or fixed scalar. The aggregate function AGGREGATE) (.) is the sum operation and the MLP function used for the combine function UPDATE (-).
D Additional Experimental Results
We also conduct experiments to validate the effectiveness of our LLaMoo on chemical reaction prediction, such forward reaction prediction and retrosynthesis tasks in Table The table demonstrates that our LLaMo still performs well on the chemical reaction tasks. LLaMo achieves the best performance on Levenshtein, RDK FTS, MACCS FTS, and Morgan FTS metrics across diverse baselines in all tasks, which indicates that LLaMo successfully comprehends molecular graph structure. We conjecture that our multi-level graph projector helps the large language model understand the molecular graph structure by representing multi-hop structural information to the molecular graph tokens.
E Detailed Experimental Settings
E.1
Implementation Details
Our code is implemented based on PyTorch (55 library. Also, we adopt PyTorch Geometric (PyG) [56]. and Huggingface transformers [57 to utilize the graph architectures and Large Language Models (LLMs). PEFT (58) and OpenDelta [59] libraries are used for paramneter-efficeent fine-tuning of LLMs, i.e., LoRA. We use LLaMA 2 chat 7B model [9] and Galactica 1.3B [2] as our base language model. We leverage GIN [21] with five layers initialized based on the MoleculeSTM graph encoder [60]. which is pre- trainad with text- contrastive learning [61] We use LoRA to train large language model stage 2. We use OGB (62) smiles2graph function, to convert SMILES representations to 2D graphs. Our experiments are run on 4 x A6000 GPUs or 4 V100 GPUs and x A6000 GPUs for LLaMA2 and Galactica, respectively. stage 1, the Adam 63 optimizer adapted with an initial learning rate of lc-4 (minimum learning rate is 1e-5 and warmup learning rate
16

---

is lc-6). The warmup step is 1,000 and the cosine scheduler is applied. In stage 2, the initial learning rate is set 5e-5 (minimum learning rate is 5e-6 and warmup learning rate is 5e-7).
To evaluate the efficacy of the proposed method, we fine-tune baseline models and evaluate them for three tasks such as 1) molecule description generation, 2) IUPAC name prediction, and 3) property question answering. We conducted experiments under two major settings: generalist and specialist models. In the generalist setting, one model handles all three tasks, whereas in the specialist setting, we train a model for each downstream task. For generalist experiments, we use datasets derived from PubChem and QM9. To be specific, for 'molecule description generation' and 'property prediction' we use the datasets derived from PubChem and QM9 of MoleculeNet [64] as in Mol-Instructions [48]. For IUPAC name prediction, a dataset derived from 3 used. To train the generalist variant of in stage 1. stage 2, the model is instruction-tume tuned with a training split of description generation and property prediction instruction dataset of Mol-Instructions, IUPAC name prediction from [3] and our GPT-generated instruction-followee
LLaMo, we use a training split of molecular description generation dataset of
data. For the evaluation of molecular description generation and property question answering tasks, use the test split of Mol-Instruction. molecular description generation and property prediction datasets, which are sampled from PubChem (44) and QM9 dataset of MoleculeNet [64], respectively. For ablations, we use a short training schedule (epoch 1 pre- training. epoch I instruction tuming). For the final models, we adopt long training schedule (epoch 1
pre-training, epoch instruction tuning).
For training the specialist variant of LLaMo, we follow MolCA to train the model with the pretrain split of PubChem324k in the stage phase and fine-tune the model for each specific downstream task in stage For the inference under specialist experiments, where a model is individually finetuned for a specific downstream task, we use test split of PubChem324k 3 ChEBI-20 (29) and IUPAC name prediction dataset from 3
E.2 Metries
We report BLEU (65] and METEOR [66] for the molecule description generation and IUPAC name prediction tasks. MAE is reported for property QA.
BLEU. The BLEU metric measures the quality of generated text by comparing n-gram sequence between the generated text and the reference text, which can be formulated as:
N
BLEU- = BP x exp(
Σ 7=
log Pn),
(8)
N
where N is the number of n-grams and Pr is the precision, i.e., the ratio of the number of n-grams in the generated text appearing in the reference text. The BLEU score also takes into account sequence length with Brevity Penalty (BP) as:
ifc> r BP (1(1-
if
where e and r are the lengths of generated and reference texts, respectively. This encourages the model to avoid generating short sequences. In our experiments, we use BLEU-4 as the default BLEU metric.
METEOR. The METEOR metric is proposed to consider both precision and recall between the generated text and the reference text, which is follows:
number of matched words
number of matched words
P number of words in generated text'
R number of words in reference text'
(10)
10PR
F == 9P + R'
(11)
number of chunks
Penalty 0.5
(12)
number of matched words
METEOR F. FF-1- (1- -Penalty),
(13)
17

---

where chunk is set of uni- grams which are adjacent in the generated text and in the reference text. Similar to BLEU, the Penalty reflected take into account the length of generated text. Therefore, the METEOR metric is specialized to measure the morphology, fluency, and adequacy of text rather than sequence order since does not use n-grams.
MAE. Mean Absolute Error (MAE) aims to measure the average magnitude of errors between the ground-truth values and predicted values, which is defined as:
MAE
Σ 1=
In
nnl.
(14)
N
where yn is a ground-truth and in. is a model prediction.
F
Baselines
For the generalist models, we train our LLaMo based on Llama-2-7b-chl
as a backbone language model for a fair comparison with Mol-Instructions [8]]. We compare our LLaMo with (1)
LLM-based generalist models including Galactica [2], Llama2-7b-cha 2.. GPT-3.5, and GPT-4,
(2) Molecule instruction-tuned generalist model such as Mol-Instructions 48 Since GPT-3.5 and
GPT-4 have difficulty solving the tasks without in-context learning, we additionally measure the performance of GPT-3.5 and GPT-4 with 4-shots in-context learning, which are GPT-3.5 (ICL) and
GPT-4 (ICL). The 4-shot exemplars are selected by computing the Tanimoto similarity [67] using 2048-bit Morgan Fingerprint 668] with RDKi choosing the four molecules most similar to the target molecule from the train split of each dataset. For the specialist models, we train our LLaMo with Galactica 1.3B 2 for fair comparison with MolCA []. We use single-task specialist models
as baselines, including MolT5 [29]. MoMu (4, and MolCA S
G Benchmarks
In this section, we provide a brief introduction to each task and dataset utilized in our research.
Molecular description generation. Generating the description of a molecule is considered one of the most important tasks molecular language models. For given molecule, we aim for the model to generate an accurate and informative description including various chemical properties, functional groups, biological roles, and real-world applications of the molecule. Developing a model capable of generating such descriptions is highly valuable because it has the potential to discover information about molecules that is currently unknown or very expensive to find out, thus serving as powerful
assistant for various tasks in chemistry and biology. Therefore, various works [3] 4 29 69 77 71]
have tried to enhance the ability to generate appropriate descriptions of chemical compounds using language models multi-modal language models.
For testing the performance of molecule description generation of LLaMo and previous models, we utilize the molecular description generation dataset of Mol-Instructions 48 based on PubChem database 44) for generalist models, and both PubChem324k B and ChEBI-20 8 for specialist models. PubChem332k is constructed by collecting 324k molecules and their associated text informa- tion from the PubChem database. ChEBI-20 is the most commonly utilized benchmark in this task, consisting of selected 33,010 pairs of molecules and descriptions from ChEBl (72). Each description of ChEBI-20 contains more than 20 words and includes various and rich information about molecules, such conjugate base/acid, functional parent, and enantiomer of molecules. We employ a test split of each dataset: 1,000 samples of Mol-Instructions and 2,000 3,300 samples of PubChem324k / ChEBI-20. Similarly, we filter out samples of which SMILES representation cannot be converted into the molecular graph via RDKit.
IUPAC name prediction.IUPAC (International Union of Pure and Applied Chemistry) nomen- clature 7 provides a systematic naming convention for molecules based on pre-defined rules, eliminating the ambiguity in molecular names. It is the standard method for naming molecular
https://gitthu/ wwctlauu.oomm/eeeeeeeeeeeeeeeeee com/rdkit/rdee and Julie Penzotti and others. Licensed under BSD 3-Clause license CCoyyright (c) 2023 ZJUNLP. Licensed under CC-BY 4.0 license
Copyright (c) 2006-2015, Rational Discovery LLC, Greg Landrum.
18

---

compounds in chemistry, and as such, numerous pieces of chemical literature utilize IUPAC names to represent molecules. Consequently, previous studies [2] have adopted the task of predicting IUPAC names from SMILES representations to evaluate the chemical understanding capability of molecular language models. We also follow this approach in our research. To assess the performance of LLaMo and baseline models, we use the PubChem324k dataset [3 again. The test split of PubChem324k offers curated high-quality samples of 2,000 molecules, each with corresponding SMILES represen- tations and IUPAC names. We provide the SMILES representation of each molecule to the model along with a prompt and then compare the model-generated IUPAC names with the ground-truth IUPAC names to evaluate performance.
Property prediction. In chemical and biological domains, particularly in drug discovery, exploring chemical compounds that satisfy specific chemical properties is crucial. Therefore, the ability to esti- mate the chemical and physical properties is essential for chemical foundation models. Additionally, leveraging machine learning-based based approaches for predicting molecular properties has proven to be much more efficient than traditional approaches in computational chemistry. Inspired by these facts, property prediction with LLMs has recently drawn attention from researchers 12 69 70
So, we also assess the property QA performance of LLaMo and baselines using the dataset introduced by [48. This dataset is a subset of QM9 dataset from MoleculeNet [64], a widely used benchmark for various chemistry machine learning tasks. The original QM9 dataset contains numerical values for 19 chemical properties per molecule, but [48] focuses three properties related to molecular orbital energy: HOMO (Highest Occupied Molecular Orbital) energy. LUMO (Lowest Unoccupied Molecular Orbital) energy, and the HOMO-LUMO gap (in Hartree units). [48] also created distinct question-form instructions for each property to make language models understand the task and accurately generate continuous values. Consequently, using the given instruction for each molecule in the 2,000 test samples and its SMILES representation as input, we have LLaMMo and baseline generalist models predict the properties of the given molecules. Similarly, we filter out samples of which SMILES representation cannot be converted into the molecular graph via RDKit.
Forward reaction prediction. Understanding chemical reactions is crucial for advancing various chemical and biological fields, including pharmaceuticals, materials science, and environmental technology. This understanding enables researchers to design efficient synthesis pathways and develop new chemical compounds. Therefore, a molecular model can effectively comprehend these chemical reactions, it serve as a powerful assistant in the research and development process. This concept has emerged through the task of forward reaction prediction, and several prior models [48] [54] have conducted experiments address this task. This task focuses the proactive determination of potential products resulting from a chemical reaction based on given reactants and reagents. This approach is also significant from the perspectives of efficiency and environmental sustainability, as it reduces the need for experimental trial and error in chemical development and research by using these models to virtually conduct simulated experiments. We utilize the forward reaction prediction segment of Mol- Instrnnstusssoons [48), based on the USPTO (74) database, to assess the performance of LLaMo and other models. This dataset contains question-form instructions for predicting reaction products, with reactants and reagents separated by a period (+.) as input, and the corresponding product of the reaction as the target output. It also doesn't 't specify what the reactants and reagents are to create a task that more closely resembles real-world scenarios. Finally, using the test samples from this dataset, we provide the SMILES representations of the reactants and reagents to the model and their molecular graphs with prompt, aiming to predict the product molecule in SMILES format as well.
Retrosynthesis. Retrosynthesis shares similar context with forward reaction prediction but ap- proaches chemical reactions from a different perspective. The retrosynthetic analysis begins with the target compound and works backward to identify potential reactant molecules for its synthesis. This reverse approach is as valuable as forward reaction synthesis because aids researchers in discovering effective and efficient synthetic methodologies for generating target molecules. This is particularly important in various chemical applications, such as drug discovery, where identifying chemically valid and economical processes for synthesizing target drug molecules is essential. As a result, several previous studies [48 54] have adopted this task to analyze the capability to deeply understand chemical knowledge and take advantage of this knowledge to specify the chemical reactants of given product. We also follow this approach and demonstrate its capability using the retrosynthesis section of the sample consists of a product molecule and the required reactant molecules, separated by a period (*").
(48) dataset, which originates from the USPTO 500MT (75). Each
19

---

Table 8: Example prompt for GPT in molecular description generation task. The top block shows the instruction part the prompts. It includes original instruction from the dataset and additional instruction added only for in-context learning. The bottom block shows optional few-shot examples and target SMILES.
You are an expert chemist. Please strictly follow the format, no other information can he provided. Given the molecular SMILES, could you provide a description of this molccule? You will be provided with several examples of molecules and their descriptions.
Molecule SMILES: wtt.o//www............o..ooooorr
o2)ec1
Molecular Description: The molecule natural product found in Selaginella tamariscina, Taxodium distichum, and other organisms with data available.
Molecule SMILES:
Molecular Description: The molecule is natural product found in Piper aequale with data available.
Molccule SMILES: wtt.aa/wwww...ccccccoooooooooooo
Molecular Description:
Table 9: Example prompt for GPT in IUPAC name prediction task. The top block shows the instruction part of the prompts. includes additional instruction added only for in-context learning. The bottom block shows optional few-shot examples and target SMILES.
You are an expert chemist. Please strictly follow the format, no other information can he provided. Given the molecular SMILES, your task is to predict the IUPAC name using your experienced chemical IUPAC name knowledge. You will be provided with several examples of molecules and their IUPAC names.
Molecule SMILES: COc1ce([C@ cccpllccceeeecccccccccccccceeeea The molecule's 's IUPAC name is (38)-3-(4-h)g)
didhiih- 2H-chromen-77
Molecule SMILES: oattleeeeeeeooooooeeeeeeeeeeoooo The molecule's IUPAC name is (BB------seelooyyy--dddethhhhhhe dihyddro- chromen-7-ol
Molecule SMILES: COC1:cCC00000cccoooaaaa0CCCCCCCa
The molecule's IUPAC name is
For each test sample, we provide the product's SMILES representation, its molecular graph, and the given instruction prompt as input to the model to generate possible reactants in SMILES format.
H
Prompts
In this section, we provide input prompts used to evaluate GPT-3.5 and GPT-4 on molecular descrip- tion generation task (T IUPAC name prediction task (Table and property question answering task (Table 10). We also provide the input prompts for machine- -eeeereeggeeerteed instruction-tuning data generation (Table 12).
I Over-smoothing Problems
We provide additional samples to show the over-smoothing problems in Figure
20

---

Table 10: Example prompt for GPT in property question answering task. The top block shows the instruction part the prompts. It includes original instruction from the dataset and additional instruction added only for in-context learning. The bottom block shows optional few-shot examples and target SMILES.
You are an expert chemist. Please strictly follow the format, no other information can he provided. Given the molccular SMILES, what is the energy separation between the HOMO and LUMO of this molecule? You will be provided with several examples of molecules and their HOMO-LUMO gap values.
Molecule SMILES: COCC120C3CCIC32 Output Value: 0.2967
Molecule SMILES: OCCC122CC3((11)C32 Output Value: 0.305
Molecule SMILES: CCCIC20C3CIC33C Output Value:
Table 11: Input prompt for generating multi-turn conversation data based SMILES and caption.
You are Al chemical assistant, and you are secing single molecule. What you see is provided with SMILES representation of the molecule and sentences describing the same molecule you analyzing. Answer all questions you seeing the molecule.
Ask diverse questions and give corresponding answers.
Include questions asking about the detailed information of the molecule, including the class, conju- gate acid/base, functional groups. chemical role, etc. Do ask any question that cannot be answered confidently. Molecule SMILES: (SMILES) Caption: (CAPTION) Conversation:
Table 12: Input prompt for generating multi-turn conversation data based on SMILES, caption, and IUPAC.
You are an Al chemical assistant, and you are seeing single molecule. What you see is provided with SMILES representation of the molecule and sentences describing the same molecule you are analyzing. In addition, the IUPAC name of the molecule is given. Answer all questions you seeing the molecule.
Ask diverse questions give corresponding answers. Include questions asking about the detailed information of the molecule, including the class, conju- gate acid/base, functional groups. chemical role, etc. Do ask any questions that cannot be answered confidently. Molecule SMILES: (SMILES)
Caption: (CAPTION)
IUPAC: (IUPAC)
Conversation:
J More Qualitative Samples
We provide further qualitative results of LLaMo w/ and w/o graph in Figure g and LLaMo w/ and wo MGProj in Figure respectively.
21

---

(8) I-layer
(b) 2-layer
(c) 4-layer
(d) 5-layer
(n) I-Iayer
(b) 2-layer
(c) 4-layer
(d) 5-layer
(a) I-layer
(b) 2-yyer
(c) layer
(d) Sayer
Figure Node representations of graph encoder with 1,2,4,5 layers. As the number of layers increases, node representations collapse.
13.144dihhydooprstaaaaddii FZalpha
w/o graph: 2-hydraxy-3-oxo- teyryyt-taaa
graph: 15-deoxy- prossagladnn J2
OH
GT: The molecule is prostaglandin carboxylic acid anion that is the conjugate base of 14- dihydroprostaglandin F2alpha, obtained by deprotonation of the carboxy group; major species at pH 7.3. It is a conjugate base of a 13,14- dihydroprosteelandin F2alpha.
LLaMo w/ graph: The molecule is prostaglandin carboxylic acid anion that is the conjugate base of 15- deoxy-Delta( Delta(12,14 2,4) 14)-prrottaglanin J2, obtained by deprotonation of the carboxy group; major species at pH 7.3. is conjugate base of 15- decxy- Delta(12,t1)) pro-pr..
w/o graph: molecule
1H-pyran. conjugate acid 2-hydroxy-3-awe 3-0,--3-,00--,,,,,----1 1.ttt,yy, 1H-pyran(2-).
Figure 8: An example of molecular description generation results of LLaMo w/o graph and LLaMo w/ graph given the molecule "CCCCcICe @HI[CCICCH]]ICCe @HI](CIC@ (CteeeCHH@CCH CCCC=C
22

---

GT: hexacyclonic acid
LLaMo w/o MGProj: 3-hydroxybutyric
LLaMo w/ MGProjj 4-hydroxy cyclhhexanecarboxylic acid
OH
LLaMo MGProj: The molecule monocarboxylic acid anion that is the conjugate base of 4- hydrooyyylohesaeecarboyyy acid, obtained by deprotonation of the carboxy group; major species at pH 7.3. It is conjugate base hydrroyyycooheanecctoooyyl acid.
GT: The molecule is hydroxy monocarboxylic acid anion that the conjugate base hexacyclonic acid, arising from deprotonation carboxy group; Major species 7.3, conjugate base hexacyclonic acid.
LLaMo w/o MGProj: The molecule is hydroxy fatty anion conjugate base of 3-hydroxybutyric acid. conjugate base of hydroxybutyric acid.
(a) Molecule "CICCCCCCCCCCCC:)00000""""""""""
GT: sn-glycero-3- phosphoethanolmmine
LLaMo w/o MGProj: 1-12-hydroxy-3- [ZZ]:cttdeec1-eeooyjprroooill phosphoethanolamine
LLaMo MGProj: sn-glycero-3-
GT: The molecule 1-falk-1eenyl)-at elycero-3 phosphoethanollmine zwitterion which the alk-1-enyl group is specified as octadec-l-enyl. It is a tautomer of a 1-(cttade--1- sn-llye--ly....-2. phosphoethanolmmin.
LLaMo MGProj: The molecule a 2-acyl-an-glycero-3
LLaMo w/o MGProj: The molecule glycerol ether conjugate base a 1-d2-hydroxy-3-
phosphoethanolamen mine. It is tautomer of a 1-0-acyl-sn-glycero--
phosphoethanolmmine.
(b) Molecule www.eeeee
Figure 9: Examples of molecular description generation results of LLaMo w/o MGProj and LLaMo w/ MGProj.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state the main claims in the introduction and abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
Our paper does not include any theoretical analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the implementation details in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
We make our code and dataset publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.cc/pullic//uuii wwwticccguie//eeeeeeeeeis CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the implementation details in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: We report the performances with single-run experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the implementation details in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp:://// httpp:::///urriss... cttpsurrssscccppppeeeee http:///cccp...ccettcccccccceeee
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethics in our research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss both positive and negative societal impacts in Appendix B
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
27

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We release the code and checkpoint with the safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We explicitly mention the license used assets in the supplement.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswwihcoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
28

---

Answer: [Yes]
We explain the data generation process in Section 4 and make the data publicly available.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not use crowdsourcing this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not use crowdsourcing this paper.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "XWyVDhyxMB", "reviewer": "Reviewer_P4Gm", "summary": "This paper presents LLaMo, a novel enabling LLMs and instruction tuning in molecular domain. To bridge the gap between different modalities, the paper also proposes a projector that transforms the graph representations into graph tokens level by level. The authors conduct extensive experiments and compare LLaMo with several proper baselines. The results are convincing.", "questions": "1. For line 138-139, could you explain in detail for the $\\[p^{(l)}_1, .... , p^{(l)}_b\\]$, such as what is the learnable prompts and how do you initialized it?\n2. For the few-shot prompts part, how do you choose the few-shot examples?", "limitations": "The authors address the limitations of this work.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method makes a good contribution to enable LLMs into the molecular domain. It successfully bridges the gap between language and graph modalities.\n2. In terms of quality, the experiment setup is sound. Also, the authors conduct detailed and extensive experiments. It seems evident that LLaMo performs better than the baselines mentioned in the paper.\n3. The paper is well-written and well-structured overall.", "weaknesses": "1. Though the authors conduct several experiments, they still lack some comparison between different GNN and LLM backbones. In section 5.3(Impact of Multi-level Graph Projector), will different GNN backbones harm or improve the results? Also, under the same experiment settings, the paper lacks the comparison between different LLM backbones. Such as the settings in Table 2, will the results be better for LLama-7b?\n2. With several layers of GNNs, the efficiency of this projector may not be good.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] More performance comparison based on different GNN and LLM backbones.**\n\nGood question. We conduct additional experiments to evaluate performance based on different GNN and LLM backbones. Specifically, we compare pretrained Graph Convolutional Networks (GCNs)  with our base GNN backbone (GIN) and use LLaMa-7B as an alternative LLM backbone, as suggested. We use the same experimental setting for reporting Table 2. The experimental results for molecule description generation are reported in the table below. \n\n- **Different base GNN (GCN v.s. GIN (ours))**\n    \n    The table demonstrates that the pre-trained GIN shows better performance than the pre-trained GCN, with a 4.8 improvement in the BLEU metric for the molecule description generation task. This result shows the superior expressivity of the GIN model, which aligns with the result in Table 1 of [1]. \n    \n- **Different base LLM (LLaMa-7B v.s. LLaMa2-7B (ours))**\n    \n    The table shows that the LLaMa2-7B achieves a 6.9 performance improvement in the BLEU metric compared to LLaMa-7B. This indicates that a more powerful LLM enhances the performance of our LLaMo.\n    \n\n| GNN | LLM | BLEU | METEOR |\n| --- | --- | --- | --- |\n| GCN | LLaMa2-7B | 33.0 | 61.8 |\n| GIN | LLaMa-7B | 30.9 | 60.9 |\n| GIN | LLaMa2-7B | 37.8 | 63.2 |\n\n**[W2] Efficiency of multi-level graph projector.**\n\nGood point. Since we do not increase the number of graph tokens and GNN layers, our multi-level graph projector maintains efficiency comparable to other projectors. To showcase it, we measure the inference time (second per generated token) of the model with various projectors, including our multi-level graph projector in the table below. From the table, our MGProj has similar efficiency to other projectors. Interestingly, the inference time difference between the setups without graphs and with graphs is minimal. We think that this is because the number of input text tokens is 108.87 in average, which is significantly larger than the number of graph tokens (32).\n\n| Projector | Time (sec/generated token) |\n| --- | --- |\n| w/o Graph | 0.056 |\n| MLP (w/ low-level) | 0.059 |\n| MLP (w/ high-level) | 0.059 |\n| MLP (w/ concat) | 0.059 |\n| Resampler | 0.058 |\n| MGProj (w/o motif) | 0.060 |\n| MGProj (ours) | 0.060 |\n\n**[Q1] Details of learnable prompts (L138-139).**\n\nThe learnable prompts means learnable tokens $\\mathbf{P}^{(l)} = \\left[\\mathbf{p}_1^{(l)}, \\dots, \\mathbf{p}_b^{(l)} \\right]$ in L138. We initialize them with values drawn from the normal distribution.\n\n**[Q2] Details of choosing the few-shot examples.**\n\nAs mentioned in the Section F of the supplement (L885-890), we select the exemplars from the train split of each dataset, based on their similarity to the target molecule. To calculate the similarity, we use the Tanimoto similarity metric, a widely-used metric for comparing molecular structures. We select the four molecules with the highest similarity scores to the target molecule.\n\n[1] Hu, Weihua, et al. \"Strategies for pre-training graph neural networks.\" ICLR 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vJPkd5UM4S", "reviewer": "Reviewer_rvcw", "summary": "This paper proposes a Large Language Model-based Molecular graph assistant (LLaMo), which can enhance the molecular graphs's general-purpose understanding and generation capabilities. By integrating molecular graph encoders with large language models, LLaMo enables instruction-following responses in the molecular domain. The paper conducts extensive empirical studies and justifies the effectiveness of the proposed method.", "questions": "1. Could GPT -4 perform better with prior information instilled, such as function groups?\n2. What is the purpose of using different LLM for LLaMo tasks (Tables 2 and 3)? Could better LLM result in better performance? Would there be some upper bounds?\n3. Could LLaMo design or edit SMILES based on the user's instructions?", "limitations": "The idea is novel and effective. It would be better if the author could make LLaMo able to design or edit SMILES based on the user's instructions.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed LLaMo has a clear and effective design, making it powerful in molecular description and property prediction.\n- Extensive experiments have been conducted to provide a good insight into the components of the proposed method.\n- The paper is generally well-written, with clear illustrations and tables.", "weaknesses": "- The multi-level graph projector shares an idea similar to JKNet[1]. In addition, Graph Transformers [2] can also handle the over-smoothing problem, especially for small molecules. The author could consider discussing how the proposed techniques differ from these referenced works.\n- The paper uses function groups as prior information for training; would this cause information leakage? The downstream task is to predict these groups or generate molecular captions based on these motifs.\n- The GPT-4 generates samples for instruction-tunning, but the samples's qualities are not assessed properly.\n\n[1] Representation Learning on Graphs with Jumping Knowledge Networks. In ICML, 2018.\n\n[2] Representation Learning on Graphs with Jumping Knowledge Networks. In NeurIPS, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] Discussion of how multi-level graph projector differs from JKNet and Graph Transformers.**\n\nWe appreciate your constructive comment. Our multi-level graph projector differs from JKNet and Graph Transformers (GTs) in two aspects: main purpose and architecture.\n\n*Main purpose*\n\nAs shown in Figure 1, our multi-level graph projector functions as a **projector (yellow box)** that converts the graph encoder’s output representations into graph tokens. In contrast, both JKNet and GTs are designed as **graph encoders (green box)**. Thus, our multi-level graph projector is orthogonal to these encoders and can be used in conjunction with them.\n\n*Architecture*\n\n- JKNet\n\nOur projector encodes each graph level into **separate tokens**, while JKNet aggregates all GNN layers into **a single representation**. The separated level-aware tokens allow the attention module in the LLM to softly select the appropriate graph level, leading to better **performance (Table 4, Multi-level graph projector (MGProj) v.s. JKNet with MLP projector (MLP w/ concat)) and interpretability (Figure 4)**.\n\n- GTs\n\nOur projector does not modify the GNN architecture, unlike GTs, which adopt **self-attention operations** instead of GNNs.\n\nWe will include this discussion in the camera-ready version if the paper gets accepted. \n\n**[W2] Would the use of functional groups as prior information cause information leakage?** \n\nWe use **simple hand-crafted features derived from the molecular graph** as functional group information to minimize the potential risk of information leakage. They are **not tailored to specific datasets or tasks**. Specifically, we represent the functional group through one-hot vector encoding the number of key atoms (e.g., carbon) and bonds (e.g., single bond) in each substructure (e.g., rings). This representation relies solely on the inherent structure of the molecular graph.\n\nAlso, we check the performance improvement by the functional groups in our ablation study of the functional group (Table 4). The table indicates that our LLaMo without motif (functional group) still shows good performance (-0.6 BLEU, +0.2 METEOR). This suggests that the performance improvements are primarily attributed to the architecture and learning scheme of LLaMo, rather than the use of functional group information.\n\n**[W3] Quality of GPT-4 generated samples.**\n\nThank you for your constructive feedback. During the generation, we implement a multi-step assessment process, detailed below:\n\nStep 1: We prompt GPT-4 to generate multi-turn conversation instruction data about molecules using captions and IUPAC names from a well-established dataset, PubChem, without any demonstrations (zero-shot).\n\nAssessment 1: We sampled over 100 subsets of data and observed that GPT-4 frequently generated **incomplete conversations and refused generation.**\n\nStep 2: To address these issues, we first sample high-quality demonstrations from a small set of complete conversations generated by GPT-4 (zero-shot). Subsequently, we prompt GPT-4 to generate data with these demonstrations.\n\nAssessment 2: We sampled 500 subsets generated via in-context learning. We found that conversations with a higher number of turns were more **prone to generating incomplete and inaccurate outputs**.\n\nStep 3: We filter out incomplete conversations and those with many turns. Approximately 5% of the data was filtered out.\n\nAssessment 3: We sampled 500 subsets from the filtered data and manually assessed their quality. We verified that the generated data contained accurate information about the given molecule, with no issues of incompleteness.\n\nIn addition, our ablation studies (Table 5) validate the effectiveness of GPT-4 generated instruction dataset. Instruction tuning with this data improves the performance of LLaMo, providing the model with more detailed and instruction-following guidance.\n\nWe believe that our data is high-quality and will make our instruction data publicly available.\n\n**[Q1] Performance of GPT-4 with prior information.**\n\nGood question. We test GPT-4's performance by adding the molecule's functional group information (FG info.) to the input prompts. We use the same FG info. used in our model. The results (below table) show that adding FG info. does not consistently improve GPT-4’s performance, which means that it is not always helpful on the molecular description task.\n\n|Model|BLEU|METEOR|\n|---|---|---|\n|GPT-4|0.8|16.7|\n|GPT-4 (ICL)|27.0|52.2|\n|GPT-4 + FG info.|0.5|16.8|\n|GPT-4 (ICL) + FG info.|24.8|50.0|\n|LLaMo|37.8|63.2|\n\n**[Q2] Purpose of using different LLM on Table 2 and Table 3. Could better LLM result in better performance?**\n\nTable 2 and Table 3 report the performance of generalist and specialist models, respectively. The best-performing baseline for the generalist model, Mol-Instruction, uses LLaMa2 as its base model, while the best baseline for the specialist model, MolCA uses Galactica-1.3B. Thus, for a fair comparison with the established baselines, we employ different LLMs in each table.\n\nTo study whether a better LLM improves performance, we conduct additional experiments with LLaMo using LLaMa and LLaMa2 under the same setting as in Table 2. The result (below table) shows that LLaMa2 achieves a 6.9-point improvement on the BLEU metric compared to LLaMa, meaning that a better LLM results in better performance.\n\n|Base LLM|BLEU|METEOR|\n|---|---|---|\n|LLaMa|30.9|60.9|\n|LLaMa2|37.8|63.2|\n\n**[Q3] SMILES edit tasks.**\n\nThank you for the suggestion. We conduct additional experiments on the forward reaction prediction task to show LLaMo’s capability in editing SMILES as suggested. We use the forward reaction prediction dataset from Mol-Instructions. The results (below table) indicate that our LLaMo consistently outperforms other models in all metrics.\n\n|Model|BLEU&uarr;|Levenshtein dist.&darr;|Tanimoto Sim.&uarr;|\n|---|---|---|---|\n|Galactica|46.8|35.02|0.16|\n|Text+Chem T5|78.2|20.41|0.71|\n|Mol-Instructions|65.4|27.26|0.31|\n|LLaMo|82.4|6.81|0.76|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tpo6LvSD01", "reviewer": "Reviewer_W78D", "summary": "This paper proposes a molecule-text model, LLaMo, which aligns text and molecule modalities to tackle diverse downstream tasks with a language interface. Through two-stage alignment: training a graph encoder and then tuning the LLM through instructions, LLaMo aligns text and molecular representations well, and achieves promising performance on diverse downstream tasks.", "questions": "Please refer to the questions given in the weakness part", "limitations": "Limitations and broader impacts are discussed in the appendix. Guidelines should be removed from the checklists.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The writing and presentation of this paper are clear. It is well-motivated and easy to read.\n2. Code implementation and comprehensive details are provided, which ensures reproducibility.\n3. The design of learnable query tokens is interesting.\n4. I observe an interesting phenomenon in the experiments. Low-level information is sometimes overlooked for graph-level prediction, and it is mainly studied for node-level predictions. However, for molecule-text models, some tasks require both local and global-level information, and it presents more importance for tasks like IUPAC prediction.", "weaknesses": "1. The implementation details of functional groups are not given in the main text nor appendix. Could you share more details on this part?\n2. How is the graph modality treated in the instruction tuning phase? Is it fixed, and the graph token is augmented into the text token inputs?\n3. I am curious about the design of the learnable query tokens. Why are the motif and graph-level representations modeled separately?\n4. One factor that prevents me from giving a higher score is that this pipeline is widely adopted for vision-language models, so it does not surprise me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] Details of functional group representations.**\n\nWe appreciate your feedback and provide a comprehensive explanation regarding functional representations. We use simple hand-crafted features derived from the molecular graph as functional group information.\n\nTo construct functional group representations, we initially identify functional groups from the given molecular graph following [1]. We define three types of substructures (rings, non-cyclic FGs, and carbon-carbon single bonds) as functional groups. \n\nThen, we vectorize the main characteristics of each functional group. Specifically, we one-hot encode the number of important atoms (e.g., carbon, oxygen, etc) and bonds (single, double, triple, aromatic bonds) contained in each functional group to represent the functional group. Each functional group is represented as $\\mathbf{z}_{\\text{FG},i}$, a vector encoding of its main characteristics.\n\nFinally, the functional group representations $\\mathbf{Z}\\_{\\text{FG}}$ are constructed by concatenating all individual functional group representations $\\mathbf{z}\\_{\\text{FG}, i}$, which is formulated as $\\mathbf{Z}\\_{\\text{FG}} = \\left[\\mathbf{z}\\_{\\text{FG},0}, \\dots, \\mathbf{z}\\_{\\text{FG},M}\\right],$ where $M$ indicates the number of the functional groups in the given molecular graph.\n\nWe hope that this explanation clarifies the details of the functional groups. We will include these details in our camera-ready version if the paper gets accepted.\n\n**[W2] How is the graph modality treated in the instruction tuning phase?**\n\nAs mentioned in Section 3.2, in the instruction tuning phase, the input molecular graph is first represented with a sequence of graph tokens and then the sequence of graph tokens is augmented into the text token inputs. In this phase, we train both multi-level graph projector and LLM, while keeping the graph encoder frozen.\n\n**[W3] Why are the motif and graph-level representations modeled separately?**\n\nGraph-level representations $\\hat{\\mathbf{P}}^{(0)}, \\dots \\hat{\\mathbf{P}}^{(L)}$ are constructed **based on outputs of each layer of GNNs**, whereas motif-level representations $\\hat{\\mathbf{P}}^{(\\text{motif})}$ are constructed based on **functional group representations** $\\mathbf{Z}_{\\text{FG}}$ as detailed in **[W1] Details of functional groups.** Due to these distinct construction methods, we model them separately.\n\n**[W4] Comparisons with vision-language model pipelines.**\n\nCompared to the Large Vision-Language Models (LVLMs) such as LLaVA [2], Instruct-BLIP [3], and mPLUG-owl [4], etc., our proposed LLaMo has several unique components tailored to the molecule domain, as summarized below:\n\n- **Multi-level graph projector**: While most LVLMs usually construct visual tokens **solely based on the outputs from the final layer** of a visual encoder, our multi-level graph projector leverages **node representations from all layers of a GNN** to construct molecular graph tokens. This method enables the tokens to encapsulate richer information, reflecting the molecular graph structure at multiple levels of abstraction. This multi-level approach has been shown to improve the performance in various molecular graph-related tasks.\n- **Molecular graph specialized instruction data**: We also present **molecular graph specialized multi-turn instruction data** automatically generated by GPT-4. This specialized multi-turn instruction data aims to improve the performance of the model in tasks related to molecular graphs by providing detailed and more contextually relevant examples. We believe that this data contributes to future research concerning the molecule domain.\n- **Functional groups (Motifs)**: LLaMo **integrates molecule-specialized functional group information**. The functional groups are statistically important subgraph patterns within a molecule exhibiting consistent chemical behaviors across different compounds. By incorporating the functional groups, our LLaMo has shown the best performance in Table 2 and 3. We have also demonstrated the effectiveness of the functional groups in Table 4.\n\nWe hope that this summarization clarifies the uniqueness of our work compared to the existing LVLM works. \n\n[1] Ji, Zewei, et al. \"ReLMole: Molecular representation learning based on two-level graph similarities.\" *Journal of Chemical Information and Modeling* 2022.\n\n[2] Liu, Haotian, et al. \"Visual instruction tuning.\" NeurIPS 2023.\n\n[3] Dai, Wenliang, et al. \"Instructblip: Towards general-purpose vision-language models with instruction tuning.\" NeurIPS 2023.\n\n[4] Ye, Qinghao, et al. \"mplug-owl: Modularization empowers large language models with multimodality.\" arXiv 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fLUKCBFkuf", "reviewer": "Reviewer_3eLt", "summary": "The paper introduces LLaMo, a Large Language Model-based Molecular graph assistant. LLaMo is a model that integrates a GNN encoder, a multi-level graph projector, and a language model. The projector uses a cross-attention mechanism to convert graph representations into graph tokens by abstracting outputs from each GNN layer and motif representations. Additionally, the authors use machine-generated molecular graph instruction data, leveraging GPT-4, for instruction-tuning. Extensive experiments show that LLaMo excels in some tasks.", "questions": "The two-step training process involves initially training the graph encoder in Stage 1, followed by fine-tuning the Large Language Model using LoRA in Stage 2. Why is the multi-level projector set as trainable during both training stages?", "limitations": "The experiments predominantly concentrate on a few tasks such as molecular description generation, property prediction, and IUPAC name prediction. This limited focus may not fully reveal the model’s versatility and potential shortcomings across a wider spectrum of molecular tasks.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The framework incorporates a multi-level graph projector that effectively captures multi-scale information. This design mitigates the over-smoothing problem. A figure is provided to illustrate the significant impact of this design on model performance and attention distribution.\n\nThe paper offers a comprehensive, step-by-step explanation of each component within the LLaMo framework. Additionally, a lot of typical examples are presented to clearly demonstrate the framework's design and functionality in practical applications.", "weaknesses": "In order to perform instruction-tuning, the authors utilized GPT-4 to generate molecular graph- text instruction-following data using graph-text pair datasets with human-written instructions. Although this approach enhances the results, it lacks a robust evaluation of the generated data's quality and validity. It is crucial to develop methodologies to rigorously assess the accuracy and relevance of the GPT-4 generated instruction data to ensure the reliability of the model's performance improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] Evaluation of GPT-generated quality.**\n\nThank you for your constructive feedback. During the generation, we implement a multi-step assessment process, detailed below:\n\nStep 1: We prompt GPT-4 to generate multi-turn conversation instruction data about molecules using captions and IUPAC names from a well-established dataset, PubChem, without any demonstrations (zero-shot).\n\nAssessment 1: We sampled over 100 subsets of data and observed that GPT-4 frequently generated **incomplete conversations and refused generation.**\n\nStep 2: To address these issues, we first sample high-quality demonstrations from a small set of complete conversations generated by GPT-4 (zero-shot). Subsequently, we prompt GPT-4 to generate data with these demonstrations.\n\nAssessment 2: We sampled 500 subsets generated via in-context learning. We found that conversations with a higher number of turns were more **prone to generating incomplete and inaccurate outputs**.\n\nStep 3: We filter out incomplete conversations and those with many turns. Approximately 5% of the data was filtered out.\n\nAssessment 3: We sampled 500 subsets from the filtered data and manually assessed their quality. We verified that the generated data contained accurate information about the given molecule, with no issues of incompleteness.\n\nIn addition, our ablation studies (Table 5) validate the effectiveness of GPT-4 generated instruction dataset. Instruction tuning with this data improves the performance of LLaMo, providing the model with more detailed and instruction-following guidance.\n\nWe believe that our data is high-quality and will make our instruction data publicly available for future research in the molecular domain.\n\n**[Q1] Why is the multi-level projector set as trainable during both stages?**\n\nWe set the projector as trainable both stage 1 and stage 2 to follow existing Large Vision-Language Models such as LLaVA [1].\n\n**[Limitations1] Other tasks.**\n\nThank you for the suggestion. We conduct additional experiments on the forward reaction prediction task to evaluate LLaMo’s ability in other spectrum of molecular tasks, as suggested. The forward reaction prediction is the task that targets the prediction of chemical reactions given the reactants. We utilize the forward reaction prediction dataset from Mol-Instructions. The experimental results are in the table below. From the table, our LLaMo shows the best performance compared to other models in all metrics.\n\n| Model | BLEU&uarr;| Levenshtein dist.&darr;| Tanimoto Sim.&uarr;|\n| --- | --- | --- | --- |\n| Galactica | 46.8 | 35.02 | 0.16 |\n| Text+Chem T5 | 78.2 | 20.41 | 0.71 |\n| Mol-Instructions | 65.4 | 27.26 | 0.31 |\n| LLaMo | 82.4 | 6.81 | 0.76 |\n\n[1] Liu, Haotian, et al. \"Visual instruction tuning.\" NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all the reviewers for their time and efforts in reviewing our paper and insightful comments and questions. We are encouraged that the reviewers recognize multiple strengths in our paper, including:

- **Clear and effective design** that enables LLMs to operate within the molecular domain (P4Gm, rvcw, W78D) by successfully bridging the gap between language and graph modalities with the multi-level graph projector (P4Gm, 3eLt).
- **Extensive and detailed experiments** (P4Gm, rvcw, 3eLt) that demonstrate the effectiveness of the proposed model and the contribution of each component (rvcw).
- **A lot of qualitative examples** (3eLt), including illustrations of attention distribution that highlight the impact of the multi-level graph projector (W78D, 3eLt).
- **Sound and detailed experimental setups** (P4Gm) with code implementation (W78D).
- **Well-structured and clear writing** (P4Gm, rvcw, W78D), including comprehensive, step-by-step explanations of each component (3eLt).

We have tried our best to address the reviewers’ questions and concerns within the available time. We believe that incorporating this constructive feedback significantly enhances the quality of the paper. We sincerely thank all the reviewers for their valuable contributions. Please find individual responses to the comments below.

Meta-review: The paper proposes a multi-level graph projector to create tokens which can complement LLM instructions. The multi-level nature is obtained by applying the embeddings from different GNN layers. It integrates well domain knowledge (e.g., different molecule modalities) and novel technology, contains an extensive evaluation, and shows good performance using state-of-the art LLMs. The paper also proposes a novel molecule + text dataset.

All four reviewers recommend to accept the paper and I recommend that as well.
Decision: Accept (poster)