LLaMo: Large Language Model-based Molecular Graph Assistant

Authors: jinyoung  park
minseong  bae
dohwan  ko
hyunwoo j kim
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 10:30:20

Abstract: Large Language Models (LLMs) have demonstrated remarkable generalization and instruction-following capabilities with instruction tuning. The advancements in LLMs and instruction tuning have led to the development of Large Vision-Language Models (LVLMs). However, the competency of the LLMs and instruction tuning have been less explored in the molecular domain. Thus, we propose LLaMo: Large Language Model-based Molecular graph assistant, which is an end-to- end trained large molecular graph-language model. To bridge the discrepancy between the language and graph modalities, we present the multi-level graph projector that transforms graph representations into graph tokens by abstracting the output representations of each GNN layer and motif representations with the cross-attention mechanism. We also introduce machine-generated molecular graph instruction data to instruction-tune the large molecular graph-language model for general-purpose molecule and language understanding. Our extensive experiments demonstrate that LLaMo shows the best performance on diverse tasks, such as molecular description generation, property prediction, and IUPAC name prediction. The code of LLaMo is available at https://github.com/mlvlab/LLaMo.

Main paper: 
 introduction: 
 In recent years, molecular machine learning [1, 2, 3, 4] has received significant attention, addressing diverse tasks in the chemical domain. The predominant approach for molecular tasks is graph machine learning [5, 6, 7] that leverages the molecular graph structure, which is a natural and expressive representation of molecules. Although graph-based methods have successfully represented molecules, they have limited interpretability and incompatibility to solve multi-modal molecular tasks dealing with pairs of texts and molecules. To address these issues, recent works [4, 8] train both a language model and a graph encoder with cross-modal contrastive learning. However, the models trained with cross-modal contrastive learning are insufficient to perform open-ended molecule-to-text generation tasks [3] , which are more applicable to practical use.Large Language Models (LLMs) [2, 9, 10, 11] have shown impressive progress and accomplished human-like open-ended text generation with the power of billions of parameters. To leverage the instruction-following capability of LLMs, many works employ instruction-tuning approaches [12, 13, 14] for general-purpose language models. Motivated by the development of LLMs and instruction tuning, Large Vision-Language Models (LVLMs) have recently been explored and achieved success on image comprehension and image-to-text generation tasks [10, 11, 15, 16, 17, 18] . Despite the success of LLM-based approaches on natural language processing and machine vision domains, the research on the integration of language models and molecular graphs has been less studied due to the lack of consideration of the architecture design of Large Molecular Graph-Language Models (LMGLMs) and the molecular graph instruction data.In this paper, we propose LLaMo: Large Language Model-based Molecular graph assistant, which seamlessly integrates a molecular graph encoder and a large language model to enable the instructionfollowing response generation in molecular domain. Specifically, LLaMo consists of the molecular graph encoder, large language model, and multi-level graph projector that bridges the graph encoder and large language model. The multi-level graph projector abstracts the representation of each GNN layer and motif representation using a cross-attention mechanism, ensuring a thorough understanding of molecular structures. Furthermore, we introduce machine-generated molecular graph instruction data through the pipeline to convert molecular descriptions and IUPAC names into a multi-turn conversation format. The generated instruction-following data enhances the model's ability to perform general-purpose molecule and language understanding, bridging the gap between molecular graph analysis and language-based tasks. Our proposed LLaMo outperforms the LLM-based works such as GPT-4 across diverse tasks, including molecular description generation, property prediction, and IUPAC name prediction .Our contributions are summarized as follows:• We propose LLaMo: Large Language Model-based Molecular graph assistant consisting of graph encoder, language model, and multi-level graph projector equipped with a multi-level graph projector that captures rich information of the graph structure at multiple levels.• We introduce GPT-4 generated molecular graph-text multi-turn conversation data to address the data scarcity problem of molecule-text datasets and improve the instruction-following capabilities of a large molecular graph-language model.• Our experiments demonstrate that LLaMo achieves the best performance on various tasks such as molecular description generation, property prediction, and IUPAC name prediction. 
related works: 
 Molecular graph modeling. Molecular graphs serve as a natural and expressive representation of molecules, effectively capturing the structural information. Graph neural networks [19, 20, 21, 22] are commonly utilized architectures for molecular graph representations. To learn graph neural networks with the limited molecular graph data [23] , self-supervised learning has been explored. For example, various approaches [24, 25] have been developed to capture multi-level features of molecular graphs, such as node-level masked atom modeling [24] , motif-based self-supervised learning [25, 26] , and graph-level contrastive learning [27, 28] . With the advance of multi-modal large language models, molecule-language tasks such as molecule-text retrieval [8] or molecule captioning [29] have recently drawn significant attention. Recent works [3, 4, 30] have attempted to enable language models to understand molecular graphs. [30] treated nodes of molecular graphs as tokens of language models. Some works have adopted GNN-based encoders, either by propagating their outputs to language models through MLP [4] or employing cross-modal projectors [3] . However, these methods fail to consider molecular graphs at multiple levels and are hindered by inherent limitations of graph encoders, such as the over-smoothing problem [31] . To address these challenges, we propose a novel architecture, LLaMo, which effectively propagates multi-level information of molecular graphs to language models.Instruction tuning. Recent advancements of LLMs lead to extensive research on instruction tuning, aimed at improving the model's capability to follow human instructions [12, 32, 33, 34, 35] . To construct high-quality instruction tuning data, a line of previous approaches [34, 35] has adopted existing human-annotated datasets and integrated them with a new structure and template. On the other hand, recent studies [12, 36, 37] on instruction tuning have collected data samples from strong LLMs like GPT-4 [10] . These works first manually construct annotated seed instruction samples and expand them by prompting LLMs. As a result, several instruction-tuned LLMs [14, 16, 37] have been proposed from the open-source LLMs, e.g., LLaMA [9] and shown generalizability across a wide range of instructions. More recently, those studies on instruction tuning have been expanded to visual instruction tuning in image [15, 17, 38] and video [39, 40] domain to enable the model to understand the visual contents. Inspired by the instruction tuning for multi-modal LLMs in other domains, in this work, we study instruction tuning specifically for molecule graphs, which has been underexplored in the literature.Instruction ⋯ ⋯ ⋯ ⋯ ⋯ ⋯ ⋯ ⋯The molecule is a 2-hydroxycarboxylate that is …Figure 1 : Overall framework of LLaMo. LLaMo consists of a graph neural network, a multi-level graph projector, and a large language model. It first encodes an input 2D molecular graph with the graph neural network and then converts the encoded graph into molecular graph tokens with the multi-level graph projector. Finally, the large language model generates the instruction-following response given the input SMILES, graph tokens, and the instruction.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Please refer to Appendix A.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
llamo: large language model-based molecular graph assistant: 
 The primary goal is to seamlessly integrate a molecular graph encoder and a Large Language Model (LLM) to generate instruction-following responses to the input texts and molecules. To achieve it, we propose LLaMo: Large Language Model-based Molecular graph assistant, a generalpurpose Large Molecular Graph-Language Model (LMGLM) equipped with a multi-level graph projector. Specifically, the proposed framework utilizes three input modalities: 1D SMILES [41] , 2D molecular graph, and text (instruction). SMILES [41] is a 1D representation of a molecule, and a 2D molecular graph is processed by a GNN. The three input modalities are fed as a sequence of tokens and our LLaMo autoregressively generates text responses. Formally, given SMILES S, molecular graph tokens G, and text (instruction) T, the proposed method renders the response Y = {y i } K i=1 as:EQUATIONwhere y <i indicates generated token sequences until i-th token.The overall architecture of LLaMo is illustrated in Figure 1 . LLaMo consists of a graph encoder, a multi-level graph projector, and a backbone large language model. The graph encoder g(•) takes a 2D molecule graph as an input and outputs their node representations as a sequence of tokens. The multi-level graph projector Proj MG (•) transforms the sequence of node representations into molecular tokens to align them with the LLM. Then, the LLM f (•) processes molecular and text tokens and provides a response in an autoregressive manner.Graph encoder. We adopt Graph Neural Networks (GNNs) as a molecular graph encoder. Given the graph G, graph neural networks g (•) iteratively update node representation z(l) v ∈ R d (l)via the message-passing framework. With the message-passing, L-layer GNN provides node representations z (L) v that express an L-hop ego-graph given the node v as a center node. More details about graph neural networks are in the Appendix C. graph into a sequence of molecular graph tokens H graph . It enables the language model to utilize graph information. In the literature, projectors have been proposed mainly for Large Vision-Language Models (LVLMs) [17, 18, 38, 42, 43] . They are usually implemented using a linear projection [38] or an abstraction of visual features [17, 42] , which are outputs of the final layer of a visual encoder given input image. Analogously, we can design the projector for large molecular graph-language models with a linear projection or an abstraction of high-level node representations from the pre-trained graph encoder, which is formulated as:EQUATIONwhereZ (L) graph = z (L) 0 , . . . , z (L) |V| ∈ R |V|×d (L) is the concatenation of node representation z (L) v ∈ R d (L)from L-th layer GNN and Proj (•) is the projector.However, we observe that the high-level representation is not effective in capturing the local information due to the over-smoothing problem [31] , which means that the node representations become indistinguishable, as the number of layers in the GNN increases. Figure 2 H graph = Proj MG Z (l) graph L l=0, whereEQUATIONThe method captures multi-hop graph information by leveraging node representations from all layers of a GNN. To handle an arbitrary number of nodes, yielding a variable length |V| × L features, we adopt the cross-attention with learnable tokensP (l) = p (l) 1 , . . . , p (l) b ∈ R b×d for l = 0, . . . , L,where b is the number of learnable prompts. Here, [•, •] indicates the concatenation operation. The learnable tokens aggregate l-th layer GNN representations into a fixed number of tokens as:EQUATIONwhere Attn (Q, K, V ) is the attention operation with query Q, key K, and value V .For more detailed representations of the input molecule, LLaMo also has learnable tokens P (motif) for motif-level representations. We use the functional groups as motifs, which are the statistically important subgraphs in the molecular graphs. To construct functional group representations Z FG , we initially identify functional groups, following [23] . Then, we vectorize the main characteristics of each functional group, which is represented as z FG,i . Finally, the functional group representations Z FG Large Language Model Then, we obtain the graph-level representations by applying MLP to the multi-hop and motif-level representations, i.e., [ P0 , . . . , P(L) , P(motif) ]. It is formulated as:EQUATIONwhere H graph is a sequence of graph tokens to be fed into the LLM.Large language models. After constructing tokens for encoding molecular graphs, LLaMo fuses SMILES representation, graph, and text tokens and puts them into the language model to generate an instruction-following response.Similar to most LVLMs [18, 38, 42] , we train LLaMo in the two-stage pipeline: (1) pre-training for molecular graph-language alignment, and (2) instruction-tuning end-to-end as in Figure 3 .Stage 1. Pre-training for molecular graph-language alignment. The first stage focuses on the alignment between the graph encoder and a large language model by learning our multi-level graph projector. In this stage, with the LLM frozen, we train the multi-level graph projector and the graph encoder by generating molecule descriptions. For training, we use a molecule-description pair dataset (e.g., PubChem [44] ) consisting of a 1D SMILES representation of molecule and molecular graph and its corresponding description.Stage 2. Instruction-tuning end-to-end. In the second stage, we train the LLM to enhance the instruction-following capabilities and enable a deeper understanding of molecular graphs. In this stage, we freeze the graph encoder and train both the multi-level graph projector and the LLM. Since it is too expensive to train the full LLM, we employ LoRA [45] to adapt LLM to the data. For instruction-following, we use the GPT-generated instruction-following multi-turn conversation dataset, which will be introduced in Section 4. In addition to our generated instruction-following dataset, we use a diverse set of datasets with various instructions: molecule description generation, molecular property prediction, IUPAC name generation, forward reaction prediction, and retrosynthesis datasets. graphs have been less explored in the literature since annotations require expertise. To alleviate the need for expertise and minimize the manual efforts, we utilize GPT-4 [10] to generate molecular graph-text instruction-following data using graph-text pair datasets.Inspired by previous results [46, 47] , we construct multi-turn conversation datasets, which are more diverse and effective for instruction tuning compared to simple pairs of questions and answers. We leverage GPT-4 to generate multi-turn conversations with tailored contexts/prompts that consist of two representations for molecular graphs and description: (i) SMILES representation that describes the chemical structures with special strings, (ii) captions that explain the molecule, and (iii) IUPAC name that describes the molecule based on its chemical composition and structure. These representations enable the GPT-4, which inherently lacks in-depth molecular knowledge, to understand and generate a diverse and high-quality set of examples. One example of the input representations is shown in the top block of Table 1 .Specifically, we generate the multi-turn conversation data in three steps: 1) select exemplar conversations among machine-generated instruction-tuning data, 2) generate multi-turn conversations via in-context learning with the exemplar conversations as prompts, and 3) filter out incomplete conversations and those with many turns. In the first step, we generate exemplars with a brief human-written instruction as shown in Appendix H. However, we found that GPT-4 frequently fails to generate complete multi-turn conversations without the exemplars. To address this issue, we generate the instruction data with in-context learning. We sample exemplars from a small set of complete conversations generated by GPT-4 in the first step. Then, GPT-4 generates the complete multi-turn conversation data for the instruction tuning guided by the prompts wrapped with the generated exemplars. To validate the quality of the generated conversation, we sample 500 subsets generated via in-context learning. We find that some conversations consisting of a large number of turns are prone to generating incomplete and inaccurate outputs. So, we filter out incomplete conversations and those with many turns. The example of the generated multi-turn conversation is in the bottom block of Table 1 . In total, we generate 12K unique molecular graph-language instruction-following samples using PubChem324k dataset [3, 44] . Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss both positive and negative societal impacts in Appendix B. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental settings: 
 Benchmarks. To evaluate the efficacy of the proposed method, we evaluate the model for three tasks such as 1) molecule description generation, 2) IUPAC name prediction, 3) property prediction (regression). We conducted experiments under two major settings: generalist and specialist models. In the generalist setting, one model handles all three tasks, whereas in the specialist setting, we train a model for each downstream task. More details about benchmarks are in Appendix G.For the generalist models, we train our LLaMo based on Llama-2-7b-chat [9] for a fair comparison with Mol-Instructions [48] . For the specialist models, we train our LLaMo with Galactica 1.3B [2] for a fair comparison with MolCA [3] . To train the generalist variant of LLaMo, we use a training split of molecular description generation dataset of Mol-Instruction [48] in stage 1. In stage 2, the model is instruction-tuned with a training split of description generation, property prediction, forward reaction, and retrosynthesis instruction dataset of Mol-Instruction [48] , IUPAC name prediction from [3] , and our GPT-generated instruction-following data. To train the specialist variant of LLaMo, we follow MolCA [3] to train the model with a pretraining split of PubChem324kV2 in the stage 1 phase and fine-tune the model for each specific downstream task in the stage 2. We adopt a long training schedule (epoch 1 pre-training, epoch 3 instruction tuning) for the final models. For analysis, we use a short training schedule (epoch 1 pre-training, epoch 1 instruction tuning). For further implementation details, refer to Appendix E.1.Baselines. For the generalist models, we compare our LLaMo with (1) LLM-based generalist models including Galactica [2] , LLaMA2-7B [9] , GPT-3.5, and GPT-4, (2) Molecule-specialized LLM such as Text+Chem T5 [49] , and (3) Molecule instruction-tuned generalist model such as Mol-Instructions [48] . Since GPT-3.5 and GPT-4 have difficulty in solving the tasks without in-context learning, we additionally measure the performance of GPT-3.5 and GPT-4 with 4-shot in-context learning, which are GPT-3.5 (ICL) and GPT-4 (ICL). For the specialist models, we use single-task specialist molecule-language models as baselines, including MolT5 [29] , MoMu [4] , and MolCA [3] .Generalist models. We provide the experimental results of generalist models in molecular description generation, IUPAC name generation, and property prediction tasks. Our LLaMo is built on LLaMA-7B and it is fine-tuned by our instruction-tuning method. Table 2 shows that our LLaMo achieves the best performance in all three tasks. In comparison to GPT-4 (ICL), which is GPT-4 with in-contextlearning, LLaMo shows a performance improvement of 11.9 in BLEU-4 and 14.9 in METEOR for molecular description generation. Furthermore, LLaMo outperforms Mol-Instructions, an instructiontuned model with molecular data, by a substantial performance gain of 41.7 in METEOR for molecular description generation and a 0.007 performance gain in MAE on the property prediction task. More experimental results on forward reaction prediction and retrosynthesis are in Appendix D. graph and high-level representation Z (L) graph , respectively. MLP (w/ concat) indicates the MLP projector with the concatenated representations of all GNN layers as an input. Resampler denotes the cross-attention based resampler projector designed in Qwen-VL [50] . MGProj (w/o motif) and MGProj are our multi-level graph projector without and with motif tokens P(motif) .Table 4 shows that our multi-level graph projector (MGProj) achieves the best performance across all three tasks. Specifically, the multi-level graph projector achieves 49.6 BLEU and 70.9 METEOR scores with a significant improvement compared to MLP projectors in the IUPAC prediction task. These experimental results demonstrate that our multi-level graph projector is more effective than conventional projectors by capturing multi-scale information, including atom, atomic group, and molecule-level information. The molecule is a member of morpholines.The molecule is a prostaglandin carboxylic acid anion that is the conjugate base of 13,14-dihydroxyprostaglandin F2alpha, obtained by deprotonation of the carboxy group. It is a conjugate base of a 13,14dihydroxyprostaglandin F2alpha. Impact of GPT-generated instruction-tuning data. In Table 5 , we provide the ablation studies of each training stage and our GPT-generated instruction dataset. The experimental results reveal that the instruction tuning with our generated multi-turn conversation data enhances the performance of LLaMo compared to the models trained via one or two-stage training without our GPT-generated instruction data. This indicates that instruction tuning with our GPT-generated multi-turn conversation data provides the model with more detailed and instruction-following guidance.Instruction tuning v.s. multi-task learning. Table 6 shows the advantages of instruction-tuning based on task instructions compared to multi-task learning using the simple task identifier. We use the task name as a simple task identifier for multi-task learning. From the table, the model without instruction tuning (Stage 1) achieves BLUE score of 35.5 and 7.3 on molecule description and IUPAC prediction tasks, respectively. The multi-task learning approach improves the scores to 36.9 for molecule description and 49.4 for IUPAC prediction. However, the instruction-tuning method demonstrated the most significant enhancement, achieving the highest scores of 37.8 for molecule description and 49.6 for IUPAC prediction. These results indicate that instruction tuning outperforms both the baseline and multi-task learning methods, suggesting its effectiveness in improving model performance on general-purpose training.Visualization of attention maps. We visualize the attention map to explore the effect of the multilevel graph projector in Figure 4 . The figure illustrates the attention maps of graph tokens for generating coarse-grained (left) and fine-grained (right) descriptions. Interestingly, the attention scores of the low-level are relatively higher than the high-level when generating fine-grained captions, whereas the attention value of the high levels is high when generating coarse-grained captions. This indicates that both low and high-level graph structural information is crucial in expressing the molecules, and the attention matrix is adaptive to the caption types.Qualitative analysis.  
conclusion: 
 We propose LLaMo: Large Language Model-based Molecular graph assitant, an end-to-end trained Large Molecular Graph Language Model, to perform various molecule-related tasks with a single model. For the projector, we newly introduce a multi-level graph projector, which addresses the over-smoothing problem of the graph encoder and captures multi-hop graph information. We also present machine-generated instruction-following data in the form of multi-turn conversations to improve the instruction-following capabilities of the large language model. GIN [21] with 5 layers for the graph encoder, which has widely been applied to molecule graph understanding tasks [4] . Formally, GIN updates node representations zv with the following equation:EQUATIONwhere ϵ (l+1) is a learnable parameter or a fixed scalar. The aggregate function AGGREGATE (•) is the sum operation and the MLP function is used for the combine function UPDATE (•). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We release the code and checkpoint with the safeguards. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We explicitly mention the license of used assets in the supplement. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: We also conduct experiments to validate the effectiveness of our LLaMo on chemical reaction prediction, such as forward reaction prediction and retrosynthesis tasks in Table 7 . The table demonstrates that our LLaMo still performs well on the chemical reaction tasks. LLaMo achieves the best performance on Levenshtein, RDK FTS, MACCS FTS, and Morgan FTS metrics across diverse baselines in all tasks, which indicates that LLaMo successfully comprehends molecular graph structure. We conjecture that our multi-level graph projector helps the large language model understand the molecular graph structure by representing multi-hop structural information to the molecular graph tokens.Our code is implemented based on PyTorch [55] library. Also, we adopt PyTorch Geometric (PyG) [56] , and Huggingface transformers [57] to utilize the graph architectures and Large Language Models (LLMs). PEFT [58] and OpenDelta [59] libraries are used for parameter-efficient fine-tuning of LLMs, i.e., LoRA. We use LLaMA 2 chat 7B model [9] and Galactica 1.3B [2] as our base language model. We leverage GIN [21] with five layers initialized based on the MoleculeSTM graph encoder [60] , which is pre-trained with text-graph contrastive learning [61] . We use LoRA to train the large language model in stage 2. We use OGB [62] , a smiles2graph function, to convert SMILES representations to 2D graphs. Our experiments are run on 4 × A6000 GPUs or 4 × V100 GPUs and 2 × A6000 GPUs for LLaMA2 and Galactica, respectively. In stage 1, the AdamW [63] optimizer is adapted with an initial learning rate of 1e-4 (minimum learning rate is 1e-5 and warmup learning rate is 1e-6). The warmup step is 1,000 and the cosine scheduler is applied. In stage 2, the initial learning rate is set to 5e-5 (minimum learning rate is 5e-6 and warmup learning rate is 5e-7).To evaluate the efficacy of the proposed method, we fine-tune baseline models and evaluate them for three tasks such as 1) molecule description generation, 2) IUPAC name prediction, and 3) property question answering. We conducted experiments under two major settings: generalist and specialist models. In the generalist setting, one model handles all three tasks, whereas in the specialist setting, we train a model for each downstream task. For generalist experiments, we use datasets derived from PubChem and QM9. To be specific, for 'molecule description generation', and 'property prediction', we use the datasets derived from PubChem and QM9 of MoleculeNet [64] as in Mol-Instructions [48] .For IUPAC name prediction, a dataset derived from [3] For training the specialist variant of LLaMo, we follow MolCA [3] to train the model with the pretrain split of PubChem324k in the stage 1 phase and fine-tune the model for each specific downstream task in stage 2. For the inference under specialist experiments, where a model is individually finetuned for a specific downstream task, we use a test split of PubChem324k [3] , ChEBI-20 [29] and IUPAC name prediction dataset from [3] .We report BLEU [65] and METEOR [66] for the molecule description generation and IUPAC name prediction tasks. MAE is reported for property QA.BLEU. The BLEU metric measures the quality of generated text by comparing n-gram sequence between the generated text and the reference text, which can be formulated as:EQUATIONwhere N is the number of n-grams and p n is the precision, i.e., the ratio of the number of n-grams in the generated text appearing in the reference text. The BLEU score also takes into account sequence length with Brevity Penalty (BP) as:EQUATIONwhere c and r are the lengths of generated and reference texts, respectively. This encourages the model to avoid generating short sequences. In our experiments, we use BLEU-4 as the default BLEU metric.METEOR. The METEOR metric is proposed to consider both precision and recall between the generated text and the reference text, which is as follows:, R = number of matched words number of words in reference text ,EQUATIONEQUATIONEQUATIONwhere a chunk is a set of uni-grams which are adjacent in the generated text and in the reference text. Similar to BLEU, the Penalty is reflected to take into account the length of generated text. Therefore, the METEOR metric is specialized to measure the morphology, fluency, and adequacy of text rather than sequence order since it does not use n-grams.MAE. Mean Absolute Error (MAE) aims to measure the average magnitude of errors between the ground-truth values and predicted values, which is defined as:EQUATION)where y n is a ground-truth and ŷn is a model prediction.For the generalist models, we train our LLaMo based on Llama-2-7b-chat [9] as a backbone language model for a fair comparison with Mol-Instructions [48] . We compare our LLaMo with (1) LLM-based generalist models including Galactica [2] , Llama2-7b-chat [9] , GPT-3.5, and GPT-4, (2) Molecule instruction-tuned generalist model such as Mol-Instructions [48] . Since GPT-3.5 and GPT-4 have difficulty in solving the tasks without in-context learning, we additionally measure the performance of GPT-3.5 and GPT-4 with 4-shots in-context learning, which are GPT-3.5 (ICL) and GPT-4 (ICL). The 4-shot exemplars are selected by computing the Tanimoto similarity [67] using a 2048-bit Morgan Fingerprint [68] with RDKit2 , choosing the four molecules most similar to the target molecule from the train split of each dataset. For the specialist models, we train our LLaMo with Galactica 1.3B [2] for a fair comparison with MolCA [3] . We use single-task specialist models as baselines, including MolT5 [29] , MoMu [4] , and MolCA [3] .In this section, we provide a brief introduction to each task and dataset utilized in our research.Generating the description of a molecule is considered one of the most important tasks in molecular language models. For a given molecule, we aim for the model to generate an accurate and informative description including various chemical properties, functional groups, biological roles, and real-world applications of the molecule. Developing a model capable of generating such descriptions is highly valuable because it has the potential to discover information about molecules that is currently unknown or very expensive to find out, thus serving as a powerful assistant for various tasks in chemistry and biology. Therefore, various works [3, 4, 29, 69, 70, 71] have tried to enhance the ability to generate appropriate descriptions of chemical compounds using language models or multi-modal language models.For testing the performance of molecule description generation of LLaMo and previous models, we utilize the molecular description generation dataset of Mol-Instructions 3 [48] , based on PubChem database [44] for generalist models, and both PubChem324k [3] and ChEBI-20 [8] for specialist models. PubChem324k is constructed by collecting 324k molecules and their associated text information from the PubChem database. ChEBI-20 is the most commonly utilized benchmark in this task, consisting of selected 33,010 pairs of molecules and descriptions from ChEBI [72] . Each description of ChEBI-20 contains more than 20 words and includes various and rich information about molecules, such as conjugate base/acid, functional parent, and enantiomer of molecules. We employ a test split of each dataset: 1,000 samples of Mol-Instructions and 2,000 / 3,300 samples of PubChem324k / ChEBI-20. Similarly, we filter out samples of which SMILES representation cannot be converted into the molecular graph via RDKit.IUPAC name prediction.IUPAC (International Union of Pure and Applied Chemistry) nomenclature [73] provides a systematic naming convention for molecules based on pre-defined rules, eliminating the ambiguity in molecular names. It is the standard method for naming molecular compounds in chemistry, and as such, numerous pieces of chemical literature utilize IUPAC names to represent molecules. Consequently, previous studies [2, 3] have adopted the task of predicting IUPAC names from SMILES representations to evaluate the chemical understanding capability of molecular language models. We also follow this approach in our research. To assess the performance of LLaMo and baseline models, we use the PubChem324k dataset [3] again. The test split of PubChem324k offers curated high-quality samples of 2,000 molecules, each with corresponding SMILES representations and IUPAC names. We provide the SMILES representation of each molecule to the model along with a prompt and then compare the model-generated IUPAC names with the ground-truth IUPAC names to evaluate performance.Property prediction. In chemical and biological domains, particularly in drug discovery, exploring chemical compounds that satisfy specific chemical properties is crucial. Therefore, the ability to estimate the chemical and physical properties is essential for chemical foundation models. Additionally, leveraging machine learning-based approaches for predicting molecular properties has proven to be much more efficient than traditional approaches in computational chemistry. Inspired by these facts, property prediction with LLMs has recently drawn attention from researchers [2, 69, 70, 71] . So, we also assess the property QA performance of LLaMo and baselines using the dataset introduced by [48] . This dataset is a subset of QM9 dataset from MoleculeNet [64] , a widely used benchmark for various chemistry machine learning tasks. The original QM9 dataset contains numerical values for 19 chemical properties per molecule, but [48] focuses on three properties related to molecular orbital energy: HOMO (Highest Occupied Molecular Orbital) energy, LUMO (Lowest Unoccupied Molecular Orbital) energy, and the HOMO-LUMO gap (in Hartree units). [48] also created distinct question-form instructions for each property to make language models understand the task and accurately generate continuous values. Consequently, using the given instruction for each molecule in the 2,000 test samples and its SMILES representation as input, we have LLaMo and baseline generalist models predict the properties of the given molecules. Similarly, we filter out samples of which SMILES representation cannot be converted into the molecular graph via RDKit.Forward reaction prediction. Understanding chemical reactions is crucial for advancing various chemical and biological fields, including pharmaceuticals, materials science, and environmental technology. This understanding enables researchers to design efficient synthesis pathways and develop new chemical compounds. Therefore, if a molecular model can effectively comprehend these chemical reactions, it can serve as a powerful assistant in the research and development process. This concept has emerged through the task of forward reaction prediction, and several prior models [48, 54] have conducted experiments to address this task. This task focuses on the proactive determination of potential products resulting from a chemical reaction based on given reactants and reagents. This approach is also significant from the perspectives of efficiency and environmental sustainability, as it reduces the need for experimental trial and error in chemical development and research by using these models to virtually conduct simulated experiments. We utilize the forward reaction prediction segment of Mol-Instructions [48] , based on the USPTO [74] database, to assess the performance of LLaMo and other models. This dataset contains question-form instructions for predicting reaction products, with reactants and reagents separated by a period ('.') as input, and the corresponding product of the reaction as the target output. It also doesn't specify what the reactants and reagents are to create a task that more closely resembles real-world scenarios. Finally, using the test samples from this dataset, we provide the SMILES representations of the reactants and reagents to the model and their molecular graphs with a prompt, aiming to predict the product molecule in SMILES format as well.Retrosynthesis. Retrosynthesis shares a similar context with forward reaction prediction but approaches chemical reactions from a different perspective. The retrosynthetic analysis begins with the target compound and works backward to identify potential reactant molecules for its synthesis. This reverse approach is as valuable as forward reaction synthesis because it aids researchers in discovering effective and efficient synthetic methodologies for generating target molecules. This is particularly important in various chemical applications, such as drug discovery, where identifying chemically valid and economical processes for synthesizing target drug molecules is essential. As a result, several previous studies [48, 54] have adopted this task to analyze the capability to deeply understand chemical knowledge and take advantage of this knowledge to specify the chemical reactants of a given product. We also follow this approach and demonstrate its capability using the retrosynthesis section of the Mol-Instructions [48] dataset, which originates from the USPTO_500MT [75] . Each sample consists of a product molecule and the required reactant molecules, separated by a period ('.'). You are an expert chemist. Please strictly follow the format, no other information can be provided. Given the molecular SMILES, could you provide a description of this molecule? You will be provided with several examples of molecules and their descriptions. You are an expert chemist. Please strictly follow the format, no other information can be provided. Given the molecular SMILES, your task is to predict the IUPAC name using your experienced chemical IUPAC name knowledge. You will be provided with several examples of molecules and their IUPAC names.The molecule's IUPAC name is (3S)-3-(4-hydroxy-3-methoxyphenyl)-3,4-dihydro-2H-chromen-7-ol ...Molecule SMILES: COc1c([C@@H]2COc3cc(O)ccc3C2)ccc2c1C=CC(C)(C)O2 The molecule's IUPAC name is (3R)-3-(5-methoxy-2,2-dimethylchromen-6-yl)-3,4-dihydro-2H- chromen-7-ol Molecule SMILES: COC1=CC(=O)C(C2COc3cc(O)ccc3C2)=CC1=O The molecule's IUPAC name isFor each test sample, we provide the product's SMILES representation, its molecular graph, and the given instruction prompt as input to the model to generate possible reactants in SMILES format.In this section, we provide input prompts used to evaluate GPT-3.5 and GPT-4 on molecular description generation task (Table 8 ), IUPAC name prediction task (Table 9 ), and property question answering task (Table 10 ). We also provide the input prompts for machine-generated instruction-tuning data generation (Table 11, 12 ).We provide additional samples to show the over-smoothing problems in Figure 7 . J More Qualitative SamplesWe provide further qualitative results of LLaMo w/ and w/o graph in Figure 8 , and LLaMo w/ and wo MGProj in Figure 9 , respectively. GT: The molecule is a prostaglandin carboxylic acid anion that is the conjugate base of 13,14dihydroprostaglandin F2alpha, obtained by deprotonation of the carboxy group; major species at pH 7.3. It is a conjugate base of a 13,14dihydroprostaglandin F2alpha.GT: 13,14-dihydroprostaglandin F2alpha Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.GPT-assisted Molecular Graph Instruction Data GenerationInstruction data are essential for improving the instruction-following capabilities of LLM-based models. Despite active research on instruction-tuning, the instruction-following data for molecularhttps://github.com/rdkit/rdkit. Copyright (c) 2006-2015, Rational Discovery LLC, Greg Landrum, and Julie Penzotti and others. Licensed under BSD3-Clause license 3 Copyright (c) 2023 ZJUNLP. Licensed under CC-BY 4.0 license 


Decision: The paper proposes a multi-level graph projector to create tokens which can complement LLM instructions. The multi-level nature is obtained by applying the embeddings from different GNN layers. It integrates well domain knowledge (e.g., different molecule modalities) and novel technology, contains an extensive evaluation, and shows good performance using state-of-the art LLMs. The paper also proposes a novel molecule + text dataset.

All four reviewers recommend to accept the paper and I recommend that as well.

Reviews: review: paper_summary: paper_summary: This paper presents LLaMo, a novel enabling LLMs and instruction tuning in molecular domain. To bridge the gap between different modalities, the paper also proposes a projector that transforms the graph representations into graph tokens level by level. The authors conduct extensive experiments and compare LLaMo with several proper baselines. The results are convincing.
strength_weakness: strength_weakness: 1. Though the authors conduct several experiments, they still lack some comparison between different GNN and LLM backbones. In section 5.3(Impact of Multi-level Graph Projector), will different GNN backbones harm or improve the results? Also, under the same experiment settings, the paper lacks the comparison between different LLM backbones. Such as the settings in Table 2, will the results be better for LLama-7b?
2. With several layers of GNNs, the efficiency of this projector may not be good.
questions: questions: 1. For line 138-139, could you explain in detail for the $\[p^{(l)}_1, .... , p^{(l)}_b\]$, such as what is the learnable prompts and how do you initialized it?
2. For the few-shot prompts part, how do you choose the few-shot examples?
limitations: limitations: The authors address the limitations of this work.
paper_summary: paper_summary: This paper proposes a Large Language Model-based Molecular graph assistant (LLaMo), which can enhance the molecular graphs's general-purpose understanding and generation capabilities. By integrating molecular graph encoders with large language models, LLaMo enables instruction-following responses in the molecular domain. The paper conducts extensive empirical studies and justifies the effectiveness of the proposed method.
strength_weakness: strength_weakness: - The multi-level graph projector shares an idea similar to JKNet[1]. In addition, Graph Transformers [2] can also handle the over-smoothing problem, especially for small molecules. The author could consider discussing how the proposed techniques differ from these referenced works.
- The paper uses function groups as prior information for training; would this cause information leakage? The downstream task is to predict these groups or generate molecular captions based on these motifs.
- The GPT-4 generates samples for instruction-tunning, but the samples's qualities are not assessed properly.

[1] Representation Learning on Graphs with Jumping Knowledge Networks. In ICML, 2018.

[2] Representation Learning on Graphs with Jumping Knowledge Networks. In NeurIPS, 2022.
questions: questions: 1. Could GPT -4 perform better with prior information instilled, such as function groups?
2. What is the purpose of using different LLM for LLaMo tasks (Tables 2 and 3)? Could better LLM result in better performance? Would there be some upper bounds?
3. Could LLaMo design or edit SMILES based on the user's instructions?
limitations: limitations: The idea is novel and effective. It would be better if the author could make LLaMo able to design or edit SMILES based on the user's instructions.
paper_summary: paper_summary: This paper proposes a molecule-text model, LLaMo, which aligns text and molecule modalities to tackle diverse downstream tasks with a language interface. Through two-stage alignment: training a graph encoder and then tuning the LLM through instructions, LLaMo aligns text and molecular representations well, and achieves promising performance on diverse downstream tasks.
strength_weakness: strength_weakness: 1. The implementation details of functional groups are not given in the main text nor appendix. Could you share more details on this part?
2. How is the graph modality treated in the instruction tuning phase? Is it fixed, and the graph token is augmented into the text token inputs?
3. I am curious about the design of the learnable query tokens. Why are the motif and graph-level representations modeled separately?
4. One factor that prevents me from giving a higher score is that this pipeline is widely adopted for vision-language models, so it does not surprise me.
questions: questions: Please refer to the questions given in the weakness part
limitations: limitations: Limitations and broader impacts are discussed in the appendix. Guidelines should be removed from the checklists.
paper_summary: paper_summary: The paper introduces LLaMo, a Large Language Model-based Molecular graph assistant. LLaMo is a model that integrates a GNN encoder, a multi-level graph projector, and a language model. The projector uses a cross-attention mechanism to convert graph representations into graph tokens by abstracting outputs from each GNN layer and motif representations. Additionally, the authors use machine-generated molecular graph instruction data, leveraging GPT-4, for instruction-tuning. Extensive experiments show that LLaMo excels in some tasks.
strength_weakness: strength_weakness: In order to perform instruction-tuning, the authors utilized GPT-4 to generate molecular graph- text instruction-following data using graph-text pair datasets with human-written instructions. Although this approach enhances the results, it lacks a robust evaluation of the generated data's quality and validity. It is crucial to develop methodologies to rigorously assess the accuracy and relevance of the GPT-4 generated instruction data to ensure the reliability of the model's performance improvements.
questions: questions: The two-step training process involves initially training the graph encoder in Stage 1, followed by fine-tuning the Large Language Model using LoRA in Stage 2. Why is the multi-level projector set as trainable during both training stages?
limitations: limitations: The experiments predominantly concentrate on a few tasks such as molecular description generation, property prediction, and IUPAC name prediction. This limited focus may not fully reveal the model’s versatility and potential shortcomings across a wider spectrum of molecular tasks.

Comments: title: Official Comment by Reviewer P4Gm
comment: Thank you for the clarification. It addresses my concerns, so I will raise my score to 7.
We sincerely appreciate the reviewer's comments and positive score.

As the reviewer suggested, we will include the discussion on the quality of GPT-generated data in the final version.
We sincerely appreciate the reviewer's positive comments and increased rating.

As the reviewer suggested, we will provide more details of the functional group representation in the final version.
We sincerely appreciate the reviewer's positive comments and increased rating.

As the reviewer suggested, we will include a discussion on over-smoothing, the quality of GPT-generated data, and other tasks in the final version.
I thank the authors for the rebuttal. It addressed most of my concerns and I tend to keep my score.
Thanks for your detailed responses. It addresses all my problems. I raise my score from 6 to 7.
Thanks for the clarification and additional results. I have raised my score to $7$.