Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planning
Authors: Gang Liu
Michael Sun
Wojciech Matusik
Meng Jiang
Jie Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Languge Models
Large Languge Models
Graph Diffusion Models
Inverse Molecular Design
Retrosynthesis
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose Llamole, a multimodal LLM that integrates a base LLM with the Graph Diffusion Transformer and Graph Neural Networks for multi-conditional molecular generation and multi-step reaction inference within texts.

Abstract: While large language models (LLMs) have integrated images, adapting them to graphs remains challenging, limiting their applications in materials and drug design. This difficulty stems from the need for coherent autoregressive generation across texts and graphs. To address this, we introduce Llamole, the first multimodal LLM capable of interleaved text and graph generation, enabling molecular inverse design with retrosynthetic planning. Llamole integrates a base LLM with the Graph Diffusion Transformer and Graph Neural Networks for multi-conditional molecular generation and reaction inference within texts, while the LLM, with enhanced molecular understanding, flexibly controls activation among the different graph modules. Additionally, Llamole integrates A* search with LLM-based cost functions for efficient retrosynthetic planning. We create benchmarking datasets and conduct extensive experiments to evaluate Llamole against in-context learning and supervised fine-tuning. Llamole significantly outperforms 14 adapted LLMs across 12 metrics for controllable molecular design and retrosynthetic planning. Code and model at https://github.com/liugangcode/Llamole.

Published as a conference paper at ICLR 2025
MULTIMODAL LARGE LANGUAGE MODELS FOR IN- VERSE MOLECULAR DESIGN WITH RETROSYNTHETIC PLANNING
Gang Liu'; Michael Sun"; Wojeiech Matusik", Meng Jiang', Jie Chen
University of Notre Dame
MIT CSAIL MMIT-IBM Watson Al Lab, IBM Research
{gliu?, mjt end)nnnnn.. edu, (msun415, wojciech)@ mim tuiiteed eiv. edu, chenji eeuus. bmm com
ABSTRACT
While large language models (LLMs) have integrated images, adapting them graphs remains challenging, limiting their applications in materials and drug design. This difficulty stems from the need for coherent autoregressive generation across texts and graphs. To address this, introduce Llamole, the first multimodal LLM capable interleaved text and graph generation, enabling molecular inverse design with retrosynthetic planning. Llamole integrates base LLM with the Graph Diffusion Transformer and Graph Neural Networks for
molecular generation and reaction inference within texts. while the LLM, with enhanced molecular understanding. flexibly controls activation among the different graph modules. Additionally, Llamole integrates A search with LLM- basdddaaedd- cost functions for efficient retrosynthetic planning. We create benchmarking datasets and conduct extensive experiments to evaluate Llamole against in-context learning and supervised fine-tuning. Llamole significantly outperforms 14 adapted LLMs across 12 metrics for controllable molecular design and retrosynthetic planning. Code and model at
com 1 wtrp///iganggoodeeee Llmmole
INTRODUCTION
The potential of LLMs for molecular discovery has been actively explored (Jablonka et al. 2023). However, LLMs struggle in the chemical domain, exhibiting poor generation quality and planning capability (Guo et al. 2023). This is due to the unique graph structures of molecular data, which are challenging for LLMs that typically handle sequential texts.
Drug (Small Molecule) Design
Inverse molecular design requires LLMs to be controllable for generating molecular structures that meet multi-property and synthesizability requirements (Chen et al.) 2020 Gao al.
Balanced Accuracy
0.7
0.662
0.5
0.6
0.502
0.486
0.537
202T). These requirements can be detailed as questions for
0.4
0.3
LLM input, shown in Figure Answering these questions demands a comprehensive understanding of molecular struc- tures and their relationship properties. However, sequence- based LLMs struggle with this because they are pre-trained or fine-tuned solely on text representations of molecules, e.g.,
0.2
0.1
0.0
Best ICL Best SFTGraphGA Llamole
Material (Polymer) Design
Mean Absolute Errer
2.0
1.632
SMILES (Wemiinger 1988). To illustrate this, we investigate 14 LLMS for molecular generation in Figure [ across 10K drug and material questions: ten using in-context learning (ICL) and four with supervised fine-tuning (SFT). LLMS generate molec- ular structures based on the questions and their properties obtained through oracles Details of the experimental set-ups the
1.5
1.372
1.0
0.5
Best SFTGraphGA Liamole Figure Comparison Control- lability: Results are averaged from the best numbers from Table [ simple yet effective graph-based method, in designing molecules with satisfactory properties.
0.0
and results can be found in Section 5 In summary, best LLMs perform worse than GraphGA (Gao et al. 2022).
"This work done while GL. and MS interned the MIT-IBM Watson AT Lab, IBM Research.

---

Published as a conference paper at ICLR 2025
"Can design molecule that inhibits both HIV and Beta-Secretase 1, with molecular weight around 284.33, and 2 ragg including 1 oromatic and I aliphotic ring, and outline its synthesis pathwary?
Question
(a) In-Context Learning (b) Supervised Fine-Tinne Tuning
(c) Multimodal Supervised Fine-Tuning
Question
Demonstrations (Text Only)
Question
Instruction Data
Question
Instruction Data
(Text
+
(Text Only)
R (Finetune)
I (Finstune)
Pre-trained
Fine-tuned
Pre-trained
Liamola
Pre-trained
LLM
LLM
LLM
LLM
Integrrate w/
Pre-trained
Text Only Generstion
Fext Only Generetion
Text and Graph Goneration
Modela
"Io synthesiie it, follow these procedures: "To synthesize follow these proceduures The molecule is complex, A solution of ... The applied reaction is" solution of The applied reaction is aromatic componnd... It
"To satisffy the reguirements:
Po
Answer
"The first reactord is commercially available "All reactants avatlable now.
Figure 2: Three LLM-based methods for molecular design. The question outlines requirements for properties, structures, and synthesis, addressed as follows: (a) In-Context Learning and (b) Supervised Tiniin use text-only data for demonstrations and instruction tuning, respectively. (c)
The proposed Llamole uses graph-text multimodal data to fine-tune the LLM, integrating parameter- frozen graph models for interleaved text and graph generation with reaction inference.
As illustrated in Figure 2] practical answers for molecular design more complex than what can be achieved by using graph methods or LLMs alone. The generation begins with a paragraph describing the intended molecule for multi-conditional generation, followed by retrosynthetic planning, detailing each synthesis step-one reaction per paragraph- in reverse order, from the target molecule to purchasable reactants. Thus, multimodal LLMs (MLLMs) are essential, with LLMs handling text generation and graph models managing molecular design.
In this work, propose the multimodal Large language model for molecular discovery (Ilamole). As shown in (ig the model seamlessly integrates LLMs and graph models within multi- modal autoregressive framework, enabling the interleaved generation of text, molecules, and reactions. It predicts the next token across both word and chemical spaces, framed as multi-class prediction tasks for word vocabulary, atom/bond types, and reaction templates. For retrosynthetic planning. Llamole integrates A# search to efficiently identify synthesis pathways for the designed molecule.
To implement Llamole, we augment a base LLM with two pre-trained graph modules: the Graph Diffusion Transformer (Graph DiT) for multi-conditional molecule generation (Liu et al. 2024c) and a GNN for reaction template prediction. The base L.M controls the generation flow using dieger--uery-predictiin approach with two sets of trigger tokens for the Graph DiT and GNN, respectively. Upon predicting a trigger token, one or a few query tokens summarize the prior text as vectors, activating the corresponding graph modules and generating molecules predicting reaction templates. Afterward, the base LLM can resume text generation, aided by a graph encoder that encodes the previously generated molecule. In retrosynthetic planning. the LLM computes heuristics to efficiently assist the A* search in navigating the vast reaction space for multi-step generation.
Our work has several highlights. First, Llamole is the first MI.MM capable of inverse molecular design with the interleaved generation of text and graphs. Second, we curated a dataset along with fine- tuning instructions to benchmark complex yet realistic molecular design outcomes, including human conversation. Third, we present compelling experimental results that demonstrate the competitiveness of Llamole against 14 LLMs and GraphGA, as shown in Figure T With details in Tables and 2 Llamole improves LLM performance by up to 80.9% across 12 metrics for controllable molecular generation and increases the success rate for retrosynthetic planning from 5.5% to 35%.
2
PRELIMINARIES
2.1
AUTOREGRESSIIE LANGUAGE MODELING
Given sequence of word tokens W = [W1, W2,... wr} } of length L from the vocabulary W, LLMs parameterized by 0, decompose the joint distribution as po, (W) I[ Po, (wal Wex), where We;
2

---

Published as a conference paper at ICLR 2025
represents the tokens preceding the i-th position. These models are optimized by minimizing the negative log- ligeliidl between their predictions and the empirical data distribution, resulting in:
LLM =
Σ -logpo, (wil Wex).
2.2 MOLECULAR DESIGN WITH GRAPH DIFFUSION MODELS
Molecular graphs can be modeled through diffusion in discrete spaces (Austin et al. 2021 Vignac
et al., 2022 Liu et al. 2024c). Given a one-hot encoded data point R² with F categories (e.g., a node or an edge), discrete models perform diffusion using a transition matrix Q, where
[Q']s² [j (x} x -x)))) for i,j E []. The forward diffusion with Q is: q(x" x
Cat(x;; P = x x²)) Q'), where Cat((;; p) denotes the categorical distribution over x with probabilities given by p. Starting from the original data point x", we have q(x* | xx) = Cat (x"; p = x')
where Q' = Ilge Q' The forward diffusion gradually corrupts data points. When the total timestep Tis large enough, q(x") converges to stationary distribution. The reverse process samples from q(x)) and gradually removes noise. The posterior distribution 9 x') is calculated as q(x-11xx x² x²) ((Q²) @ x xQQ Using a denoising model parameterized by 02, this posterior can be approximated by po, (x" -xx x²). For inverse molecular design with multi-property constraints, the denoising model can be optimized by minimizing the negative oogllikeldooo for x":
LDM Rxy(x)) Be(e))))) [- [-logg log Po,, (x") [1,, (2,,2;. 3 CM Cuext, x')]
where M molecular properties are denoted by {cz M and the text embedding is Clext- These conditions can be handled by Graph DiT (Liu et al. 2024c) without introducing additional predictors for guidance (Ho Salimans 2022).
2.3 ONE-STEP REACTION PREDICTION WITH GRAPH NEURAL NETWORKS
Retrosynthesis needs predict the reverse of synthetic reaction, which decomposes chemical products into reactants. A GNN parameterized by 0g takes the product Product to predict the label E R in the reaction space R. This label is interpreted the template and determines the reactants. With the text condition Cixx,, we minimize the negative lig-lioel of the label distribution qr)):
Spredictor (r)(() [- log pog (r Crexts Gpreduet)]
2.4 RETROSY PLANNING WITH A SEARCH
Given molecules from the structure space G, a subset Gevail represents available molecular structures that can be purchased as building blocks for synthesis. For any target Guuget, one-step prediction of the reversed reaction may not yield reactants within Gavai. Thus, retrosynthesis typically requires multi-step planning to find pathways from building blocks to the target in reverse order. The search space of chemical reactions can be navigated using A* an AND-OR tree T. with Gurger the root. Reaction nodes follow an "AND" relation, requiring all child reactants, while molecule nodes follow an "OR" relation, meaning the product can be synthesized by any child reaction (Chen et al. 2020).
Selection: We select nodes from the frontier F(T) containing unexplored molecule nodes to expand the tree. Given an oracle cost function J(.), the next node is selected as Gnext arg minGe J[G) to minimize the cost. A well-designed (.) improves search efficiency and aids in global optimality.
After selecting Gnext a single GNN predictor call can generate many one-step ret- rosynthesis proposals. The GNN provides top-candidate reaction templates, each linked to different reactants. Thus we can form molecule nodes under the reaction node as an AND-OR stump.
Expansion:
Update and Cost: After expanding Gnext- the tree becomes T'. We update the nodes in T' for the next iteration. * selects the path that minimizes J(-) = -Jeurent (.) + -heuriste (-), which includes the cost from the start to the current node Jeunear(-) and a heuristic estimate of the cost to the goal Jheurstie (-)- With the GNN predictor, the negative log- log-likel of the reaction can be used to compute path cost Jeurrene (-) to the leaf molecule node, we design Jreuristic (-) with the LLM in Llamole.
3

---

Published as a conference paper at ICLR 2025
Multimodal Autoregressive Framework
Multi-Canditional Molecular Generation
Retrosynthetic Planning
Sample
"To satisfy... ...It <design>>
Vector
G
"synthesize ... Srating
PrOD.
Llamole: Multimodal Large Language Model for Molecular Discovery
(a)
(c)
(b)
(d)
(a)
"css> To satisfy... ...i <design> Squery?
"To synthesize 61 <retm> <query>>
"To... Gs..."
Details of Active Modules
Details A*
Reaction Node
i Molecule Node
Text Vector
Text
Molacule
Template
+ewrrent
Incuristie
LLM
LLH
Gragh
GNN
"fer""
Deroder
Decoder
Predictor
Word Enooddr
Graph Encarer
Linear
Linear
8g
Llamole
Property
Graph
(8) "Entnaae vematalog
Query
Query
Text
Vector
Vector
Geusti G.GG Garnic Expansion
(a)
(b)
(c)
(d)
Selection
Figure 3: Overview of Llamole: Trigger tokens (<design> and <retro>) switch active modules
from the base I.LM the respective graph component. The subsequent <que ry> token utilizes output vectors from the L.M to summarize past texts as conditions. Using these, Llamole generates molecules and predicts one-step reactions. Enhanced with graph encoder and A search, Llamole efficiently plans synthesis routes through selection and expansion iterations on the AND-OR Tree.
LLAMOLE: LARGE LANGUAGE MODEL FOR MOLECULAR DISCOVERY
3.1 MULTIMODAL AUTOREGRESSIVE MODELING
In molecular discovery, the sequence may include molecular structures 9 and retrosynthetic reactions R with each molecule or reaction tokenized. The sequence {V1 ........N.... where y: W U G U R, combines these tokens. The sequence is interleaved with tokens in different spaces. Suppose the molecule appears at position then, we typically see:
Y1 E 9, Y+II+L E W, V++L-E
where L is the length of the text following the molecule position The sequence starts with text. If position denotes the first molecule in the sequence, then Yzs E W: otherwise, V/-1 E R. To handle non-word tokens, we integrate domain-specific Graph DiT and GNN with the LLM, forming multimodal LLM, i.e., Llamole. Parameterized by Q, Llamole unifies the cross-entropy losses from Eqs.
T to (3) into autoregressive modeling:
Clammo LIM + LDM + Epredictor
Σ-log log pe (Y.).
LDM interprets Yes as the input conditions, including desirable molecular properties and text con- ditions Mc U (Cuss) for the autoregression of Y; in G. In Lpredictors Yz; represents Gproduct and Crext- Here, Gprsduct is generated from previous diffusion models or as intermediate & Gavail in retrosynthesis. The autoregression for the label Y; is performed in the reaction space R.
We present an overview of multimodal autoregression with Llamole in Figure 3 divided into con- trollable molecular generation and retrosynthetic planning. The base LLM performs multiple roles: generating text, controlling the switch of active modules, and providing cost functions for A* search. Augmented with the graph models, the overall parameters in Llamole are θ (01, 62 fy Φ1 Φ2 Φs }. where Φ1 and Φ project text into Cux for the Graph DiT and GNN predictor, respectively. The graph encoder with @3 projects molecule tokens into the LLM. Next, we detail the design space of Llamole.
3.2 LLAMOLE DESIGN SPACE
Llamole consists of base LLM and two pre-trained graph modules: the Graph DiT for molecule generation and the GNN for one-step reaction prediction. The base LLM employs a trigger-query- prediction approach using two sets of special tokens switch between modules.
4

---

Published as a conference paper at ICLR 2025
Trigger Tokens. Llamole defines two special trigger tokens to augment the word vocabulary W: <design> for switching between the LLM and Graph DiT, and <retro> for switching between the LLM and GNN predictor. When trigger token is predicted, Llamole activates the corresponding graph model. After molecule generation or reaction prediction, the active modules revert to the I.LM.
Query Tokens. We introduce another set of special tokens, named query tokens <query> automati- cally placed after triggers. They use the LLM to query previous tokens and output hidden states as Chidden- A linear layer is applied: Ctext Linear( (hhidden. Linear(Chidden) adjusting the input size for the graph models. We use different query tokens for different triggers. Query tokens allow us to share parameters Φi and Φ2 with l1. enhancing both efficiency and effectiveness. We can apply ensemble methods by repeating the query tokens multiple times and averaging the Cnidden values (Dong et al.) (003)).
Besides the special tokens, Llamole enhances molecule understanding with graph encoder and uses the LLM to provide the cost function in A search for retrosynthetic planning.
Graph Encoder. The graph encoder parameterized by Φ3 replaces the word encoder in the LLM tokenizer for molecule tokens. The LLM decoder takes molecule embeddings from the graph encoder, along with text embeddings from the tokenizer, into the Transformer layers for next token generation. We use pre-trained Graph Isomorphism Network (GIN) (Xu et al. 2018) as the graph encoder, optimized via molecule-text contrastive learning similar to CLIP (Radford et al. 2021).
A Cost Function with LLM. We define Jneuristic as multi-choice problem, where each choice, assigned a score, represents synthesis complexity, from few to many steps. The LLM estimates the remaining synthesis steps for the leaf molecule node G E F(T) Gavait in the search tree T. It outputs probabilities for each choice, and Jneuristi is computed as the weighted score by averaging the scores with their probabilities. For G E F(T) n Gevail, heunistic = 0.
3.3 END-TO-END MODEL FINE- TUNING AND GENERATION
Supervised Tunnng We use multimodal SFT to connect the base L.M and other graph modules in Llamole (Ouyang et al. 2022). Specifically, we freeze the parameters for the graph modules (02 and 0g) and fine-tune the LLM parameters 01, the learnable special tokens, and the linear layers for the query tokens (Φ) and Φ2). We freeze the parameters of the pre-trained graph encoder (Φ3) and add a tunable linear layer between it and the LLM decoder. The optimization can be conducted end-to-end with Eq. 4 The SFT aligns the LLM with domain-specific graph models. To maintain generality in the base LLM, we employ parameter-efficient LoRA (Hu et al. (021).
Interleaved Generation. Given a question as shown in Figure 2 I.lamole performs controllable and synthesizable molecular designs, as presented in Figure For the controllable generation, Llamole uses the base LLM to analyze the requirements and switches to the Graph DiT for generating Guaget when the trigger is predicted. For the synthesizable generation, Llamole plans synthesis routes for Guaget- A search on the AND-OR tree T aids in multi-step generation, interleaving molecule and reaction nodes, with Giarget the root. During each celection-expasiion iteration, A selects Gnext = arg minGEF(T) J(G from the leaf nodes F(T). The graph encoder embeds molecule tokens into the LLM, which generates reaction conditions until the token <retro> is triggered, activating the GNN predictor. The predictor then predicts the top-50 templates as reaction nodes, along with corresponding reactants as molecule nodes for the next iteration. A stops after finding a route from Gunget to Gavail with satisfying all AND-OR constraints, or if it fails after 30 seconds or 300 iterations. Upon success, the text with the corresponding reaction along the route returned for retrosynthesis, otherwise, the base LLM directly generates texts.
4 BENCHMARKING FOR MULTIMODAL MOLECULAR DESIGN
To train Llamole, we need instruction data that provide detailed language supervision and evaluation covering synthetic complexity, drug and material utility, and reaction conditions. However, existing data based on PubChem (Kim et al. 2021 are only usable for small molecules and lack such details. Thus, we create MolQA, large-scale graph-text multimodal instruction dataset for systematic LLM benchmarking used in Section We also create MolPair with graph-text and reaction-text pairwise data to pre-train graph modules, as detailed in appendix D To this end, we first collect multisource molecule data (Figure 4). with details in appendix C. Then create MolQA and MolPair.
5

---

Published as a conference paper at ICLR 2025
MoiQA
MolPair
(500K Graph-lext Pairs)
(1.BM Reaction-Test Pairs!
Find Syn. Routes
139K QA
Thaining: 125K QAs
Generste
Graph
aosdenate gasteetc
"The suclentle dan
Text Pair
complecity scove and
relutionly single siramtue,
Anmotate
Property
Evaluation 11K Qus:
Annotate
Property
auseentar wright it relatively
Suntring on afihhani chate. Ji
Jone which could pusiively
orgiilonide (60 si and the subiure
oniie (12 gi was odled phosphoru
reflored ank stiiing hour
in pharmarskinolic
This prodhes shen
Denerate Answeer
Generste Question
Create
prapartios rack Ovanait
secrsstlllzdd mokee from
Reaction
the
properties
pendd F-ckioro f-wethyi-
Pair
sherpessti agead..
boquieolia aulling I22"C."
....e...........................
Supported by Multisource Collection and Quality Control
MoleculeNet (492K)
ThemCoon [=1K]
MSA (=1K)
USPTO (3.0M)
CHEMBL (2.2M)
ZINC (4M)
(11
FFV [0K)
Small Molecule
Polymeric Materials
Reaction
Figure 4: Creation of MolQA and MolPair: MolQA comprises two sets: a training set for ICL and (multimodal) SFT, and test set for evaluation. MolPair consists of graph-text and reaction-ten pairs, with red highlights indicating synthetic complexity, structure, and properties information.
MoIQA: Instruction Data Creation. USPTO reactions include text descriptions. We use Enamine's
1.3 million small molecules as Gavail The depth-first search identifies routes from reaction products (i.e. target molecules) to molecules within Gavill, resulting in about 139K routes with lengths ranging from 1 to 10. We sample around 1 1K routes (750 for materials and 9986 for drugs) for testing and use the rest for instruction tuning. We focus on eight popular properties for benchmarking (i.e. e. M 8
for { M in Eq. ) They include three drug-re related categorical properties (Wu al.
2018): (1)
HIV virus replication inhibition (HIV), (2) blood-brain barrier permeability (BBBP), and
(3) human
3-secretase inhibition (BACE) and five continuous material properties (Thornton et al. 2012): (4)
CO2 permeability, (5) N2 permeability, (6) O2 permeability, (7) fractional free volume (FFV), ), and (8)
thermal conductivity (TC). Not all target molecules have these properties. To enrich properties and texts, two supervised GNNs predict drug and material properties with confidence scores as in Liu et al. (2022 2023a). Only high-confident predictions are selected for annotation. Llama-3-700 then generates descriptions using template that incorporates these properties with structural and synthesis information from toolkits like RDKit. There are no polymerization reactions; we consider the monomer structure of the polymer as the synthesis target. We assemble molecule descriptions, text, and reactions from synthesis routes as answer data. Then Llama- 3-BB is prompted to generate questions, resulting MolQA with the example as shown in Figure Details are in appendix [C22
MolPair: Pairwise Data Creation. After excluding the target molecules from the instruction data, we use the remaining text-reaction data from USPTO to pre-train the GNN reaction predictor. Similarly, we utilize all other small molecules and polymers to pre-train the Graph DiT and graph encoder. For generalization, we expand beyond the eight properties used in the instruction data. For drug utility, we train another GNN to predict 41 properties, including toxicity, safety, enzyme interaction, absorption, distribution, metabolism, excretion (ADME), and biological activity (Swanson et al. 2024). For material utility, consider 14 properties, such thermal, physical, thermodynamic, permeability, solubility, and dielectric properties. Llama-3-70B generates related texts for these properties, incorporating structural and synthetic information. Finally, there are around 600K graph- text pairs for both small molecules and polymers to support pre-training. Details are in appendix C.3]
5 EXPERIMENT
We conduct a systematic evaluation demonstrate Llamole's superior performance in controllable and synthesizable molecular design (RQ1). We investigate Llamole's performance in controllable molecular generation through ablation and case studies (RQ2). We analyze retrosynthetic performance of LLMs, focusing on error analysis and the efficiency and effectiveness of Llamole (RQ3).
Set-ups: We include LLM baselines from 7B to 70B, such as Llama, Mistral, Qwen, Granite, and Flan-T5, using either ICL or LoRA-based SFT. We also include domain-specifie methods, GraphGA (Gao et al., 2022). DiGress (Vignac et al. 2022), and BioNavi (Zeng et al., 2024), for comparison. The MolQA test set contains 9,986 QA pairs for material design and 750 for drug design. LLMs are prompted with questions to generate responses for texts, molecules, and reactions. For controllability, evaluate up to 12 metries across four aspects: (1) chemical validity, (2)

---

Published as a conference paper at ICLR 2025
Table Multi-Conditional Molecular Design with LLMs: Best overall results in each metric are in
bold
best baseline results are in italic Balanced Accuracy (BA) True Pocitive Jute+Trie Negative Rate
2
Base LLM or Method
Structure (1)
Text (1)
Drug (BA f)
Muterial (MAB
Validity
Similarity
BLEU-4 ROUGE-L
HIV
BBBP BACB CO2Pern NyPerm OyPerm
FFV
TC
GraphGA
0.885
0.112
NA
NA
0.536
0.515
0.560
0.847
1.556
0.747
0.020
0.042
DiGress
0,375
0.046
NA
NA
0.515
0.522
0.580
0,635
1.884
0.680
0.020
0.049
In-Context
Learning
Llama-2-7B
0.167
0.024
0.030
0.141
0.051
0.060
0.053
5.463
3,982
4,943
0.308
0.199
Mistral-7B
0.251
0.044
0.066
0.203
0.163
0.153
0.200
5.062
3.824
4.657
0.289
0.186
Qwen2-7B
0,180
0.012
0.030
0,147
0.089
0.091
0.085
5.552
4.251
5.068
0.322
0.211
Llama-3-8B
0.656
0.112
0.155
0.307
0.471
0.473
0.562
3.233
3.106
2.924
0.171
0.123
Plan-TS-XXL
0.570
0.094
0.226
0.388
0.329
0.333
0.403
2.869
3.039
2.799
0.165
0.120
Granite-133 13B
0.498
0.079
0.170
0.326
0.260
0.293
0.285
2.994
3.165
2,993
0.180
0.123
Llama-2-13B
0.346
0.058
0.121
0.279
0.236
0,250
0.259
5.031
4,285
4.816
0.291
0.184
Mistral-8x7B
0.546
0.181
0.345
0.345
0.388
3.695
3.150
3,440
0.191
0.138
Llama-2-70B
0.299
0.045
0.099
0.222
0.237
0.242
0.274
5.368
4,336
5.017
0.319
0.202
Llama-3-70B
0.706
0.124
0.210
0.367
0.415
0.403
0.484
2.659
2.848
2,421
0.135
0.099
Supervised Fine-tuning
Mistral-7B
0.718
0.125
0.105
0.216
0.460
0.483
0.515
3:269
3.094
2,985
0.184
0.128
Qwen2-7B
0.768
0.133
0.221
0.377
0.436
0.457
0.457
2.691
2.562
2.721
0.147
0.106
Llama-3-8B
0.797
0.136
0.093
0.206
0.426
0.445
0,440
2.222
2.322
2.119
0.110
0.086
Llama-3.1-8B
0.692
0.121
0.121
0.250
0.417
0.432
0.433
3.210
2.991
2,974
0.179
0.122
Llamole
Mistral-7B
0,900
0.139
0,262
0.434
0.596
0.617
0.740
0.593
1.409
0.565
0.021
0.028
Qwen2-7B
0.888
0.135
0.261
0.432
0.600
0.639
0.746
0.645
1.452
0.581
0.021
0.026
Llama-3.1-8B
0.913
0,142
0.254
0.427
0.623
0.629
0.713
0.653
1.344
0.549
0.021
0.030
Improvement of Llamole (%)
vs. All
+3.2
+4.4
+15.9
+11.9
+16.2
+22.4
+31.7
+9.5
+6.7
+19.3
-5.0
+28.6
vs. LLMS
+14.6
+4.4
+15.9
+11.9
+32.3
+32.3
+32.7
+70.6
+37.5
+72.6
+80.9
+65.1
similarity to the reference based on Morgan fingerprints (Rogers & Hahn 2010), (3) BLEU-4 and ROUGE-L. scores against reference texts, and (4) deviation from desired properties. We follow Gao ct al. (2022) to use well-trained random forests as the oracle functions for obtaining properties of designed molecules. We focus three drug-related categorical properties assessed by balanced accuracy (BA) and five continuous material properties assessed by mean absolute error (MAE). For retrosynthesis, evaluate the success rate of designed molecules against those available in Gavail from Enamine. Details are in appendix E.
5.1 RQI: LLMs FOR CONTROLLABLE AND SYNTHESIZABLE MOLECULAR DESIGN
Table 1 and Table detail LLM performance in controllability and retrosynthesis success rate. The overall performance rankings are summarized in Figure 5 Our key observations are:
(1) Llamole significantly outperforms other LLMS in text generation, controllable molecule generation, and retrosynthetic planning. Llamole fine-tuned on various 7B-parameter LLMs, as shown in Table 2 results in top-3 rankings, surpassing 70B models that are 10x larger across all 12 metrics for controllability and planning success. Specifically, Llamole enhances chemical structure validity by 14.6%, structure controllability by 4.4%. and text generation by 11.9%-15. 15.9%. Additionally, Llamole improves property controllability by 32% to 80%. In retrosynthesis, Table 2 indicates Llamole increases the success ratio from 5% to 35% for drugs and to 17.9% for polymers.
(2) SFT improves molecular design but may not always enhance retrosynthesis. According to Figure and and Table I SFT enables 7B LLMs to achieve chemical validity, structure, and property control comparable to 70B LLMs with ICL. However, it offers minimal improvement in planning ability for the generated target molecule. A notable example Llama-3-85 from Table 2 where SFT reduces its retrosynthesis planning success from 5.5% to below 1%. Except for Llama-3-8B. connect LLM performance with the same baseline but different learning methods in Figure The results show that SFT methods still outperform ICL with the same base 7B models in most cases.
7

---

Published as a conference paper at ICLR 2025
3. Llamole
Laanz---aamaa Llamole
1
Qwen2-78e
Retrosynthesis Success Rate (Ranking)
QwenZ-7B
Retrosynthesi Success Rate (Ranking)
Llama-3 1 1-88
In-Context Learning
In-Context Learning Mistral-78
Llama-3-8B
Mistra
Llama-3-708
L-8
5
Mistral-8x78
Granite-138
Mistral-8x78
Llama-2-138
Mistyal-78
7
Llama-2-138
Jamaa Teaa--77000 70B
Lama 277
ama-3.1-8p
9
Flan-TS-XON 5-XLL
Mistral-7B
Grahite-138
11
Lama-2-708
Zlama- 1-1a-331-8B
ma--3-88
13
Alan-T5XXLL
Llama-3-88
13-
Mistrall77B
Mistral-7B
15
Qwen2-286
15
Llama-2-7B
Llama-2-78
Qwen2-79e 79
17
@ Supervised Fine-Tuning
17
@wenz-7B Supervised Fine-Tunine
17 15
13
11 9
5
3
17
15
13 11
Mulpi-Property Controllability (Ranking)
Multi-Propeerty Controllability (Ranking)
(a) LLM for Drug (Small Molecule) Design
(b) LLM for Material (Polymer) Design
Figure 5: Overall Comparison of LLMs for Controllability and Synthesizability: Performance is
ranked by averaged BA/MAE (X-axis) and retrosynthesis success rate (y-axis). Circle size indicates model size. LLMs with ICL, SFT, and Llamole are highlighted in blue, orange, and red, respectively.
Table 2: Retrosynthetic Success Rate: Best results in bold best baseline results in italic
In-Context Learning
Llama-2-78 Mistral-7B Qwen2-7B Llama-3-8B Flan-T5-XXL Granite-13B Llama-2-138 Mistral-8x7B Llama-2-708
Drug (%) Material (%)
0.1
0.2
0.4
1.2
1.0
0.3
0.4
0.8
1.2
0.8
Supervised Fine-tuning
BioNavi
Llamole
Mistral 7B Qwen2-7B 7B Alamma-3-8B Llarna-31 3.1-8B
DiGress
Mistral-7B 7B Qwen2-7B 7B Llama-3.1 3.1-8B.
Drug (%)
1.5
0.2
0.6
0.8
18.0
29.9
33.7
35,1
Material
0.8
0.8
15.4
17.9
17.6
(3) Larger models without domain-specific adaptation do not necessarily perform better in molecular designs. We calculate the average Pearson correlation coefficient between model size and molecular design metrics, yielding value of 0.366, indicating weak correlation (below 0.5) between size and performance. We also compare LLM performance with GraphGA, which has been shown to be simple yet powerful (Gao al. 2022 Liu al. (024c). Our observations confirm that GraphGA serves as strong molecular design baseline, challenging most L.LM models with ICL and SFT in generating molecules with precise multi-condition control.
5.2 RQ2: DISCUSSION ON CONTROLLABLE MOLECULAR GENERATION
5.2.1 ABLATION STUDIES ON LLM AND GRAPH DIT SYNERGY
We investigate the synergy effect of Graph DiT and LLM in Llamole for molecule controllability. We first remove text conditions Crex- this case, Graph DiT uses a learned "null" embedding to represent the dropped condition Ctex 0. Next, we remove the drug material property conditions {c;) MM associated with the question. Results in Figure show that text instructions enhance the chemical structure understanding ability of Graph DiT, while Llamole leverages Graph DiT's capabilities with property inputs to generate molecules with desirable properties.
5.2.2 CASE STUDIES FOR PROPERTY AND STRUCTURE CONTROLLABILITY
In Figure 7 Llamole can design satisfactory molecule that meets both functional and structural constraints. Functionally, the oracle function confirms that the properties of BACE and HIV align with the criteria. Structurally, all key criteria satisfied, including molecular weight, "two aromatic rings." and "connected to aliphatic chains." " Llamole also adds details for structure design, such as carboxyl (- (-COOH) group and amino group (-NH)). While the amino group is present in the
8

---

Published as a conference paper at ICLR 2025
Validity
Structure Similarity
Drug Avg. BA
Material Avg. MAE
0.913
0.16
0.142
0.72
0.655 0:664
0.72
0.687
0.60
0.75
0.90
0.822
0.735
0.10
0.14
0.12
0.124
0.098
0.48
0.56
0.64
0.505
0.56
0.64
0.48
0.5190.534
0,08
0.40
0.40
0.45
0.30
0.06
0.04
0.24
0.16
0.32
0,24
0.16
0.32
0.15
0.02
0,08
0.08
Llamole (Llama-3.1-88)
w/o text conditions
Wo text and property conditions
Figure 6: Ablation Studies for the Graph DiT Module Llamole: First, we remove the text conditions from the input, i.e., Ctext = 0. Next, we remove both text and property conditions, {c,) U Crext- There are learned embeddings that represent the "null" value for different conditions.
J.
Question "Can you design and synthesize a molecule that inhibits both Human Immunodeficions Virus and Beta-Secretase I. with molecudar weight of around 335 and aromatic rings connected aliphatic chains?"
Liamole Generation
Emolerate 1>
One Step Reacriin One Stap Reacrion Molecull
"Po satisfy the requiroment:: The molecule has complex structure with aromatic rings connected to alihaticc chalos, featuring carbexyl group and an amms group, which couributes to itr moderate synthetic complexit score. Therefore, the designed molecule is <molicuie J2."
prsceslurey: The electolyte solution was prepared by dissmboigg 2 of 1. 1,4-bis(3,S-aii) h-diioccllphhayllleeeee (ohoyyyeoegh (compound 14) in 26 ul of tolanne. The solution was stired ar room tomparative for 3 hours. The soivent
"Th synthesise smolecule follow
MoNW: HIV 1, BACE:
Sreaction
Stemplate
evaporaned under reluned prezzure. and the residue pus ffeed by column docoaaoogrppyy fhezameicthyl acetute- 1/3) to give
<undoccuee 20
Lmolerule 3>
1.5 g (88%) of the tiele compound as colorless oil. The applied reaction is srrsctiinn
Creaction
Stemplate 1' which requirer the reactanty; Smolecule (available). smotecuie 3>.
"To synthenize smolecule 3-, follow these proceduurs:: 100 meldrum acid discolved in 100 ml of methandd 100 g S-byyinghennaaddeool were added and the mixhire was stireed for i hour room iemperature. 190 ml of sanurated sodium bonate solution
<mofecuic do smoiscule 5
teuppooee 2:02
added and the sultuure extracted with 100 of odyy acctate. The organic phase was dried over sodium sulhare, filtred and concentrated to dryuess. The residue was purified by chromotoraapyy hexano/colyl acctute as elucat, stolding g of 2-op-l---ao--2. 1.-00-2--lllysooomaaa ddlllsooodoolnn--i. -oablooo----aboylllee with the tomplose cuomplate 27 wkich requires the readones <modocuie 4- (avoilable) <modecule 3=favoilable)
silico get with
The applied reuction by: "reaction
Figure 7: Interleaved generation with the base Qwen2-7B: Red indicates positions where molecules and reactions (with templates) are generated, forming three parts. The properties of the designed molecules are obtained from the oracle. Reference and other LLM responses are shown in Figure
structure, it is connected to the carbonyl group (-C(=0)-) instead of the carboxyl group. This subtle difference may require precise control based on the text condition. More results are in appendix E.3
5.3 RQ3: DISCUSSION ON YNTHESY PLANNING
Retrosynthesis challenges LLMs in two aspects:
In- In-Context Learning
Supervised Fine-tuning
(I) one-step reaction generation and (2) multi- step planning. Table highlights the weaknesses of LLMs with ICL and SFT in overall planning ability and the promise of Llamole. We examine the failure reasons in LLMs and the synergy between the GNN and LLMs to avoid them.
Reaction
39.4%
3.5%
Text
Invaid Reaction
6.1%
3.7%
Format
No Template
2.3.95 23.99%
50.00 30.0%
Template Symtax
11.8%
17.0%
5.3.1 ONE-STEP REACTION GENERATION
Templete Mismatcch
19.7%
25.5%
We conduct error analysis for LLMs in reac- tion generation. Results in Figure g average performance across all LLMs using ICL or SFT methods. We identify five types of errors related to instruction adherence, format compliance, and template matching. We find that LLMs using ICL frequently fail to follow instructions for generating reactions in text format, with a high probability (68 (68.4%) of not producing valid formats and templates. In contrast, LLMs with SFT reduce this probability to 57.6%. However, neither ICL nor SFT guarantees that the templates are correct or match the generated reactions. In comparison, Llamole avoids these errors by using GNN predictors, which estimate probabilities for over 300K templates derived from USPTO reactions. This enables I Llamole to apply templates directly to derive reactions in retrosynthesis, avoiding hallucination.
Figure 8: Error Analysis in Reaction Generation

---

Published as a conference paper at ICLR 2025
5.3.2 MULTI-STEP RETROSYNTHETE PLANNING
From the success cases in Table 2 we find that 96.40% of 777 success cases in ICL-adapted LLMs and 14% of 324 success cases in SFT-adapted LLMs arise from one-step reaction generation. However, not all designed molecules can be synthesized via one-step reactions. Compared to LLMs. Llamole achieves over 10K success cases, with 40.48% resulting from two or more steps. Figure 7 illustrates a two-step planning case for the designed molecule. The generation interleaves reaction conditions and specific formulas based on the template in both steps.
Llamole is influenced by two factors for retrosynthe- sis: (I) the size of the search space and (2) the quality of the cost Jneuristie The results reported Table limited the total planning time to 30 seconds (on an A6000 card). We remove this time constraint and report comparisons for material tasks in Table 3 We find that success rates for all base LLMS significantly improve, but this comes at the cost of long inference time. While there is a trade-off between efficiency and effectiveness, rates for finding synthesis paths. In Table 3 we also compare the Theuritiics designed by LLMs (default) with the domain model trained from Chen et al. (2020). we find that LLMs are competitive with these domain models in providing the cost function for A contrasting with previous observations where LLMs struggled with retrosynthetic planning.
Table 3: Analysis of Jneuristies and Planning
Time on Material Questions
Base LLM
w/ Domain Heuristics
w/ Unlimited
Default
Time
Llama-3.1
0.176
0.176
0.312
Mistral
0.143
0.147
0.273
Qwen2
0.179
0.181
0.273
often acceptable to extend response time by few minutes enhance success
RELATED WORK
Since the emergence of ChatGPT (Achiam et al.) 2023). LLMs (Dubey et al. 2024) have become foun- dation models for text-based problems and are revolutionizing domains like vision and speech (Dong
ct al. 2023 Wu et al. (024). These advancements extend to chemistry, biology, and material sciences, focusing on molecules (Guo et al., 2023 Jin et al.] 2023). Prior work explores LLMs in molecular generation, property prediction, and one-step reaction prediction in retrosynthesis (Guo Jablonka et al. 2023). A key lesson is the limitation of LLMs in sequential modeling of molecules (e.g., SMILES or SELFIES) (Guo al. 2023). Multimodal LMs have been developed for molecular tasks (Edwards et al. 2022; Liu et al. 2023b), but they either do not treat molecules as graphs (Edwards et al. 2022) or do not focus on inverse molecular design. Additionally, LLMs struggle with planning tasks (Kambhampati et al. 2024), which are essential for retrosynthesis. We address these issues using graph-text multimodal LLMs, augmented by A* for efficient planning.
itt al. 2023
Domain-specific molecular design methods have evolved from sequential models (Segler et al. 2018) to graph diffusion models (Vignac et al., 2022: Weiss et al., 20233 Liu et al. 2024c). Studies show that older graph methods like GraphGA remain competitive (Gao et al. 2022). To incorporate
property constraints, one can use Bayesian optimization or REINFORCE (Gao et al. 2022), or employ diffusion models with without predictor guidance (Vignac et al. 2022 Liu al. 2024c). For synthesizable molecular design, prior work has focused on bottom-up methods (Gao et al. 2021 [Sun al., 2024). These methods explore chemical space defined by a discrete action space of reaction templates and purchasable starting materials, which may limit flexibility. Thus, retrosynthesis algorithms (Chen al., 2020;; Han al. 20222] Zeng et al. (024) are also studied as separate solutions to find synthesis routes for generated molecules in a top-down manner.
7 CONCLUSION
We have presented the first graph-text MILM, Llamole, for multi-conditional molecular generation and retrosynthetic planning. By integrating base LLM with specialized graph modules, Llamole interleaved the generation of text, molecular graphs, and reactions, enabling controllable and synthe- sizable designs. Extensive benchmarking against 14 LLMs revealed their limitations in controlling molecular structures and planning synthesis routes. In contrast, Llamole significantly outperformed these LLMs. These findings underscored the value of multimodal approaches in molecular discovery and highlighted Llamole's 's potential to connect text and chemical structures. The new benchmarking dataset also laid the groundwork for future MILM research in molecular applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was in part supported by NSF IIS-2142827, I1S-2146761, IIS-2234058, CBET-2332270, and ONR N00014-22-1-2507.
REFERENCES
Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, GP Bhargav, Maxwell Crouse, Chulaka Gunasekara, et al. Granite-function calling model: Introducing function calling abilities via multi-task learning of granular tasks. arXiv preprint arXiv:2407.00122 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altan,, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint wwXiv:2333.08772 2023.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-17993 2021.
Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: pretrained language model for scientific text. arXiv preprint arXiv: 10676, 2019.
Binghong Chen, Chengtao Li, Hanjun Dai, and Le Song. Retro*; learning retrosynthetic planning with neural guided a search. In International conference machine learning, pp. 1608-1616.
PMLR, 2020.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Connor W Coley, Luke Rogers, William H Green, and Klavs Jensen. Sescore: synthetic complexity learned from a reaction corpus. Journal of chemical information and modeling, 58(2):252-22 2018.
Connor W Coley, William H Green, and Klavs F Jensen. Rdchiral: An rdkit wrapper for han- dling stereochemistry in retrosynthetic template extraction and application. Journal of chemical information and modeling, 59(6):2529-2533 2019.
Rumpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamlim: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.1 11499, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 2rXiv:2407.21780 2024.
Carl Edwards, Tuan Lai. Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation between molecules and natural language. arXiv preprint arXiv:2204.11817 2022.
Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based molecular complexity and fragment contributions. Journal of cheminfor- matics, 1:1-11, 2009.
Wenhao Gao, Rocío Mercado, and Connor W Coley. Amortized tree generation for bottom-up synthesis planning and synthesizable molecular design. arXiv preprint arXiv:2110.06389, 2021.
Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor Coley. Sample efficiency matters: benchmark for practical molecular optimization. Advances in neural information processing systems, 35: 11352-21357, 2022.
Taicheng Guo. Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, Xiangliang Zhang, et al. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. Advances in Neural Information Processing Systems, 15556662-56888 2023.

---

Published as a conference paper at ICLR 2025
Peng Han, Peilin Zhao, Chan Lu, Junzhou Huang, Jiaxiang Wu, Shuo Shang, Bin Yao, and Xiangliang Zhang. Gnn-retro: Retrosynthetic planning with graph neural networks. Proceedings of the AAAI conference artificial intelligence, volume 36, pp. 4014-4021, 2022.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 2021.
Kevin Maik Jablonka, Qiamxiang Ai, Alexander Al-Feghali, Shruti Badhwar, Joshua D Bocarsly, Andres Bran, Stefan Bringuier, L. Catherine Brinson, Kamal Choudhary, Defne Circi, al. 14 examples of how lims can transform materials science and chemistry: reflection on large language model hackathon. Digital Discovery, 2(5):1233-1255) 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji, and Jiawei Han. Large language models on graphs: A comprchensive survey, arXiv preprint SxXi:2312.02788, 2023.
Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. Llms can't 't plan, but can help planning in llm-modulo frameworks. arXiv preprint urXiv:240..01817, 2024.
Sunghwan Kim, Jie Chen, Tięjun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin Shoemaker, Paul A Thiessen, Bo Yu, et al. Pubchem in 2021: new data content and improved web interfaces. Nucleic acids research, 49(D1):D1388 10N0:DD1388--199,,.. 2021.
Gang Liu, Tong Zhao, Jiaxin Xu, Tengfei Luo, and Meng Jiang. Graph rationalization with environment-based augmentations. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1069-1078, 2022.
Gang Liu, Tong Zhao, Eric Inae, Tengfei Luo, and Meng Jiang. Semi-supervised graph imbalanced regression. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 1453-1465, 2023a.
Gang Liu, Eric Inae, Tengfei Luo, and Meng Jiang. Rationalizing graph neural networks with data augmentation. ACM Transactions Knowledge Discovery from Data, 18(4):1-23, 2024a.
Gang Liu, Eric Inae, Tong Zhao, Jiaxin Xu, Tengfei Luo, and Meng Jiang. Data-centric learning from unlabeled graphs with diffusion model. Advances in neural information processing systems, 36, 2024b.
Gang Liu, Jiaxin Xu, Tengfei Luo, and Meng Jiang. Inverse molecular design with multi-conditional diffusion guidance. arXiv preprint arXiv:2401. 13858, 2024c.
Pengfei Liu, Yiming Ren, Jun Tao, and Zhixiang Ren. Git-mol: A multi-modal large language model for molecular science with graph, image, and text. Computers biology and medicine, 171: 108073, 2024d.
Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 15623-15638, 2023b.
Daniel Lowe. Chemical reactions from US patents (1976 Sep2016). 6 2017. doi: 10. 40844/m9.igssaae551100000
URL /tttps://igghaaee hwwp:shaarr..cc.caaaoaaeeeeeeaae com htt.m//rtiieeae/emm att.ateeeddaaaeeee///ta dataset/ Chemi cal_raaaalllinnaaeaeaaaaa from w___UuuS______________ 1976 Sep201pp2221-e htt.eeeess__1177--eee1ee_____
Ruimin Ma and Tengfei Luo. Pilm: a benchmark database for polymer informatics. Journal of Chemical Information and Modeling, 60(10):46844666 2020.
12

---

Published as a conference paper at ICLR 2025
Shingo Otsuka, Isao Kuwajima, Junko Hosoya, Yibin Xu, and Masayoshi Yamazaki. Polyinfo: Polymer database for polymeric materials design. 2011 International Conference on Emerging Intelligent Data and Web Technologies, pp. 22-29. IEEE, 2011.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PMLR, 2021.
David Rogers and Mathew Hahn. Extended-cooneccivity fingerprints. Journal of chemical information and modeling, 50(5):742-75,, 2010.
Marwin HS Segler, Thierry Kogej, Christian Tyrchan, and Mark P Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS central science, 4(1): 120-131, 2018.
Teague Sterling and John J Irwin. Zine 15-ligand discovery for everyone. Journal of chemical information and modeling, 55(11 55(11):2324 (1:1):22223333 2337, 2015.
Michael Sun, Alston Lo, Wenhao Gao, Minghao Guo. Veronika Thost, Jie Chen, Connor Co- ley, and Wojciech Matusik. Syntax-guided procedural synthesis of molecules. arXiv preprint arXiv:2409.05873, 2024.
Kyle Swanson, Parker Walther, Jeremy Leitz, Souhrid Mukherjee, Joseph C Wu, Rabindra V Shiv- naraine, and James Zou. Admet-ai: machine learning admet platform for evaluation of large-scale chemical libraries. Bioinformatics, 40(7):btae416, 2024.
A Thortoo,, L Robeson, B Freeman, and D Uhlmann. Polymer gas separation mem- brane database, 2012. URL
Clement Vignac, Igor Krawezuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint 22Xi::2209.1477 14734, 2022.
David Weininger. Smiles, chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
Tomer Weiss, Eduardo Mayo Yanes, Sabyasachi Chakraborty, Luca Cosmo, Alex M Bronstein, and Renana Gershoni-Poranne Guided diffusion for inverse molecular design. Nature Computational Science, 3(10):873-882, 2023.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. NExt-GPT: Any-to-any multimodal LLM. In Forty-first International Conference on Machine Learning, 2024. URL forum? dd=i::idddNZZk
htttss://operevete wwt.:neetoooteeeo
net
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9():513-530, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv: 1110000800 2018.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li. Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1066 10671, 2024.
13

---

Published as a conference paper at ICLR 2025
Barbara Zdrazil, Eloy Felix, Fiona Hunter, Emma J Manners, James Blackshaw, Sybilla Corbett, Marleen de Veij. Harris loannidis, David Mendez Lopez, Juan F Mosquera, et al. The chembl database in 2023: drug discovery platform spanning multiple bioactivity data types and time periods. Nucleic acids research, 12(D1))D1800-D1192 192 2024.
Tao Zeng, Zhehao Jin, Shuangjia Zheng, Tao Yu, and Ruibo Wu. Developing bionavi for hybrid retrosynthesis planning. JACS Au, 4(7):2492-2502 2024.
Haiteng Zhao, Shengchao Liu, Ma Chang, Hannan Xu, Jie Fu, Zhihong Deng, Lingpeng Kong, and Qi Liu. Gimlet: A unified graph-text model for instruction-based molecule zero-shot learning. Advances in Neural Information Processing Systems, 36:5850-5887, 2023.
14

---

Published as a conference paper at ICLR 2025
CONTENTS
Introduction
2
Preliminaries
2.1
Autoregressive Language Modeling
2.2
Molecular Design with Graph Diffusion Models
One-Step Reaction Prediction with Graph Neural Networks
2.4
Retrosynthetic Planning with A Search
3
lamole: Large Language Model for Molecular Discovery
3.1
Multimodal Autoregressive Modeling
3.2
Llamole Design Space
3.3
End-to-End Model Fine- TuninT and Generation
Benchmarking for Multimodal Molecular Design
5
Experiment
5.1
RQI: LLMs for Controllable and Synthesizable Molecular Design
5.2
RQ2: Discussion on Controllable Molecular Generation
5.2.1
Ablation Studies on LLM and Graph DIT Synergy
5.2.2
Case Studies for Property and Structure Controllability
8
5.3
RQ3: Discussion on Retrosynthetic Planning
5.3.1
One-step Reaction Generation
5.3.2
Multi-step Retrosynthetic Planning
10
Related Work
10
Conclusion
10
More Related Work on Multimodal Language Modeling
17
Additional Details for Llamole
17
B.1 Details of Special Tokens
17
Details of LLM- based * Heuristics
18
Additional Benchmarking and Datasets Details
18
C.1 Details of Quality Control
18
Details on the Creation MolQA
19
C.2.1
Creation of Synthesis Routes
19
C.2.2 Creation of Property Annotations
19
C.2.3
Creation of Text Data for Molecular Description
19
15

---

Published as a conference paper at ICLR 2025
.,244 Creation of Question Answering Data
21
Details on the Creation MolPair
D Additional Pre-training and Fine-tuning Details
22
D.I Pre-training of Graph Diffusion Transformer
22
D.2 Pre-training of GNNs
................................
23
D.3 Fine-tuning of Llamole
23
Additional Experimental Details and Discussions
23
E.
Additional Details on Experimental Set-ups
23
E.I.1 Set-ups for Figure
24
E.
1.2 Extraction SMILES from LLM Responses
24
E.2
Additional Discussion One-Step Generation
24
Additional Discussion on Case Studies
25
16

---

Published as a conference paper at ICLR 2025
A MORE RELATED WORK ON MULTIMODAL LANGUAGE MODELING
Emerging approaches focus on multimodal graph and language modeling for tasks such molecular property prediction (Zhao et al. 2023), captioning (Edwards et al., 2022 Liu et al. (024d), and retrieval (Liu et al. 2023b; 2024d). The task most similar to inverse molecular design is text-based molecular generation (Edwards et al. 2022 Liu et al. 2024d). this work, inverse molecular design is framed as a question with specific requirements for properties and synthesis paths. Unlike text- based generation, which takes descriptions of molecules as input, inverse molecular design requires fewer details the molecule, focusing instead on satisfying the specified requirements. Additionally, text-based generation produces molecular structures without considering symthesizability. whereas designed molecules are often expected to be synthesizable (Gao al. 2021), involving retrosynthesis.
Table 4: Balanced Accuracy Averaged Across Three Drug Design Properties
MolT5-small MolT5-base MolTS--lare
Best LLM with ICL. Llamole
0.150
0.232
0.264
0.502
0.662
To explore the difference between inverse molecular design and text-based generation, we use the decoder model from (Edwards et al. [2022] Liu et al. (024d) (i.e., MolT5) and test questions from the MolQA benchmark compare the performance of MolT5, LLMs, and Llamole in drug design. Results on balanced accuracy are shown in Table 4 We find that even the largest MolT5 underperforms the best LLM (from ICL) in drug design. This illustrates that text-based molecular generation, which takes descriptions of molecules as input, may not perform well in inverse molecular design, which requires satisfying specific properties with synthesis paths and a lack of details about the molecule in texts. For material design, we find that MolT5 cannot generate valid polymer structures due to its lack of knowledge about polymerization points, typically represented by the asterisk symbol in SMILES strings. As result, valid MAE error is reported. Additionally, existing multimodal language models have not addressed the retrosynthetic planning problem.
B ADDITIONAL DETAILS FOR LLAMOLE
B. DETAILS OF SPECIAL TOKENS
In total, there nine special tokens divided into three groups. These tokens augment the word vocabulary W, enabling flexible control of the generation flow:
Trigger and Query tokens:
<design.start>, <design <design.ends <retro.start>, <retro.bedy>, <retrolend>
Molecule token: <molecule>
. Callback tokens: sccallbacksstarte <callback_end>
The tokens <des gtwstarttttttooommmmm and <retro <retro-start> start> switch between the LLM and the Graph DiT or GNN, respectively. The tokens blesign_body and <retro be_dyo_ serve as query tokens, repeated eight times. After tokenization, the LLM takes their embeddings as input and outputs vector from the last layer. The tokens <design.end> and <retro_end> indicate the end of these switches.
The <moleculs token marks the position of the molecular graph where the graph encoder is applied. In the instruction dataset, the segment Emol.start>SSil "mmnd denotes the position and identity of the molecule. SMILES will be converted molecular graphs using RDKit, and this segment will be replaced by the <moleculs token for Llamole inputs.
Finally, callback tokens control the LLM to generate backup results as complements to the specialized graph modules. For instance, if the Graph DiT fails to produce a valid molecule, the base LLM can generate alternative, regardless of validity.
17

---

Published as a conference paper at ICLR 2025
B.2 DETAILS OF LLM-BASEE A HEURISTICS
Llamole models Jheurisiics in A search as multi-choice problem, filling in information from the molecule node, its parent reaction nodes and siblings using the template below. Parameters such as step, reaction template, and reactants are optional.
Estimate remaining steps for the target (smiles) given the
following parameters:
Current step (step),
Current template: (template),
Reactants: (reactants).
Consider the following factors:
1. Intermediate complexity
2. Reagent availability
3. Side reactions
4. Stereochemistryy chal lenges.
Using this question to estimate remaining steps, we input the text into the base L.M and formulate five choices with corresponding scores:
A. All readily available // Score: 0
B. Some commercial, some need 1-2 steps // Score: 1
C.
Mix of commercial and multi-step synthesis // Score: 2.5
D. Mostly require complex syrthesis // Score: 4.5
E. All require extensive multi-step synthesis // Score:
The LLM outputs logits for the next token, which we average for each choice to obtain overall probabilities. The Jucuristies calculated as the weighted score using these probabilities.
C ADDITIONAL BENCHMARKING AND DATASETS DETAILS
We collect small drug molecules from PubChem (Kim et al.
2021
MoleculeNet (Wu et al.
2018) 8).
ChEMBL (Zdrazil al. 2024), and ZINC (Sterling & Irwin 2015). Polymers macromo
with one repeating unit called monomers. We collect volymers from PIIM (Ma & Luo 2020).
the Membrane Society of Australia (MSA) (Thornton al. 2012), and others (Liu 2024b).
Additionally, we collect 3.8 million patent chemical reactions with descriptions from USPTO (Lowe 2017,, spanning from 1976 2016.
C.1 DETAILS QUALITY CONTROL
After collecting molecules and polymers from various sources, we deduplicate and merge the label information for identical molecules. We use RDKit obtaín canonical SMILES. For small molecules. we calculate the first 14 characters of the InChiKey the unique identifier, while for polymers, where the polymerization point represented by "g* we use the canonical SMILES directly.
For drug-like small molecules, we apply the following rules to filter out alert structures, known the Rule of Five (Ro5):
Molecular Weight (MW): Must be < 500 Da.
Hydrogen Bond Acceptors (HBA): Must not exceed 10.
Hydrogen Bond Donors (HBD): Must not exceed
LogP: Must be < 5, indicating lipophilicity.
molecule passes the Ro5 test if at least three these four conditions are met, indicating potential oral bioavailability.
We also apply 15 filter rules from the RDKit package, including the following from the Fil- terCatalogs Class: BRENK, CHEMBL, CHEMBL.BMS, CHeMBL.Dundee, CHEMBL.Glaxo. CHEMBL Inpharmatica, CHEMBL LINT, CHEMBL MLSMR, CHEMBL SurCChEMBL, NIH, PAINS, PAINS A, PAINS B, PAINS C, and ZINC.
18

---

Published as a conference paper at ICLR 2025
C.2 DETAILS ON THE CREATION OF MOLQA
C.2.1 CREATION OF SYNTHESIS ROUTES
The USPTO has 3.7 million reactions. There are approximately 1.3 million unique product molecules. The purchasable compounds come from the Enamine Building Block (June 2024 version), supple- mented with other common ions and starting materials, totaling around 1.3 million. We check each product from USPTO as a target molecule in the retrosynthesis task, exploring whether they can be synthesized using existing USPTO reactions through depth-firs first search (DFS). Ultimately, we identify about 139K target molecules with synthesis routes, supporting the creation of MolQA.
Since there no polymerization reactions, we consider only monomer structures by replacing the * point with hydrogen. Among the 139K small molecules with synthesis routes, 2196 fit the monomer structures and serve target molecules for polymer retrosynthesis. The length of synthesis routes ranges from to 10. For each length of the routes, we split half of the molecules into the testing set, with maximum of 3000, while the remainder is retained in the training set.
results in around 1 11K routes (750 for materials and 9986 for drugs) for testing and 126K target molecules for training.
C.2.2 CREATION OF PROPERTY ANNOTATIONS
We focus on eight benchmarking properties: three drug-related categorical properties (Wu al. 2018) (1) HIV virus replication inhibition (HIV), (2) blood-brain barrier permeability (BBBP), and (3) human 3-secretase inhibition (BACE) and five continuous material properties (Thornton
et al. 2012) (4) CO2 permeability (CO,Perm), (5) N₂ permeability (NyPerm). (6) O2 permeability (O2Perm), (7) fractional free volume (FFV), and (8) thermal conductivity (TC).
First, we check existing sources for annotations of these properties. To enrich the label space, we use well-trained GNN models (Liu et al. 2022) to generate confident pseudo-labels, following the method in (Liu et al. 2023a). We collect all labeled data to train two supervised multi-task GIN models for drug and material property annotation. The GIN models employ rationalization techniques (Liu et al.) 2024ab to split the molecular graph into rationale and environment subgraphs in the latent space, predicting labels from the rationale subgraph. The confidence score is computed by combining the rationale subgraph with various environment subgraphs, using the reciprocal of prediction variance. We annotate properties when prediction confidence exceeds the median threshold.
C.2.3 CREATION TEXT DATA FOR MOLECULAR DESCRIPTION
In addition to property annotations, we consider structural and synthesis information of the molecules using RDKit and heuristic complexity estimation scores. First, for any molecule, extract the following structural information:
Scaffold: Extracted scaffold from the molecule structure.
Molecular Weight: Calculated using the molecular weight descriptor.
Number of Rings: Total number of rings in the molecule.
Number of Aromatic Rings: Total number of aromatic rings in the molecule.
Number of Aliphatic Rings: Total number of aliphatic rings in the molecule.
Number of Rotatable Bonds: Total number of rotatable bonds in the molecule.
Number of Hydrogen Bond Donors: Total number of hydrogen bond donors.
Number of Hydrogen Bond Acceptors: Total number of hydrogen bond acceptors.
Next, we compute the synthetic accessibility score (SAScore) (Eri & Schuffenhauer, 2009) and SCScore (Coley et al. 2018). Based this information, we use the following template:
Generate a summary description that starts directly with "The molecule/polymen ... based the predicted chemical properties, synthetic complexity scores, and structural information for the molecule with SMILES: ((smiles)). Use your own knowledge, focus
19

---

Published as a conference paper at ICLR 2025
functions, and avoid using numbers, redundant words, mentioning SMILES. Ensure the output sentence is complete and ends with period. This is for Drug/Material Utility of Molecule SecculeePolyerr
The structural context of molecule includes its scaffold, which is the cere structure around which the molecule is built. Key structural features include the presence of aromatic rings, aliphatic chains, and common functional groups such as hydroxyl, carboxyl, and amino groupa. The complexity of the molecule's s structure can significantly influence its physical and chemical properties.
Scaffold: ((scaffold))
Molecular Weight: ((mw)}
Number of Rings: { (inumrrings))
Number of Aromatic Rings:
Number of Aliphatic Rings: (fum_aliphh_riggs)
Number of Rotatable Bonds: ([nun_rot_bondss]
Number of Hydrogen Bond Donors: ((num sam__h_doooss]
Number of Hydrogen Bond Acceptors: ((nm acceptors)
(utilityyccnnexx) ((properties))
The pre-defined utility context for the small molecule is as follows:
The drug utility of a molecule is assessed based on its potential to serve as therapeutic agent. Key properties considered include pharmacokinetics, which encompasses absorption, distribution, metabolism, excretion (ADME), and toxicity. Bioactivity is another critical factor, measured by the molecule's ability to interact with biological targets, typically through binding affinity. Additionally, drug-likeness, which refers to the molecule's adherence to established rules such Lipinski's Rule of Five, is essential. This rule evaluates molecular weight, hydrogen bond denors and acceptors, and Lipophilicity to predict a molecule's suitability as an oral drug.
The pre-defined utility context for the polymer is follows:
The material utility of molecule, particularly for creating polymeric materials, is evaluated based on properties like mechanical strength, flexibility, and thermal and electrical behavior. For polymer membranes used in gas separation, crucial factors include gas permeability, which determines the efficiency of gas diffusion, and chemical stability, ensuring resistance to degradation. Additionally; thermal properties such as melting point and thermal conductivity are vital, as they affect the material's performance under various temperature conditions. Electrical properties, such as conductivity and dielectric constant, may also be significant depending on the intended application.
For the property variable, we include the property name with values, as well as the minimum, maximum, and percentile among the labels in the template. We repeat all annotated properties in the property variable. The estimated synthesis complexity scores are included among them.
We also prompt Llama-3-700 3-BB to generate short responses of 50-70 words, producing molecular description for each molecule based on its properties, structures, and synthesis estimation. If a molecule has description from PubChem (Kim et al. 2021). concatenate these descriptions.
20

---

Published as a conference paper at ICLR 2025
The generated texts may not always be meaningful valid. We establish filter rules based on patterns observed in poorly generated texts to remove them. We then regenerate texts for these items. After several iterations, we obtain the final text data for molecular utility descriptions, improving overall text quality. We also apply this strategy to other steps that involves prompting L.LMs for synthetic data creation.
C.2.4 CREATION OF QUESTION ANSWERING DATA
After annotating molecular description texts from appendix C.2.3 combine them with reaction descriptions, including the reaction formula and template from synthesis routes in appendix C.2.1 This forms the answer data in QA data pair.
Next, we prompt -lame-3-708B to generate questions for each answer based on the following template.
I'm creating question ivessioo-answer dataset for LLM fine-tuning, The question is about designing molecule/polyme with these properties: (property_info) and the following structure information: (structure_info)
The expected answer for the question is: (answer) Generate a SINGLE question about designing and synthesizing such moleeuleepoolyerr that meets these criteria:
(1) Start with 'Question:'; (2) End with question mark;
(3) Sound natural; (4) Be diverse; (5) Avoid redundancy and introductory words (like 'Here is a question that meets the criteria:')
(6) Do not include the answer; (7) Do not include incorrect information.
Example questions:
(1) How can I design and synthesize a molecule with X, Y, and z properties?
(2) What is the best way to create a polymer with X, Y, and z characteristics?
(3) How to design a molecule with x, Y, and & features and synthesize it?
(4) I want molecule with x, Y properties and Z structures. Please design it and describe the synthesis path.
The template is applied to any answer with the corresponding structure, property information, and complete answer texts.
C.3 DETAILS ON THE CREATION OF MOLPAIR
MolPair consists of two parts: reaction-tex. pairs and graph-text pairs. We curate reaction-text pairs from USPTO (Lowe 2017). pairing each reaction with its corresponding description of the reaction conditions. We first deduplicate product molecules in reactions, obtaining input data as the product molecule alongside the reaction condition texts. Next, we extract reaction templates from the reaction formula using rdchiral (Coley et al. 2019), resulting in approximately 300K templates, which will serve as labels for predictions. Finally, we have approximately 1.6 million training examples.
For the graph-text pairs, we use small molecules and polymers from the multisource collection, excluding those in MolQA. We follow the same pipeline used to create property and text annotations for the MolQA data, focusing broader properties that describe drug-related utility with 41 small molecule properties (Swanson et al. 2024). Besides the three used in MolQA, others include:
Toxicity and Safety: AMES, Carcinogens Lagunin, ClinTox, DILI, Skin Reaction, hERG
Enzyme Interaction: CYP1A2 Veith, CYP2C19 Veith, CYP2C9 Substrate CarbonMangels, CYP2C9 Veith, CYP2D6 Substrate CarbonMangels, CYP2D6 Veith, CYP3A4 Substrate CarbonMangels, CYP3A4 Veith
21

---

Published as a conference paper at ICLR 2025
Absorption, Distribution, Metabolism, and Excretion (ADME): BBB Martins, Bioavailability
Ma, Caco2 Wang. Clearance Hepatocyte AZ, Clearance Microsome AZ, HIA Hou, Half
Life Obach, Hydration Free Energy FreeSolv, Lipophilicity AstraZeneca, PAMPA NCATS,
PPBR AZ, Pgp Broccatelli, Solubility AqSolDB, VDss Lombardo
Stress Response: SR-ARE, SR-ATADS, SR-HSE, SR-MMP, SR-p53
Nuclear Receptor Interaction: NR-AR-LBD, NR-AR, NR-AHR, NR-Aromatase, NR-ER- LBD, NR-ER, NR-PPAR-gamae gaamma
We describe polymeric material utility based on 14 polymer properties collected from Otsuka et al. (0011)
Thermal Properties: Melting temperature [] Specific heat capacity at constant pressure (C,) [cal/(g- "C)] Specific heat capacity at constant volume (C,) [cal/(g. CC); Thermal conductivity [W/[m-K]]
Physical & Thermodynamic Properties: Density [g/em"]; Fractional Free Volume (dimen- sionless); Radius of Gyration (Rg) [nm]
Permeability Properties:
Gas diffusion coefficient (D) [em²/s]; Gas permeability coefficient
(P) [em (STP))cm/(cmm Oxygen (02) Gas Permeability (Barrer); Nitrogen (N2)
Gas Permeability (Barrer); Carbon Dioxide (CO2) Gas Permeability (Barrer)
Solubility Properties: Gas solubility coefficient (S) [em (STP):cm/(cm²>>)
Dielectric & Optical Properties:
Dielectric constant.
We train two multi-task GIN models based on the rationalization method (Liu et al. 2022) using all existing labeled data for drug and material property prediction, respectively. We these models to predict properties for millions small molecules and polymers, retaining the top ten thousand predictions by confidence score for each property. These are then used to prompt Llama-3-70B to create molecular descriptions, using the same prompt template as in appendix C.2.3 Additionally, we apply the same strategy as appendix C.2.3 to annotate labels for the eight studied properties, which can serve as input for pretraining the multi-conditional Graph DiT. Finally, we have approximately 300K graph-text pairs for small molecules and 300K graph-text pairs for polymers.
D ADDITIONAL PRE-TRAINING AND FINE-TUNING DETAILS
We pre-train three graph models including Graph DiT (Liu et al. 2024ch for multi-conditions molecular generation, GIN-based GNN predictor for reaction template prediction, and a GIN-based graph encoder for molecule understanding (Xu et al. 2018).
D.1 PRE-TRAINING OF GRAPH DIFFUSION TRANSFORMER
Suppose the node has Fy categories and the edge has Fg categories (including non-bond). Graph DiT models the node token by concatenating all its edge configurations to other nodes. For each node x E RR,, we have F Fv + NG FE, where NG denotes the graph size. This facilitates defining the transition matrix Q for the joint distribution of nodes and edges (Liu et al. (024c). Graph DiT uses Transformer layers, replacing layer normalization with adaptive layer normalization (AdaLN):
(h) (h)
AdaLN (h, c) = (C(c) O
+ Ba(c),
where h denotes the hidden state of x and c is the vector representing the input conditions.
Given multiple conditions with categorical, continuous properties, and text, Graph DIT uses one-hot encoding for categorical properties and a clustering-based approach with Linear (Softmax (Linear(c)) to embed continuous condition values We employ pre-trained SciBERT (Beltagy et al. 2019) to embed input texts into a 168-dimensional vector by averaging the representations of all text tokens in the sentence, then using linear layer to adjust the dimension for Graph DiT. For each condition, the model also learns drop embedding. The drop embedding is used when no values provided. Finally, the model sums the representation vectors of different
22

---

Published as a conference paper at ICLR 2025
conditions input for c. In the reverse diffusion process, the denoising model uses predictor-free guidance to sample molecular graphs given multiple conditions. We pre-train the denoising model with the loss function in Eq. 2 using 600K graph-text pairwise data and the eight properties defined in appendix C.33 The model employs the following hyperparameters: depth of 28, hidden size of 1024, 16 heads, and MIP hidden size of 4096. The total model size is around 574 million parameters. We pre-train the model for 45 epochs, which takes approximately one week on a single A100 card.
D.2 PRE-TRAINING OF GNNS
We pre-train a three-layer GIN to predict reaction templates among 30, labels, using a hidden size of 512. Reaction template prediction is multi-class classification task. Given reaction-text pairs from MolPair, we extract the product molecular graph from the reaction formula, using the reaction condition text input. SciBERT (Beltagy et al. 2019) used as the text encoder with frozen parameters. We average the text representations to obtain a sentence-level representation. The prediction target the reaction template extracted from the reaction (Coley et al., 2019). GIN naturally uses molecular graphs, employing the AdaLN approach as the normalization layer added after each passage-passing layer to incorporate text conditions. We pre-train the model for 5 epochs on single V100 card, with 632 million parameters. This model serves as the reaction predictor to suggest reaction templates for Llamole.
For molecular understanding, ERT (Beltagy et al. 2019) is used as the text encoder with frozen parameters. We average the text representations to obtain sentence-level representation, while the GIN model uses sum pooling to produce the graph representation. For each graph-text pair from MolPair, we optimize the graph encoder using the CLIP loss (Radford et al. 2021) for 40 epochs. The CLIP loss consists of two contrastive losses: it first computes the similarity score between graph-text pairs, then contrasts it with all other similarity scores by pairing the graph with other texts and pairing the text with other graphs as negative pairs. The model has around 43 million parameters. The model can be pre-traines on a single V100 card in few days. This graph encoder will replace the word encoder in the LLM tokenizer module for molecules indicated by the token <molecul le> as shown in appendix B
pre-train a five-layer GIN model with hidden size of 768. SciB-
D.3 FINE-TUNING OF LLAMOLE
Llamole is fine-tuned on graph-text multimodal instruction data, freezing the parameters of the Graph DIT, GNN predictor, and graph encoder. automatically adds eight query tokens to the sequence once the trigger tokens are predicted, allowing the base L.M to continue autoregression and output vectors for all eight query tokens. We average these output vectors as queries for prior generated texts and use them as input text vectors for the subsequent Graph DiT or GNN predictor module via tunable linear layer. For the <molecule> token, we add a tunable linear layer on top of the token embedding after the graph encoder outputs it. Without loss of generality, we study three variants of Llamole with different base LLMs: Llama-3.1-88 (Dubey et al. 2024). Mistral-7B (Jinng et al. 2023), and DB (Yang et al.) 2024 All LLMs are fine-tuned using LoRA (Hu et al. (021 for four epochs, taking approximately two days on single A 100 card.
E ADDITIONAL. EXPERIMENTAL DETAILS AND DISCUSSIONS
E.1 ADDITIONAL DETAILS ON EXPERIMENTAL SET-UPS
In Tables [ and 2 and Figures 5a and 5b
Llamole is compared with fourteen LLMs with sizes rang-
ing from 7B to 70B, including Lama
Dubey et al. 2024), Mistral (Jiang et al. 2023), Qwen (Yang
2024), Granite (Abdelaziz et al. 2024), and Flan-T5 (Chung et al. 2024). We prefer the instruct le al. version of the model when available.
Using the MolQA training set, previous work can implement these LLMs in two ways: in-context learning (ICL) and text-only supervised fine-tuning (SFT). For ICL, we retrieve five closest QA pairs from the training set based on the average property difference from desired properties. The template used to construct the prompt with demonstrations is:
23

---

Published as a conference paper at ICLR 2025
I' working designing and syathesizing molecules. Here are some example questions and answers about molecular zequirenents, design, and synthesis: [[examples]]
Now, based on these examples; please answer the following question about molecular design gn and synthesis: ((quastion)]
For SFT, we fine-tune the LLMs with LoRA after converting molecules into SMILES strings.
The MolQA test set contains 9,986 QA pairs for small molecules in drug applications and 750 pairs for polymeric materials. The questions serve as imput to prompt the LLMs to generate responses.
For the controllability of multi-conditional molecular generation, evaluate up 12 metrics across four aspects: (1) chemical validity, (2) similarity to the truth based on Morgan fingerprints, (3) BLEU- 4 and ROUGE-L scores compared to reference texts, and (4) deviation from desired properties. For polymer validity, we further examine whether the generated molecular structures contain at least two polymerization points ("*"). To obtain the properties of the designed structure, define an oracle function based on well-trained random forests from all annotated molecules, following previous work (Gao et al. 2022, Liu et al., (202cc). We evaluate three drug-related categorical properties using balanced accuracy (BA) and five continuous material properties using mean absolute error (MAE). As a baseline, we consider GraphGA (Gao et al. 2022) to reference the performance of LLMMs compared to domain-specific methods.
For retrosynthesis, we evaluate the success rate from the designed molecule to those available in Gwvil purchasable from the Enamine Building Block (June 2024 version), supplemented with other common ions and starting materials, totaling around million.
E.1.1 SET-UPS FOR FIGURE
For Figure ! we average the balanced accuracy for three drug-related properties and five MAEs for the polymeric material properties. We then select the model with the best performance in each category based on these average metrics. For drug tasks, the best ICL model Llama-3-8B-ICH the best SFT model is Mistral-7B-SFT. and the best Llamole variant is based on Qwen2-7B. For material tasks, the best ICL model Llama-3-70B-ICH the best SFT model is Llama-3-BB-SFT, and the best Llamole variant is based on Llama-3. Alamm-331--BB. 1-8B. Their average performance is visualized in Figure in comparison with GraphGA
E.1.2 EXTRACTION OF SMILES FROM LLM RESPONSES
ICL or SFT-based LLMs generate free-form text that includes both natural language and SMILES- represented molecular structures. We need a method to automatically extract SMILES strings from LLM outputs for evaluation. Practically, one can observe generation patterns to summarize rules for regular expressions to accomplish this. In the MolQA training set, the designed molecular structures typically follow the phrase "the designed molecule is:" as shown in examples Figures and I0 LLMs may not always adhere strictly to this pattern, In the future, more sophisticated regular expressions could be developed to extract SMILES strings from text directly. However, these will still need to be combined with additional rules to identify the designed molecules, as L.MS may generate intermediate SMILES strings before and after the designed molecule. Compared to them, Lllamole <des gn_start> <retro-start>> to indicate the position of generated molecular structures.
cases. may need to extend this rule cover
E.2 ADDITIONAL DISCUSSION ON ONE-STEP GENERATION
We further examine the text generation results for reaction conditions. Since the answer represents just possibility in retrosynthesis, condition descriptions as references for T based on the available templates within the USPTO reaction space. One template may correspond to thousands of reactions, items to manage costs while identifying the best matching generated and reference pairs.
use the template to retrieve the best-matching reaction
limit our search to five
The results of generating reaction texts shown in Table where Llamole achieves the highest ROUGE-L but low BLEU-4 scores. The best ROUGE-L score for Llamole indicates its capacity to understand and maintain the overall structure of the answer after fine-tuning. The lower BLEU-4
24

---

Published as a conference paper at ICLR 2025
Table 5: Text Generation for Reaction Conditions: Best results and best baselines are highlighted.
In-Context Learning
1oma-2-77
Misemi-78
Qwen2-7B Elama-3-88 Ellma-3-88R
Flun-TS-XXI. Gounie-113 138 U1lma-2-138 Mistral-8x78
1lmma-2-70R
1lmma-2-70R
BLEU-4
0,021
0.006
0,005
0.107
0.130
0.077
0.051
0.048
0.136
0.004
0.009
ROUGEF-
0.112
0.141
0.095
0.205
0.250
0,202
0.159
0.149
0.248
0.152
0.164
Supervised Fine- Fine-tuning
Llamsde
Mistral-7B
Qwen2-78 7B Llama-3-88 3-813 Llama-3.1-88 3.1-8B8
Mastral-88 Qwee2-7B 7B Liana 3. 3-B-
BLEU-4 ROUGE
DAOKS
0.141
0.114
0.111
(L00))
0,0774
0,085
0.191
0.201
0.192
0.262
0.268
scores may result from the A search nature in Llamole, which explores a vast space (300K) of possible reactions, leading to fewer exact n-gram matches with reference sentences. The many- to-many relationships between products and reactants, along with various conditions for the same reaction, diminish BLEU-4's effectiveness in evaluating Llamole's 's capabilities. Overall, Llamole is not merely memorizing reaction conditions but actively exploring possibilities, yielding more contextually coherent and meaningful outputs.
E.3 ADDITIONAL DISCUSSION ON CASE STUDIES
We present case studies for baseline LLMs using the same question as in Figure 7 Results are shown in Figure 7 The reference indicates one possible ground truth for molecular design with retrosynthetic pathways, noting that many alternatives exist. Compared to the reference, results in Figure demonstrate that Llamole designs another molecule with similar structures, properties, and shorter synthesis routes, showcasing its potential for controllability and generating synthesizable molecules. Using ICL., Qwen2-7B fails to generate meaningful responses, despite indicating possesses rich knowledge about molecular design. SFT allows Qwen2-7B more strictly follow instructions, producing meaningful responses. However, text-only generation leads to hallucinations, as the generated templates do not yield expected products in retrosynthetic planning.
Another example based on Llama-3.1/2 3.1-388B is provided in Figure I0 The ICL method may copy from the demonstrations get the SMILES string CC (=O) C-Col ce (C1) ccelCl. It also includes one SMILES string before the designed molecule, such as CN (C) cl coa (C=NNc2ccc (I) cc2) ccl. However, it does not follow the instruction pattern and therefore not automatically extracted for evaluation, as illustrated in appendix 1. SFT follows the instructions through fine-tuning, using the pattern "the designed molecule is:" but generates invalid structures with meaninglessly repeated sentences. In contrast, Llamole generates meaningful and valid molecular structures that generally satisfy the question's 's requirements. During text generation for molecular design, Llamole analyzes the question and includes more details about desirable structures, such as "aromatic rings" and "aliphatic chains". Some functional groups, like hydroxyl, may not be precisely represented in the structure. This indicates need for enhanced text instruction adherence in Graph DIT.
In addition to small molecules, we present a polymer inverse design case in Figure 11 based on Qwen2-7B. The polymer has polymerization points denoted by "g* in the reference structure. Since polymerization reactions are not considered, we focus on the retrosynthetic routes to the monomer structures by replacing polymerization points with hydrogen atoms. In this case, ICL-based Qwen2- 7B fails molecular design due to the same issue as in Figure 9 not following instructions to generate polymer structures. SFT-based Qwen2-7B generates polymer in SMILES format but invalid in chemical space. In contrast, Llamole successfully generates valid molecular structures through Graph DIT, satisfying the requirements of "high CO2 permeability and low permeability to N22 and O2," " and suggests a two-step retrosynthetic pathway for the monomer structure.
25

---

Published as a conference paper at ICLR 2025
Question
"Can you design and synthesize a molecule that inhibits both Human Inmunodeficiency Virus and Beta- Secretase 1, with molecular weight around 335 and two aromatic rings connected to aliphatic chains?"
Reference Answer
"To satisfy the vequirements: The molecule has modorasely complex structure, consisting of aromatic rings connected to aliphatic chains, and featuring carboxyl and hydrozy! finnctional groups. Therefore, the designed molecule is: <molecule 17.
This is step I of 5 in the synthesis process. To synthesize <molecule 12. follow these procedures: A solution
of (R)-acetic acid 3-ftoluene-4- cooourne--sufffoyloo sullfonlooxy--uut ester 89 g. 21 mmal) and 4-chloro- --hoooo--phenory-eee 2-phenory-phooo
15.0g 23 mmol) in DMF (50 mL) is treated with cesiumi carbonate (7.4 g. 23 mmol). The sobition is heated to
60tm00000 C. and stirred owenight. The reaction cooled and quenched with IN HCL The solution is partitioned in FrOAc and water. The orgonic separated. washed with brine, and dried over sodium sulffate. The organic is filtere, and the solvent removed afford acctic 2- ppennxy-phenoxy phenoxy)-buut
3-(6-chloro
ester, which then diluted in methanol (100 mL) and treated with potassium carbonate (5. (3.68 g. 40 mmol). The reaction is stirred for 2 hours at r1. The reaction is then partitioned EIOAc and water. The organic layer is separated, washed with brine, and dried over sodiumm sulfate. The organic filtered and the solvent removed afford the crude product. The crude purified by silica gel columa
chromatography using 1/1 hexanes/EEOAA to clute the pure product. The solvent is removed to afford 4.35 g (72%) of the desired product. 1H NMR (409 MH., CDCB3, MS (ES-) mis mass caled for C1GH17CI03 292, found 293 (M+1, 1007)). The applied reaction is Sreaction 12, which veguiiees the reactants; <molocule 2> (avoilable). <molecide 3> (avatlable), <molocule 4> ovailable, modss he synfhestzed), [Cst] (avatilable) and [Cr+] (available).
PROO OA emotectde 25 SmOLCuu, SPecnieraate dt
This is step 2 of 3 in the synthesis process. To symthesize <molecule 47, follow these procodures: A (2222 242000000 C. solution of
t-chloro-2 pheeno----- (methaxy-beezzene (7.16 g. 30.5 mmol) dry CHZCI2 (70 mL) treated dropwise with borontribramide (22.9 g. mool) and then wormed to 0u000b0 C, and stived for h under N2. The reaction is dilated with E120 and quenched with water. The organic layer is dried (Ma2504), and the solvent is removed in vacuo to afford 7.11 g (100%) of the title compound. Rf-0.35 0.30 (4/I henanesiacetone) HH NMR (400 MHts. CDCB3, MS (ESu2212) mis mass caled C12H902CI 220, found 219 (Mu22121 100%). The applied reaction <reaction 22, which requires the reactants: smolocule 5* (not available, needs to be synthosized).
This is step 3 of 3 in the synthests process. To synthesize emolocule 3-. follew these procedures: mesture of 2- --homo-4-chlroo- c-hro----------mmm-e- Aolaro---methory mmethory-benznn (8.0 g. 36.1 mmol), phonol (6.80 g. 72.2 mmol), cestum carhonate
(23.54 g. 72.2 mmol). copper (I) chloride (1.79 g. 18.1 mmolj 2.2.6,6 5.66.tterrmmellll.. 1erramethyl-35-hhptaae 3,5.heptamedione (1.66 g. 9.00 mimott
Connction 2
emoleouts Eo
in dry 1-methyt-22 -pyrrooiiiii (80 mL) is heated 120 12020000 C. for 20 hours under N2. The reaction cooled. filtered and the filtrate quenched with I N HCI (50 mL)) The filtrate dibuted with EX20 and extracted with water. The organic layer is dried (Ha22504) and the solvent is removed in vacuo to afford crude product that absorbed on silica gel and purified by flash chromatogrup uhyh using 9/1 hexanes/FiO. to afford 7.42 g (88%) of the ttle The applied reaction is: creaction 3>"
compound. Rf-0.37 37 (4/l hexames which requires the reactians: <molerule 6> (available), emolocuie 7> (available),
Spuaciin 3P
<molecule smclecuda_ smolecda
<molecule 30 (available). [Cs J (avaitable). [Cst] (availablee) and <molecule 8 (available))"
ICL
"Derisiny zontherising a erdicaic isbivii Havas deicion Firr (813) and Sertiio i (BACEl) sould mnchee costing a soupio strotve mil specilic foctioall wiiecal
Hoe's sipfoli chaio "Funciooal RACE) the BACBI ieblition) "Rjshppuuccip ". biddig Paredal Moireslr
grogs daat bind rial seth
"Mddccaa Feight" Araud tree
Rogs"
Licty inclss
dhe H/V italiii
for inharellle
"donettily"" "*
mabilyy
"Rosnalle Bood t feshiliy oklnng
4 pussily
rost
for all
BACI The
rag night iv convened trachai aljhate rkale hetuter lpupiiliciyy sid sprapnu rebiitantl for ungate
fissklin AN Susheas Aunt SSug Suafes of droall Core+. "Oljemme** Frone saall cosatic
interstion.
mmpprittion besk the nevesrry aomaic For essplc staiing milk olow olly composiil, perfore Dis's Siker resation chelloptiic oddlin reactis inboche fort arssuni
"Mashod"" Use mull-mg sybbou incliitg condenstion restiiss kalagenttoo, whler fictioa
Salnopaest sigo roel isslle fetke wullication sach Be carbay fi midh and onie factistallies SSg 2 Pastiooalaaionn and Living "OMjoiie"" H2, houoher fescionll grogg shat on orreee wit HIF oud BACE. ""Mohhu"" Us satleghhie adluitationn mas fist, odbils
halggassion, intoobation Janciion
dieet coppiig woldd d.g. rick cheaisty is atach de resiiolf Jactionll gragu the anosatii coes. Tais inaae stuching gouss the suoeemsss for HIT whiiii hodyy eners scep' pags for 84C)) whihiian "SSp Conteciing Alpealic Oniut* ""gaermn Juis no sosall gasee do alpkati chilse ctna yutial confiumatiee cutabl for auyse fisting "Mook""" Carry ou coniss of moriuss macions caspopae aáite mariau coternt fartioealeed amsutt ragr dragt alphaii brdgy The clstce allpph cboter will deposi balawing detied bydephatartt and Eestiby "eaop Protecioon and Depreerttis** *Ugyctyy Pranec mottr foaciose grugr daveg racher and deprotest thom jur hofe finat partirmtion "*ileoo"" sunder protennng group maiegen sptepsani frncioal grapu der right react undetrally dring pameest Conseas proteting gropp whhsh 6Sh, Posc fic props, depending on the
fectioal pugp intelast "Suy 5. Parifcation Choveriiationn "OONjecti"" folis torel esfeeak coffee partt und Wastber neight Exptoy iecheigee posterprrisetsury bologial actvtry
choonoogrgpyyy HPIC. prguraaivv
puij ustocals geevell antys (WUR MS sorll) mistive confee though
soggs sugring repication porotially apeirances ""Cpnscrutoe"" emboon
comity
sajey prolin for
driggs nystous regate plantig sathen' Addmsnally so yarkons dngs that caget malipll pathrage couple and opticall mndie multdteppiiury opreach
pooeecy spetteeyy
and rppertio
contiong machine chenuory, balngy, and pharaacology.
(Error: Repeating many times with invalid structure) SFT
To satiefy the requirements: The molecude is moderately complex structure comprising two aromatic rings connected to aliphatic chains, which may influence its physical and chemical properties. Therefore, the designed motecule is: (NNjecce(()joo200P))
(F).)). This is follow these procedures: In like
the preparation of NN-44-benyyloxee connyloxyccrbbonl- 2-cyno-----o---e- 3- J.kyyruxspphnn/ 5 S-melhylizatin, 2-amino-5-12 5-2,,2,2 2r222:inllorocchha))/oee beezoniriie (0.698 g; 0.46 mmol) was roacted with 6-meshyl-ppan www.yyl-pordiiie--.oom 3-caroomyl chloride (0.073 ml: 0.35 mmol) to obtain N. N.(5-methyl-2--22 2-brffurroohhoxs (pridwoo)) 5-merhylisotin as on off-white powder (0, 062 gi. The applied reaction it:
step I in the synthesis procese. To syuthesize ColevcCC(-
(elcee((-O)C(()eea (NC-OOCCNNeeeeCCNCcccc.. 2OCC (F))ee/))) Celleee(C--OOCieeee [C:HO:D3, (0:))/00000000000000 (0: -::-:://--....--:66--->::: HH:IC: -0010:02000011(1000:::::. (1::::.)))-----)--e---e- ( which requires the reacionts:
NDCC)))) with the template ([::4:)) (avallbble), and (CCcccc(OOC/F)(F)Fo... (ovoilabie).
(Error: Retrosynthesis falls since the template application did not yleld the expected products)
Generated Reaction
Generated Template
Mismatch
Figure 9: Reference Answer with Generation Results from ICL or SET Using the base LLM Qwen2- 7B. The question is the same as in Figure 7
26

---

Published as a conference paper at ICLR 2025
Question
What is a possible molecular structure that can penetrate the blood-brain barrier and inhibit Human Immunodeficiency Virus, given a molecular weight of 209.249 and specific ring and bond characteristies?
Reference Answer
"To satisfy the requirements: The molecule has an aromatic ring and common functional group of curboxyl, which suggests a certain level of structural complexity, likely influencing its physical and chemical properties, and indicating a moderate syntheti complexity score. Therefore, the designed molecale is: <molecule 1>. This is step of in the
synthesis process. To synthesize <molocule 1>, follow these procedures: To a suspension of (aa.bboo)o--((-nniiii (f-ninro2-pyyrdddi
(0.27 g. 1.13 mmol) in methanol (2 ml.), ethyl acetate (4 mL) and TEA (0. 16 mL) was added 10% Pd/C (60 mg, 0.056 nonol) under argon.
<molecule 1>
The reaction mixture was hydrogenated under I atm H2 H2 20 kr. filtered through Celite and concentrated in vacuo to give N- 15--mino((2-ppp pyridyyy) (tert- babraiiarboxa (0.226g 97%).. The applied
reaction is: <reaction 1> with the template <template 12, which requires the reactants: <molecule 2> (available)."
<reaction_ 12
<molecule 2>
ICL
Rosdd the gius aode wlar waihh and rin and heed charorortotrs, a posaille resievslor imcuue that car prrvenate dhr Moutt harrior and iubisi thaean inmouiidittcc fino is CWCClluterC Nlicbwsfbeeeeee This audensl kes a wookrat gybbat compleniit und uccesiiiityy scons, velicsing weigh to retatined challonging to saflesic is soficallr noight and wetts of snomalic rios costrilate is is moderstt cosplestt. Tie presonee of Ayshougl yod unisn fouciions grogs saggeio posille lystnges loosing oxpsbilties. The noshesic's stafiidd componat of noo sussaiic rog Vibkst by on pais goug, whish cosld bighowee is plosical and clessial progesios The welccator strectue designed to be oble to poncioite thosd lwass bavrics and wilth Mlams teovosiieereeey Vius. The appind reccios in CWIC teeett Kieboethe beeeeeeeettssteee coccs Gppe Nilloodtee which regaieer the murtuss CNICploodt--Gn
<template... 1>
Gesiisshy, Nileoostleei inmilaates Question: What a feasible molecular design that incorporates an aromatic ring and carbory! functional group, with molecular weight around 215 and only rotatable bonds? Answer: To sutisfy the requirements: The molecule has a relatively low synthetic complexity, featuring an aromatic ring and carboxyl functional group, and likely to designed molecule is: OCC=O)C-Celee([ii
be easily synthesized. Therefore, ocr oic CcheeChree!.. CC,, jollos toos pescebbrs. Is ran (599 wil vas duselved solins lyshosile (2.0g). and to the witstrt war wbite unchooe (100 sil, and She 23 25-Jiciinohcnnnbbllee 05.9 u. The wimoe was sireed nom nosperowe for hase und oochor war exagermed seler rechool prssore The reribee mo cairached with slly uostatt and the orgonic layer was wached with eaur and sataated brive, and con cnmaoo ualer rebsood prursae to glve 4:2 SchAhinghhet is F-baon ao (35 dgg. The spolled ensction
Cl. This is mp of is the spashods pricees cysthorto
Diat fia+),000 with somplae 10.D1.H0 (0--H955)--C (C:4) D1--CC--2:-02 022---2--------020-------- 1020=00- (CH.) w.-1-- (CHED), -0 (C- ():4))(001 19-51, whick requiees the reacisstr O- Celee fanatiatfa), (HH)CC(2H//2H)CC -OC.-02 (ematisble), [Diat]
Ch-Celool CCC.CCCCCCC.CCCC111 (oMi)s2taisitaa (11 (0
tarallabee JOB3 temilalli
(Error: Copy the answer from the demonstrations)
SFT
This is a moderately complex molecule with a scaffold consisting of on aromatic ring and an aliphatic chain. The presence of a carbaxy! functional group and an amino group contributes to its overall synthetic complexity. Therefore, the designed molecule is: CCeln CCein[nHJof [nH]ol-222
Sactioo grap c2cccce2(nn C. Tae wodenasly compler wplocate vifk a regfisl constving of os ansuatc and alpiatte chett. The presene conbbary group
spstooe complait Thanfire, desggeed molecsle CCcinjalfj---eeneee (Ervor: Repeating many times with invalid structure)
trspect
Llamole
To satisfy the requirements: The molecule is characterized by the presence of aromatic rings, aliphutic chains, and common
functional groups such as hydroxyl, carboryl, and amino groups, which contribute to its moderate synthetic complexity.
Therefore, the designed molecule is: <molecule 1> To synthesize "molecule 1>, follow these procedures: $ g of 3-ih-ro
2H-1, 2H-1,4-beme 4-benzoxazin-4-ont (prepared according to the procedure of Example I. step I) was dissolved in 30 ml of ethanol and 1.0 g 10% Pd/C was added. The mixture was hydrogenated under atmospheric pressure was filtered off and the filtrate was evaporated under reduced pressure.
<molecule_ 11
BB8P: 1
temperature for 24 h. The catalyst
HIV. 1
Moll: 289.08
The residue was crystallized from ethanol to give 2.5 g of the fitle compound as a yellow solid. The applied reaction is: <reaction 1> with the template <template_ 1> which requires the reactants: <molecule 2>. To synthesize smolecule 2>, follow these proceduee::
<reaction 1>
<moleccle 2>
3-1 +-(2-MMehhh (2-Methoxy-5-methe 5-mmedyyppeeny)amnnno Sactiooo---ppppp -propanno (3.0 g. 14.6 mmol) and triethylamine (2.8 g. 28 numol) were dissolved in methanol (20 ml) and cooled to 0la0060 C. 3-Bromopropavoic acid (1.0 g. 7.3 mmol) was added and the reaction stirred at 0'u00b0 C. for 2 h. The reaction was then allowed to warm to room temperature and stirred for 18 h. The reaction was concentrated and the residue was difuted with ethy! acetate (100 mi) and washed with IN HCI (100 mil and brine (100 ml). The organle layer was dried (Mg504) and concentrated to give (((-mmethxxy-5e Serghhhxy---mehhllppe S-meehylphenyl)nmee propoppoooii acid (2.4 g, 12.2 nunol, 84% yieldy. The applied reaction is: <reaction 2> with the template stemplate 2> which requires
<template_ 1>
<reaction 2>
<molecule 3> <molecule 4>
the reactants: <molecule 3> (available). <molecule 4> (available).
<template 2>
Figure 10: Reference Answer with Generation Results from ICL with Llama-3-88 3-BB and SFT and Llamole with Llama-3.1-8B.
27

---

Published as a conference paper at ICLR 2025
Question
What is the optimal molecular design and synthesis roude for a polymer with high CO2 gas permeability and low permeability to N2 and 02. featurrng an aromatic ring and specific functional groups?
Reference Answer
"To satisfy the requirements: The polymer,featuring featuring on aromatic ring und functional groups such as carboxyl and hydraxyl, exhibits a moderate synthetic complexity score and high gas permeability. indicating potential for use gas seporation membranes with good efficiency. Therefore, the designed molecude Spolymer I>.This is step of 1 synthesis process. To synthesize <polymer 12. folloolow these
procedures: Ethanol (3.2 mL.,, 54 mmuli and 4-dimethylamino pyridine (DMAP) (54 mg, 0.45 mmoll) were added mixture of pyrroe-3--carbopp
Spolymer_ 1>
acid (500 mg. 4.5 mmol) and dicyeloheryccaroddiiiie (1.11 g. 5.4 mmol) (Aldrich. Milwaukee, Wis.) tetrahydrofuran (THF) (15 mL). After heating at 600u0000 C. for 10 hours, the reaction was cooled. The precipitate was filtered off. washed with etly! acetate,
P.a. <molecule 25 <molecule 3 smolecule 4>
Sreaction 1>
the combised filtrate filvatt was concentrated and purifted on silica
gel columa give 500 mg (8 (81%) 1H -yrroll--33 carboxylic acid othyl ester
colorless oil. The applied reaction is: sreaction 1> with the template <template 12, whick requires the reactants; <molecude 2> (available), smolecule 3> (available), and <molecule 4> (available) ICL
stemplate 1>
"n duige
palyaer CO2 gu perssabiity los paraasbiitt and fromnhng
rhg guegle grogpe
aloo any CO2 insegort hat resiees O2 paraas This unell iusiive of fartie siecture that ollor cuirr diloi frougt poijner cvsiiesl with dee
agongs
mrap
spepie
0Z, bacdaae, stich ixpesse subilyy the poinner doe iv iakoraat riphir asi elschank properies. *"Fanctioal Grage
eslacolar
couaierd
**
""obnner farlone ** loket
His right tehaas youge lk carvvyl (COOR) =C-G exi which have lvas sionen endasse C22 porseolityy is cow sareial Adhiinally deveporating whlee antee coving bonaactisss dhrugh INe anaaaat rags or onglggad grage codd help rige CT7 offemnly aihher Adng et divethe for matler coppttr X2 and 02 seleesits "* bting and Shaectigg 460igg crue lods aud bezeiing in de polnter santt on rtodyy the euopphsigg and indor the netaonts dendl orlons vy perveshitty wiik pooilf iisingg de difisi path for X2 usd O2. *1S "tark Rose 1o) "PPrawation of Howwery. Syshoux or olou museeee cottir coppoiill with mumut basilve,, eumple ssiuht palveoriatiiv condersatioo partisd reuiifiatiio
Jstwaher feettnd
can eveaee CO2 pronziiiiy skic siciniing pormadhitt atet
deatt Junciionl grass
contitinng canbuyfic and (CTOH)) atilldd O)) crsfonatii poù vevication dysslingg the
resliv slee dar can tgee
sire ofte pro) seriiatis d'arioptic properiie Farttooaliatlee "I
radical
father
fanctions!
polyer hash Art and O3, is excore that doe cuteeveer detge wore the inqatedd rinoro Ie hespur the pormeabilty turr gar adoppts and suemoningy could by and jvr 2iss parpert 1 "Cpuxroccc"" Hoad on caroeerrntti rendee aptiatt the noieeske duigg exeleets cootitiee roeded to masterr C02 prosecndly shilr wetroning or caniattng XC ond (22 porwoolity is
pervostton ckeattristiie 4 ** Son as Aealy nxderchal pol
peraeelilt) sponiicllly toeend Cot
evessury, dee optinal osfecshor dorgs couid inceponale at ansaatic baghbas sil shalogaall ylocad fonctioal graps tha facitoo C02 tsuggee whit Ansberr in dillnos of N2 and the
ooliesing the desiee gas permention progeries. The syslesis mase iastee peoprrig the right easomeey, sclectng on apgingisst polpeeriatiinn wallod, petontially contusiing pooigel wortisiss chanscweting tor pahyoet, tanabialy sporering desige geverrry (Errror: Molecudor design failed due
following instructions)
SFT
"The pofymer has velatively low synthetic complexity score, indicating it can be easify syuthesized. Therefore, the designed molerule is: (Olte//CCO))CCCCCCCCCCCCC 1C. This is step of in the syuthesis process. To synthesize
Celezx(()ccCO CCCCCCOC(-OOCCC) follow these procedures: 30 g (0.33 mol) of 3-flurro methhylbenn alcohol are dissollvd at 5.00000 C. in 200 má of dickloromehhnee and 55.6 g 40 mol) of titanium tetrachloride and then stirred with cooling S'u00000 C. for 15 minutes, A solution of 27.9 (0.33 mol) of 2-chloroncetyl chloride in 50 ml of dichooromethane is odded dropwise and the mixture is allowed to warm to room temperature and stirring is continued for 2 hours. The reaction mixture is extracted with water and the organic phase is washed with saturated sodium hydrogen carhonste sobution, vater once again and an aqueous saturated sudium chloride solation. The organic phase is dried over magnesium sulphate and concentrated. The residue chromatographed on silica gel using gyloherone/ethyl aceiate (8:2). This gives the desired product in a yield of 31.3 g (76% of theory). The applied reaction is:
Cet (eeecc()e) 10a1-CCOCCCeeeeee CCicceeePPeCc CCO.. C0..CCCOOCCCCCCC O-CICICCC with the template (C-2]-(CH2--D2:--1 (]H[O::: +::::: 0::J==CC (3-/)=CCC(CH2DDD. 10:1]- [C:2) D1; :1:3:::::::::3C) -C:) D3: Q:4][-/C:DI (-----1-1--C- HO)-HC; D3: (0) -0,8), which requires the reactants:
Cel Celecco(F)eICC (available), and O-C(CI)CC (available), Banel on peeooide information, the aversl soahene lentty sove for this modeeale us Are, inficsing that
woee Iy wary nechaniadd Tunnon, du soiged mohecdle i. CCCCCCOCI DjCk... (irehenan))
(Error: Molecular design failed due to invalid structure and not not following instructions)
Llamole
"To sotisfy the requirements: The polymer has relatively low syuhetic complexity score, indicating that can be easily synthesized. The presence of an aromatic ring and common
Co, 18.3
NyPerm: 8.2
O,Perm: 10.7
functional groups such as hydrosyl and carboxy! groups contributes to its structural
complexiry. Therefore, the designed molecule is: <polymer__ 1>. To synthesize
<polymer_
"potymer_ follfollow these procedures: Mix 100.0 gms of meldrum acid hemiacetal and 150.0 gms of 10% pallodium ch-chare in methanol and hydrogenate atmospheric pressure for hours. Remove hydrosyl group and yields 134.2 gms of I.I. with the template <template 1> which requires the reactants: <inllecule 2>.
2-propyn.. The applied reaction is: =reaction 1>>
<reaction
<molecule
To synthssize <mollcude 2>, follow these procedures: g (0.004 molj of acid were suspended in 100 ml of ethanol and 0.5 ml of acetic acid were added. The reaction mixture was heated
2. -oryddrrr-----o--------
<template 1>
reftux for 24 hours. The reaction mixture was cooled
temperature and concentrated under reduced pressure. The residue was porified by column chromatography silica gel (eluent: ethyl acetate/hesane 1:1) give 1.0 g (51%) of 2.3 d-ihydrro-iHHH -H---Hol-------- -nn-oo----ybbrrnic acid a colortess powder.
creaction
2
<molecule 3> <molecule 4>
The applied reaction is: Sreaction 2n with the <template 2> which requires the reactonts: <molecule 3> (available). <molecule 4> (available)."
<template 2>
Figure 11: A Case Study for the Polymer: We include the reference answer and the generation results from ICL, SFT, and Llamole with Qwen2-7B.
28


Reviews:
[{"review_id": "xFutzWmDNC", "reviewer": "Reviewer_DEua", "summary": "This paper propose a multimodal large language model framework to jointly model molecules and texts for drug discovery and retrosynthesis planning tasks, which is motivated by the fact that the joint modeling of texts and molecules remain challenging.  To address this, the framework combines an existing LLM with two pre-trained graph encoders: one for conditional molecule generation and another for predicting reaction templates for retrosynthesis planning. The paper further proposes to apply A* search to ensure the target product could be synthesized from reactants that are purchasabe on the market. To summarize, the goal of this paper is to propose a MLLM-based framework that is able to build a pipeline from drug design to retrosynthesis planning for the discovered molecules. To evaluate the proposed framework, the paper proposes a dataset to benchmark in such task. From the empirical experiments, the paper argues that the proposed framw3ork Llamole outperforms the baselines of ICL or SFT with only LLMs, as well as GraphGA. The paper also justifies their deisgn of the framework based on ablation studies.", "questions": "What is the computational cost and inference time of the proposed methods, compared to GraphGA?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea proposed in the paper on jointly model texts and molecules with MLLM for inverse molecular deisgn and retrosynthesis planning pipeline is novel. The created dataset could also contribute to the area;\n2. From the experiments the proposed framework shows strong performance.\n3. The paper is clearly written for readers to understand the proposed method.", "weaknesses": "The baselines selected in the experiments are mainly LLMs with ICL or SFT. The only baseline with similar purpose is GraphGA. It would be interested to see the comaprisons of more baselines of the combination of existing inverse molecular design [1] and retrosynthesis planning [2,3] methods.\n\n1. Weiss, Tomer, et al. \"Guided diffusion for inverse molecular design.\" Nature Computational Science 3.10 (2023): 873-882.\n2. Zeng, Tao, et al. \"Developing BioNavi for Hybrid Retrosynthesis Planning.\" JACS Au 4.7 (2024): 2492-2502.\n3. Han, Peng, et al. \"Gnn-retro: Retrosynthetic planning with graph neural networks.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 36. No. 4. 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OFIIUaGs92", "reviewer": "Reviewer_F4DQ", "summary": "The paper introduces Llamole, a novel multimodal large language model (MLLM) designed for inverse molecular design. It is capable of autoregressive generation of interleaved text and graphs, which is a significant advancement in the field. Llamole integrates a base LLM with a Graph Diffusion Transformer and GNN model to enable molecular generation and molecular inference within textual contexts. Additionally, it incorporates an A* search algorithm with LLM-based cost functions for efficient retrosynthetic planning. The authors have further created benchmarking datasets, MolQA and MolPair, and conducted extensive experiments comparing Llamole against various state-of-the-art LLMs, demonstrating its superiority in controllable molecular design and retrosynthetic planning.", "questions": "I've expressed all my concerns in the weaknesses session.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents an autoregressive MLLM capable of understanding and generating interleaved text and graphs, which has the potential to be applied to a wide range of molecule-text downstream tasks.\n- The authors have collected and processed new datasets, MolQA and MolPair, which allow for more complex molecule-text downstream evaluations. These datasets will be valuable resources for future research in the field if they can be made open-source.\n- The experiments in controllable molecule generation are particularly compelling.\n- The integration of the A* search algorithm with LLM-based cost functions is a practical and effective approach to retrosynthetic planning", "weaknesses": "- The framework of this work (Figure 3) is somewhat difficult to understand due to its tight layout and restricted color use. - Rearranging the layout and using a more varied color scheme could improve the clarity of the framework.\n- The first claim about this paper's contribution is a bit vague. Although this model is indeed a novel MLLM (Multimodal LLM), there are already some existing MLLMs for molecular generation and understanding. The key innovation of this work lies in the autoregressive generation of interleaved text and graphs with GNN and Graph Diffusion Transformer. This should be emphasized in the introduction and preliminaries to distinguish it from existing work.\n- As mentioned above, there are existing MLLM methods tailored for molecule-text related tasks, some of which can be scaled up to 7B or higher parameter levels. These models have made a great contribution to molecule discovery and should be included in the introduction (Figure 1), preliminaries, and experiment sections of this paper.\n- This work conducts experiments majorly on the two new datasets proposed in this paper. While the comprehensiveness of the newly proposed datasets is commendable, additional comparisons on existing datasets such as MoleculeNet or USPTO retrosynthesis datasets would lead to better comparison with previous approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jOxEkk5vaH", "reviewer": "Reviewer_iNVA", "summary": "The paper introduces Llamole, a novel multimodal large language model (MLLM) designed for inverse molecular design that integrates graph and text generation capabilities for retrosynthetic planning. Llamole combines a base LLM with specialized graph modules (Graph Diffusion Transformer and Graph Neural Networks) to perform multi-conditional molecular generation and reaction inference within a unified framework. The model also incorporates A* search with LLM-based cost functions to enhance retrosynthetic planning efficiency. Extensive experiments demonstrate that Llamole outperforms other adapted LLMs on several benchmarks, establishing its effectiveness in controllable molecular design and retrosynthetic planning.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novel Multimodal Integration: The integration of graph-based models with a base LLM to handle both text and graph data is innovative, addressing the limitations of LLMs in generating coherent molecular structures and enabling interleaved text and graph generation.\n\n2. Efficient Retrosynthetic Planning: The use of A* search combined with LLM-derived cost functions significantly improves the efficiency of retrosynthetic planning, offering a practical solution for synthesizable molecular design.\n\n3. Good Performance: Llamole achieves substantial improvements over baseline methods across multiple metrics, demonstrating its superiority in both molecular design and planning tasks.", "weaknesses": "1. Theoretical Justification: The paper lacks a strong theoretical explanation for why the Llamole approach works effectively for inverse molecular design. While the empirical results are compelling, a deeper theoretical discussion would enhance the understanding of its effectiveness.\n\n2. Scalability Concerns: The computational cost and scalability of Llamole are not thoroughly discussed. The practicality of deploying such a large-scale multimodal model in real-world scenarios remains unclear, especially given the integration of multiple complex components.\n\n3. Comparative Analysis Gaps: The paper does not compare Llamole with some recent multimodal approaches that also attempt to bridge the gap between graph and text modalities, which would provide a more comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sNkP26mAyj", "reviewer": "Reviewer_GFHU", "summary": "Summary: In this paper, the authors address an important task of multi-modal molecule design using LLMs. In particular, they have integrated pre-trained Graph Models with pre-trained auto-regressive LLM and fine-tuned them using multi-modal data (Text+Graph) to enable interleaved generate Multi-modal output (text, molecules, and graph). Moreover, they have incorporated retrosynthetic planning during generation, where they integrate A* search to efficiently identify synthesis pathways for the designed molecule.", "questions": "Check the Weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well written. The problem statement is well-motivated. The authors did a comprehensive review of prior work on LLMs for Molecule Synthesis.\n- The authors proposed the first multi-modal LLM for molecular design. They proposed a novel idea to integrate Graph encoders with pre-trained LLMs and unified cross-entropy loss into the autoregressive modeling of LLMs. \n- They curated a dataset and developed fine-tuning guidelines for benchmarking multimodal molecular design.\n- Authors conduct extensive experiments with popular benchmark datasets and compare them with state-of-the-art models to show the effectiveness of their proposed model.\n- Authors conduct ablation studies to show the utility of different components of their model.\n- Source code is  provided", "weaknesses": "- Can the model handle any random format of the question, or is the format specified in Fig-2 predetermined? \n- What is the minimum information required by LLMole to generate an appropriate molecular structure?\n- To what extent does the information requested in the text match the generated material, and are there any qualitative results available for this?\n- For property prediction for Drugs and Materials, Can the model handle unseen datasets for property prediction in drugs and materials? Could the authors provide some insights on this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)