3M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecular Structure Generation
Authors: Huaisheng Zhu
Teng Xiao
Vasant G Honavar
Venue: COLM (2024)
Keywords: Multi-modal, Molecule Generation, Diffusion Model
TL;DR: Latent Multi-Modal Diffusion for Language-Guided Molecule Generation

Abstract: Generating molecular structures with desired properties is a critical task with broad applications in drug discovery and materials design. We propose 3M-Diffusion, a novel multi-modal molecular graph generation method, to generate diverse, ideally novel molecular structures with desired properties.  3M-Diffusion encodes molecular graphs into a graph latent space which it then aligns with the text space learned by encoder-based LLMs from textual descriptions. It then reconstructs the molecular structure and atomic attributes based on the given text descriptions using the molecule decoder. It then learns a probabilistic mapping from the text space to the latent molecular graph space using a diffusion model. The results of our extensive experiments on several datasets demonstrate that 3M-Diffusion can generate high-quality, novel and diverse molecular graphs that semantically match the textual description provided.

Published as a conference paper at COLM 2024
3M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecular Structure Generation
Huaisheng 222 Teng 112 & Vasant G Honavar
Artificial Intelligence Research Laboratory
College of Information Sciences and Technology
Center for Artificial Intelligence Foundations and Scientific Applications
#Institute for Computational and Data Sciences The Pennsylvania State University
University Park, PA, USA
{hvz5312, tengxiao, vuh1 eesus. edu
Abstract
Generating molecular structures with desired properties is critical task with broad applications in drug discovery and materials design. We propose 3M-Diffusion, novel multi-modal molecular graph generation method, to generate diverse, ideally novel molecular structures with de- sired properties. 3M-Diffusion encodes molecular graphs into graph latent space which it then aligns with the text space learned by encoder- based LLMs from textual descriptions. It then reconstructs the molecular structure and atomic attributes based on the given text descriptions us- ing the molecule decoder. It then learns probabilistic mapping from the text space to the latent molecular graph space using a diffusion model. The results of our extensive experiments on several datasets demonstrate that 3M-Diffusion can generate high-quality, novel and diverse molecular graphs that semantically match the textual description provided. The code is available github.
1 Introduction
Generating molecular structures with the desired properties is a critical task with broad applications in drug discovery and materials design (Hajduk Greer, 2007; Mandal al., 2009; Kyzer-Knapp et al., 2015). There is a growing interest in generative models to automate this task (You et al., 2018; Jin et al., 2018; 2020; Bjerrum & Threlfall, 2017). Kusner et al. (2017) proposed variational autoencoder that uses parse trees for probabilistic context free grammars encode discrete sequences that specify the Simplified Molecular-Input Line- entry System (SMILES) (Weininger, 1988) notation of molecules use the resulting parse trees to generate SMILES codes that describe molecular graphs. Edwards et al. (2021) formulated molecule retrieval from textual description cross-lingual retrieval task.
Recent advances in language models have led to innovative approaches to generate molecu- lar structures that match human-friendly textual descriptions of their properties, substruc- tures, and biochemical activity (Edwards et al., 2022; Zeng et al., 2022; Zhao et al., 2023b; Fang al., 2023a). These approaches learn mappings from textual descriptions to molecular structures using single transformer-based ation. Specifically, with the given text description of molecule, these approaches produce SMILES strings that encode the molecular structure.
Prngs-lnngual language model for molecule gener-
Despite promising initial results, such approaches suffer from some important limitations: (i) SMILES encoding is degenerate in that the a given molecular structure can have multiple SMILE encodings. For example, the structure of ethanol can be specified using CCO, OCC or C(O)C. Although in principle unique canonical SMILES representation can be produced
'Fqual contribution

---

Published as a conference paper at COLM 2024
for given molecular structure, the procedure used to do so relies several arbitrary choices. Not surprisingly, molecules with identical structure may map to different SMILES strings. This makes SMILES notation less than ideal for generating molecules from their textual descriptions (Jin et al., 2018). Important chemical properties corresponding to the substructures of molecules are more straightforward to express using molecular graphs rather than linear SMILES representations (jin et al., 2018; Du et al., 2022); (ii) Using language models to generate diverse and high-quality molecular structures matching description is challenging, because the process used to decode the learned representation into the corresponding molecular structure, e.g,, beam search or greedy search, often produces structures that are often too similar to each other (See Figure 2). Hence, there is an urgent need for better approaches for generating diverse, novel molecular structures from a given textual description (Brown et al., 2019). Against this background, this paper aims to answer the following research question: How can we generate a diverse and novel collection of molecular graphs that match given textual description?
We introduce 3M-Diffusion, novel multi-modal molecular dif- fusion method, for generating high-quality, diverse, novel molec- ular structures from a textual de- scription. Inspired by text-guided image generation using diffusion model on the latent space (Rom- bach et al., 2022), 3M-Diffusion is trained to generate molecular structures from the latent distribu- tion of molecular graph autoen- coder (See Figure 1). This enables the diffusion process to focus on the high-level semantics of encod- ings of molecular structures. We find that this approach is particularly well-suited for discrete molecular graph modality because it relegates the challenge of modeling dis- crete and diverse distribution to the autoencoder and simplifies the diffusion process by restricting it to the continuous, latent feature space. Since graph data consists of both node attributes and graph structure, which differs significantly from the sequential text data, to align the latent representations of textual descriptions and molecular graphs, 3M-difuusion employs contrastive learning on a large data set of pairs of molecular structures and their textual descriptions to pretrain molecular graph encoders and LLM text encoders. This enables 3M-difuusion to generate high-quality molecular graphs that match the given textual descriptions.
Tzaning Irdusunce
Molecute Graphç
Only Teateting
Graph Uecoder
Condition
Dittusiinn Model
Labent 2
LUM Encodex
Encodex
Reserseniation Aligmentt
f
NoLecule Graph
Figure 1: The overview of 3Mddiffusion, with molec- ular graph encoder decoder and latent diffusion model conditioned on a prior (an aligned LLM en- coder). The details of alignment in the text /itpggaphhho en- coders and diffusion model are given in Section 4.
pecoder/decoder
Key Contributions. The key contributions of this paper are as follows: (i) To the best of our knowledge, 3M-Difuusion offers the first multimodal diffusion approach to generating molecular structures from their textual descriptions, surpassing the limitations of state- of-the-art (SOTA) methods for this problem. (fi) 3M-Diffusion aligns the latent spaces of molecular graphs and textual descriptions to offer text-molecule aligned latent diffusion model to generate higher-qualityy diverse, and novel molecular structures that match the textual description provided; and (iii) Results of extensive experiments using four real-world text-based molecular graphs generation benchmarks show that 3M-Diffusion outperforms SOTA methods for both text-guided and unconditional molecular structure generation. In particular, 3M-Difuusion achieves 146.27% novelty, 130.04% diversity relative improvement over the SOTA method with maintained semantics in textual prompt on the PCDes dataset.
Related Work
2.1 Molecular Structure Generation
Existing molecular graph generation methods can be broadly grouped into two categories: text-based and graph-based models. Text-based models typically utilize the SMILES
2

---

Published as a conference paper at COLM 2024
Prompt:
MolTS-large
lege: 5.57
lopP:
logP:
logP: 5.87
logP: 1,32
3M 3M-Diffusion
loge: 8.58
logP:
logp:
logp.
lopP: 5.67
Prompt: The molecule is a member class of flavones.
MolTS-large
Reforonce
Sim: 0.74
Sim: 0.63
Sm: 0.47
Sm: 0.46
SM-Diffusion
Reforence
Figure Qualitative comparisons to the Mol15-large of generated molecules on ChEBl-20. Compared with the SOTA method MolT5-large, our results are more diverse and novel with maintained semantics in textual prompt. More results are provided in Appendix D.4
(Weininger, 1988) or SELFIES (Krenn et al., 2022) linear strings to describe each molecule (Bjerrum & Threlfall, 2017; Gómez-Bombardee et al., 2018; Kusner et al., 2017; Flam-Shepherd et al., 2022; Fang et al., 2023b; Grisoni, 2023). Because given molecular graph can have several linear string representations and small changes to the string representation can result in large changes in molecular graph being described, such linear string encodings are far from ideal for learning generative models for producing molecular graph from textual descriptions. Inspired by the success of learning enhanced graph representations using graph based models (Kipf & Welling, 2016; You et al., 2020; Xiao et al.; 2024), several authors have employed deep generative models, including graph variational autoencoders (VAEs) (You et al., 2018; Jin et al., 2018; Xiao et al., 2021; Jin et al., 2020; Kong et al., 2022; Liu et al., 2018; Diamant et al., 2023), normalizing flows (Madhawa et al., 2019; Zang & Wang, 2020; Luo et al., 2021), generative adversarial networks (Guarino
al., 2017; Maziarka et al., 2020), diffusion models (Niu et al., 2020; Vignac et al., 2022; Jo et al., 2022) and autoregressive models (You al., 2018; Xiao & Wang, 2021; Popova al., 2019; Goyal et al., 2020) to learn distributions of molecular graphs from large databases of known graphs. However, many of these studies focus on generating molecular graphs with specific low-level properties, such as logP (the octanol-water partition coefficient) as opposed to those that match high-level textual descriptions of broader range molecular properties (Edwards et al., 2022), e-g., water solubility, chemical activity, etc. However, practical applications, such as drug design, call for effective approaches to generating diverse and novel molecular graphs that match given textual description of such high-level properties.
2.2 Language-guided Molecule Generation
Advances in deep learning and language models have inspired a growing body of work on applications of such models to problems in the molecular sciences (Schwaller et al., 2019; Xiao et al., 2023; Toniato et al., 2021; Xiao et al., 2024; Zhao et al., 2023a; Seidl et al., 2023; Schwaller et al., 2021; Vaucher et al., 2020). Advances in text-guided generation of images, videos, and audios (Radford et al., 2021; Rombach et al., 2022; OpenAl, 2023; Yin et al., 2023), together with the flexibility offered by natural language descriptions of molecular graphs have inspired growing body of work on text-guided generation of molecular graphs (Christofidelliss et al., 2023; Edwards et al., 2022; Fang et al., 2023a; Zeng et al., 2022; Su et al., 2022). For example, MolT5 (Edwards et al., 2021; 2022) and
3

---

Published as a conference paper at COLM 2024
ChemT5 (Christofidellis et al., 2023) pre-train substantial volume of unlabeled language text and SMILES strings for tasks such as molecule captioning and text-based molecule generation (Christofidellis et al., 2023; Edwards et al., 2022). However, noted above, the degenerate nature of linear SMILES string representation of molecular graphs makes less ideal for generating molecules from their textual descriptions. To address this problem, we propose 3M-Diffusion, a novel method that aligns the latent representation of textual descriptions of molecules with that of the corresponding molecular structures, yielding powerful approach to text-guided molecular structure generation.
3 Preliminaries
3.1 Problem Definition
We consider generative modeling of 2D molecular graphs. Each molecule is represented as a graph g = (V,E), where = U111 is the set of [ nodes and & is the set of
edges. Let X = X1 X2, X E x be the node attribute matrix, where x, is the Dx-dimensional one-hot encoding feature vector of vi, such as atomic type and chirality type. Similarly, a tensor A E R | groups the one-hot encoding eij of each edge, where each entry is distinct edge type (bond types for molecule), with the absence of an edge encoded explicitly using a designated edge type. Given the dataset D = { Ti) } where molecule G has specific text descriptions T characterizing the features of the molecule, our goal is to learn conditional generation models Po T) for generating novel, diverse, and valid molecules while also aligning with the desired features outlined text descriptions T. For brevity, in what follows, omit the input subscript i when it is clear from context.
3.2 Diffusion Models
Diffusion models (DMs) (Ho et al., 2020; Song et al., 2020) are latent variable models that represent the data zq as Markov chains ..... .. 2o, where intermediate variables share the same dimension. DMs involve a forward diffusion process 4 (Z1.7 2n) = IT 9 (z, 2;-1) that systematically adds Gaussian noise to samples and a reverse process Pe (Zo:T)
(ZT) (ZT) Po (z--1 zz) that iteratively "denoises" samples from the Gaussian distribution to generate samples from the data distribution. A formal description of diffusion models
given in the Appendix A. To ensure training stability, we can use a simple regression objective with a denoising network 2p with input (x;
Lait
Exxo A1 /20
(Vaizo v1 vI-R(e,t) Z
(1)
where is the time step, E [0, 1] is the noise schedule and e (0, I) is Gaussian noise and A is time-dependent weighting term. Intuitively, the denoising network is trained to denoise a noisy state, z; /aizo + V1- aie, aiming reconstruct the clean data zo with Equation (1) that prioritizes specific times f. After training, we can draw samples with 2g by the iterative ancestral sampling (Ho et al., 2020):
[ir (1-aj1 20 (x1, + (a-1-i(1--(-11 I-
Zz
Zi o(t -,,te,
(2)
where Afj:-1 represents a and a(t 1,t) = (1- x111) (1 (1- 1//1-11 /(1 a1). The sampling chain is initialized from Gaussian prior ZT p (ZT) = N (Zr;
4 Latent Multi-Modal Diffusion for Molecules
3M-Diffusion (See Figure 1) aims to learn probabilistic mapping from the latent space of textual descriptions to a latent space of molecular graphs for text-guided generation of molecular graphs. However, direct mapping between these two latent representations using
4

---

Published as a conference paper at COLM 2024
diffusion model faces key hurdle, namely, large mismatch between the latent spaces of textual descriptions and of molecular graphs. To overcome this hurdle, 3M-Diffusion adopts two-step approach. The first step uses contrastive learning to produce text-molecule aligned variational autoencoder that align the representation of molecular graphs with that of their textual descriptions, and decoder that maps the latent representation back to the corresponding molecular graphs. The second step uses the text-aligned latent representation of molecular graphs to learn conditional generative model, that maps textual descriptions to latent representations of molecular graphs. As we will see later, 3M-Diffusion can produce diverse novel molecular graphs that match the given textual descriptions.
4.1 Text-Molecule Aligned Variational Autoencoder
The proposed text-molecule aligned variational autonecoder comprises of three parts: the molecular graph encoder Eg, LLM encoder E, and molecular graph decoder D. To bridge the representation gap between molecular graphs and their textual descriptions, the text and molecular graph encoders are trained on large-scale dataset of molecule-text pairs. These encoders use contrastive learning to construct well-aligned text-molecular structure encodings that align the latent representation of molecular structures by aligning it with with that of their textual descriptions. The details of this process are given below.
Molecular Graph Encoder aims to map each molecular graph 9 with the adjacency tensor A and node features matrix X into lower-dimensional latent space Z. This transformation maps the discrete space of graph structures and node attributes into a continuous latent space. Specifically, we use Graph Isomorphism Network (GIN) (Hu et al., 2019) as encoder network on the input adjacency tensor A and node features X of graph G to derive the mean and standard deviation of the variational marginals:
Mg. g = GIN(A,
(3)
where Fg and 0g represent the mean and standard deviations, respectively. Then, the latent graph representation z is sampled from Gaussian distribution as ~ 9 (z A, = N (ng, gg). This graph encoding process can be expressed as = (g) signifying the outcome of the encoding operation in the input context of G.
LLM Encoder maps the textual descriptions of molecular structures into a latent space that provides input conditional generative model of molecular graphs. This encoded information is instrumental in generating molecules that match given textual description. To inject potentially useful scientific knowledge from the literature into E, we initialize with encoder-based based LLM, i.e., Sci-BERT (Beltagy et al., 2019), which is pretrained on the text of scientific publications.
Aligning the representation of molecular graphs with that of their textual descriptions is crucial for establishing a shared representation for effective text-guided molecular graph generation. Graph data encodes both node attributes and graph structure which differs significantly from the sequential and contextual nature of text data. We aim to align the molecular representation from the molecule graph encoder Eg with text representation from the LLM encoder E, using contrastive learning. Consider a molecular graph & and its textual description T. To align these two encoders and ensure cohesive latent space, we employ the following contrastive learning loss over a batch B E DD of large-scale dataset containing molecule-text pairs to pretrain encoders:
exp (cos (z,, /T)
Leon
Σ
log
(4)
B
exp cos Zi, (,) /T)
(G.
B
where z; = Eg)),, C = (T) and c; = ET)T)) are representations of corresponding molecule graph and texts, respectively. cos(..) denotes cosine similarity and T is the temperature hyperparameter. The encoders are pretrained on molecule-text (300K) pairs from PubChem Liu et al. (2023). As we will see, this process yields well-aligned molecule- text space that can support the generation of diverse and novel molecular structures from their textual descriptions.
5

---

Published as a conference paper at COLM 2024
Molecular Graph Decoder learns to use latent representation to generate the correspond- ing molecular structure. In this work, we employ the HierVAE (Jin et al., 2020) decoder. Note however that our approach can be combined with other graph decoders (Li et al., 2018; Kong et al., 2022; Grover et al., 2019). Specifically, the molecular graph decoder generates a molecular structure using graph representation from the encoder. Given the molecular graph G, we minimize the negative Evidence Lower Bound (ELBO) (Jin et al., 2020) to train both the encoder and the decoder:
Lelbo Ey(e/Ax)p((( 9)] 4KI.[q(z ())l
(5)
Here 9 (z), in which D denotes the decoder parameterized by HierVAB (Jin et al., 2020).
(.)l(p))pp()) is the Kullback-Leii Leibler (KI.) divergence between 9(.) and p(.). We follow Jin et al. (2020) and set the weight of KL loss a 0.1. Here, p(z) = (z 0,1) the isotropic multivariate Gaussian prior, where is the identity matrix. We optimize this first expectation term concurrently using the reparameterization trick (Kingma & Welling, 2013).
4.2 Multi-modal Molecule Latent Diffusion
We next introduce latent diffusion for conditional molecular graph generation, method that leverages the aligned molecule-text latent space of encoder-decoder described above, enabling us to preserve the crucial molecular properties and aligning the textual description of G. Specifically, we learn a probabilistic mapping from the text to the text aligned molecular structure latent space, and thus generate molecular structures that match the conditional text inputs.
The denoising network 24 focuses on generating latent graph representation conditioned on the text representation c from the LLM encoder E, The objective to be optimized is:
Cafff = Ex
A |2 (Vaiz+ + Vi-N,e,t,c) -2
(6)
where A are time-dependent weights and we use MLPs for the network 2g. Specifically, we concatenate to graph latent representation z; for timestep as the input of 2g. The denoising network is trained to denoise a noisy latent, / /azz + vI a;6 to the clean z. c is the text representation from the fixed encoder E, pretrained by Equation (4).
Additionally, to improve sample quality we take advantage of classifier-free guidance (Ho & Salimans, 2022), to jointly train an unconditional network, 20( and a conditional network, 2n (z,, f, c). Conditioning information is randomly dropped with probability of = 0.1 during training. When conditioning information is dropped, we replace the embedded source text with its embedding.
4.3 Training and Inference
Training. We proceed to describe the training procedure for 3M-Diffusion. While the learning objectives for graph encoder-decoder and multi-modal latent diffusion have already been specified by Equations (5) and (6), it is unclear whether the two components should be trained sequentially. Previous research on latent diffusion models for image generation indicates that a two-stage training strategy often results in superior performance (Rombach et al., 2022). Hence, we adopt a similar two-stage strategy. The first stage consists of training an encoder-decoder pair using Equation (5), and the second stage consists of training multi-modal molecule latent diffusion with the fixed encoder-decoder from the first stage.
Inference. During inference, & Salimans, 2022):
weight w in computing the prediction follows (Ho
w2z (,) + (1- w)22 (z,,)),
(7)
where 2, represents the newly predicted results, incorporating both conditional and uncon- ditional information. Setting = 1.0 corresponds to the conditional diffusion model, while setting w > 1.0 enhances the impact of the conditioning information. A smaller value of w contributes to more diverse samples. The new sampling equation is:
Z
01-1
41
2, 2
/1 M-1)
1 -,
2t + o(t 1,t)e,
(8)
6

---

Published as a conference paper at COLM 2024
Table 1: Quantitative comparison of conditional generation on the PCDes and MoMu. 3M- Diffusion outperforms other SOTA methods in terms of novelty, diversity, and validity metrics by large margin, while maintaining a good similarity metric.
PCDes
MoMu
a Metrics
Similarity (8)
Novelty (%)
Diversity (%)
Validity (%)
Similarity (%)
Novelty (%)
Diversity (%)
Validity (%)
MalTS-small
MolTS-bose
71.71
25.85
10.30
81.92
19.76
97.78
29.98
68.84
Mol1S-lange
88.37
20.15
9:49
96.48
25.07
97.47
50.35
90140
Chem'T 15-small
N6.27
23.28
13.17
93.73
23.25
96.97
30.04
98.45
Chem 15 base
85.01
25.35
14.08
92.93
23,40
97.65
30.07
87.61
Mol-Instruction
79.19
3M-Diffusion
81.37
63.66
32:39
100.0
24.82
98.16
37.85
100.0
Table 2: Quantitative comparison of conditional generation on the ChEBI-20 and PubChem. 3M-Diffusion outperforms other SOTA methods in terms of novelty, diversity, and validity metrics by a large margin, while maintaining a good similarity metric.
ChEBI-20
PubChem
Methods
Similarity (%)
Novelty (5)
Diversity (%)
Validity (%)
Similarity (5)
Novelty (%)
Diversity (%)
Validity (%)
MollS-small
73.32
31.33
17:22
78.27
68.36
2163
9.32
78,86
MolTS base
80.75
32.83
17.66
84.63
23.85
21.86
9,89
79,88
MollI-large
91.57
21855
Chem' small
13.94
89.32
20,89
13.10
ChemiS TS base
95.48
15.12
13.91
97.15
89.42
22 40
13.98
92.43
Mol-Instruction
63.75
32.01
26.50
77.91
23.40
37
27.97
71,10
3M-Diffusion
87.09
55.36
34,03
100
87:05
64.41
33.H
100
where the sampling chain is initialized from Gaussian prior Zr ~ (2T) = A (zr;0,1), The detailed training and inference algorithms of 3M-diffusion are provided in Appendix
Experiments and Results
5.1 Experimental Setup
Datasets. We use PubChem (Liu et al., 2023), ChEBI-20 (Edwards et al., 2021), PCDes (Zeng et al., 2022) and Momu (Su et al., 2022) datasets. Following the evaluation methodology adopted with datasets (ZINC250K (Irwin et al., 2012) and QM9 (Blum Reymond, 2009; Rupp et al., 2012)) used in previous studies of molecular structure generation, we limit our training, validation, and test data molecules with fewer than 30 atoms. Statistics of datasets are shown in Table 5 and detailed descriptions are given in the Appendix C.1.
Evaluation. We evaluate the model's performance on two tasks: text-guided and uncondi- tional molecule generation. Following previous works on text-guided molecule generation, we compare our 3M-Diffusion the SOTA baselines (Edwards et al., 2022; Christofidellis et al., 2023; Fang et al., 2023a). Text-guided molecule generation measures the model's capacity to generate chemically valid, structurally diverse, novel molecules that match the textual description provided. As in previous work (Chen et al., 2021; Fu et al., 2021; Wang et al., 2022), we adopt the following metrics for this task: Similarity, Novelty, Diversity, Validity. We call a predicted molecular structure, specifically one that is chemically valid (g'), qualified p
with the ground truth graph if (g.gg)) > 0.5, where f measures the MACCS (Durant et al., 2002) cosine similarity between the two. If the f(g,g') similarity between g' and g is smaller than a threshold (0.8 in this paper), we call this predicted molecule novel. Specifically, (i) Similarity: the percentage of qualified molecules out of 20K molecules; (ii) Novelty: the percentage of novel molecules out of all qualified molecules; (iii) Diversity: the average pairwise distance f(. between all qualified molecules. (iv) Validity: the percentage of generated molecules that are chemically valid out of all molecules.
We report the results for unconditional molecule generation tasks using the GuacaMol benchmarks (Brown et al., 2019) to assess the model's ability to learn generative model of molecular structures that can be used to generate realistic, diverse molecules. Specifi- cally, Uniqueness gauges the ratio of unique molecules among the generated ones. Novelfy measures the models' capacity to generate molecules not present in the training set. KL Divergence assesses the proximity between the distributions of various physicochemical properties in the training set and the generated molecules. Fréchet ChemNet Distance (FCD) computes the proximity of two sets of molecules based their hidden representations in
7

---

Published as a conference paper at COLM 2024
Table 3: Quantitative comparison of unconditional generation. Results of Uniq, KL Div and FCD on ChEBI-20 and PubChem, which refer to Uniqueness, KL. Divergence and Fréchet ChemNet Distance, respectively. A higher number indicates a better generation quality.
ChEBI-20
PuhChem
Miethods
Uniq (5)
Novelty (%)
(%)
FCD(N)
Validity (%)
Uniq (%)
Novelty
FCD (S)
Validity
CharKNN
VAE
47.58
42.47
91.67
94.10
AAE
1.23
1,23
38.47
0.06
1.35
2.94
3.21
39.33
0.08
197
LatenIGAN
66.93
57.52
91.38
76.65
73.02
52,00
50.35
91.38
57.38
53.62
BeR
22,09
21.97
$0.99
0.26
22,66
82.35
82.34
45.53
0.31
87.73
HierVAE
82.17
72,83
93.30
64.32
100.0
75.33
72.44
89.05
50.04
100.0
IS-VAE
76.09
83.16
32.01
66.52
83.41
100.0
3M-Diffusion
81.20
92.67
ChemNet (Preuer et al., 2018). Each metric normalized to a range of 0 to 1, with higher value indicating better performance.
Baselines. For text guided molecule generation, we compare 3MDiffussin with the follow- ing baselines: MolT5 (Edwards et al., 2022), Chem (Christofidellis et al., 2023) and Mol- Instruction (Fang et al., 2023a). We consider seven representative baselines for unconditional molecule generation based on Variational Autoencoder, including CharRNN (Segler al., 2018), VAE (Kingma & Welling, 2013), AAE (Makhzani et al., 2015), LatentGAN (Prykhodko et al., 2019) BwR (Diamant al., 2023), HierVAE (Jin et al., 2020) and PS-VAE (Kong al., 2022). The details of the baselines are given in Appendix C.1.
Setup. We randomly initialize the model parameters and use the Adam optimizer to train the model. HierVAE is used our variational autoencoder for all of the following experiments. For baselines of unconditional generation that did not report results in these datasets, we reproduce the results using the official code made available by the authors. To ensure a fair comparison across all methods for unconditional generation, we select the best hyperparameter configuration solely based on the loss on the training set. We evaluate the performance of baselines for text-guided molecule generation using the pretrained parameters made available by the authors of the respective studies.
5.2 Text-guided Molecule Generation
103
40x faster
Time (ms)
Table 1 and 2 report Similarity, Novelty, Di- versity, and Validily for text-guided molecule generation four datasets. Each metric is computed as an average over 5 structures generated from given textual description. We see that 3--Dffussin is competitive with SOTA baselines with respect Similarity and outperforms them substantially with respect to Diversity and Novelty. Specifically, 3M- Diffusion achieves 146.27% (Novelty of PCDes), 130.04% (Diversity of PCDes) relative improvement over the second-best SOTA baseline while matching the textual prompt. We conclude that while the transformer based models that are pretrained on a large dataset excel in producing molecular structures that match the textual descriptions provided, 3M- Diffusion outperforms them with respect to the diversity and novelty of the molecular structures without sacrificing the match between the textual descriptions provided as input and the molecular structures generated as output. Also, Figure 3 shows, 3--Difussio enjoys a substantial advantage in terms of its speed over transformer-basee methods (See Appendix D.1 for details).
10a
10
Mol75 small
ChemTS-smal
Meff5 Mesf5-base base
ChemTS-har base
Mol large
3M- Dittusion
Figure 3: Inference time comparison for con- ditional molecule generation on ChEBI-20.
5.3 Unconditional Molecule Generation
Table 3 shows the results of unconditional molecule generation on PubChem and ChEBI-20. 3M-Diffusion consistently outperforms or matches the performance of SOTA baselines across all four metrics on both datasets, suggesting that it is able to generate realistic, diverse, and novel molecules without overfitting the training data. Furthermore, 3M- Diffusion consistently outperforms HierVAE when both methods use the same decoder.
8

---

Published as a conference paper at COLM 2024
Table 4: Results of ablation study with different model designs of 3M-diffusion for Pubchem.
Methods
Similarity (%)
Novelty (%)
Diversity (%)
Validity (%)
(i) Without tworstage training
77.54
67.05
31.05
100.0
(i1) Without representation elignment
86:11
62.81
32.78
100.0
Without two-stage training representation alignment
10.24
77.59
27.86
100.0
3M-Diffusion
100.0
This underscores the clear advantage of 3M-Diffusion in generating diverse and novel molecular graphs (See Figure 10 for 50 samples from the learned distribution).
5.4 Ablation Studies
We conduct ablation studies to assess the impact of different model designs, by joint training of the diffusion model and Variational Autoencoder ("without two-stage training") and training the model without aligning the representation space of graphs and text using fcon in Equation (4) ( ("without representation alignment"). Additionally, we perform experiments without both two-stage training and representation alignment ("without two- stage training & representation alignment") in Table 4. We find that training the aligned encoder-decoder alongside Multi-modal Molecule Latent Diffusion achieves comparable or superior performance relative to 3M-Diffusion in terms of diversity and novelty, but at the expense of reduced quality of match between the molecular structures produced and the textual descriptions provided. We find that training the model without the aligned encoder- decoder results substantial drop in performance relative to 3M--iffusion, underscoring the importance of aligning the representation of textual descriptions of molecules with that of molecular structures. Ablation of both two-stage training and representation alignment components leads to substantial drop in performance (Similarity of 10.24), rendering the resulting model unsuitable for real-world applications. The full 3M--iffusion model (last row) achieves the best performance with respect to all of the metrics. This confirms the importance of two-step training and representation alignment in training a model that can generate diverse, novel set of molecular structures from their textual descriptions.
5.5 Qualitative Analysis and Case Study
Property Conditional Molecule Generation. In traditional molecule generation models, the generation process is often conditioned on specific molecular properties such as logP (Kus- ner et al., 2017) and QED. logl' assesses a molecule's solubility, while QED evaluates its drug likeness (Bickerton et al., 2012). Previous methods have explored ways to optimize such models further in order to generate molecules with desired properties. In contrast, 3M-Diffusion utilizes textual prompts for conditional molecule generation (See Figure 2). In particular, we generated 10 samples for each prompt and subsequently select the top 5 samples based on the value of the desired property. For MolT5-large, we adjust the temper- ature of the transformer in an attempt to encourage diversity while ensuring that invalid samples are controlled during the sampling process. We applied the same methodology used in our previous experiments to maintain consistency across the comparative analysis, as shown in Figure 2. When compared to the MolT5-large model, we find that 3M-Diffusion excels in generating diverse range of molecules that also score better with respect to the desired properties, as evidenced by higher log?' values Figure 2. This underscores our 3M-Diffusion's ability to produce molecules that are not only novel and diverse but also match the textual description provided. This unique approach distinguishes 3M-Diffusion from prior methods for molecular generation, presenting fresh and promising perspective on molecular design. Additional experimental details and insights regarding prompts related to solubility and drug likeness are provided in Appendix D.2.
Qualitative Comparison. In Figure 2, we introduce more specific prompts related to molec- ular categories, such as "The molecule is a member of the class of flavones." " These prompts bias the model in favor of generating useful drug candidates. Specifically, we choose reference molecule from the training dataset that matches the given textual prompt. Subse- quently, generate ten additional molecules and select the top five molecules that exhibit the highest structural similarity (calculated by Rascal algorithm (Raymond et al., 2002)) to the reference molecule. These selected molecules are shown in Figure 2.Comparison of 3M-

---

Published as a conference paper at COLM 2024
The meteace Erasii
add,, eetadey Zem
fodiory peade and
no des fom lomal
dictal weste coraliat wou andaal alete eeland
prop
with out eireee
rolcci member clea Ldd hedrec grease
pode lios te collryy
thal loo cilad boe ti plyphend
can hychopyaiss.
Figure Molecules generated conditionally on input text by 3M-Diffusion on ChEBI-20.
Diffusion with Molge- shows that 3M-Diffusion generates more diverse molecules that exhibit higher degree of structural similarity with the reference molecule while producing novel molecular structures. Additional examples of prompts are provided in Appendix D.3.
Case Study. In Figure 4, we choose set of examples from the ChEBI-20 dataset to visually illustrate the structural similarity between the molecules generated by 3M-Diffusion and the ground truth structures corresponding to the given textual descriptions. Specifically, we show the largest substructure shared by the generated and ground truth molecules (using RDkit (Landrum, 2010)). 3M-Diffusion demonstrates the capability to generate molecules that include the relevant substructure, as illustrated in Figure 4. Simultaneously, it displays the flexibility to generate molecules with novel substructures, thereby striking balance between generating structures that are similar to those represented in the training data and suggesting novel structures that match the textual description provided. We include additional results in Appendix D.4.
6 Conclusion
Generating molecules with desired properties is a critical task with broad applications in drug discovery and materials design. Such applications call for methods that produce diverse, and ideally novel, molecules with the desired properties. We have introduced 3M-Diffusion, novel multi-modal molecular graph generation method, address this challenge. 3M-Difussion first encodes molecular graphs into graph latent space aligned with text descriptions. It then reconstructs the molecular graph and atomic attributes based the given text descriptions using the molecule decoder. 3M-Diffusion learns probabilistic mapping from the text space to the latent molecular graph space using the latent diffusion model. The results of extensive experiments show that 3M-Diffusion can generate high-quality, novel and diverse molecular structures that semantically match the textual description provided. Ablation studies underscore the critical role of aligning the latent representation of text with that of molecular graphs in 3M-Diffusion
Acknowledgments
We thank the anonymous reviewers and Prof. Dane Morgan of University of Wisconsin- Madison for suggestions that have helped us improve the paper. This work was supported in part by grants from the National Science Foundation (2020243, 2226025) to Vasant Honavar and the National Center for Advancing Translational Sciences, and the National Institutes of Health (UL1 TR002014).
Impact Statements
This paper advances machine learning approaches to text-guided generation of molecular structures, with potential application in drug design and material discovery. The result- ing molecules, if successfully synthesized and experimentally characterized in the lab, could provide new drugs for combating diseases and for new materials for wide ranging applications.
10

---

Published as a conference paper at COLM 2024
References
Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. aro preprint ar XrXi:11903.10677 2019.
G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2)990-9 2012.
Esben Jannik Bjerrum and Richard Threlfall. Molecular generation with recurrent neural networks (rnns). arXiv preprint #rXic:1705.04612, 2017.
Lorenz C Blum and Jean-Louis Reymond. 970 million druglike small molecules for virtual screening in the chemical universe database gdb-13. Journal of the American Chemical Society, (25):83333 (73328883328333 2009.
Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: bench- marking models for de novo molecular design. Journal of chemical information and modeling, 59(3);1096-1188, 2019.
Binghong Chen, Tianzhe Wang, Chengtao Li, Hanjun Dai, and Le Song. Molecule optimiza- tion by explainable evolution. In International conference on learning representation (ICLR), 2021.
Dimitrios Christofidellis, Giorgio Giannone, Jannis Born, Ole Winther, Teodoro Laino, and Matteo Manica. Unifying molecular and textual representations via multi-task language modelling, ar arviv preprint ar Arx:::2301.125: 2023.
Nathaniel Lee Diamant, Alex M Tseng, Kangway V Chuang, Tommaso Biancalani, and Gabriele Scalia. Improving graph generation by restricting graph bandwidth. In Interna- tional Conference on Machine Learning, pp. 7939-7959. PMLR, 2023.
Yuanqi Du, Tianfan Fu, Jimeng Sun, and Shengchao Liu. Molgensurvey: A systematic survey machine learning models for molecule design. arXiv preprint Xiv::22003 14500, 2022.
Joseph L Durant, Burton A Leland, Douglas R Henry, and James G Nourse. Reoptimization of mdl keys for use in drug discovery. Journal of chemical information and computer sciences, 12(6)11273 2002.
Carl Edwards, ChengXiang Zhai, and Heng Ji. Text2mol: Cross-modal molecule retrieval with natural language queries. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 595-607, 2021.
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Transla- tion between molecules and natural language. ari preprint arx::22204.188 2022.
Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions: large-scale biomolecular instruction dataset for large language models. arviv preprint arXiv:2306.088. 080188, 2023a.
Yin Fang, Ningyu Zhang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Molecular language model as multi-task generator. arviv preprint ar arXie:2301.11259 2023b.
Daniel Flam-Shepherd, Kevin Zhu, and Alán Aspuru-Guzik. Language models can learn complex molecular distributions. Nature Communications, 13(1):3293, 2022.
Tianfan Fu, Wenhao Gao, Cao Xiao, Jacob Yasonik, Connor W Coley, and Jimeng Sun.
Differentiable scaffolding tree for molecular optimization. aiv preprint arXiv:2109.10469 2021.
Rafael BómezeBom Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sänchez-Lengeling Dennis Sheberla, Jorge Aguilera-Iparraguiin Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzil Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268-277 2018.
11

---

Published as a conference paper at COLM 2024
Nikhil Goyal, Harsh Vardhan Jain, and Sayan Ranu. Graphgen: A scalable approach to domain-agnostic labeled graph generation. In Proceedings of The Web Conference 2020, pp- 1253-1263, 2020.
Francesca Grisoni. Chemical language models for de novo drug design: Challenges and opportunities. Current Opinion in Structural Biology, 79:102527, 2023.
Aditya Grover, Aaron Zweig, and Stefano Ermon. Graphite: Iterative generative modeling of graphs. In International conference on machine learning, pp. 2434-2444. PMLR, 2019.
Michael Guarino, Alexander Shah, and Pablo Rivas. Dipol-gan: Generating molecular graphs adversarially with relational differentiable pooling. Under review, 2017.
Philip] Hajduk and Jonathan Greer. A decade of fragment-based drug design: strategic advances and lessons learned. Nature reviews Drug discovery, 6(3):211-219, 2007.
Janna Hastings, Gareth Owen, Adriano Dekker, Marcus Ennis, Namrata Kale, Venkatesh Muthukrishnan, Steve Turner, Neil Swainston, Pedro Mendes, and Christoph Steinbeck. Chebi in 2016: Improved services and an expanding collection of metabolites. Nucleic acids research, 4((Di):D2144D121) 2016.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:220..12598, 2022.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.
Advances in neural information processing systems,, 33:6840-6851, 2020.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXio preprint arXiv:1905.1226: 2019.
John J Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan G Coleman. Zine: free tool to discover chemistry for biology. Journal of chemical information and modeling, 52(7):1757-1768, 2012.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2323-2332. PMLR, 2018.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International conference on machine learning, pp. 4839- 4848. PMLR, 2020.
Jaehyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs vía the system of stochastic differential equations. In International Conference on Machine Learning, pp. 10362- 10383. PMLR, 2022.
Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, et al. Pubchem substance and compound databases. Nucleic acids research, 4((D1):D120-DD1213, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint ar xrXi::1412.6980 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rXi6:1312..6114, 2013.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arv preprint ar xxxe:1669.00907,, 2016.
Xiangzhe Kong, Wenbing Huang, Zhixing Tan, and Yang Liu. Molecule generation by principal subgraph mining and assembling. Adoances in Neural Information Processing Systems, 35:2550-254 2022.
12

---

Published as a conference paper at COLM 2024
Mario Krenn, Qianxiang Ai, Senja Barthel, Nessa Carson, Angelo Frei, Nathan C Frey, Pascal Friederich, Théophile Gaudin, Alberto Alexander Gayle, Kevin Maik Jablonka, et al. Selfies and the future of molecular string representations. Patterns, 3(10), 2022.
Matt J Kusner, Brooks Paige, and José Miguel Hemándee-Lobato. Grammar variational autoencoder. In International conference machine learning, pp. 1945-1954. PMLR, 2017.
Greg Landrum. Rdkit: Open-source cheminformatics, 2010. URL http:
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs. ar preprint ar 1rXx:1803.003222 2018.
Qi Liu, Miltiadis Allamanis, Marc Brockschmidt, and Alexander Gaunt. Constrained graph variational autoencoders for molecule design. Advances in neural information processing systems, 31, 2018.
Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. ar preprint wrXim:2310.127333 2023.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Dan S Weld. S2orc: The semantic scholar open research corpus. arXiv preprint ar arXio:1911.02788 2019.
Youzhi Luo, Keqiang Yan, and Shuiwang Ji. Graphdf: discrete flow model for molecular graph generation. In International Conference on Machine Learning, pp. 7192-7203. PMLR, 2021.
Kaushalya Madhawa, Katushiko Ishiguro, Kosuke Nakago, and Motoki Abe. Graphnvp: An invertible flow model for generating molecular graphs. arXiv preprint #XXiv1905.11100) 2019.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, lan Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXio:1511.05644 2015.
Soma Mandal, Sanat K Mandal, et al. Rational drug design. European journal of pharmacology, pp. 90-100, 2009.
Lukasz Maziarka, Agnieszka Pocha, Jan Kaczmarczyk, Krzysztof Rataj, Tomasz Danel, and Michat Warchot. Mol-cyclegan: a generative model for molecular optimization. Journal of Cheminformatiic, 12(1):1-18, 2020.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Er- mon. Permutation invariant graph generation via score-based generative modeling. In International Conference on Artificial Intelligence and Statistics, pp. 4474-4484 PMLR, 2020.
R OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article 2:13, 2023.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladin- skiy, Mark Veselov, et al. Molecular sets (moses): benchmarking platform for molecular generation models. Frontiers in pharmacology, 11:565644, 2020.
Mariya Popova, Mykhailo Shvets, Junier Oliva, and Olexandr Isayev. Molecularrnn: Gener- ating realistic molecular graphs with optimized properties. ar preprint arXiv:1905.1337,, 2019.
Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Gunter Klambauer. Fréchet chemnet distance: a metric for generative models for molecules in drug discovery. Journal of chemical information and modeling, 58(9):1736-1741 2018.
13

---

Published as a conference paper at COLM 2024
Oleksii Prykhodko, Simon Viet Johansson, Panagiotis-Christe Kotsias, Josep Arús-Pous, Esben Jannik Bjerrum, Ola Engkvist, and Hongming Chen. A de novo molecular gen- eration method using latent vector based generative adversarial network. Journal of Cheminformatics, 11:1-13, 2019.
Edward O Pyzer-Knapp Changwon Suh, Rafael Gömez-Bombarelli, Jorge Aguilera- Iparraguirre, and Alán Aspuru-Guzik. What high-throughput virtual screening? a perspective from organic materials discovery. Annual Review of Materials Research, 45: 195-216, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, Pp. 1748-8763 PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 2020.
John W Raymond, Eleanor Gardiner, and Peter Willett. Rascal: Calculation of graph similarity using maximum common edge subgraphs. The Computer Journal, 45(6))6311644 2002.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Matthias Rupp, Alexandre Tkatchenko, Klaus-Robert Müller, and O Anatole Von Lilienfeld. Fast and accurate modeling of molecular atomization energies with machine learning. Physical review letters, 108(5)005830,, 2012.
Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher A Hunter, Costas Bekas, and Alpha A Lee. Molecular transformer: model for uncertainty-calibraeed chemical reaction prediction. ACS central science, 5(9):1572-158,, 2019.
Philippe Schwaller, Daniel Probst, Alain C Vaucher, Vishnu H Nair, David Kreutter, Teodoro Laino, and Jean-Louis Reymond. Mapping the space of chemical reactions using attention- based neural networks. Nature machine intelligence, 3(2):144-152, 2021.
Marwin HS Segler, Thierry Kogej, Christian Tyrchan, and Mark P Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS central science, 4(1):120-131, 2018
Philipp Seidl, Andreu Vall, Sepp Hochreiter, and Günter Klambauer. Enhancing activity prediction models in drug discovery with the ability to understand human language. arx preprint ar arXiv:2303.03366, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arxiv preprint ar arXiv=2010.02502, 2020.
Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ri-R Wen. A molecular multimodal foundation model associating molecule graphs with natural language. arXiv preprint ar arXiv:2209.054 2022.
Alessandra Toniato, Philippe Schwaller, Antonio Cardinale, Joppe Geluykens, and Teodoro Laino. Unassisted noise reduction of chemical reaction datasets. Nature Machine Intelli- gence, pp. 485-494, 2021.
Alain C Vaucher, Federico Zipoli, Joppe Geluykens, Vishnu H Nair, Philippe Schwaller, and Teodoro Laino. Automated extraction of chemical synthesis actions from experimental procedures. Nature communications, 11 (11:33601, 2020.
14

---

Published as a conference paper at COLM 2024
Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint, 2022.
Zichao Wang, Weili Nie, Zhuoran Qiao, Chaowei Xiao, Richard Baraniuk, and Anima Anand- kumar. Retrieval-based controllable molecule generation. arXiv preprint ar XrXto:2208.11126 2022.
David Weininger. Smiles, chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28 (1):31-36, 1988.
Teng Xiao and Donglin Wang- A general offline reinforcement learning framework for interactive recommendation. In Proceedings of the AAAI Conference on Arfificial Intelligence, volume 35, pp- 4512-4520, 2021.
Teng Xiao, Huaisheng Zhu, Zhiwei Zhang, Zhimeng Guo, Charu C Aggarwal, Suhang Wang, and Vasant G Honavar. Efficient contrastive learning for fast and accurate inference on graphs. In Forty-first International Conference on Machine Learning.
Teng Xiao, Zhengyu Chen, Donglin Wang, and Suhang Wang. Learning how to propagate messages in graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1894-1903, 2021.
Teng Xiao, Chao Cui, Huaisheng Zhu, and Vasant Honavar. Molbind: Multimodal alignment of language, molecules, and proteins. arXiv preprint, 2023.
Teng Xiao, Huaisheng Zhu, Zhengyu Chen, and Suhang Wang. Simple and asymmet- ric graph contrastive learning without augmentations. Advances in Neural Information Processing Systems, 36, 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint ar arXiv:2306.13549 2023.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolu- tional policy network for goal-directed molecular graph generation. Advances in neural information processing systems, 31, 2018.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812-5823,, 2020.
Chengxi Zang and Fei Wang. Moflow: an invertible flow model for generating molecuclar graphs. In Proceedings of the 26th ACM SIGKDD international conference knowledge discovery & data mining, pp. 617-626, 2020.
Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. A deep-learning system bridg- ing molecule structure and biomedical text with comprehension comparable to human professionals. Nature communications, 13(1):862, 2022.
Haiteng Zhao, Shengchao Liu, Chang Ma, Hannan Xu, Jie Fu, Zhi-Hong Deng, Lingpeng Kong, and Qi Liu. Gimlet: unified graph-text model for instruction- molecule zero-shot learning. bioRxiv, pp. 2023-05, 2023a.
Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang, Pei Cheng, Bin Fu, Tao Chen, Gang Yu, and Shenghua Gao. Michelangelo: Conditional 3d shape generation based on shape-image-tex aligned latent representation. arXiv preprint ar111:2306.17115, 2023b.
15

---

Published as a conference paper at COLM 2024
A Diffusion Models
We present a formal description of diffusion (Ho et al., 2020; Song et 2020). Diffusion mod- els are latent variable models that represent the data Za Markov chains ZT Zo, where intermediate variables share the same dimension. DMs involve forward diffusion process q (Z1.7 zo) IT 1 9 (z: Z1-1) and reverse process pa (Zo.T) = p (ZT) II pa (Zi-1 21). The forward process starting from Zo can be written as:
g (z xo) == (Z xo) dz 1) = N (zu / (1 (1-21)1 I),
(9)
where the hyperparameter &1:T determines the magnitude of noise added at each timestep t. The values of #1:7 are selected to ensure that samples z7 converge to standard Gaussians, ie,, 9 (ZT) ~ (0, I). This forward process q is usually predefined without trainable parameters.
The generation process of DMs is defined as learning a parameterized reverse denoising process, which incrementally denoises the noisy variables zy:1 to approximate clean data x in the target data distribution:
Po 2 11)) = - 2 1:40 ( (,--1,,)))((111111))
(10)
where the initial distribution p(zr) is defined as N I). The means He typically are neural networks such as U-Nets for images or Transformers for text, and the variances o² - 1, typically are also predefined. As latent variable models, the forward process 4 zo)
can be seen as a fixed posterior. The reverse process (o(:)))) is trained to maximize the variational lower bound of the likelihood of the data:
log 9 (ZT
Zo) Pe (ZT)
q (Z)-1
Z 2t)
log pe (zo 21)
Lvbb = Eg Eggzerlza 20)
+
log
(11)
Pe
(Z:-1 2t)
To ensure training stability, a simple surrogate objective up to irrelevant constant terms is introduced (Song et al., 2020; Ho et al., 2020; Nichol & Dhariwal, 2021):
Laifi = Eizo Ae 1/1 (Vazzo + v1 (1-a,e,t) z
(12)
where z; is used to denote / aazzo + v aie for simplicity. Intuitively, we train a neural network én (z) to approximate the original data given some noisy latent and the timestep through Equation (12), 2o (2,, f) z z. With trained denoising network, we define the mean and variance function in Equation (10) following the previous work (Ho et al., 2020) as:
No (zi, 1- 1,1) Vari ( - ai|(-1) (1-djl-1))
/allt (1 a(-1)
20 (Zi,
Zi,
1-a; a:
8:
(13)
o2t (,t) (1 - 01 14-1) (1- -1/l//1-1)
where A is the time-dependent weight and atj[-1 21/4411- For generation, we employ the standard DDPM sampler, also referred to as the ancestral sampler (Ho et al., 2020). The process involves sampling initial noise Zz (0 and iteratively applying the update rule:
Va: Va:-1 a(|1-1)
Aappt11 (1 &t
2,
(0(z,, t)+ +
2, + - 1,t)e,
(14)
1- ai
where (0, We use T 50 for the sampling timesteps.
16

---

Published as a conference paper at COLM 2024
Algorithm 1 Training Algorithm of 3M-Diffusion
1: Input: Pairs of Molecule Graph data G = (V,E) and Textual Description T
2: Initial: Graph Encoder network Eg. LLM Encoder network E, and Decoder network D, denoising network 2,
3: Pretraining Stage: Aligned Text and Graph Encoder Training
4: while Eg and E, have not converged do
(x)(c 5: Lcon [8] L(G.TT) log -
6: Eg F. +- optimizer(Loom
7: end while
8: First Stage: Autoencoder Training
9: Initialize Eg from Pretraining Stage
10: while Eg and D have not converged do
11:
z = (g)
12:
D(z)
Decoding
13: Lelbo - [pG 9)] + <KL(q(z ()]lp(z]]
Eg, < Aptimizer(Lettnn
15: end while
Multi-modal Molecule Latent Diffusion Training 17: Fix Graph Encoder Eg from Stage 1 and LLM Encoder E, from Pretraining Stage while Zą have not converged do
Second Stage:
19:
2 < (g)
20:
f ~~ U),,T), - I(,,
Laitf = EiA A |20 (Varzo + VI- ate,t) zo 2012
21:
22:
0 + *oopimmierr(Ai (fair)
end while
24: return Eg, D
Algorithm Sampling Algorithm of 3M-Diffusion
1: Input: decoder network D, denoising network 2.
2: zz NO, I)
3: for f in T, T 1,
do
4: e~ N0 I)
Latent Denoising Loop
5: 2: w2g (Zz, 1, )) + (1 w 20)2g((
vay (1 aq 2, + /ajee (1 -l---Ba- 11-1)
6:
1
2, + +o(tt 1,
7: end for
8: G ~ ~p(g]z))
Decoding
9: return h
B Training and Sampling Algorithm
The complete training algorithm is depicted in Algorithm Firstly, the process begins by training the graph and LLM encoders using contrastive learning. Following that, the Variational Autoencoder trained to facilitate the reconstruction of molecule graphs. Lastly, the Multi-modal Molecule Latent Diffusion Model is employed to train the denoising network for the sampling process. This structured approach forms the core of our training procedure. Moreover, we present the sampling method of 3M-Diffusion in Algorithm 2.
17

---

Published as a conference paper at COLM 2024
Table 5: Statistics of all datasets.
Dataset
#Training
#Validation
#Test
ChEBI-20
15,409
1,971
1,965
PubChem
6,912
571
1,162
PCDes
7,474
1,051
2,136
MoMu
7,474
1,051
4,554
C Experimental Details
C.1 Datasets Details and Statistics
The statistics of datasets, including their the number of samples for training, validation and test sets are given Table
ChEBI-20 (Edwards et al., 2021): This dataset is generated by leveraging PubChem (Kim et al., 2016) and Chemical Entities of Biological Interest (ChEBI) (Hastings et al., 2016). compiles ChEBI annotations of compounds extracted from PubChem, comprising molecule- description pairs, To ensure less noise and more informative molecule descriptions, includes samples with descriptions exceeding 20 words.
PubChem (Liu et al., 2023): This dataset comprises 324k molecule-text pairs gathered from the PubChem website. As the dataset contains numerous uninformative texts like "The molecule is a peptide," it curates a high-quality subset of 15k pairs with text longer than 19 words for downstream tasks. This refined subset is then randomly split into training, validation, and test sets. The remaining dataset, which is more noisy, is utilized for pretraining. To prevent data leakage, we exclude samples that appear in the testing set of other datasets from the pretraining dataset.
PCDes (Zeng al., 2022): This dataset compiles molecule-text pairs sourced from Pub- Chem (Kim et al., 2016), encompassing names, SMILES notations, and accompanying paragraphs providing property descriptions for each molecule.
MoMu (Su et al., 2022): This dataset comprises paired molecule graph-text data, with the textual information for each molecule extracted from the SCI paper dataset (Lo et al., 2019).
For all datasets, inspired by previous datasets (ZINC250K (Irwin et al., 2012) and QM9 (Blum Reymond, 2009; Rupp et al., 2012)) on molecule generation, we preserve molecules with fewer than 30 atoms, creating new training, validation, and test sets from their original counterparts.
C.2 Baselines
We firstly introduce the baselines used for text-guided molecule generation in this section.
MolT5 (Edwards et al., 2022): MolT5 addresses the difficult problem of cross-domain generation by linking natural language and chemistry, tackling tasks such as text-conditional molecule generation and molecule captioning. It presents several 15-based LMs (Raffel et al., 2020) by training on the SMILBS-to-text and text-to-SMILES translations.
Chem T5 (Christofidellis et al., 2023): Chem'T5 introduces a versatile multi-task model for seamless translation between textual and chemical domains. This innovative approach
leverages transfer learning in the chemical domain, with particular emphasis on addressing cross-domain tasks that involve both chemistry and natural language.
Then, will also include seven baselines for unconditional generation to verify the effec- tiveness of our model.
CharRNN (Segler et al., 2018): CharRNN models distribution over the next token based on the previously generated tokens. We train this model by maximizing the log-likelihood of the training data, which is represented as SMILES strings.
18

---

Published as a conference paper at COLM 2024
VAE (Kingma & Welling, 2013): Variational Autoencoder (VAE) consists of two neural networks: an encoder and a decoder. The encoder infers a mapping from high-dimensional data to a lower-dimensional space, while the decoder maps this lower-dimensional repre- sentation back to the original high-dimensional space. It optimizes reconstruction loss and regularization term in form of Kullback-Leebler divergence. We SMILES input and output representations as discussed in (Polykovskiy et al., 2020).
AAE (Makhzani et al., 2015): Adversarial Variational Autoencoder (AAB) replaces the Kullback-Leibler divergence in VAE with an adversarial objective. An auxiliary discrimi- nator network is trained distinguish between samples from a prior distribution and the model's 's latent codes. Similar to VAE, we use SMILES as input and output representations as shown in (Polykovskiy et al., 2020).
LatentGAN (Prykhodko et al., 2019): The Latent Vector Based Generative Adversarial Network (LateniGAN) integrates an autoencoder with generative adversarial network. It first pretrains an autoencoder to map SMILES structures to latent vectors. Subsequently, a generative adversarial network trained to generate latent vectors for the pre-trained decoder.
BwR (Diamant et al., 2023): BwR aims decrease the time/space complexity and output space of graph generative models, demonstrating notable performance improvements in generating high-quality outputs on biological and chemical datasets.
HierVAE (Jin et al., 2020): HierVAB employs hierarchical encoder-decoder architecture to
generate molecular graphs, leveraging structural motifs as fundamental building blocks.
PS-VAE (Kong et al., 2022): PS-VAE automatically identifies regularities within molecules, extracting them as principal subgraphs. Utilizing these extracted principal subgraphs, proceeds to generate molecules in two distinct phases.
Note that we use the code for CharRNN, VAE, AAE, LatentGAN by the Molecular Sets (MOSES) Benchmark Polykovskiy et al. (2020).
C.3 Setup and Hyper-parameter settings
We use official implementation publicly released by the authors of the baselines or imple- mentation from Pytorch. Note that we use the decoder of HierVAE for our model. We run experiments on machine with a NVIDIA A100 GPU with 80GB of GPU memory. In all experiments, we use the Adam optimizer (Kingma & Ba, 2014). In particular, for our 3M-diffusion. we set the dimension of latent representation as 24, the number of layers for the denosing networks (MLP) as 4, a as 0.1, T as 100 for training and 50 for sampling process and the probability generation, we set 7 as 10 or for the sampling process.
0.8 for text-guided molecule generation. For unconditional
D More Experimental Results
D.1 Average Inference Time Comparison
In this section, we compare the average inference times of 3M-Diffusion, MolT5, and T5 by generating 1,000 samples, as shown in Figure 3. The average time to generate each molecule calculated and presented. We observe that 3M-Diffusion achieves speed 45 times faster than baseline methods, confirming the efficiency of our proposed model. Our proposed model not only generates molecules more quickly than previous approaches but also maintains a high level of semantic alignment with textual descriptions.
D.2 Property Conditional Molecule Generation
In this section, we provide more prompt examples and more experiment details for Property Conditional Molecule Generation. We generate 10 samples for each prompt and subse- quently select the top 5 samples based on the desired property values, as shown in Figure 2, 5, 6 and 7. For instance, when using the prompt "This molecule is not like drug," we
19

---

Published as a conference paper at COLM 2024
Prompt: This molecule soluble water.
M
Mó
Nil
MolTS-large
loge: -2.62
loge: -2.62
-2.62
logP: -2.62
loge: -1.06
logP: -1.06
HO
3M-Diffusion
loge: -8.87
logP: -5.76
logp: -4,8
logF: -4.76
logP: -3.97
Figure 5: Qualitative comparisons to the Moll5-large in terms of generated molecules CheBI-20. Compared with the SOTA method MolT5-large, our generated results are more diverse and novel with maintained semantics in textual prompt.
select the top 5 molecules with the smallest QED values, as indicated. Conversely, for the prompt "This molecule is like drug," we opt for the top 5 molecules with the largest QED values. This same approach is applied to solubility, where we select molecules based logP values. Specifically, model to promote the generation of diverse molecules. We carefully selected an appropriate temperature threshold that struck balance between diversity and maintaining the valid- ity of the generated molecules. Moreover, when dealing with MolT5, it is worth noting that it may occasionally generate multiple unconnected molecules within single SMILES representation. In such cases, we select the molecule among these unconnected ones that best aligns with the top desired attribute. From all figures, it becomes evident that our model excels in generating molecules with improved desired properties when compared to MolT5-large.
adjust temperature setting within the MolT5 transformer
Prompt: This molecule is like drug.
MolTS-large
QED: 0.79
QED: 0.79
QED: 0.77
CED: 0.77
QED: 0.63
SM-Diffusion
QFI:: 0.83
QED:
QED: 0.76
Figure 6: Qualitative comparisons to the MolT5-large in terms of generated molecules on CheBI-20. Compared with the SOTA method MolT5-large, our generated results are more diverse and novel with maintained semantics in textual prompt.
D.3 Qualitative Comparison
In this section, we introduce additional prompt example: "The molecule is an antho- cyanidin cation." " This prompt is used to generate molecules belonging to specific categories. We have the same observation that our model excels in generating molecules that not only closely resemble the reference molecule but also encompass some diverse and novel structures.
20

---

Published as a conference paper at COLM 2024
Prompt: This molecule not like drug.
MolTS-large
QED: 0.77
QED: 0.77
QED: 0.89
QED: 0.89
QEC: 0.88
QED: 0.05
QED: 0.05
QEC:
QED: 0.11
QED: 0.13
Figure 7: Qualitative comparisons to the Moll5-large in terms of generated molecules CheBI-20. Compared with the SOTA method Moll5-large, our generated results are more diverse and novel with maintained semantics in textual prompt.
Prompt: The molscule is an anthocyanidin cation.
MofTS-large
Reference
Sim: 0.74
Sim: 0.74
Sim: 0.74
Sim: 0.73
3M-Diffusion
Figure 8: Qualitative comparisons to the MolT5-large in terms of generated molecules on CheBI-20. Compared with the SOTA method MolT5-large, our generated results are diverse and novel with maintained semantics in textual prompt.
D.4 Case Study
In this section, we choose 20 more examples from the ChEBI-20 dataset to visually illus- trate the structural similarity between the molecules generated by 3M-Diffusion and the ground truth molecules corresponding to the given text descriptions. All results are shown in Figure 4. We also observe that our model is proficient at generating molecules that share significant common structural elements, while simultaneously obtaining diverse and innovative molecular structures.
D.5 Unconditional Molecule Generation
As shown in Figure 10, our sampled molecules present rich variety and structural complexity. This demonstrates our model's ability to generate diverse, novel, and realistic molecules.
21

---

Published as a conference paper at COLM 2024
The relacdeli t/acdallahpcl yrua tackay cropee
call evlid vaie recundee chaiiareel member of ploceeitullten
Pesiollo specis, ana
unibarart
celooaaoi alnenta slartwa witk cocarellle
wesstteeeiie Aannii of
The peppode compored of retaereee Lideeene ahe maberle cerve fome
eesanatun Juan st rothhe
The relocaeli alo msarting fon mualy aodura detusaieleeeieeeeeeee tydrovylanne.
and en tockl airisn ofritiuuiee dentar
indolmannd
te snio lem renituuee te sevesver mele sote mersee to varkkiio oviletion. skibt
malecuee shediay drsece
rocee mesulleneooe Reeosuutteenn rndoooornnee lesi aucoaetiio anaik deeves vesitee
The reaccal suualllii aisal yehoo Mociae anai unk thase seliccaks
The rolecde ls acomatiz Lild Lethheenthe 3-0 eiieeyyaes apecrssapph thes ate Crnrbbyyppp..
melecae bovyla oesigne medibvtti. di 2-tndo dy lecio add.
group
staruc troipheryttuaii subal una cionns paiiure and
bar ca plrt resalael and de skilei doc sypice wpher/ettylee
Figure 9: More molecules generated conditionally on input text by 3M-Diffusion on the ChEBI-20 dataset.
22

---

Published as a conference paper at COLM 2024
too-do'
RR
dago
oirdra
238
LOT
xon
Imand
bugte
X0
900
Figure 10: 50 molecules sampled from the prior distribution (0, I).
23


Reviews:
[{"review_id": "EE2N47nNeN", "reviewer": "Reviewer_8fJY", "summary": "This paper proposes a language model on molecule generation, well combining the existing practices in multi-modal generation: (1) train a VAE model for molecule graph generation and text descriptor generation; (2) align the representation of molecule graph and text descriptors; (3) introduce a latent diffusion model for mapping the text spaces to the latent molecular graph spaces, which then can be decoded to a molecular graph.\nExperiment results show the proposed method is effective, outperforming SOTA methods for both text-guided and unconditional molecule generation.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's perception of our contributions to both empirical analysis and motivations, and are grateful for the encouraging comments. Please see our responses below:\n\n**Q1. It is more natural for me to understand the necessity of molecule-to-text generation instead of text-guided molecular generation.**\n\n**A1.** Thank you for your comments. Traditional Molecule generation based on value guidance specifying a single or a handful of target properties might be insufficient to capture intricate conditions. Using natural language can greatly utilize biomedical knowledge from humans for molecule generation [1], [2]. Specifically, text-guided molecule generation allows us to encompass conditions adeptly and flexibly, like substructure (benzene ring) or the novel properties like categories of the molecule (“flavones” as shown in Figure 2).\n\n\n**Q2. The technical novelty is limited, but it is acceptable.**\n\n**A2.** Thank you for your feedback on the technical novelty of our work! We believe that our approach introduces several innovative aspects. Specifically, our paper highlights:\n\n- **novel problem**: we explore the previously unaddressed problem of text-guided molecular graph generation.\n\n- **novel perspective**: we present an approach from a novel perspective based on diffusion models for the text-guided molecular graph generation.\n\n- **novel results**: our 3M-Diffusion model successfully generates diverse and high-quality molecules that accurately match textual descriptions, which outperform other baselines for both text-guided and unconditional molecule generation.\n\n\n**Q3. How about using the 3D geometry for molecular modeling and introducing the protein or pocket information as a prompt for drug discovery?**\n\n**A3.** Thanks for your questions! We agree with your opinions. However, our current paper is focused on text-guided molecule generation and this concept falls outside our scope. We acknowledge the potential of this approach and suggest exploring these promising settings in future work.\n\n[1] Edwards, Carl, et al. \"Translation between Molecules and Natural Language.\" EMNLP-2022.\n\n[2] Christofidellis, Dimitrios, et al. \"Unifying molecular and textual representations via multi-task language modeling.\" ICML-2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Effectiveness: The proposed method is effective and shows strong performances in both text-guided and unconditional molecule generation.\n2. Well-motivated: Introducing molecular graph modeling in text-guided molecule generation is well-motivated.\n3. Well writing.", "reasons_to_reject": "1. Aligning the concept of molecular to text generation is useful, which will help us to understand the properties of unknown or new molecules. In such a case, it is more natural for me to understand the necessity of molecule-to-text generation instead of text-guided molecular generation.\n2. The technical novelty is limited, but it is acceptable.", "questions_to_authors": "1. How about using the 3D geometry for molecular modeling and introducing the protein or pocket information as a prompt for drug discovery?", "extra_scores": {}}, {"review_id": "vhzKdYS0uN", "reviewer": "Reviewer_H3jC", "summary": "This paper studies language-guided molecule generation given text description. The proposed method consists of multiple neural networks, including a graph encoder, decoder, text encoder, and a diffusion model in the latent space. To align the representations of text and graphs in the latent space, the authors utilized contrastive loss. Experiments on conditional and unconditional molecule generation were conducted to demonstrate the effectiveness of the proposed framework.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully appreciate your time in reviewing our paper. \n\n**Q1. Overclaim and SOTA.**\n\n**A1.** Thanks for your comments. We add Mol-instruct (ICLR 2024) [1] as a baseline. The results show that 3M-Diffusion outperforms Mol-instruct (Due to word limits, we just put results on PCDs, all other datasets have similar conclusions).\n\n|          |   | Similarity (%) | Novelty (%) | Diversity (%) | Validity (%) | \n|-----------|---|------|--------|--------|---------|\n| Mol-instruct | | 60.86 | 35.60 | 24.57 | 79.19 | \n| 3M-Diffusion  | | **81.57** | **63.66** | **32.39** | **100.0** | \n\nFor the unconditional generation results, we put them in Table 5 of the Appendix. \n\n**Q2. No technical contribution.**\n\n**A2.** We respectfully disagree that our model isn't novel or has no technical contributions because it is not surprising, and uses components in previous works. We believe that discouraging authors from building on others' work due to fear of reviewer criticism hinders community progress. We argue that we explore text-guided molecule generation from a novel perspective that involves generating molecular graphs based on texts with diffusion models. \n\nWhile the diffusion model has shown promise in various fields, it has not yet been applied to the novel problem of text-guided molecular graphs. We stress the fact that our method can be used to discover novel, diverse and high-quality molecules.\n\n**Q3. Trade-off rather than improvement.**\n\n**A3.** We respectfully disagree that our model's performance is merely a trade-off rather than an improvement. Trade-offs are inherent in molecule generation. Generating molecules that match text is important, but creating diverse and novel molecules is also crucial in drug discovery. A model that only reconstructs known molecules but cannot discover new ones fails to meet drug discovery needs. Therefore, the target of text-guided molecule generation maintains an acceptable level of similarity while ensuring diversity and novelty. Overall, our average results on all metrics outperform other methods as shown below (just put PCDes for world limit and all datasets have similar conclusion) :\n\n|         | Avg |\n|---------------|--------|\n| MolT5-small   | 50.06    |\n| MolT5-base    | 53.96    |\n| MolT5-large   | 54.77    |\n| ChemT5-small  | 55.1    |\n| ChemT5-base   | 55.42    |\n| 3M-Diffusion   | **69.12**    |\n\n[1]Mol-instructions: A large-scale biomolecular instruction dataset for large language models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **Less explored yet important topic.** The molecule generation using language descriptions is relatively less explored topic and it has many practical applications such as drug design and material discovery. In addition, the graph generation is relatively challenging since graphs are irregular without clear directions and they have a various number of nodes and edges.\n2. **Ablation study.** The authors provide the ablation study to show the contribution of each component, including two-stage training and representation alignment.", "reasons_to_reject": "1. **Overclaim and SOTA.** The key contribution, (iii) the proposed method outperforms SOTA methods, is questionable. First of all, the baselines did not include recent methods, and unconditional generation results are not provided in the table.\n2. **No technical contribution.** The proposed pipeline is a combination of existing modules. HierVAE, Latent diffusion, GIN for graph encoder and contrastive loss for text and graph latent space alignment. Also, no significant modification/adaption of existing methods for text-guided molecule generation.\n3. **Trade-off rather than improvement.** Table 1 and Table 2 show that the proposed method shows a noticeable degradation in similarity and, instead, performance gains in other metrics. This could be viewed as trade-off between metrics. In addition, Novelty, Diversity and Validity are not related to language-guidance. They are not specifically designed to evaluate the fidelity between conditions and generated samples.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3wmlsjKY9N", "reviewer": "Reviewer_c27h", "summary": "The paper introduces a diffusion model that generates molecular graphs using textual descriptions of molecules as conditions. This framework first aligns the representations of molecular graphs and text descriptions. It then proceeds to train in two phases: an encoder-decoder and a multi-modal latent diffusion model.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments.\n\n**Q1. Figure 1 is confusing, the legends in Table 1 are incorrect and Table 1 and 2 could be merged.**\n\n**A1.** Thanks for your comments! We will update the Figure and Tables following your suggestion.\n\n\n**Q2. How to obtain the ground truth graph of a generated molecule if two texts are very similar.** \n\n**A2.** We believe there may be some misunderstandings. Each sample in the dataset consists of a text paired with a **molecular ground truth graph (included in the dataset)**. \n\n**Q3. Retrieval tasks should be included.**\n\n**A3.** We present the molecule-to-text retrieval results on PubChem, comparing them with Sci-BERT:\n\n\n|         | Acc  | R@20  | \n|------------|------|-------|\n| Sci-BERT   | 39.7 | 85.8  | \n| 3M-Diffusion   | **61.7** | **90.8**  | \n\nWe measure performance with accuracy and recall@20, where our model outperforms Sci-BERT.\n\n**Q4. Does the case study from the training set?**\n\n**A4.** No, we select the samples from the test set.\n\n**Q5. It is inappropriate to express improvements as the percentage increases.** \n\n**A5.** Thank you for your comments. We will use absolute relative improvement.\n\n**Q6. Plot the distribution of logP and QED.**\n\n**A6.** Thanks for your comments! We have plotted the logP and QED distribution on the ChEBI-20 dataset for ground truth data, MolT5-large and 3M-Diffusion in the [link](https://anonymous.4open.science/r/3M-Diffusion-2217/distribution.pdf). Our model shows smaller or similar distribution differences with the ground truth compared to MolT5-large, highlighting its effectiveness.\n\n**Q7. Why is there such a big difference between the prompts in Figure 2 and 3?** \n\n**A7.** Thanks for your questions! The text in Figure 3 is derived from the original dataset, and the property-related words used in Figure 2 is designed by ourselves. We try to explore our model's ability for text-guided molecule generation of different cases with different prompts. \n\n**Q8. The paper's statement requires supporting evidence. If the method generate consistent SMILES, would this issue not exist?**\n\n**A8.** Thanks for your question! The structure of ethanol can be specified using CCO, OCC, or C(O)C,' is in the [link](https://en.wikipedia.org/wiki/Simplified_molecular-input_line-entry_system). While the model delivers consistent results, variations in SMILES can yield molecules with distinct properties, as noted in JT-VAE, making it also unsuitable to use SMILES for molecule representation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a novel method called 3M-Diffusion for generating diverse and novel molecular graphs from textual descriptions. The multi-modal diffusion approach that operates on an aligned latent space of graphs and text appears innovative and well-motivated to address limitations of prior work.\n- The experiments on multiple real-world datasets demonstrate that 3M-Diffusion outperforms state-of-the-art baselines, especially in terms of the novelty and diversity of generated molecules, while still maintaining semantic consistency with the text prompts. For example, it achieves over 146% improvement in novelty and 130% in diversity on the PCDes dataset.", "reasons_to_reject": "- Figure 1 is confusing; the encoder and condition should be inputted together into the diffusion model.\n- The legends in Table 1 are incorrect for the datasets ChEBI-20 and PubChem. \n- Tables 1 and 2 could be merged to reduce redundancy.\n- How to obtain the ground truth graph of a generated molecule when calculating similarity if the descriptions of the two texts are very similar.\nAdditional experiments, such as retrieval tasks, should be conducted to validate the alignment of representations between text and molecular graphs.\n- Does the case study use examples taken from the training set?\n- It is inappropriate to express improvements in novelty and diversity as the percentage increases. The statement '3M-Diffusion achieves 146.27% novelty and 130.04% diversity relative improvement over the SOTA method' should be rephrased to better reflect these metrics.\n- Due to the lack of distinctiveness in the similarity, novelty, diversity, and validity metrics, it would be beneficial to identify additional indicators for assessing the algorithm, such as plotting the distribution of logP and QED.\n- Why is there such a significant difference between the prompts in Figure 2 and Figure 3? The prompt in Figure 3 includes the molecular components, whereas Figure 2 only provides a simple description of properties. The disparity is too significant.\nThe paper's statement, 'For example, the structure of ethanol can be specified using CCO, OCC, or C(O)C,' requires supporting evidence. If the method for generating SMILES is consistent, would this issue not exist?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n22edcXQoU", "reviewer": "Reviewer_xhrH", "summary": "This paper proposes a method for text-guided molecule generation. The method leverages a latent diffusion model to generate latent representations of molecular graphs. Further, a variational graph autoencoder is employed to decode a molecular graph from the latent representation. For text-guided generation, an LLM encoder is trained to align with the graph encoder by cross-modal contrastive learning. The output of the LLM encoder is also incorporated as a condition to control the latent diffusion model.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and feedback! \n\n**Q1. These results on the GuacaMol benchmarks are absent from the subsequent sections.**\n\n**A1.** Thanks for your comments! We believe there are misunderstandings. The results of unconditional molecule generation with GuacaMol benchmarks are put in Table 5 of Appendix. \n\n**Q2. Why are the evaluation metrics from MolT5 missing?**\n\n**A2.** Thanks for your question! The metrics like BLEU, Exact, Levenshtein employed in MolT5 are used to assess similarity based on SMILES representations rather than graph similarity. **We have used MACCS similarity metrics in our paper.** Additionally, our paper conduct molecule generation with a focus not just on reproducing known molecules, but on generating diverse and novel molecules that have not yet been discovered, aiming to advance drug discovery.\n\n**Q3. The comparison should include baselines in [1]. Table 5 should be moved to the main body.**\n\n**A3.** Thanks for your comments! We have included the baseline from the source code provided in [1]. The results presented below demonstrate that 3M-Diffusion outperforms all baselines across all evaluated metrics.\n\n|   | ChEBI-20 |       |       |       |       | PubChem |       |       |       |\n|-----------|----------|-------|-------|-------|-------|---------|-------|-------|-------|\n|           | Uniq (%) | Novelty (%) | KL Div (%) | FCD (%) | Validity (%) | Uniq (%) | Novelty (%) | KL Div (%) | FCD (%) | Validity (%) |\n| CharRNN   | 72.46    | 11.57  | 95.21 | 75.95 | 98.21 | 63.28   | 23.47  | 90.72 | 76.02 | 94.09      |\n| VAE       | 57.57    | 47.88  | 95.47 | 74.19 | 63.84 | 44.45   | 42.47  | 91.67 | 55.56 | 94.10      |\n| AAE       | 1.23     | 1.23   | 38.47 | 0.06  | 1.35  | 2.94    | 3.21   | 39.33 | 0.08  | 1.97       |\n| LatentGAN | 66.93    | 57.52  | 94.38 | 76.65 | 73.02 | 52.00   | 50.36  | 91.38 | 57.38 | 53.62      |\n| 3M-Diffusion | **83.04** | **70.80** | **96.29** | **77.83** | **100.00** | **85.42** | **81.20** | **92.67** | **58.27** | **100.00** |\n\n\nOur paper mainly focuses on text-guided molecule generation. While unconditional generation isn't our main focus, it serves to validate our model's effectiveness in learning diverse and novel molecules. Therefore, we include the results of unconditional molecule generation in the Appendix rather than in the main body.\n\n\n**Q4. Table 1 and Table 2 have the same caption.**\n\n**A4.** Thanks for your comments! We will correct this problem in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The method is overall sound and novel. Applying diffusion models for text-guided molecule generation is an important application.\n- The literature review is comprehensive and effectively positions this work within the context of existing studies.", "reasons_to_reject": "* In Introduction, the authors promised results on the GuacaMol benchmarks, yet these results are absent from the subsequent sections. \n* The evaluation metrics in Table 1 and Table 2 are insufficient. Why are the evaluation metrics from MolT5 (*e.g.*, BLEU, Exact, Levenshtein, MACCS) missing in this work?\n* **Unconditional molecule generation (Table 5)**\n  * Since no textual condition is used, the comparison should include baselines in [1]. \n  * Table 5 should be moved to the main body of the paper.\n* Table 1 and Table 2 have the same caption, yet the contents are different. The authors should specify the distinct contents of each table to avoid confusion.\n\n[1] Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models. In Frontiers in Pharmacology, 2020.", "questions_to_authors": "cf. above.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work presents a method to guide molecular generation with NL. The key concern being improving on SOA through improving the diversity of molecules generated. the work accomplishes its task, but the requires several different models, each doing a specific task to be trained together. This is is unsurprising because the work builds on pretrained models (graph/llm) which raises the question of what becomes more efficient if you build end-to-end.  

As reviewer H3jC points out, the changes wrt SOA methods need to be evidenced/explained more, especially given the substantial change in the fidelity of the model (c.f. Mol-instruct numbers shared in the rebuttal). These big changes should be explained in terms of what this model is fixing/ignoring wrt the baseline, for example are there particular types of structure/strategies that increase the diversity for 3M compared to Mol-instruct?  Why is Validity always 100% for 3M, are there cases where that is not the case? Have you done any human eval on a small subset to understand how well aligned generated molecules are with descriptions? 

Overall, this is a nice work and the thorough baselines that are useful to the community. I would encourage authors to share as many artifacts from data to models as possible
Decision: Accept