LDMol: A Text-to-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
Authors: Jinho Chang
Jong Chul Ye
Venue: ICML 2025 poster (2025)
Keywords: Diffusion models
Molecule generation
Representation learning
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: We present a state-of-the-art text-to-molecule diffusion model by a novel construction of structurally informative molecule latent space.

Abstract: With the emergence of diffusion models as a frontline generative model, many researchers have proposed molecule generation techniques with conditional diffusion models. However, the unavoidable discreteness of a molecule makes it difficult for a diffusion model to connect raw data with highly complex conditions like natural language. To address this, here we present a novel latent diffusion model dubbed LDMol for text-conditioned molecule generation. 
By recognizing that the suitable latent space design is the key to the diffusion model performance, we employ a contrastive learning strategy to extract novel feature space from text data that embeds the unique characteristics of the molecule structure. 
Experiments show that LDMol outperforms the existing autoregressive baselines on the text-to-molecule generation benchmark, being one of the first diffusion models that outperforms autoregressive models in textual data generation with a better choice of the latent domain.
Furthermore, we show that LDMol can be applied to downstream tasks such as molecule-to-text retrieval and text-guided molecule editing, demonstrating its versatility as a diffusion model.

LDMol: A Text-to-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
Jinho Chang
Jong Chul Ye
Abstract
graphs (Lim et al., 2020), and point clouds (Hoogeboom et al., 2022). The attributes controlled by these models evolved from simple chemical properties (Olivecrona et al., 2017; Kómez-Bombarele et al., 2018) to complex biological activity (Staszak et al., 2022) and multi-objective condition- ing (Li al., 2018; Chang & Ye, 2024). More recently, as deep learning models' natural language comprehension ability has rapidly increased, there's 's a growing interest in molecule generation controlled by natural language (Ed- wards et al., 2022: Pei et al., 2023; Liu et al., 2024a; Su et al., 2022) which encompasses much broader and user- friendly controllable conditions.
With the emergence of diffusion models as front- line generative model, many researchers have pro- posed molecule generation techniques with con- ditional diffusion models. However, the unavoid- able discreteness of molecule makes it difficult for diffusion model to connect raw data with highly complex conditions like natural language. To address this, here we present novel latent dif- fusion model dubbed LDMol for text-conditioned molecule generation. By recognizing that the suit- able latent space design is the key to the diffu- sion model performance, we employ contrastive learning strategy extract novel feature space from text data that embeds the unique characteris- ties of the molecule structure. Experiments show that LDMol outperforms the existing autoregres- sive baselines on the eex-to-mmlecull generation benchmark, being one of the first diffusion models that outperforms autoregressive models in textual data generation with a better choice of the latent domain. Furthermore, show that LDMol can be applied to downstream tasks such as molecule- to-text retrieval and text-guided molecule editing, demonstrating its versatility as a diffusion model.
Meanwhile, diffusion models (Song & Ermon, 2019; Ho et al., 2020) have emerged frontline of generative mod- els over the past few years. Through simple and stable training objective of predicting noise from noisy data (Ho et al., 2020), diffusion models have achieved highly realistic and controllable data generation (Dhariwal & Nichol, 2021; Karras et al., 2022; Ho & Salimans, 2021). Furthermore, leveraging that the score function of the data distribution learned in their training (Song et al., 2021b), state-of-the- art image diffusion models enabled various applications on the image domain (Saharia et al., 2022; Kim & Ye, 2021; Chung et al., 2023). Inspired by the success of diffusion models, several papers suggested diffusion- based molecule generative models on various molecule domains including a molecular graph (Luo al., 2023), strings like Simplified Molecular-Input Line-Entry System (SMILES) (Gong et al., 2024), and point clouds (Hoogeboom et al., 2022).
1. Introduction
Designing compounds with the desired characteristics is the essence of solving many chemical tasks. Inspired by the rapid development of generative models in the last decades, de novo molecule generation via deep learn- ing models has been extensively studied. Diverse models have been proposed for generating molecules that agree with given condition various data modalities, includ- ing string representations (Segler et al., 2017), molecular
However, a discrepancy between molecule data and com- mon data domains like images makes it hard to connect the diffusion models to molecule generation. Whereas diffusion models are deeply studied on continuous data domain with Gaussian noise, any molecule modality has inevitable discreteness such atom and bond type, connectivity, and SMILES tokens (Figure 1-(a)). As result, diffusion models trained on raw molecule data often failed to faithfully follow the given conditions or showed poor data quality (e.g., in- valid molecules) as the condition became more sophisticated like natural language. Most molecule diffusion models pre- sented so far have used few, relatively simple conditions to control, while major developments in texl-to-molecule
'Graduate School of Artificial Intelligence, Korea Advanced Institute of Seience and Technology, Scoul, South Korca, Corre- spondence to: Jong Chul Ye yc ye @ kr.>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada, PMLR 267, 2025. Copyright 2025 by the author(s).

---

LDMel: A Text-to-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
(c)
text-to-molccule generation
64 n1 cc ccl discrete
nleceecl
"It's a benzere ring with nitrogen."
nicccee1
chemically informative encader
(b)
nicceee1
molecule-text retrieval
"It's carbon chain..
encoder
LDMol
"I't's carboxylic
HNN
+It's aroeatic compound_"
decoder
text-guided molecule editing
decoder
nicecec1
"It's an ester."
H6N
nicecce1
domain that diffusion models aim
Figure 1. Different strategies of data domain selection for molecule diffusion models. (a) The model directly learns the raw representation of the molecule such string tokens. (b) An autoencoder be employed to let the generative model learn its latent distribution. (c) regularized, chemically pre-trained encoder can provide a latent space readily learnable by external generative models.
generative models were based autoregressive models.
We propose latent diffusion model LDMol for text- conditioned molecule generation to generate valid molecules that are better aligned to the text condition. This approach demonstrates the potential of generative models for chemical entities in latent space.
To overcome this gap, we suggest that a latent domain (Vah- dat et al., 2021; Rombach et al., 2022) essential to train effective diffusion models for complex molecule genera- tion tasks. Moreover, beyond the limitation of the previous works (Xu al., 2023) that mainly focused on resolving the discreteness with naive reconstruction loss (Figure 1- (b)), we report that latent encoder extracting rich and refined information about the molecule structure can further improve the generative model performance (Figure 1-(c)). Specifically, we design novel Latent Diffusion Molecular generative model (LDMoll) for rext-conditioned molecule generation, trained on the latent space of the separately pre- trained molccule encoder. By preparing encoder to pro- vide chemically useful and interpretable feature space, our model can more easily connect the molecule data with the highly complicated condition of natural text. In the process, we suggest a novel contrastive encoder training strategy by minimizing mutual information between positive SMILES pairs encode unique structural characteristic.
We report the importance of preparing a chemically informative latent space for the molecule latent diffu- sion model, and suggest novel contrastive learning method to train an encoder that captures the molecular structural characteristic.
LDMol outperforms the texl-to-molecule generation baselines, and its modeled conditional score function enables the advanced attributes of diffusion models including various applications like molecule-to-te retrieval and text-guided molecule editing.
2. Background
Diffusion generative models. Diffusion models first de- fine forward process that perturbs the original data, and generates the data from the known prior distribution by the learned reverse process of the pre-defined forward process. Ho et al. (2020) fixed their forward process by gradually adding Gaussian noise the data, which can be formalized as follows:
Extensive experimental results show that LDMol can out- perform many state-of-the-an autoregressive models and generate valid SMILES that meet the input text condition. Considering SMILES as a variation of text data, we report one of the first diffusion models that successfully surpassed autoregressive models in textual data generation. This may suggest the possibility of improving existing diffusion mod- els (Lovelace et al., 2024) for natural language through careful design of the latent space. Furthermore, LDMol can leverage the learned score function and be applied to several multi-modal downstream tasks such as molecule-to- trieval and text- text-guided molecule editing. without additional task- sss-sppeiffic training. We summarize the contribution of this work as follows:
(( = N(x /I 3,0 B.I)
(1)
where 3,, 1 1,.., is a noise schedule. This definition of forward process allows us to sample x, directly from q (2)) as follows, where a = 1 3r and a. = II aq
re-
vāLzo -I- aje, where (0, (2)
The model eg learns the reverse process p(x,-1 (2:) by approximating g(x+-11x;) with Gaussian distribution
2

---

LDMol: A Text-to-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
po( (xg-1ill00ee (),
where
models that could generate molecules with simple chem- ical and biological conditions (Hoogeboom et al., 2022; Luo et al., 2023; Trippe et al., 2023). Gong et al. (2024) al- tempted text-conditioned molecule diffusion model trained on the sequence of tokenized SMILES indices. However, these models treated discrete molecules with continuous Gaussian diffusion, introducing arbitrary numeric values and suboptimal performances. Xu al. (2023) employed an autoencoder to build diffusion model on smooth latent space, but its controllable conditions were still limited to several physiochemical properties.
1 vI
no (31,1t)
xt
co (o(
(3)
you v
Cxe
which can be trained by minimizing the difference between e aadd c0 t):
0" = arg min E.e...elle ( 1)
(4)
0
Once 0 is trained, novel data can be generated with the learned reverse process pe; starting from the random noise Dr ~ N (0, T), the output can be gradually denoised accord- ing to the modeled distribution of po( (zze).
3. Methods
Various real-world data generation tasks require to generate data Zo with a given condition c. To build diffusion models that can generate data from the conditional data distribution ((za cc,, the model that predicts the injected noise should also be conditioned by c.
In this section, we explain the overall model architecture and training procedure of the proposed LDMol, which are briefly illustrated in Figure 2.
3.1. Extracting structure-awars SMILES latent space
0* = arg min Exolo,t,ell- c0( L, cc1l
(5)
The primary goal of introducing autoencoders for im- age latent diffusion models is to map raw images into low-dimensional space, which reduces the computation cost (Vahdat et al., 2021; Rombach et al., 2022). This is plausible because a high-resolution image has an enormous dimension in the pixel domain, yet each pixel contains little information.
0
After the success of diffusion models in the image and video domain, various works tried to build diffusion models to generate text data. While many suggested training dif- fusion models on text tokens (Austin al., 2021), word embedding (Li et al., 2022), text autoencoder latent space (Lovelace et al., 2024), their performance has been suboptimal compared to autoregressive models (Brown et al., 2020). We assume that this can be resolved with a better latent space design that reflects the characteristics of the data domain, and suggest latent diffusion model that outperforms autoregressive models for textual data.
In this work for molecule generation, we utilize string- based notation SMILES, one of the most popular molecule representations in text-molecule pair databases and bench- marks. We built SMILES encoder to map raw SMILES strings into a latent vector. In this case, the role of our SMILES autoencoder has to be different from that of the autoencoders for images; a molecule structure can be fully expressed by only sequence of L integers for SMILES to- kens, where L is the maximum token length. However, each token carries significant information, and hidden interac- tions between these tokens are much more complicated than interactions between image pixels. Therefore, the SMILES encoder should focus more on extracting chemical meaning into the latent space, even if results latent space with more dimensions than the raw SMILES string.
Conditional molecule generation. As promising tool for many important chemical and engineering tasks like de novo drug discovery and material design, conditional molecule generation has been extensively studied with various models including recurrent neural network (RNN)s (Segler et al., 2017), bidirectional RNN (Grisoni et al., 2020), graph neu- ral networks (Lim al., 2020), and variational autoen- coders (oóbez-Bombarelli et al., 2018; Lim et al., 2018). With the advent of large and scalable pre-trained models with transformers (Vaswani et al., 2017), the controllable conditions became more abundant and complicated (Bagal et al., 2021; Chang& & Ye, 2024). Recent works reached a texteg molecule generation (Edwards et al., 2022; Su al.. 2022: Liu et al., 2024a) leveraging a deep compre- hension ability for natural language, especially with recent emergence of large language model (LLM)s (Liu et al., 2024b).
A number of molecule encoders (Wang et al., 2019; Liu et al., 2024a; Zeng et al., 2022: Liu et al., 2023a) have been presented that can extract various useful chemical features, including biochemical activity or human-annotated descriptions. Nonetheless, these molecule encoders aim to extract certain desired features rather than encode all the information about the molecule structure. Therefore, the input cannot be fully restored from the model output.
Recent works attempted to import the success of the diffu- sion model into molecule generation. Several graph- based and point cloud-basee works have built conditional diffusion
Although autoencoders with appropriate regularization (e.g., KL-divergence loss (Kingma, 2014; GómezzBombareell et al., 2018)) provide a continuous and reconstructible
3

---

LDMol: A Text-to-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
(a)
projection norm.
(c)
t~U(O,T)
"Ie's benzene zinç with nitrogen_.
SMIILES enncoder
nicccoo1
Diffusion Transformer(Din)
fg.
SMILES
In-loach indlse
dnt products
cinceoc1 mg
SMILES encoder
Traiiing
Inference
ascimilates
distances
sampling Iteration
nétrogen.""
(b)
compression layer
text encoder
SMILES encoder
SMILLS ES decoder transformer)
*
Diffusion Tranffomme(DDT)
Eg. Ee
SMILES decoder
d2
T. d.
[aS]
[CLS]
Figure Overview the proposed molecule autoencoder and the latent diffusion model. (a) SMILES encoder is trained with contrastive learning to construct latent space that embeds structural characteristic. (b) After the SMILES encoder is prepared, linear compression layer and an autoregressive decoder are trained restore encoder input. (c) The training and inference process of latent diffusion model is conditioned by the output of the frozen external text encoder.
molecular latent space, their encoder output is not guaran- teed to possess the characteristic of the underlying molecular structure, beyond the minimal information to reconstruct the input string. To visualize this, we prepared a trained 3-VAE (Higgins et al., 2017) and measured the feature distance between two SMILES from the same molecule obtained via SMILES enumeration (Bjerrum, 2017). Here, SMILES enumeration is the process of writing out all possi- ble SMILES of the same molecule, illustrated in Figure 3-(a). Figure 3-(b) shows that 3-VAB had difficulties assim- ilating features from the same molecule compared to the one between random SMILES pairs, indicating that it couldn't 't capture the intrinsic features beneath the SMILES string. This inconsistency makes it difficult for later models that learn this latent space figure out the connection between the latent and the molecule, which could eventually degrade the performance as the condition gets more complex like natural texts. Assuming most of the controllable conditions has unavoidable correlation with the molecule structure, we insist that latent domain where the feature proximity is more structurally meaningful would benefit the conditional generative model.
taining similar information (i.e. positive pair) and distancing semantically unrelated features (i.e. negative pair). We de- fine two enumerated SMILES from the same molecule as positive pair and two SMILES from different molecules as a negative pair.
Here, we argue that the proposed contrastive learning with SMILES enumeration can train the encoder to encapsulate the unique structural characteristics of the input molecule: Contrastive learning learns invariant for the augmenta- tions applied on positive pairs (Zhang & Ma, 2022), and it is known that good augmentation should reduce as much mu- tual information between positive pairs as possible while pre- serving relevant information (Tian et al., 2020). Meanwhile, enumerated SMILES of the same molecule are obtained by traversing the nodes and edges in the molecular graph with different visiting order. Therefore, to detect all possible enu- merated SMILES and find the model has to understand the entire connectivity between atoms. This makes the encoder output a unique characteris- tic that captures the overall molccular structure. Compared to the hand-crafted augmentations previously presented for molecule contrastive learning (You al., 2020), enumerated SMILES pairs have minimal mutual information since utilize all possible variations in the SMILES format. And since all enumerated SMILES are guaranteed to represent an identical molecule, there no relevant information loss during the augmentation. Figure 3-(a) and Appendix B. demonstrate that our LDMol trained with contrastive learn- ing with SMILES enumeration now correctly assimilates features from the same molecule, and its latent space cap-
Accordingly, here propose three conditions that our SMILES autoencoder's latent space has to satisfy: enable reconstruction of the input, have small dimensions as possible, and embed molecular structural information that can be readily learned by diffusion models.
Encoder design. In this respect, we train our SMILES en- coder with contrastive learning (Figure 2-(a)), which aims to learn better representation by assimilating features con-

---

LDMel: A Text-to-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
(a)
OH
CANONICAL
SMILES enumerated SMILES pairs
Density
CC(C)Cclcccec1((=000
10
ENUMERATED
0.6
0.5
0.4
0.3
0.1
Euclidean distance of B-VAE encoder features
O=CCO))lccccccIC((C)
random SMILES pairs
enumerated SMILES pairs
OH
ENUMERATED
Dersity
cICcCCO)OdcCCOO=))ccccl
1.2
1.0
3.8 0.5 0.4
0.2
0.0
Euclidean distance LDMol encoder features
Figure 3. Behaviour of encoder features SMILES enumeration. (a) Examples of SMILES enumeration with node traversal order. (b) Euclidean distance between features from 8-VAE (3 = 0.001) and our proposed encoder, with 1,000 random SMILES pairs and 1,000 enumerated SMILES pairs. The distance rescaled by V where d is latent dimension size.
tures meaningful structural information of the molecule.
range this linear layer output is a target domain of our latent diffusion model.
Specifically, a SMILES string is fed into the SMILES en- coder E-) with special "[SOS]" token which denotes the start of the sequence. For a batch of N input SMILES M {yTi, 172..... m.,. MIN J. prepare positive pair SMILES m'. for each m& by SMILES enumeration to ,''x' }. After M and M' are passed through the SMILES encoder, we feed each SMILES' output vector corresponding to the [SOS] token into an additional linear projection and normalization layer, denoting its output e/ and v/ (k: = 1, 2, ........ N). Assimi- lating un and v/ from the positive pairs and distancing the others can be done by minimizing the following InfoNCE loss (Oord et al., 2018).
Decoder design. When SMILES is passed through the SMILES decoder and the compression layer. the SMILES decoder reconstructs coder knows nothing about SMILES distribution or its corre- lation with natural texts, and any design would be acceptable as long as it recovers SMILES from its latent. Following many major works that treated SMILES variant of lan- guage data (Segler et al., 2017: Chang & Ye, 2024), we built autoregressive transformer (Vaswani et al., 2017) as our decoder (Figure 2-(b)) which is widely used to success- fully generate sequential data with varied length (Brown et al., 2020). Specifically, starting from the [SOS] token, the decoder predicts the next SMILES token using infor- mation from ʃ(E (m)) with cross-attention layers. When (to, t1, ....... t.. } the token sequence of m, the decoder trained minimize the next-token prediction loss described as Eq. (8). Here, the decoder and the compression layer are jointly trained while the encoder's parameter is frozen. Af- ter being fully trained, the decoder was able to reconstruct roughly 98% of the SMILES encoder input.
from f(E(m)). Note that the de-
construct M' = [mm,, m.,.... m'......
exp(Uk d///T)
Leon (M,
Σ log
(6)
-= exp(vx 1//7)
Here, is a positive temperature parameter. To utilize a symmetric loss against the input, we trained our encoder with the following loss function.
(M, (M, M') Loon (MMM)) + Loon (MM',
Compressing the latent space. The pre-trained SMILES encoder maps a molecule into vector of size [L x dene], where dene is the feature size of the encoder. To avoid the curse dimensionality and construct more learnable fea- ture space for diffusion models, we additionally employed linear compression layer f(-) (Figure 2-(b)) to reduce the dimension from [L x dene] to [L d.). Here, we build our compression layer as simply as possible to prevent its out- put from deviating from the previous structure-awand aware and regulated features (See A.2 for further justification). The
Láce Σ
log (o0p((
(8)
3.2. Text-conditioned latent diffusion model
As shown in Figure 2-(c), our diffusion model learns the conditional distribution of the SMILES latent whose di- mension [L x dg]. In the training phase, a molecule the training data mapped to the latent and applied a forward noising process into 2 with randomly sampled diffusion
5

---

LDMol: A Text-te-Molccule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
timestep l and injected noise e. A diffusion model predicts the injected noise from 2t, conditioned by the paired text description via frozen external text encoder. In the infer- ence phase, the diffusion model iteratively generates new latent sample z' from a given text condition, which is then decoded to a molecule via the SMILES decoder.
lines (Edwards al., 2022; Pei et al., 2024) with millions of unimodal and multimodal data from various databases.
The latent diffusion model was trained with the training loss suggested by Dhariwal & Nichol (2021). To take advan- tage of classifier-free free guidance (Ho & Salimans, 2021), we randomly replaced 3% of the given text condition with the null text during the training. The sampling iteration in the inference stage used DDIM-based (Song et al., 2021a) 100 sampling steps with a classifier-fres guidance. More detailed training hyperparameters can be found in Appendix A.2.
Since most contributions to diffusion models were made in the image domain, most off- she-ll diffusion models have the architecture of convolution- Unet (Ronneberger et al., 2015). However, introducing the spatial inductive bias of Unet cannot be justified for the latent space of our encoder. Therefore we employed DiT (Peebles & Xie, 2023) architecture, one of the most successful approaches to transformer-basee diffusiio models for more general data domain. Specifically, we utilized a DiT model with min- imal modifications to handle text conditions with cross- attention, where more details can be found in Section A.
The code for LDMol training and mell-lo- sampling is available at https//gghubbbooo/jjojjj111/////
4. Experiments
4.1. Text-conditioned molecule generation
In this section, we evaluated the trained L.MMol's ability to generate molecules that agree with the given natural lan- guage conditions. First, we generated molecules with LD- Mol using the text captions in the ChEBI-20 test set and compared them with the ground truth. The metrics we' ve used are as follows: SMILES validity, BLEU score (Pap- ineni et al., 2002) and Levenshtein distance between two SMILES, Tanimoto similarity (Bajusz et al., 2015) between two SMILES with three different fingerprints (MACCS, RDK, Morgan), the exact match ratio, and Frechet Chem- Net Distance (FCD) (Preuer et al., 2018). We tested different scales for the classifier-free guidance scale w in the sampling process and found w = 2.5 works best (See Section B.2).
Text-based molecule generation requires a text encoder to process natural language conditions. Existing text- molccule generation models trained their text encoder from scratch (Pei et al., 2023), or utilize separate encoder model pre-trained on scientific domain corpora (Beltagy et al., 2019). In this work, we took the encoder part of Moll3 (Edwards et al., 2022) as our text encoder.
3.3. Implementation details
The pre-training of the SMILES encoder and the correspond- ing decoder was done with 10,000,000 general molecules from PubChem (Kim et al., 2023). The SMILES tokenizer vocabulary consists of 300 tokens, which were obtained from the pre-training data SMILES corpus using the BPE algorithm (Gage, 1994). We only used SMILES data that does not exceed a fixed maximum token length L. To ensure enough batch size for negative samples (He et al., 2020), we build a memory queue that stores Q recent input and use them for the encoder training. We found that if the training data have a stereoisomer, considering it as "hard-negative" samples and including it in the loss calculation batch helps the encoder training to differentiate different stereoisomers.
Table contains the performance of LDMol and other base- lines for text-to-molecule generation on the ChEBI-20 and PCDes test set. Including both autoregressive models and diffusion-basee based models, LDMol outperformed the existing models in almost every metric. While few models showed higher validity than ours, they showed a lower agreement between the output and the ground truth, which we insist is a more important role of generative text-to-molecule models. Also, MolTStarge. uses the same text encoder as LDMol, yet there's significant performance difference between the two models. We believe this is because our continuous and structure- latent space is much easier to learn and align with the same textual information, compared to the raw token sequence for transformer-basee models.
To train the text-conditioned latent diffusion model, we gathered three existing datasets of text-molecule pairs: Pub- chemSTM curated by Liu et al. (2023a), ChEBI-20 (Ed- wards et al., 2021), and PCdes (Zeng et al., 2022). Only train split for each dataset was used for the training, and pairs that appeared in the test set for the experiments are additionally removed. We also used 10,000 molecules from ZINCI5 (Sterling & Irwin, 2015) without any text descrip- tions, which helps the model learn the common distribution of molecules. When these unlabeled data were fed into the training model, we used a pre-defined null text for the ab- sence of text condition. We utilized 320,000 training data in total, much smaller than recent transformer- baadddd base-
To demonstrate the LDMol's molecule generalization ability with more broad and general text inputs, we analyzed the generated output with several hand-written prompts. These input prompts were not contained in the training data and were relatively vague and high-level so that many different molecules could satisfy the condition. Figure 4 shows the samples of generated molecules from LDMol with several input prompt examples. We found that LMDol can generate molecules with high validity that follow the various levels of
6

---

LDMol: A Text-te-Molccule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
Table Benchmark results of text-to-moleeule generation ChEBI-20 and PCDes test set. The hest performance for each metric was
written in bold. The "Family" column denotes whether the model is AR model) or DM(diffusion model).
Dataset
Model
Family
Validitt
BLEU
Levenshieing
MACCS FTSt
RDK FTS1
Morgan FTS
Matcht
PCD]
Transformer
AR
0.906
0.499
57.660
0.480
0.320
0.217
0.000
11.32
GIT-Mel (Läu et al., 2024a)
AR
0.928
0.756
26.315
0.738
0.582
0.519
0.051
Tht.re (Raffel al., 2020)
AR
0.660
0.765
24.950
0.731
0.605
0.545
0.069
2.48
MofIS (Bdwards
2022)
AR
0.772
0.769
24.458
0.721
0.588
0.529
0.081
2.18
TStarae
AR
0.902
0.854
16.721
0.823
0.731
0.670
0.279
1.22
ChEBI-20
MolTS
AR
0.905
0.854
16.071
0.834
0.746
0.684
0.311
1.20
MolXPT (Liu al. 20236)
AR
0.757
0.667
0.215
0.45
bioT5 (Pei et 2023)
AR
1,000
0.867
15.097
0.886
0.801
0.734
0.413
0.43
bioI5+ (Pei at, 2024)
AR
1,000
0.872
12.776
0,907
0.835
0.779
0.522
0.35
TGM-DLM (Gong ol., 2024)
DM
0.871
0.826
17.003
0.854
0.739
0.688
0,242
0.77
LDMel
DM
0.941
0.926
6,750
0,973
0.950
0.931
0,530
0.20
MorTS
AR
0.944
0.692
18.481
0,810
0.741
0.699
0.440
0.70
bioTS
AR
1,000
0.754
15.658
0.797
0.726
0.677
0.455
0.69
PCDes
bioT5+
AR
0.999
0.677
20.464
0.743
0.615
0.541
0.266
1.09
LDMol
DM
0.944
0.857
8.726
0.885
0.817
0.780
0.464
0.32
(a) his molectte containe allions
(b) This mocecele phoepholppii
Table 2. 64-way accuracy on molecule-to-text retrieval task.
For LDMol, is a number of iterations where ||és <l12 was calculated. The best performance for each task written bold. Model
PCdes test
MoMu test
sentence
paragraph
sentence
paragruph
SciBERT (Reltagy et al., 2019)
50.4
82.6
1.38
1.38
Nalidiry: BMTH
Vallsiey: HURR
KV-PLM (Zeng al., 2022)
55.9
77.9
1.51
MoMu-S (Su et al., 2022)
58.6
80.6
39.5
45.7
(c) This molecute las long carbon chain.
MoMu-K
58.7
81.1
39.1
46.2
MoleculeSTM (Liu 2023a)
81.4
67.6
MolCA (Liu et al., 2024c)
86.4
73.4
LDMol(v=10)
60.7
90.2
66.4
84.8
Nalidiry: BUSS
LDMol(v=25)
62.2
90.3
78.4
87.1
(il) This conlooule inclades beausstaruup.
(c) This moloukk a tyvosine desivation.
downstream tasks. The state-of-the-art image diffusion mod- els have shown their versatility in image editing (Meng et al., 2022; Hertz et al., 2023), classification (Li et al., 2023), re- trieval (Jin et al., 2023), inverse problems like inpainting and deblurring (Chung et al., 2023), image personalization (Ruiz et al., 2023), etc. To demonstrate LMDol's potential versatil- ity as a diffusion model, we applied the pre- pre-trained LDMol to the molecule-to-tex retrieval and text-guided molecule editing. See Appendix A.3 for a more detailed procedure for each downstream task.
Nelistiey
Yalidity: 8.996
Figure 4. Examples of the generated molecules by LDMol with various text conditions, with validity on 1,000 generated samples.
input conditions for specific atoms(a), compound class(b), molecular substructure(c), functional groups(d), and sub- stance names(e). The validity was calculated by the number of valid SMILES over 1,000 generated samples, above 0.9 for most scenarios we tested. Considering that these short, broad, and hand-written text conditions are distinct from the text conditions in the training dataset, we' 've concluded that our model is able to learn the general relation between natural language and molecules. We conducted quantitative analyses on the case studies and additional examples with hand-written prompts, which can be found in Appendix B.3.
Molecule-to-text retrieval. Our approach to molecule-to- text retrieval is similar the idea of using pre-trained diffusion model a classifier (Li et al., 2023): LDMol takes each candidate text with query molecule's noised latent, and retrieves the text that minimizes the noise esti- mation error ||2, - <<1i between the injected noise c and the predicted noise én. Since this process has randomness due to the stochasticity of and e, we repeated the same process times with resampled t and and used mean error to minimize the performance variance.
We measured a 64-way in tn-batch retrieval accuracy of LD- Mol using two different test sets: PCdes test split and MoMu retrieval dataset curated by Su et al. (2022), where the result with other baseline models are listed in Table 2. Only one
4.2. Applications toward downstream tasks
Well-trained diffusion models learned the score function of a data distribution, which enables high applicability to various
7

---

LDMol: A Text-to-Molccule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
1.0
MaleruleSTM(Graphh)
Table 3. Quantitative results of the ablation study. The best perfor-
LDMol
mance for each metric is written in bold.
0.8
models
Autoeneoder
ChEBIZU generation
Recon. Acc
Validity
Match1
FCD,
LDMol w/o contrastive learning L.Maol (A-AA (8=0.001)
1,000
0.019
0.000
58.60
0.6
Hit Hatroto
0.999
0.847
0.492
0.34
LDMol
0.983
0.941
0.530
0.20
0.4
structed latent space, to emphasize the benefit of the sug- gested encoder training. Each model is pre-trained with the same number of DiT training iterations. We've also per- formed an ablation study on more detailed design choices of LDMol, which can be found in Appendix B.6.
0.0
Is 1pp-- 5 lkke
snot ar
has Non
has low has morehhas more mean billy acceptir. donor.
solutee
When we remove the contrastive encoder pre-training ob- jective and construct the molecule latent space with a naive autoencoder, the diffusion model completely fails to learn the latent distribution to generate valid SMILES. On the other hand, 3-VAE with KL--diverencce regularization has reconstructible latent space and showed text-to-molecule generation match ratio of 0.492, which already outperforms the previous diffusion model TGM-DLM and several autore- gressive models in Table 1. This demonstrates the necessity of diffusion models in the continuous data domain, with their potential to be successful on discrete molecule data comparable the autoregressive models. Nonetheless, its overall metric is still worse than the proposed LDMol, with notably low validity and FCD. We insist that this gap comes from the structurally informative latent space of LDMol which is easier for the diffusion model to learn the correla- tion between the latent space and the condition.
Figure 5. Hit ratio of molecule editing by LDMol and MoleculeSTM (Liu et al., 2023a) in eight scenarios. In the fig- ure, we omitted "This molecule" in front of the actual prompts, and abbreviated "hydrogen bonding" to "H-bond".
randomly selected sentence in each candidate description was used for the retrieval in the "sentence" column, and all descriptions were used for the "paragraph" column. LDMol achieved a higher performance in all four scenarios com- pared to the previously presented models and maintained its performance on a relatively ditor--distribution MoMu test set with minimal accuracy drop. LDMol became more accu- rate as the number of function evaluations increased, and the improvement was more significant at the sentence-level re- trieval and out-of-distribution dataset. The actual examples from the retrieval result can be found in Appendix B.4.
Text-guided molecule editing. We applied a method of Delta Denoising Score (DDS) (Hertz et al., 2023), which was originally suggested for texteguided image editing, to see whether LDMol can be used to optimize a source molecule to match given text. Using two text prompts that describe the source data zarc and the desired target, DDS presents how a text-conditioned diffusion model can modify Eare into a new data Ztge that follows the target text prompt.
5. Conclusion
In this work, we presented text-to-molecule diffusion model LDMol that runs on chemical latent space reflect- ing structural information. By introducing the deeply stud- ied paradigm of the latent diffuusinn model with carefully designed latent encoder, LDMol retains many advanced at- tributes of diffusion models that enable various applications.
We imported a method of DDS on LDMol's molecule latent to edit a given molecule to math the target text, with several prepared editing prompts that require the model to change certain atoms, substructures, and intrinsic properties from the source molecule. Figure 5 shows that LDMoo had compa- rable performance with a previously suggested text-guided molecule editing model (Liu et al., 2023a), with a higher hit ratio in five out of cight scenarios. Several editing examples with mordtwrttten scenarios are shown in Appendix B.5.
Despite the noticeable performances of LDMol, it still has limitations that can be improved, LDMol still often strug- gles to follow some text conditions such as complex biolog- ical properties. Nonetheless, we expect that the LDMol'ss performance could be improved further with the emergence of richer text-molecule pair data and more powerful text encoders. Moreover, combining physiochemical and bio- logical annotations top of the structurally informative latent space is promising future work that can ease the connection between molecules and text conditions.
4.3. Effectiveness of the suggested latent space
We believe that our approach could inspire tackling various chemical generation tasks using latent space, not only text- conditioned but also many more desired properties, such as
We' ve conducted an ablation study in Table 3 that compares LDMol with latent diffusion models trained on naively con-
8

---

LDMol: A Text-to-Molccule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
biochemical activity. Especially, we expect LDMol to be starting point bridge achievements in the state-of-the-art diffusion model into the chemical domain.
Askell, A., et al. Language models few-shot learners. Conference Neural Information Processing Systems, 33:1877-1901, 2020.
Chang, J. and Ye, J. C. Bidirectional generation of structure and properties through a single molecular foundation model. Nature Communications, 15(1):2323, 2024.
Acknowledgments
This work was supported by the National Research Foun- dation of Korea under Grant No. RS-2024-00336454, and the Institute of Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea (MVS)) (RS-2025-02304967, Al Star Fellow- Ship(KAIST)).
Chung, H., Kim, J., Mccann, M. T., Klasky, M. L., and Ye, J. C. Diffusion posterior sampling for general noisy inverse problems. International Conference Learning Representations, 2023.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Conference on Neural Information Processing Systems, 34:8780-8799 2021.
Impact Statement
This paper presents work whose goal advance the field of machine learning and its chemical applications. There many potential socictal consequences of our work, in- cluding the possibility of the molecular design for harmful or inappropriate properties.
Edwards, C., Zhai, C., and Ji, H. Text2mol: Cross-modal molecule retrieval with natural language queries. In Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 595-607, 2021.
Edwards, C., Lai, T., Ros, K., Honke, G., Cho, K., and Ji, H.
References
Translation between molecules and natural language. Em- pirical Methods in Natural Language Processing. 2022.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few- shot learning. Advances in neural information processing 2022.
Gage, P. A new algorithm for data compression. C Users Journal, 12(2):23-38, 1994.
systems,
R., Wei, J. N., Duvenaud, D., Hermández-Lobato, J. M., Sánchez-Lengeling, B., She- berla, D., Aguilera-Iparraguirre J., Hirzel, T. D., Adams, P., and Aspuru-Guzik, A. Automatic chemical de- sign using a data-driven continuous representation of molecules. ACS central science, 4(2):268 276, 2018.
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces. Conference in Neural Information Processing Systems, 34:17981-17999 2021.
Bagal, V., Aggarwal, R., Vinod, P., and Priyakumar, U. D. Molgpt: molecular generation using a transformer- decoder model. Journal of Chemical Information and Modeling, 62(9):2064-2076, 2021.
Gong, H., Liu, Q., Wu, S., and Wang, L. Text-guided molecule generation with diffusion language model. AAAI Conference on Artificial Intelligence 2024.
Bajusz, D., Rácz, A., and Héberger, K. Why is tanimoto in- dex appropriate choice for fingerprint-based similarity calculations? Journal of cheminformaties, 7:1-13, 2015.
Grisoni, F., Moret, M., Lingwood, R., and Schneider, G. Bidirectional molecule generation with recurrent neural networks. Journal of chemical information and modeling, 60(3):1175-1183, 2020.
Beltagy, I., Lo, K., and Cohan, Scibert: pretrained language model for scientific text. Empirical Methods in Natural Language Processing, 2019.
Gómez-Bombarelli, R., Wei, J. N., Duvenaud, D., Hernández-Lobato, J. M., Sánchez-Lengelingg B., She- berla, D., aggullera-pparaggnn P. and Aspuru-Guzik, A. Automatic chemical de- sign using a data-driven continuous representation of molecules. ACS Central Science, 4:268 2288-276,66 01 2018. doi: 10. 102 (0000///ccceeooo 1/acscentsci.76005772
J., Hirzel, T. D.. Adams,
Bemis, G. W. and Murcko, A. The properties of known drugs. 1. molecular frameworks. Journal of medicinal chemistry, 39(15):2887-289) 1996.
Bjerrum, E. J. Smiles enumeration as data augmentation for neural network modeling of molecules. arXiv preprint arXiv: 1703.07076 2017.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In JEEE/CVF Conference on Computer Vision and Pattern Recognition 2024, pp. 9729 9729-9738. 2020.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,

---

LDMol: A Text-to-Molccule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
Hertz, A., Aberman, K., and Cohen-Or, D. Delta denoising score. International Conference on Computer Vision, pp- 2328-2337, 2023.
Lim, J., Ryu, S., Kim, J. W., and Kim, W. Y. Molecular generative model based on conditional variational autoen- coder for de novo molecular design. Journal of chemin- formatics, 10(1):1-9, 2018.
Higgins, I,, Matthey, L.., Pal, A., Burgess, C. P., Glorot, X., Botvinick, M. M., Mohamed, S., and Lerchner, A. beta-vae: Learning basic visual concepts with con- strained variational framework. International Conference on Learning Representations. 3, 2017.
Lim, J., Hwang. S.-Y., Moon, S., Kim, S., and Kim, W. Y. Scaffold-based molecular design with graph generative model. Chem. Sci., 11:1153-1164 2020, doi: 10.1039/ C9SC04503A.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. Conference on Neural Information Processing Systems, 2021.
Liu, P., Ren, Y., Tao, J., and Ren, Z. Git-mol: A multi- modti large language model for molecular science with graph, image, and text. Computers in Biology and Medicine, 171:108073, 2024a.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Conference on Neural Information Pro- cessing Systems, 12 2020.
Liu, S., Nie, W., Wang, C., Lu, J., Qiao, Z., Liu, L.., Tang, J., Xiao, C.,, and Anandkumar, A. Multi-modal molecule structure-tex model for text-based retrieval and editing. Nature Machine Intelligence, 5(12): 1 1447-1457 2023a.
Hoogeboom, E., Satorras, V. G., Vignac, C., and Welling, M. Equivariant diffusion for molecule generation 3d. In International Conference on Machine Learning, pp. 8867-8887, PMLR, 2022.
Liu, X., Guo, Y., Li, H., Liu, J., Huang, S., Ke, B., and Lv, J. Drugllm: Open large language model for few-shot molecule generation. arXiv preprint 2xXi:22405006990 2024b.
Jin, P., Li, H., Cheng, Z., Li, K., X., Liu, C., Yuan, L., and Chen, J. Diffusionret: Generative text-video retrieval with diffusion model. In International Conference on Computer Vision, pp. 2470-2481, 2023.
Liu, Z.,, Zhang, W., Xia, Y., Wu, L., Xie, S., Qin, T., Zhang, M., and Liu, T.-Y. Molxpt: Wrapping molecules with text for generative pre-training. Association for Compu- lational Linguistics, 2023b.
Karras, T., Aittala, M., Aila, T., and Laine, Elucidating the design space of diffusion-based generative models. Conference on Neural Information Processing Systems, 35:26565-2653 26577, 2022.
Liu, Z., Li, S., Luo, Y., Fei, H., Cao, Y., Kawaguchi, K., Wang, X., and Chua, T.-S. Molca: Molecular graph- language modeling with cross-modal projector and uni- modal adapter. Empirical Methods in Natural Language Processing, 2024c.
Kim, K. and Ye, C. Noise2score: tweedie's approach to self-supervised image denoising without clean images. Conference on Neural Information Processing Systems, 34:864-88 884,-774777 2021.
Lovelace, J., Kishore, V., Wan, C., Shekhtman, E., and Wein- berger, K. Q. Latent diffusion for language generation. Advances in Neural Information Processing Systems, 36, 2024.
Kim, S., Chen, J., Cheng, T., Gindulyte, A., He, J., He, S., Li, Q.. Shoemaker, B. A., Thiessen, P. A., Yu, B., et al. Pubchem 2023 update. Nucleic acids research, 51(DI): D1373-D1380, 2023.
Luo, T., Mo, Z., and Pan, S. Fast graph generation via spectral diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Kingma, D. P. Auto-encoding variational bayes. Interna- tional Conference on Learning Representations, 2014.
Li, A. C., Prabhudesai, M., Duggal, S., Brown, E.,, and Pathak, D. Your diffusion model is secretly zero-shot classifier. In International Conference on Computer Vi- sion, pp. 2206-2217, 2023.
McInnes, L., Healy, J., Saul, N., and Grossberger, L. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018.
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and edit- ing with stochastic differential equations. International Conference on Learning Representations, 2022.
Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and Hashimoto, T. B. Diffiuson-Im improves controllable text generation. Advances in Neural Information Process- ing Systems, 35:4328 4343, 2022.
Olivecrona, M., Blaschke, T., Engkvist, O., and Chen, H. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatics, 9, 09 2017. doi: 10.1186/s 1331-0077-023-
Li, Y., Zhang, L.,, and Liu, Z. Multi-objective de novo drug design with conditional graph generative model. Journal of cheminformatics, 10:1-24, 2018
10

---

LDMol: A Text-to-Molccule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
Oord, A. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint 1XX7:1807.0377 2018
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence, 45(4):4713-4726 2022.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J Bleu: method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. International Conference on Learning Representations, 2022.
Segler, M. H. S., Kogej, T., Tyrchan, C., and Waller, M. P. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS Central Science, 4: 120-131, 12 2017. doi: 10. 1000///cccentsi.7000000
Peebles. W. and Xie, S. Scalable diffusion models with transformers. In International Conference on Computer Vision, pp. 4195-4205, 2023.
Pei, Q., Zhang, W., J., Wu, K., Gao, K., Wu, L., Xia, Y., and Yan, R. Biot5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations. Empirical Methods in Natural Language Processing, 2023.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. International Conference on Learning Representations, 2021a.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Conference on Neural Information Processing Systems, 32, 2019.
Pei. Q.. Wu, L., Gao, K., Liang, X., Fang. Y., Zhu, J., Xie, S., Qin, T., and Yan, R. BioTS+: Towards generalized biolog- ical understanding with IUPAC integration and multi-task tuning. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 1216-1240, Bangkok, Thai- land, August 2024. Association for Computational Lin- guistics. doi: 10. 8653/v1/2024. http3//w//222..nnngg--aoo// findings-acl.71.
Song, Y., SohhlDickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021b.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, 1. Con- sistency models. Fortieth International Conference on Machine Learning, 2023.
Preuer, K., Renz, P., Unterthiner, T., Hochreiter, S., and Klambauer, G. Fréchet chemnet distance: a metric for generative models for molecules in drug discovery. Jour- nal of chemical information and modeling, 58(9):1736- 1741, 2018.
Staszak, M., Staszak, K., Wieszezycka, K., Bajek, A., Roszkowski, K., and Tylkowski, B. Machine learning in drug design: Use of artificial intelligence to explore the chemical strueture structure-biologien biological activity relationship. Wi- ley Interdisciplinary Reviews: Computational Molecular Science, 12(2):e 1568, 2022.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 1140):1-67, 2020.
Sterling, T. and Irwin, J. J. Zinc 15-ligand discovery for everyone. Journal of chemical information and modeling, (5(11):2324-233 2015.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024, pp. 10684-10695, 2022.
Su, B., Du, D., Yang. Z., Zhou, Y., Li, Rao, A., Sun, H., Lu, Z., and Wen, J.-R. A molecular multimodal foun- dation model associating molecule graphs with natural language. arXiv preprint arXiv:2209.0525 005411 2022.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. Medical Image Computing and Computer-Assiste Intervention-MiCCA. 2015: 18th International Confer- ence, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241, 2015.
Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., and Isola, What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6833 2020.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to- image diffusion models for subject-driven generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024, pp. 22500-226 22510, 2023.
Trippe, B. L., Yim, J., Tischer, D., Baker, D., Broderick, T., Barzilay, R., and Jaakkola, T. Diffusson probabilistic mod- eling of protein backbones 3d for the scaffoldan problem. International Conference on Learning Repre- sentations, 2023.

---

LDMol: A Text-to-Molccule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
Vahdat, A., Kreis, K., and Kautz, J. Score-based gener- ative modeling in latent space. Conference Neural Information Processing Systems, 34:1 34:11287-11302, 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E.,, and Polosukhin, I. Atten- tion is all you need. Advances in neural information processing systems, 30, 2017.
Wang, and Liu, H. Understanding the behaviour of con- trastive loss. In Proceedings of the IEEE/CVF conference computer vision and pallern recognition, pp. 2495- 2504, 2021.
Wang, S., Guo, Y., Wang. Y., Sun, H., and Huang. J. Smiles- bert: large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM in- ternational conference on bioinformatics. computational biology and health informatics, pp. 429-436, 2019.
Xu, M., Powers, A. S., Dror, R. O., Ermon, S., and Leskovec, Geometric latent diffusion models for 3d molecule gen- eration. In International Conference Machine Learn- ing, pp. 38592-38610. PMLR, 2023.
You, Y., Chen, T., Sui, Y., Chen, T., Wang. Z., and Shen, Y. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812-5822 2020.
Zeng. Z., Yao, Y., Liu, Z., and Sun, M. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable human professionals.
Nature communications, 13(1):862, 2022.
Zhang. J. and Ma, K. Rethinking the augmentation module in contrastive learning: Learning hierarchical augmen- tation invariance with expanded views. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16650- 16650-16659, 2022.
12

---

LDMol: A Text-to-Molccule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
A. Experimental details
A.1. DIT block for SMILES latent diffusion model with text conditions
The DiT block architecture of the class-conditioned image diffusion model published by Peebles & Xie (2023) is shown in Figure 6-(a). The noised input image latent 2, is passed through a patch embedding layer and spatially flattened be fed into the DiT block. The condition embedding y and diffusion timestep embedding / are incorporated into the model prediction via adaptive layer norm. The dimension of and both [B F), where B is the batch size and F is the number of features.
In the case of LDMol, the input latent with dimension [B L x F] is already spatially one mee-imensiional, we simply pass through a linear layer to prepare DiT' block input. Also, the text condition feature we'r ve used has a much higher dimension of [B L' F] where L' the token length of the text condition. Therefore, we stacked a cross-attention layer for text condition features after each self-attention layer, as shown in Figure 6-(b).
(8)
Output
Output (catures
features
Scale
Fandffrward
Fendforward
Scale Shift
Scale - Shift
LayerNorm
LayerNorm
Scale
Multi-
Shift
LayerNorm
Scale
Scale
Multi Head Self Attention
Multi Head Self Attencion
Scale Shift
Scale + Shift
LeyenNorm
LayerNorm
MLP
Inpor
t,y
Iopet
y
Patch Emocdding
Linear
Xy
Figure Input embedding layer and DiT blocks in the (a) originally published DiT and (b) LDMol.
A.2. Model hyperparameters and training setup
The LDMol encoder and decoder consist of 12 transformer layers BERT BERTiases base, where the decoder has a causal mask in its self-attention layers and includes cross-attention layer after each self-attention layer to receive latent information. Detailed hyperparameters on the model architecture are listed in Table with settings on the model training procedure. Here, we provide the results of several ablation studies support our selection of the hyperparameters.
It is known that lower temperature in contrastive learning penalizes hard negatives more strongly, making the learned feature more sensitive to fine-grain details (Wang & Liu, 2021). We considered this desirable property for our latent space and used small tau of 0.07. When we used too big of 0.15 as shown in Figure 7, it reduced the model's ability to distinguish different molecules and made the training loss converge to a much higher value.
Table 5 lists the L.Mol autoencoder's SMILES reconstruction accuracy in various autoencoder training strategies. When we
13

---

LDMol: A Text-te-Molccule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
Table 4. The choice of the model hyperparameters and training setup.
hyperparameters
L
128
dene
1024
dz
64
0.07
Q
16384
training setup
optimizer learning rate batch size per GPU training resources
autoencoder: Adam DiT: Adam
autoencoder: cosine annealing( 1e-4-+e-5), DiT: lc-4
encoder: 64, decoder: 128, DiT: 64
8 NVIDIA A1M0(VRAM:0GGB)
14
0=0.44
Encoder
8
Methods
Recon. Ace.
Contrastive learning compressed latent with d,=64
0.084
compression with d,=32
0.948
4
Contrastive learming on dem:=1024
compression with d,=64
0,980
2
compression with d,=128
0.989
2500
5000
7500 10000
17500 20000 Iterations
12500 15000
Table S. SMILES reconstruction accuracy of various trained autoencoders,
1,000 unseen SMILES.
Figure 7. The encoder loss convergence with different tem- perature parameter r.
apply contrastive loss directly into the compressed latent domain, the encoder fails capture informative features, makes the decoder couldn't reconstruct the input molecule. In the scenario of adding linear compression after contrastive training with de24= we observed an етrоr rate of more than 5% for the compression size of d.232. Compression with d,= =128 slightly increased the reconstruction accuracy compared to d, 64, but the training time for the subsequent diffusion model rapidly increased. Considering that the failed 2% for the current model were mostly very long molecules, we concluded that d, = is sufficient for our model.
A.3. LDMol's application on downstream tasks
Algorithm 2 Text-guided Molecule Editing with LDMol
Algorithm Molecule-to-Tex Retrieval with LDMol
Require: Zores Carc: Ctgt: N E N+ 7 0, 2 1,
Require: 2, C = [-]] E N
1: Initialize Ztgt Zare
1: Initialize Errors[c] = 0 for = 1 to B
2: for iter = to N do
2: for iter = to do
3: t t~ U T1. (0.
3: U|0, [], ~ 1),
4: 2t,rre V Th + v1 Tist
4: 2, = / V -I- arE
Zt,tt yai Eigtt + VI -I-a atE
5: for - to B do
6: Errors/ci += |1ão(21, cc1
6:
6g,orc (1 w) 1,2) wep t, Caro)
du
7:
C (1 (20, ())+ + we (2 t,, Cegt)
7: end for
8: Ztgt Ztgt 7(6 egt (.,
8: end for
9: end for
9: Return
10: Return D(zegt)
14

---

LDMel: A Text-to-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
"It's benzene ring with nätrogen..""
DIT
E
0.0038 (retrieved)
t-u(0,7)
"It's carbon chaán..."
2
DIT
Ea
11co
c11
0.1209
"It's amino acid._"
e-N((.1)
DIT
Eg
0. 0714
(b)
iterative optimization "It has a benzene ring...
DIT
Éggsre
Zarc
e (0,
ggt < Ztgt y(E εg,
t))
DIT
Zegs
Zr
"It has a pyridine ring.."
Figure 8. Overall pipeline for the downstream task applications of LDMol. (a) Molecule-to-text retrieval. (b) Text-guided molecule editing. The SMILES autoencoder and the text encoder are not drawn in this figure.
Figure 8-(a) and Algorithm show the LDMol's molecule- retrieval process with given query molecule and text candidates C = [c A given query molecule is converted to a latent z, and then forward noise process is applied with randomly sampled timestep t and noise e. This 2, is fed to LDMol with each candidate C,, and the candidate that minimizes the loss [ Fe, 1, c4) - <l12 between and the output noise (2 I, (.) is retrieved. To minimize the variance from the stochasticity and c, the same process can be repeated n times with resampled t and to use a mean loss.
Figure 8-(b) and Algorithm 2 illustrate the DDS-based molecule editing with L.Mol.. Specifically, DDS requires source data Zares target data Zigt which initialized to Zero. and their corresponding source and target text descriptions {Ceres Ctar). We apply the forward noise process 2are and Zigt using the same randomly sampled and get 2t,are and 21, These are fed into the pre-trained LDMol with their corresponding text, where we denote the output noise êg,sro and ég. Finally, Ztgt is modified towards the target text by optimizing it to the direction of (Ég.1gt - ég.e with learning rate Here, ég. and ég.egs can be replaced with the classifier-free-guidde free-guided noises, utilizing the output with the null text and the guidance scale w. Zigt is decoded back as the editing output after the optimization step is iterated N times. In Figure 5, where we applied the same scenario to a batch of molecules, we used null text as Care since it's impractical to prepare source prompt for each molecule. The hyperparameters {N, ), are fixed for each scenario, where every choice is in the range of 100 < N < 200, 0.1 < 7 S 0.3 and 2.0 < 4.5. Following MoleculeSTM, each scenario applied 200 randomly sampled molecules from ZINC15, and the mean and standard deviation three separate runs were plotted.
B. Additional results
B.I. Visualization of the LDMol latent space
To visualize the structural information encoded in the latent space of our encoder, we prepared 10 molecular clusters that contain 100 molecules, each sharing the common Murcko scaffold (Bemis & Murcko, 1996). Then, we obtained their
15

---

LDMol: A Text-te-Molccule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
(8)
(b)
UMAP Projection Colored by Murcko Scaffold
UMAP Projection Colored by Murcko Scaffold
25
scaffold o
20
scaffold 2
sceffold
h
scattold
15-
scaffold 5
UMWP2
UMMP
10-
scaffold
sceffold
4.
-10
-10
UMAP-1
UMAP-1
Figure 9. UMAP visualization of LDMol encoder output (a) and the final latent space after the linear compression layer (b), from 10 groups containing 100 molecules each with shared Murcko scaffold(colored) and 5,000 general molecules(light grey).
Validity
0.24
FCD
0.940
0.52
0.938
0.50
0.936
0.22
0.934
0.21
0.932
0.46
--- Exact Match
0.930
0.20
1
3
1
3
3
Classifier-Free Guidance Scale
Figure 10. Texl-lo-molecule generation performance LDMol against different classifier-free guidance scales.
latent vector from the LDMol encoder and visualized them in 2D via UMAP (McInnes et al., 2018). Note that As shown in Figure 9, the molecules with shared Murcko scaffold have formed clusters in the latent vector space.
B.2. Effect of classifier-free guidance scale
Figure 10 plots the LDMol's texl-to-molecul generation performance on the ChEBI-20 test set, with different classifier-fie frree guidance scale w in the sampling process. Starting from w 1.0, which equivalent to a naive conditional generation, we observed that the overall sample quality improved as w increases but collapses for too big w. This agrees with the well-known observation on image diffusion models, and we decided use 2.5 for the texl-to-molecule generation with LDMol.
B.3. Text-to-molecule generation
We measured the uniqueness, novelty, and prompt alignment score for the prompts in Figure using 1,000 samples. Validity the proportion of generated SMILES that are valid. Uniqueness the proportion of valid SMILES that are unique. The "align" score is the proportion of unique SMILES that match the given prompt. Novelty is the proportion of the unique SMILES that are not included in the training dataset. The alignment score was measured with SMILES pattern matching with the substructure described by the prompt. We observed that even when stochastic sampling was enabled, AR models struggled to generate various samples from a single prompt. LDMol can generate molecules that align better with various hand-written prompts. Furthermore, its outputs were much more diverse than the previous AR models.
Figure shows the behavior LDMolls mexlelee generation with several exceptional scenarios. When we fed
16

---

LDMel: A Text-to-Molccule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
Table 6. Quantitative results of the case studies in Figure 4. The best performance for each metric written in bold.
Models
Validity(V)
Uniqueness(U)
Align(A)
VxUxA
Novelty
molTStarge
0.996
0.006
1.000
0.006
Case (a)
bioTS+
0.846
0.028
0.625
0.015
LDMol
0.910
0.951
1.000
0.865
0.988
mol
0.927
0.012
0.818
0.009
Case (b)
bioTS+
1.000
0.573
0.782
0.448
LDMol
0.989
0.960
0.906
0.860
0.958
molT3
0.783
0.072
0.643
0.036
Case (c)
bioT5+
1.000
0.160
0.750
0.120
L.Mol
0.955
0.861
0.688
0.566
0.780
molT5
0.995
0.002
0.500
0.001
Case (d)
bio
1.000
0.015
0.733
0.011
L.Mol
0.956
0.849
0.703
0.571
0.842
mol13
0,956
0.015
0.571
0.008
Case (c)
bioTS+
1.000
0.035
0,086
0,003
LDMol
0.996
0.187
0.595
0.111
0.667
Table 7. Quantitative results of the ablation study. The best performance for each metric is written bold.
Autocncoder
ChEB1-20
text-to-molecule generation
models
Recon. Acc.f
Validity
Matchf
FCDJ
LDMol w/o compression layer
0.964
0.022
0.000
67.93
LDMol w/ transformer compression layer
0,986
0.565
0.084
2.19
LDMol w/o stereoisomer hard-negative
0.891
0.939
0.278
0.24
LDMol
0.983
0.941
0.530
0.20
completely ambiguous input such as "beautiful" or "important", the model spits out a variety of different molecules without any consistency. When we fed contradictory inputs that could not be satisfied, the outputs were chimeric between contradictory prompts, with a clearly decreased validity.
B.4. Molecule-to-text retrieval
Figure 13 contains examples of molecule-to-ted retrieval results with molecules from the PCdes test set. The retrieval was done at the sentence level, and the top three retrieval outputs for each query molecule are described. The corresponding description from the data pair was correctly retrieved first for all cases, and the other retrieved candidates show a weak correlation with the query molecule.
B.5. Text-guided molecule editing
Figure 12 illustrates several case studies with hand-writter editing prompts and results, where the editing output successfully modified the input molecule towards the target prompt with minimal corruption of the unrelated region. Here, we repeated DDS iterations with N 150, 0.1 and w 2.5.
B.6. A blation study
We've conducted an ablation study on more detailed design choices of the proposed LDMol in Table analyze and emphasize their role.
When we didn't t introduce a compression layer, the later diffusion model completely failed to learn the latent space since its dimension was too big for the diffusion model to learn. We tried to utilize a more complex compression module by transformer encoder layers Perceiver-Be Resemple (Alayrac et al., 2022; Lovelace et al., 2024) manner, but the performance was significantly decreased as shown in the second row. This is presumably because adding another complicated layer makes the latent space deviate from the former informative and well-regulated learnable space.
17

---

LDMol: A Text-to-Molccule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
Important
This molecule bas pyrazine ring.
This molecule has pyrazine ring and acctyl group.
N
Validley:: 0.948
Nalidiey: 0.918
conjugeec tryptophin, conjogatc thrveninstle.
This molecule contains fluorine. contains bromine.
Trecophun
"de Thromen
molecule carhoxylic acid. This molecule
Validity:
Figure Examples the generated molecules by LDMol, with (a, b) ambiguous text conditions and (c) contradictory and unrea- sonable input.
Figure 12. Examples text-guided molecule editing with LDMol. The difference between the source text and the target text, and the corresponding region, is colored in purple.
When stereoisomers were not utilized as hard negative samples in the contrastive encoder training, the constructed latent space was not detailed enough to specify the input, which degraded the reconstruction accuracy of the autoencoder. The similarity metric of FCD didn't decrease as much, but the exact match ratio has decreased significantly.
B.7. Computational efficiency
Table 8. Quantitative results of the ablation study.
Models
molTSStagge
bioT5+
LDMol
Required time[s]
523
180
361
VRAM usage(GB]
4.92
1.08
3.79
Table 8 compares the computational efficiency of LDMol and several baselines with state-of-the-art performance. In terms of memory usage, our model can operate with less than 4GB VRAM, which is smaller than that of molTS The required time was also comparable transformer-baseee models, even with the latent decoder and the Classificr-Fre Guidance(CFG) which doubles the diffusion model usage. Considering many works have been published reduce the inference time of diffusion models (Song et al., 2023; Salimans & Ho, 2022), one of the first successful texl-to-molecule diffusion models, we believe that the inference time can be further improved by future research.
18

---

LDMol: A Text-te-Molecule Diffusion Model with Structurally Informative Latent Space Surpasses AR Models
Query malecule
monocarboxylic acid anion oxo fatty acid anion.
A strong dicarboxylic acid occurring muny plants and vegetubles.
3. It is tautomer in zwitterion.
Query molecule
Dihydrourocanic acid monocarboxylle been replaced by imidazol-4-yl group.
propionic acid which hydrogens position
bas been found in human liver and kidney tissues, and has also been detected multiple biolluids, such urine and blood.
3. It is soluble in water.
Query
1. Outside of the human body, 2, loxirrrt can be found in herbs and spices.
Used nematocide insecticide.
from propan-2-ol.
Query molscule
1. It is member of the class of mebendazole in which the benzoyl group is replaced by p-fluorobenzoyl group.
2. It has role as a prodrug.
derives from an acctic acid.
Figure 13. The examples of molecule-to-text retrieval result the PCdes Three sentences with the lowest noise estimation were retrieved for each query molecule.
19


Reviews:
[{"review_id": "xuQ0nWRrMW", "reviewer": "Reviewer_YcAV", "summary": "- This paper proposes a SMILE-based latent diffusion method for text-driven molecule generation task.\nThis paper augments the data by aligning enumerated SMILES with the traditional contrastive learning approach at the pretraining stage to ask the model to learn the invariant features from SMILES.\nWith the pretrained molecule encoder, a decoder that can successfully reconstruct the molecule, and a frozen text encoder, this paper trains a latent diffusion model that shows competitive performance with other Auto-regressive SMILE-based models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer YcAV for the constructive feedback. Below, we provide point-by-point responses on your questions and concerns.\n\n**[YcAV] asked for the statistics of the enumerated SMILES.**\n\nPlease note that the SMILES enumeration we’ve introduced is done by different SMILES constructions of the same molecule, and does not modify the given molecule. Therefore, the molecule size, atoms, bonds, etc., are the same as in the initial dataset PubChem, which contains a wide range of general molecules. We used 10 million randomly sampled molecules from PubChem to construct contrastive latent space, where the SMILES length over 500 were removed. No paired text descriptions were used in this training phase.\n___\n\n**[YcAV] requested the analysis and ablation study on the effect of contrastive learning on the generation task.**\n\nPlease refer to the revised ablation studies below. The first two rows (a) and (b) show the model performance by replacing contrastive learning-based latent space into two common latent space construction strategies of naive reconstruction and KL-regularized autoencoder. The latent space without any regularization was unable to learn by the diffusion model, and the latent space from beta-VAE had suboptimal performance with notably low validity.\n\n| Models  | Latent space construction | stereoisomer hard-negatives | laten space compression | AE recon. acc.[↑] | ChEBI-20 Validity[↑] | ChEBI-20 Match[↑] | ChEBI-20 FCD[↓] |\n|---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| LDMol | Contrastive loss | O | Linear | 0.983 | **0.941** | **0.530** | **0.20** |\n| (a) | None | (N/A) |  | **1.000** | 0.019 | 0.000 | 58.6 |\n| (b) | KL regularization | (N/A) |  | 0.999 | 0.847 | 0.492 | 0.34 |\n| (c) |  | X |  | 0.891 | 0.939 | 0.278 | 0.24 |\n| (d) |  |  | None | 0.964 | 0.022 | 0.000 | 67.9 |\n| (e) |  |  | transformer layers | 0.986 | 0.565 | 0.084 | 2.19 |\n\nAlthough we appreciate the reviewer’s suggestion, our latent construction of contrastive learning is currently inapplicable to AR models since they do not utilize any latent domain. It might be possible to apply contrastive loss into intermediate feature space similar to [1], but we believe this is beyond the scope of our work.\n___\n\n**[YcAV] asked for further analysis on the compression layer**\n\nSince the latent space compression was done by a single linear layer, its role is to reduce the dimension for the diffusion modeling rather than the extra curation of the features. Please refer to the UMAP[1] latent visualization in the following Figure 1 [[LINK]](https://docs.google.com/presentation/d/1UV4vMpHT4hrGa1I9PdswLgZ9PJ7vlPH22xbgFKpfo1U/edit#slide=id.p), where the feature space is structurally informative before and after the compression layer. \n\nAlso, we consistently observed that the role of the compression layer should be minimized, as we had to leverage the smooth and regulated latent space from the contrastive learning. Please note that the model performance degraded as the diffusion model’s target domain deviates from the contrastive latent space, even when the compression layer capacity is increased(see row (e) of the table above).\n___\n\n**[YcAV] suggested further clarification and examination of the SMILES enumeration.**\n\nLonger SMILES have more enumerated SMILES pairs as the reviewer noted, and we randomly selected one possible enumeration for each pre-training epoch. Therefore, the number of enumeration pairs for each training data that the model encountered are mostly equal, except for the molecules that are extremely small. Even fixing the single enumeration for each training data was enough for the encoder to learn the enumeration-invariant features, as we show the histogram of enumerated SMILES pair distances in the following Figure 2 [[LINK]](https://docs.google.com/presentation/d/1UV4vMpHT4hrGa1I9PdswLgZ9PJ7vlPH22xbgFKpfo1U/edit#slide=id.g3473ce19713_0_1).\n\n___\n\n[1] UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, arxiv 2018.\n\n[2] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think, ICLR 2025", "claims_and_evidence": "- The assumption of previous methods treated enumerated SMILE differently is well proved in Figure 3 (b) which shows the distance gets diminished by contrastive learning.\n- The statistic of enumerated SMILEs needs to be discussed, need some introduction on the original datasets and the augmented datasets. For instance, what are the sizes of molecules? What are the elements and function groups of molecules? What are the lengths and the informative text descriptions? How are they aligned and how many enumerated SMILE pairs on average?\n- The assumption that contrastive learning would help the model to generate better molecules should be further discussed and needs an experiment on the final downstream tasks. Even if Figure 3 (b) shows the distance between enumerated SMILES was large without contrastive learning, it may still need to show the performance under original settings. If the main contribution is introducing contrastive pre-training, then authors may need to try the proposed method on those Auto-regressive models for showing better performance. It is hard to conclude whether the competitive result is from pre-training, compression layers, or diffusion models.", "methods_and_evaluation_criteria": "Yes, authors follow the previous works such as MolT5. They compared the performance among Auto-regressive models under same metrics.", "theoretical_claims": "Yes, I have checked the theoretical part. The major theoretical claim is that contrastive learning will help the model learn invariant features from SMILES, and it is well stated by Figure 3.", "experimental_designs_or_analyses": "I have checked the experiment design. Related concerns:\nThe paper should include a comprehensive evaluation of the model's final performance on downstream tasks within the experimental section. Simply claiming that contrastive learning benefits the final-stage performance without providing empirical evidence is insufficient. A quantitative comparison with baseline methods is necessary to demonstrate the advantages of the proposed approach.\n- The current ablation study lacks clarity and depth. While the compression layer is not presented as the primary contribution of the paper, it evidently plays a crucial role in downstream task performance. The authors should conduct a more thorough investigation into the function of this layer by analyzing the latent space representations. Providing insights into how the compression layer influences model performance would significantly improve the final performance.\n- The dataset augmentation should be explicitly examined in the experiments. Since longer SMILES strings naturally generate more enumerated pairs, is it necessary to use all of them? If not, what is the optimal amount of augmented data required for effective training. Specifically, the authors should investigate how varying the amounts of augmented SMILES pairs to abstract away from explicit SMILES grammar constraints, or is there still be grammar things left in the model?", "relation_to_broader_scientific_literature": "- The major contribution of this paper is proposing a pretraining stage with augmented dataset and contrastive learning to help model learn in-variant features from SMILE.\n- The second contribution is as one of the early latent diffusion methods in text-driven molecule generation, it shows comparable performance to AR-based methods.", "essential_references_not_discussed": "- As the one of the early latent diffusion methods in text-driven molecule generation, the related references are included.", "other_strengths_and_weaknesses": "__Strengths:__\n- Clearly written, easy to follow, and understand the paper, the figures are very clear and helpful for understanding.\n- Originality: the thought of data augmentation and contrastive learning are simple and implemented in the unconditional molecule generation before, but the originality of this paper is enough for text-driven molecule generation at this early stage.\n\n__Weakness:__ please check the __Experimental Designs Or Analyses__ and __Claims And Evidence__.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please check the __Experimental Designs Or Analyses__ and __Claims And Evidence__.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aLh7gBS8yZ", "reviewer": "Reviewer_bRG5", "summary": "The paper proposes a latent diffusion model for text-conditioned molecule generation. The authors claim that their primary innovation lies in introducing a contrastive learning approach to capture molecular structural features from SMILES sequences. In tasks such as molecule-to-text retrieval and text-guided molecule editing, this method demonstrates certain improvements compared to autoregressive-based approaches.\n\n## update after rebuttal\nI have read all the feedback and raised my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for your detailed comments and valuable suggestions. Below, we provide thorough point-by-point responses to address your concerns. \n\n**[bRG5] asked how the proposed contrastive learning can learn the structural differences.**\n\nWe would like to remind the reviewer that in order to minimize the proposed contrastive loss, the feature cosine similarity from the different molecules in the batch needs to be minimized, which enables the encoder to reflect structural differences into their latent output. Please refer to the UMAP[1] visualization of LDMol’s latent space in the following Figure 1 [[LINK]](https://docs.google.com/presentation/d/1UV4vMpHT4hrGa1I9PdswLgZ9PJ7vlPH22xbgFKpfo1U/edit#slide=id.p), where the molecules with shared Murcko molecular scaffold[2] locate closer in the latent space.\n___\n**[bRG5] asked how our latent space construction helps the alignment with the text conditions.**\n\nCompared to the raw SMILES tokens or structure-unaware latent from $\\beta$-VAE(Figure 3-(b)), our encoder provides a latent where the proximity is more structurally meaningful, hence more “interpretable” for the later diffusion model that needs to link the text information. Indeed, fingerprint similarities and FCD on the ChEBI-20 dataset were improved by 7~15%p and 40% from the previous baselines, meaning that LDMol made a reasonable guess even when it was incorrect.\n___\n**[bRG5] expressed concern about the claim on the model performance.**\n\nWhile there were many “text-conditioned” chemical diffusion models, we clarify that the term “textual data” generation refers to the text-like generation target of SMILES; most successful chemical diffusion models had targeted graph or point-cloud data, and diffusion models that generate SMILES[3] showed suboptimal performance compared to AR models. We insist that we’ve shown the potential of improving diffusion model performances on text-like data with carefully designed latent space.\n___\n**Text-guided diffusion models as baselines.**\n\nPlease note that TGM-DLM[4] is included as a text-guided molecule diffusion model baseline (line 340). TGM-DLM trained the diffusion model by naively treating the token index as a continuous real value, which led to a severe performance drop compared to LDMol.\n___\n**Ablation study on contrastive learning.**\n\n Please refer to the revised ablation studies that we included in the response to the reviewer `YcAV` above. To replace our suggested latent space construction, we tested two common strategies of naive reconstruction and KL-regularized autoencoder. The latent space without any regularization was unable to learn by the diffusion model, and the latent space from $\\beta$-VAE had suboptimal performance with notably low validity.\n___\n**[bRG5] asked the prompt alignment score in the case studies.**\n\nPlease refer to the table in the response to the reviewer `mgY4` below, where we measured the prompt alignment score for the prompts we’ve tested using 1,000 samples. The result demonstrates that LDMol showed better and more consistent alignment along various hand-written prompts compared to the AR baselines.\n___\n**[bRG5] asked for a clearer description of the molecule editing process.**\n\nOur text-guided molecule editing was done similarly to Delta Denoising Score(DDS), where the input data is optimized to match the target prompt by minimizing the difference between the model-predited noise with (source_prompt, source_data) pair and (target_prompt, target_data) pair. Please note that Supplementary material A.3 contains a detailed description and pseudocode of our DDS-based test-guided molecule editing.\n___\n**The architecture of the encoder.**\n\nThe encoder we’ve used consists of 12 bidirectional transformer layers of BERT$_{base}$, with a feature size of 1024 and 16 attention heads. We thank the reviewer for the suggestion, and we’ll include this information in Supplementary material A.2.\n___\n**[bRG5] questions the statement of minimizing mutual information.**\n\nThe mentioned statement describes our contrastive latent space construction with SMILES enumeration, compared to previously suggested molecular contrastive learning methods[4]. \nSince SMILES enumeration provides all possible variations under the SMILES grammar, the enumerated SMILES pair has minimal mutual information(i.e., the connectivity of atoms and bonds) compared to simple or local augmentations. As a result, while most contrastive learning focuses on “extracting” certain desired features, our contrastive learning can “fully preserve” the structural information as the augmentation invariant to become an autoencoder. \n___\n\n[1] UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, arxiv 2018.\n\n[2] The Properties of Known Drugs. 1. Molecular Frameworks, Journal of Medical Chemistry 1996. \n\n[3] Text-Guided Molecule Generation with Diffusion Language Model, AAAI 2024.\n\n[4] Graph contrastive learning with augmentations. NeurIPS 2020.", "claims_and_evidence": "Some of the claims in this paper are far-fetched. I have outlined specific comments in the weaknesses section.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "This paper has no theoretical proofs.", "experimental_designs_or_analyses": "Some baselines lack comparative analysis. I have detailed specific comments in the weaknesses section.", "relation_to_broader_scientific_literature": "The key contribution of this paper stems from the latent diffusion framework in the field of image generation.", "essential_references_not_discussed": "Several similar text-conditioned generative models based on diffusion frameworks have already been proposed. I have listed the relevant works in the weaknesses part.", "other_strengths_and_weaknesses": "**Strengths:**\n1. The authors’ attempt to address text-conditioned molecule generation is a worthwhile and exploratory field.\n2. The authors’ use of different SMILES sequences of the same molecule for contrastive learning is a novel approach.\n\n**Weaknesses:**\n1. Some claims in the paper are far-fetched:\n\n   a) The authors claim that the SMILES encoder can learn molecule structure information, but the proposed contrastive learning strategy relies more on whether the SMILES sequences originate from the same molecule. Structural differences are not explicitly modeled or learned.\n\n   b) The authors claim, \"By preparing an encoder to provide a chemically useful and interpretable feature space, our model can more easily connect the molecule data with the highly complicated condition of natural texts.\" How does the encoder trained with the proposed contrastive learning strategy achieve easier alignment with the text space? Additionally, the term \"interpretable\" lacks detailed explanation and validation.\n\n   c) The authors claim they proposed \"the first diffusion model that successfully surpassed autoregressive models in textual data generation.\" Such models are not novel, as many diffusion models have already been proposed for text-guided molecule generation, such as:\n\n      [1] Periodic Materials Generation using Text-Guided Joint Diffusion Model. ICLR 2025.\n\n      [2] Text-Guided Molecule Generation with Diffusion Language Model. AAAI 2024.\n\n      [3] Hierarchical Graph Latent Diffusion Model for Conditional Molecule Generation. CIKM 2024.\n\n      [4] Text-guided small molecule generation via diffusion model. iScience, 2024, 27(11).\n\n2. The experimental results are insufficient:\n\n   a) The authors only compared autoregressive models. Text-guided diffusion models should also be included as baselines.\n\n   b) An ablation study on the contrastive learning component is necessary.", "other_comments_or_suggestions": "1. The examples in Figure 4 only present the validity of the conditionally generated molecules. Whether these molecules meet the expectations of the given conditions should also be showcased. \n\n2. The description of the text-guided molecule editing task is not sufficiently clear. The authors only refer to the DDS method. A brief outline of the steps should be included in the paper.  \n\n3. The model architecture of the encoder should be presented more clearly. At least, an architectural framework should be provided in the appendix.", "questions_for_authors": "1. How to understand the statement \"we suggest a novel contrastive encoder learning strategy by minimizing mutual information between positive SMILES pairs\"?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "skIOhxrj2N", "reviewer": "Reviewer_qAD8", "summary": "This paper proposes a text-conditioned molecule (SMILES) generation model based on latent diffusion. LDMol learns a structurally informative latent space through contrastive learning, and surpasses AR models. The authors claim LDMol is the first diffusion model outperforms AR models in text-to-mol generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and for considering our work, and we appreciate your time and evaluation. Below we provide point-by-point responses on your questions and concerns.\n\n**[qAD8] asked for visualization of the latent space and its structural information.**\n\nTo visualize the structural information encoded in the latent space of our encoder, we prepared 10 molecular clusters that contain 100 molecules, each sharing the common Murcko scaffold. Then, we obtained their latent vector from the LDMol encoder and visualized them in 2D via UMAP[1]. We also plotted the latent vector of 5,000 randomly sampled molecules. As shown in the following Figure 1 [[LINK]](https://docs.google.com/presentation/d/1UV4vMpHT4hrGa1I9PdswLgZ9PJ7vlPH22xbgFKpfo1U/edit#slide=id.p), the molecules with shared Murcko scaffold[2] have formed the clusters on the latent vector space.\n___\n**[qAD8] expressed a concern on the text-to-molecule generation performance.**\n\nWe kindly note that we had discussed that MolXPT and bioT5 had higher validity than ours in the manuscript(line 307\\~310, right column), yet they shorted in every metric for the similarity between the ground truth. We insist that the primary role of a text-to-molecule model is to generate a molecule that meets the user prompt, thus, similarity metrics should be prioritized when evaluating models.\n\nWe agree that character-wise metrics like BLEU and Levenshtein distance are not appropriate for SMILES generation tasks, and we included them as a convention for many studies in this benchmark. LDMol still showed the major performance improvement from the previous AR models on chemical similarities, including a 6.7~15.2%p increases in fingerprint similarities and the state-of-the-art exact match ratio of 0.530 in ChEBI-20 benchmark.\n\nFor the case study in Figure 4, we measured the uniqueness, novelty, and the prompt alignment score using 1,000 samples. The alignment score was measured with pattern matching with the structure described by the prompt. Please refer to the table in the response to the reviewer `mgY4` below. It is shown that LDMol can generate molecules that match with various hand-written prompts, and its output is more diverse than the previous AR models.\n\n___\n**[qAD8] questioned the usefulness of molecule-to-text retrieval and suggests text-to-molecule retrieval.**\n\nWe want to emphasize that the primary contribution of LDMol is in the text-to-molecule generation, and the downstream tasks were done to demonstrate its applicability as a diffusion model compared to AR-based generative models. That said, we performed a paragraph-level text-to-molecule retrieval task on the PCdes test set and the MoMu dataset, measuring 64-way accuracy. Here, we had to estimate the ELBO of each query molecule with given text input using noise prediction error across different noise levels, and the query with the highest likelihood was selected. Although the exact calculation for ELBO would take 1,000 noise predictions per pair, LDMol already showed comparable performance with the baseline representation models with a rough estimation of NFE=25 while being a successful generative model at the same time. \n\n|  | mol-to-text(PCdes) | mol-to-text(MoMu) | text-to-mol(PCdes) | text-to-mol(MoMu) |\n|---:|:---:|:---:|:---:|:---:|\n|SciBERT|62.6|1.38|61.8|1.56|\n|KV-PLM|77.9|1.51|77.0|1.60|\n|MoMu-S|80.6|45.7|80.2|46.0|\n|MoMu-K|81.1|46.2|81.4|45.7|\n|MoleculeSTM|81.4|67.6|78.9|64.1|\n|MolCA|86.4|73.4|**84.8**|72.8|\n|LDMol(n=25)|**90.3**|**87.1**|83.3|**74.0**|\n___\n**[qAD8] questioned the usefulness of text-to-molecule generation.**\n\nAs demonstrated by the applications of LLM on various data domains[3][4], natural text is a modality that enables the incorporation of various conditions like molecular properties, interactions, etc. This makes the generative model utilizing text conditions more applicable and expandable compared to the models trained with specific condition types. While its pratical utility still needs to be improved for real-world usage, text-conditioned molecule generation has received a growing interest as we noted in Related Works, and LDMol achieved the state-of-the-art performance over the baselines.\n___\n\n[1] UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, arxiv 2018.\n\n[2] The Properties of Known Drugs. 1. Molecular Frameworks, Journal of Medical Chemistry 1996. \n\n[3] TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios, arxiv 2024.\n\n[4] Can Language Models Solve Graph Problems in Natural Language?, NeurIPS 2023.", "claims_and_evidence": "1. LDMol verifies the potential of latent diffusion in molecule generation\n2. Through comtrastive learning, LDMol learns a chemically informative latent space, which is important for molecule LDM.", "methods_and_evaluation_criteria": "See experiment review", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. Structure-aware SMILES latent space with SMILES contrastive learning.\nStructure-awareness seems an overclaim. The author can visualize the learned latent representations and see if there are meaning clusters reflecting certain chemical structures.\n2. Text-conditioned molecule generation.\nLDMol's validity is worse than bioT5, which is not discussed. BLEU score is not a good metric for molecule SMILES generation, and the exact match scores are still low. The conclusions associated with Fig 4 seem overclaimed if there is no further quantitative evidence.\n3. Molecule-to-text retrieval is not useful.\nThe reviewer concerns LDMol can truly align narrative description and chemical structures. Text-to-molecule retrieval seems more helpful in real world applications. Can the authors try text-to-molecule retrieval settings?", "relation_to_broader_scientific_literature": "In real world applications, structure- or function-conditioned molecule generation/retrieval is more helpful, especially in drug discovery or de novo design. For example, virtual screening, affinity prediction, drug design.\n\nGao, Bowen, et al. \"Drugclip: Contrastive protein-molecule representation learning for virtual screening.\" Advances in Neural Information Processing Systems 36 (2023): 44595-44614.\nWang, Renxiao, et al. \"The PDBbind database: methodologies and updates.\" Journal of medicinal chemistry 48.12 (2005): 4111-4119.\nLuo S, Guan J, Ma J, et al. A 3D generative model for structure-based drug design[J]. Advances in Neural Information Processing Systems, 2021, 34: 6229-6239.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weakness:\n1. Is text-to-SMILES generation really useful? Can you provide some real world cases?\n2. See experiment review.", "other_comments_or_suggestions": "No", "questions_for_authors": "Pairwise translation seems easier to hack than retrieval. How to guarantee the model truly aligns the narrative and chemical semantics instead of memorizing some prompts and SMILES tokens? Why not text-2-mol retrieval, which is harder but more useful than text-2-mol generation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XCc9AvLPZD", "reviewer": "Reviewer_mgY4", "summary": "The authors introduce LDMol, a latent diffusion model for generating molecules based on text inputs. By integrating a chemically informed autoencoder and utilizing contrastive learning, their proposed diffusion model ensures structural consistency in the latent space, addressing the problem of multiple SMILES representations for the same molecule. Experimental results show that LDMol outperforms existing models in terms of SMILES validity, structural accuracy, and adherence to conditions. In addition, their LDMol is versatile, supporting tasks such as molecule-to-text retrieval and text-guided molecule editing, and provides a structurally aware diffusion approach as a compelling alternative to autoregressive models in molecular generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate Reviewer mgY4 for the review and thoughtful feedback. Below, we provide detailed point-by-point responses to address your remaining concerns.\n\n**[mgY4] asked for additional metrics on text-to-molecule generation.**\n\nAs the reviewer suggested, we measured the uniqueness, novelty, and prompt alignment score for the prompts we’ve tested using 1,000 samples. Validity is the proportion of generated SMILES that is valid. Uniqueness is the proportion of valid SMILES that are unique.  The “align” score is the proportion of unique SMILES that match the given prompt. Novelty is the proportion of the unique SMILES that are not included in the training dataset. The alignment score was measured with pattern matching with the substructure described by the prompt. \n\nWe observed that even when stochastic sampling was enabled, AR models struggled to generate various samples from a single prompt. LDMol can generate molecules that align better with various hand-written prompts. Furthermore, its outputs were much more diverse than the previous AR models.\n\n| | Models  | Validity(V) | Uniqueness(U) | Align(A) | VxUxA | Novelty |\n|---:|---:|:---:|:---:|:---:|:---:|:---:|\n|Case (a)|molT5|**0.996**|0.006|**1.000**|0.006|N/A|\n| |bioT5+|0.846|0.028|0.625|0.015|N/A|\n| |LDMol|0.910|**0.951**|**1.000**|**0.865**|0.988|\n|Case (b)|molT5|0.927|0.012|0.818|0.009|N/A|\n| |bioT5+|**1.000**|0.573|0.782|0.448|N/A|\n| |LDMol|0.989|**0.960**|**0.906**|**0.860**|0.958|\n|Case (c)|molT5|0.783|0.072|0.643|0.036|N/A|\n| |bioT5+|**1.000**|0.160|**0.750**|0.120|N/A|\n| |LDMol|0.955|**0.861**|0.688|**0.566**|0.780|\n|Case (d)|molT5|0.995|0.002|0.500|0.001|N/A|\n| |bioT5+|**1.000**|0.015|**0.733**|0.011|N/A|\n| |LDMol|0.956|**0.849**|0.703|**0.571**|0.842|\n|Case (e)|molT5|0.956|0.015|0.571|0.008|N/A|\n| |bioT5+|**1.000**|0.035|0.086|0.003|N/A|\n| |LDMol|0.996|**0.187**|**0.595**|**0.111**|0.667|\n___\n\n**[mgY4] suggested the potential expansion toward different types of data and conditions.**\n\nWe appreciate the reviewer’s expectation for the potential applicability of our approach towards the other types of data, and we also regard the expanding of carefully designed latent diffusion models as possible future works. As the reviewer suggested, this can be applicable to both different target chemical data domain itself or various biochemical conditions.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Yes", "essential_references_not_discussed": "Yes", "other_strengths_and_weaknesses": "Strengths:\n1) The proposed model shows its effectiveness in text-guided molecule generation, which highlighting its potential applications across different areas of chemical and biomedical research.\n2) The source code is available, and the paper is well-written.\n3) The experiments are evaluated using a variety of measures and experimental results are compared with several baseline models.\n\nWeaknesses:\n1) In addition to the existing metrics, it would be valuable to include other statistical measures, such as uniqueness and novelty, to provide a more comprehensive evaluation of the model's performance and its ability to generate innovative and diverse results. \n2) The model demonstrates strong performance on the datasets, its ability to generalize to other types of chemical data or a wider range of text inputs warrants further investigation. Expanding the model's testing to diverse datasets and inputs would provide a clearer understanding of its versatility and robustness across different chemical and textual domains.\n3) The generation of molecules from text descriptions has been explored in numerous other studies. It would be valuable to investigate whether the model can be extended to handle other types of data, such as molecular graphs or motifs. Expanding the model's capabilities to incorporate these additional data formats could significantly enhance its versatility and enable more comprehensive molecular design, potentially improving its performance in a wider range of applications.", "other_comments_or_suggestions": "See above", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper uses existing SMILES enumerations to improve a representation via contrastive learning that's to be used in a latent diffusion to generate molecules conditioned on text. While a straightforward assembly of techniques already in the literature, the results look strong enough for this paper to be accepted. The main weakness was the writing was often vague and seemed like it overclaimed as noted by several of the reviewers. The authors should address this writing issue and work in some of the additional results and ablations in the author response in the final version of the paper.
Decision: Accept (poster)