3M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecular Structure Generation

Authors: huaisheng  zhu
teng  xiao
vasant g honavar
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 05:47:57

Abstract: Generating molecular structures with desired properties is a critical task with broad applications in drug discovery and materials design. We propose 3M-Diffusion, a novel multi-modal molecular graph generation method, to generate diverse, ideally novel molecular structures with desired properties.  3M-Diffusion encodes molecular graphs into a graph latent space which it then aligns with the text space learned by encoder-based LLMs from textual descriptions. It then reconstructs the molecular structure and atomic attributes based on the given text descriptions using the molecule decoder. It then learns a probabilistic mapping from the text space to the latent molecular graph space using a diffusion model. The results of our extensive experiments on several datasets demonstrate that 3M-Diffusion can generate high-quality, novel and diverse molecular graphs that semantically match the textual description provided.

Main paper: 
 introduction: 
 Generating molecular structures with the desired properties is a critical task with broad applications in drug discovery and materials design (Hajduk & Greer, 2007; Mandal et al., 2009; Pyzer-Knapp et al., 2015) . There is a growing interest in generative models to automate this task (You et al., 2018; Jin et al., 2018; 2020; Bjerrum & Threlfall, 2017) . Kusner et al. (2017) proposed a variational autoencoder that uses parse trees for probabilistic context free grammars to encode discrete sequences that specify the Simplified Molecular-Input Lineentry System (SMILES) (Weininger, 1988) notation of molecules use the resulting parse trees to generate SMILES codes that describe molecular graphs. Edwards et al. (2021) formulated molecule retrieval from textual description as a cross-lingual retrieval task.Recent advances in language models have led to innovative approaches to generate molecular structures that match human-friendly textual descriptions of their properties, substructures, and biochemical activity (Edwards et al., 2022; Zeng et al., 2022; Zhao et al., 2023b; Fang et al., 2023a) . These approaches learn mappings from textual descriptions to molecular structures using single transformer-based cross-lingual language model for molecule generation. Specifically, with the given text description of a molecule, these approaches produce SMILES strings that encode the molecular structure.Despite promising initial results, such approaches suffer from some important limitations: (i) SMILES encoding is degenerate in that the a given molecular structure can have multiple SMILE encodings. For example, the structure of ethanol can be specified using CCO, OCC or C(O)C. Although in principle a unique canonical SMILES representation can be produced for a given molecular structure, the procedure used to do so relies on several arbitrary choices. Not surprisingly, molecules with identical structure may map to different SMILES strings. This makes SMILES notation less than ideal for generating molecules from their textual descriptions (Jin et al., 2018) . Important chemical properties corresponding to the substructures of molecules are more straightforward to express using molecular graphs rather than linear SMILES representations (Jin et al., 2018; Du et al., 2022) ; (ii) Using language models to generate diverse and high-quality molecular structures matching a description is challenging, because the process used to decode the learned representation into the corresponding molecular structure, e.g., beam search or greedy search, often produces structures that are often too similar to each other (See Figure 2 ). Hence, there is an urgent need for better approaches for generating diverse, novel molecular structures from a given textual description (Brown et al., 2019) . Against this background, this paper aims to answer the following research question: How can we generate a diverse and novel collection of molecular graphs that match a given textual description? We introduce 3M-Diffusion, a novel multi-modal molecular diffusion method, for generating high-quality, diverse, novel molecular structures from a textual description.Inspired by text-guided image generation using a diffusion model on the latent space (Rombach et al., 2022) , 3M-Diffusion is trained to generate molecular structures from the latent distribution of a molecular graph autoencoder (See Figure 1 ). This enables the diffusion process to focus on the high-level semantics of encodings of molecular structures. We find that this approach is particularly well-suited for discrete molecular graph modality because it relegates the challenge of modeling a discrete and diverse distribution to the autoencoder and simplifies the diffusion process by restricting it to the continuous, latent feature space. Since graph data consists of both node attributes and graph structure, which differs significantly from the sequential text data, to align the latent representations of textual descriptions and molecular graphs, 3M-diffusion employs contrastive learning on a large data set of pairs of molecular structures and their textual descriptions to pretrain molecular graph encoders and LLM text encoders. This enables 3M-diffusion to generate high-quality molecular graphs that match the given textual descriptions.The key contributions of this paper are as follows: (i) To the best of our knowledge, 3M-Diffusion offers the first multimodal diffusion approach to generating molecular structures from their textual descriptions, surpassing the limitations of stateof-the-art (SOTA) methods for this problem. (ii) 3M-Diffusion aligns the latent spaces of molecular graphs and textual descriptions to offer a text-molecule aligned latent diffusion model to generate higher-quality, diverse, and novel molecular structures that match the textual description provided; and (iii) Results of extensive experiments using four real-world text-based molecular graphs generation benchmarks show that 3M-Diffusion outperforms SOTA methods for both text-guided and unconditional molecular structure generation. In particular, 3M-Diffusion achieves 146.27% novelty, 130.04% diversity relative improvement over the SOTA method with maintained semantics in textual prompt on the PCDes dataset. 
molecular structure generation: 
 Existing molecular graph generation methods can be broadly grouped into two categories: text-based and graph-based models. Text-based models typically utilize the SMILESFigure 2 : Qualitative comparisons to the MolT5-large of generated molecules on ChEBI-20.Compared with the SOTA method MolT5-large, our results are more diverse and novel with maintained semantics in textual prompt. More results are provided in Appendix D.4 (Weininger, 1988) or SELFIES (Krenn et al., 2022) linear strings to describe each molecule (Bjerrum & Threlfall, 2017; G ómez-Bombarelli et al., 2018; Kusner et al., 2017; Flam-Shepherd et al., 2022; Fang et al., 2023b; Grisoni, 2023) . Because a given molecular graph can have several linear string representations and small changes to the string representation can result in large changes in molecular graph being described, such linear string encodings are far from ideal for learning generative models for producing molecular graph from textual descriptions. Inspired by the success of learning enhanced graph representations using graph based models (Kipf & Welling, 2016; You et al., 2020; Xiao et al.; 2024) , several authors have employed deep generative models, including graph variational autoencoders (VAEs) (You et al., 2018; Jin et al., 2018; Xiao et al., 2021; Jin et al., 2020; Kong et al., 2022; Liu et al., 2018; Diamant et al., 2023) , normalizing flows (Madhawa et al., 2019; Zang & Wang, 2020; Luo et al., 2021) , generative adversarial networks (Guarino et al., 2017; Maziarka et al., 2020) , diffusion models (Niu et al., 2020; Vignac et al., 2022; Jo et al., 2022) and autoregressive models (You et al., 2018; Xiao & Wang, 2021; Popova et al., 2019; Goyal et al., 2020) to learn distributions of molecular graphs from large databases of known graphs. However, many of these studies focus on generating molecular graphs with specific low-level properties, such as logP (the octanol-water partition coefficient) as opposed to those that match high-level textual descriptions of a broader range of molecular properties (Edwards et al., 2022) , e.g., water solubility, chemical activity, etc. However, practical applications, such as drug design, call for effective approaches to generating diverse and novel molecular graphs that match a given textual description of such high-level properties.Advances in deep learning and language models have inspired a growing body of work on applications of such models to problems in the molecular sciences (Schwaller et al., 2019; Xiao et al., 2023; Toniato et al., 2021; Xiao et al., 2024; Zhao et al., 2023a; Seidl et al., 2023; Schwaller et al., 2021; Vaucher et al., 2020) . Advances in text-guided generation of images, videos, and audios (Radford et al., 2021; Rombach et al., 2022; OpenAI, 2023; Yin et al., 2023) , together with the flexibility offered by natural language descriptions of molecular graphs have inspired a growing body of work on text-guided generation of molecular graphs (Christofidellis et al., 2023; Edwards et al., 2022; Fang et al., 2023a; Zeng et al., 2022; Su et al., 2022) . For example, MolT5 (Edwards et al., 2021; 2022) and ChemT5 (Christofidellis et al., 2023) pre-train on a substantial volume of unlabeled language text and SMILES strings for tasks such as molecule captioning and text-based molecule generation (Christofidellis et al., 2023; Edwards et al., 2022) . However, as noted above, the degenerate nature of linear SMILES string representation of molecular graphs makes it less ideal for generating molecules from their textual descriptions. To address this problem, we propose 3M-Diffusion, a novel method that aligns the latent representation of textual descriptions of molecules with that of the corresponding molecular structures, yielding a powerful approach to text-guided molecular structure generation. 
problem definition: 
 We consider generative modeling of 2D molecular graphs. Each molecule is representedas a graph G = (V, E ), where V = v 1 , . . . , v |V | is the set of |V | nodes and E is the set of edges. Let X = x 1 , x 2 , • • • , x |V | ∈ R [V |×Dx be the node attribute matrix, where x i is the D x -dimensional one-hot encoding feature vector of v i , such as atomic type and chirality type. Similarly, a tensor A ∈ R |V |×|V |×b groups the one-hot encoding e ij of each edge, where each entry is a distinct edge type (bond types for molecule), with the absence of an edge encoded explicitly using a designated edge type. Given the dataset D = {(G i , T i ) N i=1 } where molecule G has specific text descriptions T characterizing the features of the molecule, our goal is to learn conditional generation models p θ (G | T ) for generating novel, diverse, and valid molecules while also aligning with the desired features outlined in text descriptions T . For brevity, in what follows, we omit the input subscript i when it is clear from context.Diffusion models (DMs) (Ho et al., 2020; Song et al., 2020) are latent variable models that represent the data z 0 as Markov chains z T • • • z 0 , where intermediate variables share the same dimension. DMs involve a forward diffusion process q (z 1:T | z 0 ) = ∏ T t=1 q (z t | z t-1 ) that systematically adds Gaussian noise to samples and a reverse process p θ (z 0:T ) = p (z T ) ∏ T t=1 p θ (z t-1 | z t ) that iteratively "denoises" samples from the Gaussian distribution to generate samples from the data distribution. A formal description of diffusion models is given in the Appendix A. To ensure training stability, we can use a simple regression objective with a denoising network ẑθ with input (x t , t):EQUATION)where t is the time step, α t ∈ [0, 1] is the noise schedule and ϵ ∼ N (0, I) is Gaussian noise and λ t is a time-dependent weighting term. Intuitively, the denoising network is trained to denoise a noisy state, z t = √ α t z 0 + √ 1α t ϵ, aiming to reconstruct the clean data z 0 with Equation (1) that prioritizes specific times t. After training, we can draw samples with ẑθ by the iterative ancestral sampling (Ho et al., 2020) :EQUATIONwhere α t|t-1 represents α t /α t-1 and σ(t -1, t) = (1 -α t-1 ) 1 -α t|t-1 /(1 -α t ).The sampling chain is initialized from Gaussian prior z T ∼ p (z T ) = N (z T ; 0, I). 
latent multi-modal diffusion for molecules: 
 3M-Diffusion (See Figure 1 ) aims to learn a probabilistic mapping from the latent space of textual descriptions to a latent space of molecular graphs for text-guided generation of molecular graphs. However, direct mapping between these two latent representations using a diffusion model faces a key hurdle, namely, large mismatch between the latent spaces of textual descriptions and of molecular graphs. To overcome this hurdle, 3M-Diffusion adopts a two-step approach. The first step uses contrastive learning to produce text-molecule aligned variational autoencoder that align the representation of molecular graphs with that of their textual descriptions, and a decoder that maps the latent representation back to the corresponding molecular graphs. The second step uses the text-aligned latent representation of molecular graphs to learn a conditional generative model, that maps textual descriptions to latent representations of molecular graphs. As we will see later, 3M-Diffusion can produce diverse novel molecular graphs that match the given textual descriptions.The proposed text-molecule aligned variational autonecoder comprises of three parts: the molecular graph encoder E g , LLM encoder E t and molecular graph decoder D. To bridge the representation gap between molecular graphs and their textual descriptions, the text and molecular graph encoders are trained on a large-scale dataset of molecule-text pairs. These encoders use contrastive learning to construct well-aligned text-molecular structure encodings that align the latent representation of molecular structures by aligning it with with that of their textual descriptions. The details of this process are given below.Molecular Graph Encoder aims to map each molecular graph G with the adjacency tensor A and node features matrix X into a lower-dimensional latent space Z. This transformation maps the discrete space of graph structures and node attributes into a continuous latent space. Specifically, we use Graph Isomorphism Network (GIN) (Hu et al., 2019) as encoder network on the input adjacency tensor A and node features X of graph G to derive the mean and standard deviation of the variational marginals:EQUATIONwhere µ g and σ g represent the mean and standard deviations, respectively. Then, the latent graph representation z is sampled from a Gaussian distribution as z ∼ q (z | A, X) ≃ N µ g , σ g . This graph encoding process can be expressed as z = E g (G) signifying the outcome of the encoding operation in the input context of G.LLM Encoder maps the textual descriptions of molecular structures into a latent space that provides input to a conditional generative model of molecular graphs. This encoded information is instrumental in generating molecules that match a given textual description.To inject potentially useful scientific knowledge from the literature into E t we initialize it with encoder-based LLM, i.e., Sci-BERT (Beltagy et al., 2019) , which is pretrained on the text of scientific publications.Aligning the representation of molecular graphs with that of their textual descriptions is crucial for establishing a shared representation for effective text-guided molecular graph generation. Graph data encodes both node attributes and graph structure which differs significantly from the sequential and contextual nature of text data. We aim to align the molecular representation from the molecule graph encoder E g with text representation from the LLM encoder E t using contrastive learning. Consider a molecular graph G and its textual description T . To align these two encoders and ensure a cohesive latent space, we employ the following contrastive learning loss over a batch B ∈ D of large-scale dataset containing molecule-text pairs to pretrain encoders:EQUATIONwherez i = E g (G i ), c i = E t (T i) and c j = E t (T j ) are representations of corresponding molecule graph and texts, respectively. cos(•, •) denotes cosine similarity and τ is the temperature hyperparameter. The encoders are pretrained on molecule-text (300K) pairs from PubChem Liu et al. (2023) . As we will see, this process yields a well-aligned moleculetext space that can support the generation of diverse and novel molecular structures from their textual descriptions.Molecular Graph Decoder learns to use latent representation z to generate the corresponding molecular structure. In this work, we employ the HierVAE (Jin et al., 2020) decoder. Note however that our approach can be combined with other graph decoders (Li et al., 2018; Kong et al., 2022; Grover et al., 2019) . Specifically, the molecular graph decoder generates a molecular structure G using a graph representation z from the encoder. Given the molecular graph G, we minimize the negative Evidence Lower Bound (ELBO) (Jin et al., 2020) to train both the encoder and the decoder:EQUATIONHere Ĝ = D(z), in which D denotes the decoder parameterized by HierVAE (Jin et al., 2020) . KL[q(•)||p(•)] is the Kullback-Leibler (KL) divergence between q(•) and p(•). We follow Jin et al. (2020) and set the weight of KL loss α = 0.1. Here, p(z) = N (z | 0, I) is the isotropic multivariate Gaussian prior, where I is the identity matrix. We optimize this first expectation term concurrently using the reparameterization trick (Kingma & Welling, 2013).We next introduce latent diffusion for conditional molecular graph generation, a method that leverages the aligned molecule-text latent space of encoder-decoder described above, enabling us to preserve the crucial molecular properties and aligning the textual description of G. Specifically, we learn a probabilistic mapping from the text to the text aligned molecular structure latent space, and thus generate molecular structures that match the conditional text inputs.The denoising network ẑθ focuses on generating latent graph representation z conditioned on the text representation c from the LLM encoder E t . The objective to be optimized is:EQUATIONwhere λ t are time-dependent weights and we use MLPs for the network ẑθ . Specifically, we concatenate c to graph latent representation z t for timestep t as the input of ẑθ . The denoising network is trained to denoise a noisy latent, z t = √ α t z + √ 1α t ϵ to the clean z. c is the text representation from the fixed encoder E t pretrained by Equation (4).Additionally, to improve sample quality we take advantage of classifier-free guidance (Ho & Salimans, 2022) , to jointly train an unconditional network, ẑθ (z t , t), and a conditional network, ẑθ (z t , t, c). Conditioning information is randomly dropped with a probability of p = 0.1 during training. When conditioning information is dropped, we replace the embedded source text with its embedding.Training. We proceed to describe the training procedure for 3M-Diffusion. While the learning objectives for graph encoder-decoder and multi-modal latent diffusion have already been specified by Equations ( 5) and ( 6), it is unclear whether the two components should be trained sequentially. Previous research on latent diffusion models for image generation indicates that a two-stage training strategy often results in superior performance (Rombach et al., 2022) . Hence, we adopt a similar two-stage strategy. The first stage consists of training an encoder-decoder pair using Equation (5), and the second stage consists of training a multi-modal molecule latent diffusion with the fixed encoder-decoder from the first stage.Inference. During inference, we use a weight w in computing the prediction as follows (Ho & Salimans, 2022) EQUATIONwhere zt represents the newly predicted results, incorporating both conditional and unconditional information. Setting w = 1.0 corresponds to the conditional diffusion model, while setting w > 1.0 enhances the impact of the conditioning information. A smaller value of w contributes to more diverse samples. The new sampling equation is: where the sampling chain is initialized from Gaussian prior z T ∼ p (z T ) = N (z T ; 0, I). The detailed training and inference algorithms of 3M-diffusion are provided in Appendix B.EQUATION 
experimental setup: 
 Datasets. We use PubChem (Liu et al., 2023) , ChEBI-20 (Edwards et al., 2021) , PCDes (Zeng et al., 2022) and Momu (Su et al., 2022) Evaluation. We evaluate the model's performance on two tasks: text-guided and unconditional molecule generation. Following previous works on text-guided molecule generation, we compare our 3M-Diffusion the SOTA baselines (Edwards et al., 2022; Christofidellis et al., 2023; Fang et al., 2023a) . Text-guided molecule generation measures the model's capacity to generate chemically valid, structurally diverse, novel molecules that match the textual description provided. As in previous work (Chen et al., 2021; Fu et al., 2021; Wang et al., 2022) , we adopt the following metrics for this task: Similarity, Novelty, Diversity, Validity. We call a predicted molecular structure, specifically one that is chemically valid (g ′ ), qualified with the ground truth graph g if f (g, g ′ ) > 0.5, where f measures the MACCS (Durant et al., 2002) cosine similarity between the two. If the f (g, g ′ ) similarity between g ′ and g is smaller than a threshold (0.8 in this paper), we call this predicted molecule g novel. Specifically, (i) Similarity: the percentage of qualified molecules out of 20K molecules; (ii) Novelty: the percentage of novel molecules out of all qualified molecules; (iii) Diversity: the average pairwise distance 1f (•, •) between all qualified molecules. (iv) Validity: the percentage of generated molecules that are chemically valid out of all molecules.We report the results for unconditional molecule generation tasks using the GuacaMol benchmarks (Brown et al., 2019) to assess the model's ability to learn a generative model of molecular structures that can be used to generate realistic, diverse molecules. Specifically, Uniqueness gauges the ratio of unique molecules among the generated ones. Novelty measures the models' capacity to generate molecules not present in the training set. KL Divergence assesses the proximity between the distributions of various physicochemical properties in the training set and the generated molecules. Fréchet ChemNet Distance (FCD) computes the proximity of two sets of molecules based on their hidden representations in ChemNet (Preuer et al., 2018) . Each metric is normalized to a range of 0 to 1, with a higher value indicating better performance.Baselines. For text guided molecule generation, we compare 3M-Diffusion with the following baselines: MolT5 (Edwards et al., 2022) , ChemT5 (Christofidellis et al., 2023) and Mol-Instruction (Fang et al., 2023a) . We consider seven representative baselines for unconditional molecule generation based on Variational Autoencoder, including CharRNN (Segler et al., 2018) , VAE (Kingma & Welling, 2013), AAE (Makhzani et al., 2015) , LatentGAN (Prykhodko et al., 2019 ) BwR (Diamant et al., 2023) , HierVAE (Jin et al., 2020) and PS-VAE (Kong et al., 2022) . The details of the baselines are given in Appendix C.1.We randomly initialize the model parameters and use the Adam optimizer to train the model. HierVAE is used as our variational autoencoder for all of the following experiments. For baselines of unconditional generation that did not report results in these datasets, we reproduce the results using the official code made available by the authors.To ensure a fair comparison across all methods for unconditional generation, we select the best hyperparameter configuration solely based on the loss on the training set. We evaluate the performance of baselines for text-guided molecule generation using the pretrained parameters made available by the authors of the respective studies. Table 1 and 2 report Similarity, Novelty, Diversity, and Validity for text-guided molecule generation on four datasets. Each metric is computed as an average over 5 structures generated from a given textual description.We see that 3M-Diffusion is competitive with SOTA baselines with respect to Similarity and outperforms them substantially with respect to Diversity and Novelty. Specifically, 3M-Diffusion achieves 146.27% (Novelty of PCDes), 130.04% (Diversity of PCDes) relative improvement over the second-best SOTA baseline while matching the textual prompt. We conclude that while the transformer based models that are pretrained on a large dataset excel in producing molecular structures that match the textual descriptions provided, 3M-Diffusion outperforms them with respect to the diversity and novelty of the molecular structures without sacrificing the match between the textual descriptions provided as input and the molecular structures generated as output. Also, as Figure 3 shows, 3M-Diffusion enjoys a substantial advantage in terms of its speed over transformer-based methods (See Appendix D.1 for details).Table 3 shows the results of unconditional molecule generation on PubChem and ChEBI-20. 3M-Diffusion consistently outperforms or matches the performance of SOTA baselines across all four metrics on both datasets, suggesting that it is able to generate realistic, diverse, and novel molecules without overfitting the training data. Furthermore, 3M-Diffusion consistently outperforms HierVAE when both methods use the same decoder. This underscores the clear advantage of 3M-Diffusion in generating a diverse and novel molecular graphs (See Figure 10 for 50 samples from the learned distribution).We conduct ablation studies to assess the impact of different model designs, by joint training of the diffusion model and Variational Autoencoder ("without two-stage training") and training the model without aligning the representation space of graphs and text using L con in Equation (4) ( "without representation alignment"). Additionally, we perform experiments without both two-stage training and representation alignment ("without twostage training & representation alignment") in Table 4 . We find that joint training the aligned encoder-decoder alongside Multi-modal Molecule Latent Diffusion achieves comparable or superior performance relative to 3M-Diffusion in terms of diversity and novelty, but at the expense of reduced quality of match between the molecular structures produced and the textual descriptions provided. We find that training the model without the aligned encoderdecoder results in substantial drop in performance relative to 3M-Diffusion, underscoring the importance of aligning the representation of textual descriptions of molecules with that of molecular structures. Ablation of both two-stage training and representation alignment components leads to a substantial drop in performance (Similarity of 10.24), rendering the resulting model unsuitable for real-world applications. The full 3M-Diffusion model (last row) achieves the best performance with respect to all of the metrics. This confirms the importance of two-step training and representation alignment in training a model that can generate diverse, novel set of molecular structures from their textual descriptions.Property Conditional Molecule Generation. In traditional molecule generation models, the generation process is often conditioned on specific molecular properties such as logP (Kusner et al., 2017) and QED. logP assesses a molecule's solubility, while QED evaluates its drug likeness (Bickerton et al., 2012) . Previous methods have explored ways to optimize such models further in order to generate molecules with desired properties. In contrast, 3M-Diffusion utilizes textual prompts for conditional molecule generation (See Figure 2 ).In particular, we generated 10 samples for each prompt and subsequently select the top 5 samples based on the value of the desired property. For MolT5-large, we adjust the temperature of the transformer in an attempt to encourage diversity while ensuring that invalid samples are controlled during the sampling process. We applied the same methodology used in our previous experiments to maintain consistency across the comparative analysis, as shown in Figure 2 . When compared to the MolT5-large model, we find that 3M-Diffusion excels in generating a diverse range of molecules that also score better with respect to the desired properties, as evidenced by higher logP values Figure 2 . This underscores our 3M-Diffusion's ability to produce molecules that are not only novel and diverse but also match the textual description provided. This unique approach distinguishes 3M-Diffusion from prior methods for molecular generation, presenting a fresh and promising perspective on molecular design. Additional experimental details and insights regarding prompts related to solubility and drug likeness are provided in Appendix D.2.In Figure 2 , we introduce more specific prompts related to molecular categories, such as "The molecule is a member of the class of flavones." These prompts bias the model in favor of generating useful drug candidates. Specifically, we choose a reference molecule from the training dataset that matches the given textual prompt. Subsequently, we generate ten additional molecules and select the top five molecules that exhibit the highest structural similarity (calculated by Rascal algorithm (Raymond et al., 2002) ) to the reference molecule. These selected molecules are shown in Figure 2 . Case Study. In Figure 4 , we choose a set of examples from the ChEBI-20 dataset to visually illustrate the structural similarity between the molecules generated by 3M-Diffusion and the ground truth structures corresponding to the given textual descriptions. Specifically, we show the largest substructure shared by the generated and ground truth molecules (using RDkit (Landrum, 2010) ). 3M-Diffusion demonstrates the capability to generate molecules that include the relevant substructure, as illustrated in Figure 4 . Simultaneously, it displays the flexibility to generate molecules with novel substructures, thereby striking a balance between generating structures that are similar to those represented in the training data and suggesting novel structures that match the textual description provided. We include additional results in Appendix D.4. 
conclusion: 
 Generating molecules with desired properties is a critical task with broad applications in drug discovery and materials design. Such applications call for methods that produce diverse, and ideally novel, molecules with the desired properties. We have introduced 3M-Diffusion, a novel multi-modal molecular graph generation method, to address this challenge. 3M-Diffusion first encodes molecular graphs into a graph latent space aligned with text descriptions. It then reconstructs the molecular graph and atomic attributes based on the given text descriptions using the molecule decoder. 3M-Diffusion learns a probabilistic mapping from the text space to the latent molecular graph space using the latent diffusion model. The results of extensive experiments show that 3M-Diffusion can generate high-quality, novel and diverse molecular structures that semantically match the textual description provided. Ablation studies underscore the critical role of aligning the latent representation of text with that of molecular graphs in 3M-Diffusion. 
 
 Appendix: We present a formal description of diffusion (Ho et al., 2020; Song et al., 2020) . Diffusion models are latent variable models that represent the data z 0 as Markov chains z T • • • z 0 , where intermediate variables share the same dimension. DMs involve a forward diffusion process q (z 1:T | z 0 ) = ∏ T t=1 q (z t | z t-1 ) and a reverse process p θ (z 0:T ) = p (z T ) ∏ T t=1 p θ (z t-1 | z t ). The forward process starting from z 0 can be written as:EQUATIONwhere the hyperparameter α 1:T determines the magnitude of noise added at each timestep t.The values of α 1:T are selected to ensure that samples z T converge to standard Gaussians, i.e., q (z T ) ≈ N (0, I). This forward process q is usually predefined without trainable parameters.The generation process of DMs is defined as learning a parameterized reverse denoising process, which incrementally denoises the noisy variables z T:1 to approximate clean data x 0 in the target data distribution:EQUATIONwhere the initial distribution p(z T ) is defined as N (0, I). The means µ θ typically are neural networks such as U-Nets for images or Transformers for text, and the variances σ 2 (t -1, t) typically are also predefined. As latent variable models, the forward process q(z 1:T |z 0 ) can be seen as a fixed posterior. The reverse process p θ (z 0:T ) is trained to maximize the variational lower bound of the likelihood of the data:EQUATIONTo ensure training stability, a simple surrogate objective up to irrelevant constant terms is introduced (Song et al., 2020; Ho et al., 2020; Nichol & Dhariwal, 2021) :EQUATION)where z t is used to denote √ α t z 0 + √ 1α t ϵ for simplicity. Intuitively, we train a neural network ẑθ (z t , t) to approximate the original data given some noisy latent and the timestep through Equation ( 12), ẑθ (z t , t) ≈ z. With a trained denoising network, we define the mean and variance function in Equation ( 10) following the previous work (Ho et al., 2020) as:EQUATIONwhere λ t is the time-dependent weight and α t|t-1 = α t /α t-1 . For generation, we employ the standard DDPM sampler, also referred to as the ancestral sampler (Ho et al., 2020) . The process involves sampling initial noise z T ∼ N (0, I) and iteratively applying the update rule:EQUATIONwhere ϵ ∼ N (0, I). We use T = 50 for the sampling timesteps. zt = wẑ θ (z t , t, c) + (1w)ẑ θ (z t , t), 6:L con = -1 |B| ∑ (G i ,T i )∈B log exp(cos(z i ,c i )/τ) ∑ |B| j=1 exp(cos(z i ,c j )/τ)L elbo = -E q(z|A,X) [p(G | Ĝ)] + αKL[q(z | G)∥p(z)]z ← E g (G) 20: t ∼ U(0, T), ϵ ∼ N (0, I) 21: L diff = E t,x 0 ,ϵ λ t ẑθ √ α t z 0 + √ 1 -α t ϵ, t -z 0 2 2 22: θ ← optimizer(L diffz t-1 = √ α t-1 (1-α t|t-1 ) 1-α t zt + √ α t|t-1 (1-α t-1 ) 1-α t z t + σ(t -1, t)ϵ 7: end for 8: G ∼ p(G|z 0 )▶ Decoding 9: return x, hThe complete training algorithm is depicted in Algorithm 1. Firstly, the process begins by training the graph and LLM encoders using contrastive learning. Following that, the Variational Autoencoder is trained to facilitate the reconstruction of molecule graphs. Lastly, the Multi-modal Molecule Latent Diffusion Model is employed to train the denoising network for the sampling process. This structured approach forms the core of our training procedure. Moreover, we present the sampling method of 3M-Diffusion in Algorithm 2. The statistics of datasets, including their the number of samples for training, validation and test sets are given in Table 5 .ChEBI-20 (Edwards et al., 2021) : This dataset is generated by leveraging PubChem (Kim et al., 2016) and Chemical Entities of Biological Interest (ChEBI) (Hastings et al., 2016) . It compiles ChEBI annotations of compounds extracted from PubChem, comprising moleculedescription pairs. To ensure less noise and more informative molecule descriptions, it includes samples with descriptions exceeding 20 words.PubChem (Liu et al., 2023) : This dataset comprises 324k molecule-text pairs gathered from the PubChem website. As the dataset contains numerous uninformative texts like "The molecule is a peptide," it curates a high-quality subset of 15k pairs with text longer than 19 words for downstream tasks. This refined subset is then randomly split into training, validation, and test sets. The remaining dataset, which is more noisy, is utilized for pretraining. To prevent data leakage, we exclude samples that appear in the testing set of other datasets from the pretraining dataset.PCDes (Zeng et al., 2022) : This dataset compiles molecule-text pairs sourced from Pub-Chem (Kim et al., 2016) , encompassing names, SMILES notations, and accompanying paragraphs providing property descriptions for each molecule.MoMu (Su et al., 2022) : This dataset comprises paired molecule graph-text data, with the textual information for each molecule extracted from the SCI paper dataset (Lo et al., 2019) .For all datasets, inspired by previous datasets (ZINC250K (Irwin et al., 2012) and QM9 (Blum & Reymond, 2009; Rupp et al., 2012) ) on molecule generation, we preserve molecules with fewer than 30 atoms, creating new training, validation, and test sets from their original counterparts.We firstly introduce the baselines used for text-guided molecule generation in this section.MolT5 (Edwards et al., 2022) : MolT5 addresses the difficult problem of cross-domain generation by linking natural language and chemistry, tackling tasks such as text-conditional molecule generation and molecule captioning. It presents several T5-based LMs (Raffel et al., 2020) by training on the SMILES-to-text and text-to-SMILES translations.ChemT5 (Christofidellis et al., 2023) : ChemT5 introduces a versatile multi-task model for seamless translation between textual and chemical domains. This innovative approach leverages transfer learning in the chemical domain, with a particular emphasis on addressing cross-domain tasks that involve both chemistry and natural language.Then, we will also include seven baselines for unconditional generation to verify the effectiveness of our model.CharRNN (Segler et al., 2018) : CharRNN models a distribution over the next token based on the previously generated tokens. We train this model by maximizing the log-likelihood of the training data, which is represented as SMILES strings.VAE (Kingma & Welling, 2013) : Variational Autoencoder (VAE) consists of two neural networks: an encoder and a decoder. The encoder infers a mapping from high-dimensional data to a lower-dimensional space, while the decoder maps this lower-dimensional representation back to the original high-dimensional space. It optimizes reconstruction loss and regularization term in a form of Kullback-Leibler divergence. We use SMILES as input and output representations as discussed in (Polykovskiy et al., 2020) .AAE (Makhzani et al., 2015) : Adversarial Variational Autoencoder (AAE) replaces the Kullback-Leibler divergence in VAE with an adversarial objective. An auxiliary discriminator network is trained to distinguish between samples from a prior distribution and the model's latent codes. Similar to VAE, we use SMILES as input and output representations as shown in (Polykovskiy et al., 2020) .LatentGAN (Prykhodko et al., 2019) : The Latent Vector Based Generative Adversarial Network (LatentGAN) integrates an autoencoder with a generative adversarial network. It first pretrains an autoencoder to map SMILES structures to latent vectors. Subsequently, a generative adversarial network is trained to generate latent vectors for the pre-trained decoder.BwR (Diamant et al., 2023) : BwR aims to decrease the time/space complexity and output space of graph generative models, demonstrating notable performance improvements in generating high-quality outputs on biological and chemical datasets.HierVAE (Jin et al., 2020) : HierVAE employs a hierarchical encoder-decoder architecture to generate molecular graphs, leveraging structural motifs as fundamental building blocks. (Kong et al., 2022) : PS-VAE automatically identifies regularities within molecules, extracting them as principal subgraphs. Utilizing these extracted principal subgraphs, it proceeds to generate molecules in two distinct phases.Note that we use the code for CharRNN, VAE, AAE, LatentGAN by the Molecular Sets (MOSES) Benchmark Polykovskiy et al. (2020) .We use official implementation publicly released by the authors of the baselines or implementation from Pytorch. Note that we use the decoder of HierVAE for our model. We run experiments on a machine with a NVIDIA A100 GPU with 80GB of GPU memory. In all experiments, we use the Adam optimizer (Kingma & Ba, 2014) . In particular, for our 3M-diffusion. we set the dimension of latent representation as 24, the number of layers for the denosing networks (MLP) as 4, α as 0.1, T as 100 for training and 50 for sampling process and the probability p = 0.8 for text-guided molecule generation. For unconditional generation, we set T as 10 or 5 for the sampling process.In this section, we compare the average inference times of 3M-Diffusion, MolT5, and ChemT5 by generating 1,000 samples, as shown in Figure 3 . The average time to generate each molecule is calculated and presented. We observe that 3M-Diffusion achieves a speed 45 times faster than baseline methods, confirming the efficiency of our proposed model. Our proposed model not only generates molecules more quickly than previous approaches but also maintains a high level of semantic alignment with textual descriptions.In this section, we provide more prompt examples and more experiment details for Property Conditional Molecule Generation. We generate 10 samples for each prompt and subsequently select the top 5 samples based on the desired property values, as shown in Figure 2 , 5, 6 and 7. For instance, when using the prompt "This molecule is not like a drug," we In this section, we choose 20 more examples from the ChEBI-20 dataset to visually illustrate the structural similarity between the molecules generated by 3M-Diffusion and the ground truth molecules corresponding to the given text descriptions. All results are shown in Figure 4 . We also observe that our model is proficient at generating molecules that share significant common structural elements, while simultaneously obtaining diverse and innovative molecular structures.As shown in Figure 10 , our sampled molecules present rich variety and structural complexity. This demonstrates our model's ability to generate diverse, novel, and realistic molecules.The molecule is a hydroxyquinolone that is 3-hydroxyuinolin-2(1H)-one which is substituted at position 4 by a phenyl groups. Isolated from the mycelium of several Penicillium species, it exhibits strong antibiotic activity against M. tuberculosis and also against B. subtilis, S. aureus and S. cerevisiae. It has a role as an antibacterial agent and a Penicillium metabolite.The molecule is a tripeptide composed of Lphenylalanine, L-alanine and L-proline joined in sequence by peptide linkages. It has a role as a metabolite. It derives from a Lphenylalanine, a L-alanine and a L-proline.The molecule is an imidazoquinoline that is 3Himidazo[4,5-f]quinoline substituted by a methyl group at position 3 and an amino group at position 2. It has a role as a carcinogenic agent.The molecule is a L-serine derivative. It has a role as an Escherichia coli metabolite. It is a conjugate acid of a N-(2,3dihydroxybenzoyl)-L-serinate.The molecule is a phenylacetaldehyde in which the 3 and 4 positions of the phenyl group are substituted by hydroxy groups. It has a role as a human metabolite, an Escherichia coli metabolite and a mouse metabolite. It is a member of catechols, an alpha-CH2-containing aldehyde and a member of phenylacetaldehydes.The molecule is a quinolinemonocarboxylate that is the conjugate base of xanthurenic acid, obtained by deprotonation of the carboxy group. It has a role as an animal metabolite. It is a conjugate base of a xanthurenic acid.The molecule is an aldoxime resulting from the formal condensation of the aldehyde moiety of indol-3-ylacetaldehyde with hydroxylamine. It is a member of indoles and an aldoxime.The molecule is a quaternary ammonium ion obtained by methylation of the tertiary amino function of reticuline. It derives from a reticuline.The molecule is an N(2)-acyl-L-ornithine where the acyl group is specified to be acetyl. It has a role as a human metabolite, an Escherichia coli metabolite, a Saccharomyces cerevisiae metabolite and a mouse metabolite. It is an acetyl-L-ornithine and a N2-acyl-L-ornithine. It is a tautomer of a N(2)acetyl-L-ornithine zwitterion.The molecule is a trihydroxyflavone with the hydroxy groups at positions C-5, -7 and -8. It has a role as an antioxidant and a metabolite.The molecule is a leucoanthocyanidin that is the 3-deoxy derivative of (2R,3S,4S)leucocyanidin. It is a pentahydroxyflavan and a leucoanthocyanidin. It derives from a (2R,3S,4S)-leucocyanidin.The molecule is a tetrahydroxyflavone that is apigenin with an extra hydroxy group at position 8. It has a role as a metabolite. It derives from an apigenin.The molecule is an aromatic L-alpha-amino acid zwitterion resulting from transfer of a proton from the carboxy to the amino group of 3-O-methyldopa. Major species at pH 7.3 It has a role as a human metabolite. It is a tautomer of a 3-O-methyldopa.The molecule is a 2-hydroxy carboxylate that is the conjugate base of 3-(indol-3-yl)lactic acid. It has a role as a human metabolite. It derives from a lactate. It is a conjugate base of a 3-(indol-3-yl)lactic acid.The molecule is a tetrachlorobiphenyl that is biphenyl in which both phenyl groups are substituted by chlorines at positions 3 and 5. It is a tetrachlorobiphenyl and a dichlorobenzene.The molecule is an aromatic ether that is the derivative of 2-phenylethylamine with methoxy substituents at the 3-and 4-positions. It is an alkaloid isolated from the Cactaceae family. It has a role as a plant metabolite and an allergen. It is a phenylethylamine, an aromatic ether and an alkaloid. It derives from a hydride of a 2-phenylethylamine.  


Decision: This work presents a method to guide molecular generation with NL. The key concern being improving on SOA through improving the diversity of molecules generated. the work accomplishes its task, but the requires several different models, each doing a specific task to be trained together. This is is unsurprising because the work builds on pretrained models (graph/llm) which raises the question of what becomes more efficient if you build end-to-end.  

As reviewer H3jC points out, the changes wrt SOA methods need to be evidenced/explained more, especially given the substantial change in the fidelity of the model (c.f. Mol-instruct numbers shared in the rebuttal). These big changes should be explained in terms of what this model is fixing/ignoring wrt the baseline, for example are there particular types of structure/strategies that increase the diversity for 3M compared to Mol-instruct?  Why is Validity always 100% for 3M, are there cases where that is not the case? Have you done any human eval on a small subset to understand how well aligned generated molecules are with descriptions? 

Overall, this is a nice work and the thorough baselines that are useful to the community. I would encourage authors to share as many artifacts from data to models as possible

Reviews: review: paper_summary: paper_summary: This paper proposes a language model on molecule generation, well combining the existing practices in multi-modal generation: (1) train a VAE model for molecule graph generation and text descriptor generation; (2) align the representation of molecule graph and text descriptors; (3) introduce a latent diffusion model for mapping the text spaces to the latent molecular graph spaces, which then can be decoded to a molecular graph.
Experiment results show the proposed method is effective, outperforming SOTA methods for both text-guided and unconditional molecule generation.
strength_weakness: strength_weakness: 1. Aligning the concept of molecular to text generation is useful, which will help us to understand the properties of unknown or new molecules. In such a case, it is more natural for me to understand the necessity of molecule-to-text generation instead of text-guided molecular generation.
2. The technical novelty is limited, but it is acceptable.
questions: questions: 1. How about using the 3D geometry for molecular modeling and introducing the protein or pocket information as a prompt for drug discovery?
paper_summary: paper_summary: This paper studies language-guided molecule generation given text description. The proposed method consists of multiple neural networks, including a graph encoder, decoder, text encoder, and a diffusion model in the latent space. To align the representations of text and graphs in the latent space, the authors utilized contrastive loss. Experiments on conditional and unconditional molecule generation were conducted to demonstrate the effectiveness of the proposed framework.
strength_weakness: strength_weakness: 1. **Overclaim and SOTA.** The key contribution, (iii) the proposed method outperforms SOTA methods, is questionable. First of all, the baselines did not include recent methods, and unconditional generation results are not provided in the table.
2. **No technical contribution.** The proposed pipeline is a combination of existing modules. HierVAE, Latent diffusion, GIN for graph encoder and contrastive loss for text and graph latent space alignment. Also, no significant modification/adaption of existing methods for text-guided molecule generation.
3. **Trade-off rather than improvement.** Table 1 and Table 2 show that the proposed method shows a noticeable degradation in similarity and, instead, performance gains in other metrics. This could be viewed as trade-off between metrics. In addition, Novelty, Diversity and Validity are not related to language-guidance. They are not specifically designed to evaluate the fidelity between conditions and generated samples.
paper_summary: paper_summary: The paper introduces a diffusion model that generates molecular graphs using textual descriptions of molecules as conditions. This framework first aligns the representations of molecular graphs and text descriptions. It then proceeds to train in two phases: an encoder-decoder and a multi-modal latent diffusion model.
strength_weakness: strength_weakness: - Figure 1 is confusing; the encoder and condition should be inputted together into the diffusion model.
- The legends in Table 1 are incorrect for the datasets ChEBI-20 and PubChem. 
- Tables 1 and 2 could be merged to reduce redundancy.
- How to obtain the ground truth graph of a generated molecule when calculating similarity if the descriptions of the two texts are very similar.
Additional experiments, such as retrieval tasks, should be conducted to validate the alignment of representations between text and molecular graphs.
- Does the case study use examples taken from the training set?
- It is inappropriate to express improvements in novelty and diversity as the percentage increases. The statement '3M-Diffusion achieves 146.27% novelty and 130.04% diversity relative improvement over the SOTA method' should be rephrased to better reflect these metrics.
- Due to the lack of distinctiveness in the similarity, novelty, diversity, and validity metrics, it would be beneficial to identify additional indicators for assessing the algorithm, such as plotting the distribution of logP and QED.
- Why is there such a significant difference between the prompts in Figure 2 and Figure 3? The prompt in Figure 3 includes the molecular components, whereas Figure 2 only provides a simple description of properties. The disparity is too significant.
The paper's statement, 'For example, the structure of ethanol can be specified using CCO, OCC, or C(O)C,' requires supporting evidence. If the method for generating SMILES is consistent, would this issue not exist?
paper_summary: paper_summary: This paper proposes a method for text-guided molecule generation. The method leverages a latent diffusion model to generate latent representations of molecular graphs. Further, a variational graph autoencoder is employed to decode a molecular graph from the latent representation. For text-guided generation, an LLM encoder is trained to align with the graph encoder by cross-modal contrastive learning. The output of the LLM encoder is also incorporated as a condition to control the latent diffusion model.
strength_weakness: strength_weakness: * In Introduction, the authors promised results on the GuacaMol benchmarks, yet these results are absent from the subsequent sections. 
* The evaluation metrics in Table 1 and Table 2 are insufficient. Why are the evaluation metrics from MolT5 (*e.g.*, BLEU, Exact, Levenshtein, MACCS) missing in this work?
* **Unconditional molecule generation (Table 5)**
  * Since no textual condition is used, the comparison should include baselines in [1]. 
  * Table 5 should be moved to the main body of the paper.
* Table 1 and Table 2 have the same caption, yet the contents are different. The authors should specify the distinct contents of each table to avoid confusion.

[1] Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models. In Frontiers in Pharmacology, 2020.
questions: questions: cf. above.

Comments: title: Kind reminder to reviewer xhrH: the author-reviewer discussion period is coming to an end.
General Response by Authors
A sincere and kind reminder to the reviewer
Further Responses and Clarifications to Reviewer H3jC
Thank you for your response to our rebuttal
Additional Experiment Results
Additional Experiment Results
comment: Dear Reviewer xhrH,

We want to sincerely thank you again for your time and comments!

We have made extensive efforts to address your comments and carefully considered your suggestions point-by-point and added unconditional molecule generation you request in the rebuttal. We would like to kindly remind you that we are approaching the end of the author-reviewer discussion. In light of our rebuttal, we kindly ask if you could consider increasing your score. Thank you very much once again, and we look forward to hearing back from you if you have further comments.

Thank you very much for your time.

Best regards,

The Authors
Dear Reviewers,

We sincerely thank all the reviewers for their thoughtful feedback and their time. We are encouraged to see all reviewers agree that our paper is a novel, interesting, and effective work with strong experimental results.

In our work, we propose the first multimodal diffusion approach to generating molecular graphs from their textual descriptions, which is critical for molecule generation or drug discovery with textual guidance. Our proposed 3M-Diffusion effectively aligns the latent spaces of molecular graphs and textual descriptions to offer a text-molecule aligned latent diffusion model to generate high-quality, diverse, and novel molecular graphs that match the textual description provided. We demonstrate that our 3M-Diffusion has great empirical performance on both text-guided molecule generation and unconditional molecule generation compared with baselines. These empirical results and a novel perspective for text-guided molecule generation were found to be particularly interesting by all reviewers.

We have made our greatest efforts to incorporate all questions, suggestions, and comments in our response to each reviewer, and have prepared a point-by-point response for each. Briefly, we have addressed key themes in the reviews, including additional comparisons with baselines, retrieval tasks, misunderstandings, and minor questions. Below, we summarize the major responses, while we address the comments of each reviewer separately.

- We added the additional comparisons with baselines for text-guided molecule generation and unconditional generation (Reviewer H3jC, xhrH).
- We added retrieval tasks for our text and graph encoder (Reviewer c27h).
- We added additional clarification about technical novelty and contributions (Reviewer H3jC, 8fJY).
- We compared 3M-Diffusion with the baseline by plotting the distribution of logP and QED. (Reviewer c27h).

For the other questions, we have provided individual responses to each one. We will update the paper to include these additional experiments and clarifications in response to your comments.

We extend our sincere thanks to all the reviewers once again for their time and efforts.

Best regards,

Authors
Dear Reviewer xhrH:

We gratefully appreciate your time in reviewing our paper and your comments.

We have made extensive efforts to address your comments and believe that they adequately address all your concerns. The reviewer's comments are mainly about some clarifications and are not fatal to the contributions of our paper; we believe that the reviewer's insightful comments can be easily and effectively addressed in the final version.

With only four days left in the discussion period, we would like to confirm whether there are any other clarifications they would like. If the reviewer's concerns are clarified, we would be grateful if the reviewer could increase the score.

Many thanks for your time; we are extremely grateful.

The authors of "3M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecule Generation"
We sincerely thank the reviewer for the additional comments and suggestions. Please see our detailed clarifications about technical novelty below:

- 3M-Diffusion studies a novel problem about text-guided molecular graph generations. **The task of generating diverse, novel molecules that match textual descriptions is challenging due to the sparsity and discrete nature of graph structures.** While there is extensive research on using variational autoencoders (VAEs) or diffusion models (DMs) for molecular graph generation to produce diverse, novel or property-related molecules, there is no existing work that leverages the capabilities of VAEs or DMs to generate molecules that are both diverse, novel, and text-matching. **We believe that the 3M-Diffusion model makes significant technical contributions and novelty by establishing a pipeline capable of generating diverse, novel and text-matching molecular graphs.** This pipeline achieves performance comparable to transformer-based models in terms of "Similarity," which measures the model's text-matching capabilities, and excels at generating diverse and novel molecules.

- Furthermore, we are the first to explore the ability of diffusion models on the latent space of molecular graphs to generate novel, diverse and text-matching molecules. **Specifically, our technical contributions and novel approaches include designing the latent space and creating a suitable scoring networks for latent diffusion tailored to the latent space of molecular graphs, enabling our model to function effectively.**

- Also, in our 3M-Diffusion model, **we do not directly utilize HierVAE's encoder; instead, we employ our own trained encoder that aligns graph and text representations, while only incorporating HierVAE's decoder.** Unlike HierVAE's encoder, which learns hierarchical graph representations but can’t understand text, we propose using an aligned encoder from GIN. This aligned encoder is designed to understand textual descriptions by contrastive learning. **Utilizing this encoder's representation for the decoder and making it effectively generate diverse, novel, and text-matching molecules, represents a significant technical novelty and contribution of our model.**
Dear Reviewer H3jC:

Thank you very much for reviewing our paper and reading our rebuttal. We sincerely appreciate your recognition of our clarifications and the increase in your score!

**Thank you also for your additional comments to facilitate further discussion, which will further improve our paper.**

**In the responses below, we have carefully addressed your remaining concerns about technical novelty.**

We are truly grateful for your time and your reply.

Best regards, Authors
I appreciate the authors' detailed responses. As the experimental results, the proposed method shows strong performance. The response to the technical novelty is not convincing and novelty cannot be addressed by the response unless there exists any misunderstanding. Although this paper has limited technical contributions, this is an early attempt for molecule generation, showing great performance. I raise my rating.
Thank you once again for your comments and suggestions! Due to word limits of previous rebuttal part, we have provided additional results of more datasets for the experiments you suggested in this part.

**Additional Experiments for Q1 on PCDes, Momu, ChEBI-20 and PubChem datasets:**

|   |  | PCDes |        |        |         | MoMu |       |        |         |
|-----------|---|------|--------|--------|---------|------|-------|--------|---------|
|     Metrics      |   | Similarity (%) | Novelty (%) | Diversity (%) | Validity (%) | Similarity (%) | Novelty (%) | Diversity (%) | Validity (%) |
| Mol-instruct | | 60.86 | 35.60 | 24.57 | 79.19 | 14.89 | 97.52 | 30.17 | 68.32    |
| 3M-Diffusion  | | **81.57** | **63.66** | **32.39** | **100.0** | **24.62** | **98.16** | **37.65** | **100.0**     |

|   | ChEBI-20 |  |        |        |   PubChem  |   |       |        |         |
|-----------|---|------|--------|--------|---------|------|-------|--------|---------|
| Metrics         | Similarity (%) | Novelty (%) | Diversity (%) | Validity (%) | Similarity (%) | Novelty (%) | Diversity (%) | Validity (%) |
| Mol-Instruct | 65.75         | 32.01       | 26.50         | 77.91        | 23.40          | 37.37        | 27.97         | 71.10        |
| 3M-Diffusion  | **87.09**         |  **55.36**       | **34.03**         | **100**          | **87.05**          | **64.41**        | **33.44**         | **100**          |

**Additional Experiments for Q3  on all datasets (average results of all metrics are put in the following table):**

| Model         | PCDes | MoMu | ChEBI-20 | PubChem |
|---------------|--------|--------|--------|--------|
| MolT5-small   | 50.06    | 51.06 | 50.06 | 44.29 |
| MolT5-base    | 53.96    | 46.37 | 53.97 | 46.37 |
| MolT5-large   | 54.77    | 54.36 | 54.36 | 54.36 |
| ChemT5-small  | 55.10    | 54.20 | 55.10 | 54.20 |
| ChemT5-base   | 55.42    | 54.56 | 55.42 | 54.56 |
|    3M-Diffusion  | **69.12** | **65.10** | **69.12** |  **71.23** |

**We can find that 3M-diffusion's average results on all metrics outperform other baselines, which show the improvement of our model.**
Thanks again for your effort on reviewing our paper. Due to the word limit in the previous rebuttal part, we provide the following additional demonstrations and experiments by your suggestions:

**Additional experiment and demonstrations for Q3**

We present molecule-to-text retrieval (M2T) and text-to-molecule retrieval (T2M) tasks by comparing with Sci-BERT [1] and KV-PLM [2] on PubChem. Sci-BERT, pretrained on scientific texts, serves as the initial text encoder for the 3M-Diffusion model. Similarly, KV-PLM, a deep learning system, can be utilized for molecule and text retrieval tasks. Both models employ SMILES to facilitate the molecule and text tasks. Here are the following results:

| Model      | M2T  |       | T2M  |       |
|------------|------|-------|------|-------|
|            | Acc  | R@20  | Acc  | R@20  |
| Sci-BERT   | 39.7 | 85.8  | 37.5 | 85.2  |
| KV-PLM     | 38.8 | 86.0  | 37.7 | 85.5  |
| 3M-Diffusion     | **61.7** | **90.8**  | **63.2** | **90.2**  |

We observe that 3M-Diffusion can outperform Sci-BERT and KV-PLM, which verifies the effectiveness of our model for the retrieval task.

[1] SciBERT: A Pretrained Language Model for Scientific Text.

[2] A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals.