Lift Your Molecules: Molecular Graph Generation in Latent Euclidean Space
Authors: Mohamed Amine Ketata
Nicholas Gao
Johanna Sommer
Tom Wollschläger
Stephan Günnemann
Venue: ICLR 2025 Poster (2025)
Keywords: Drug Design
Computational Biology
Molecule Generation
Graph Generation
Latent Diffusion Models
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose a latent Euclidean space for molecular graph generation and demonstrate that a diffusion model on such a space achieves state-of-the-art performance on common molecular graph generation benchmarks.

Abstract: We introduce a new framework for 2D molecular graph generation using 3D molecule generative models. Our Synthetic Coordinate Embedding (SyCo) framework maps 2D molecular graphs to 3D Euclidean point clouds via synthetic coordinates and learns the inverse map using an E($n$)-Equivariant Graph Neural Network (EGNN). The induced point cloud-structured latent space is well-suited to apply existing 3D molecule generative models. This approach simplifies the graph generation problem into a point cloud generation problem followed by node and edge classification tasks, without relying on molecular fragments nor autoregressive decoding. Further, we propose a novel similarity-constrained optimization scheme for 3D diffusion models based on inpainting and guidance. As a concrete implementation of our framework, we develop EDM-SyCo based on the E(3) Equivariant Diffusion Model (EDM). EDM-SyCo achieves state-of-the-art performance in distribution learning of molecular graphs, outperforming the best non-autoregressive methods by more than 26\% on ZINC250K and 16\% on the GuacaMol dataset while improving conditional generation by up to 3.9 times.

Published as a conference paper at ICLR 2025
LIFT YOUR MOLECULES: MOLECULAR GRAPH GENERATION IN LATENT EUCLIDEAN SPACE
Mohamed Amine Ketata', Nicholas Gav", Johanna Sommer,,
Tom Wollschläger, Stephan Günnemann
Department of Computer Science & Munich Data Science Institute
Technical University of Munich
(a. kekketta,, n. gao, jm. sommer, t. w.w. squennemannn @ de
ABSTRACT
We introduce a new framework for 2D molecular graph generation using 3D molecule generative models. Our Synthetic Coordinate Embedding (SyCo) frame- work maps 2D molecular graphs to 3D Euclidean point clouds via synthetic co- ordinates and learns the inverse map using an E(n)-Equivariant Graph Neural Network (EGNN). The induced point cloud-structured latent space is well-suited to apply existing 3D molecule generative models. This approach simplifies the graph generation problem into point cloud generation problem followed by node and edge classification tasks, without relying on molecular fragments nor autore- gressive decoding. Further, we propose novel momilaaity-constrane optimization scheme for 3D diffusion models based on inpainting and guidance. As con- crete implementation of our framework, we develop EDM-SYCo based on the E(3) Equivariant Diffusion Model (EDM). EDM-SYCO achieves state-of-the-art performance in distribution learning of molecular graphs, outperforming the best non-autoregressive methods by more than 26% on ZINC250K and 16% on the GuacaMol dataset while improving conditional generation by up to 3.9 times
INTRODUCTION
Machine learning- based drug discovery has recently shown great potential to accelerate the drug development process while reducing costs associated with clinical trials (Jiménez-Luna et al. 2020)] Kim et al. 2020). Among the steps of this process, generating novel molecules or optimizing properties such as likgelikeness or synthesizability are of crucial importance. Molecular graph generation is an active area of research that aims to solve these tasks (Gónez-Bombarelii et al.) 2018).
Molecular graph generation is challenging due to discrete and sparse nature of graphs as well as the presence of symmetric substructures such as rings. Many methods have been proposed to tackle this problem, offering different tradcoffs (Jin et al. 2018; Shi et al., 2020). A common approach is to train a Variational Autoencoder (VAE) to encode the graph into continuous fixed-size latent space, then decode it back oftentimes autoregressively fragment (Jin al.) 2018 2020 Maziarz et al. 202T). While such methods achieve good generation and optimization performance, they have inherent limitations. First, using a fixed-sized latent space does not account for the variable size of molecular graphs, which can go from few atoms for small molecules to thousands for macromolecules. Second, autoregressive decoding requires choosing a concrete generation order (Schneider al. (2015), which prevents learning permutation-invariiat invariant graph distributions. Fueled by recent developments in diffusion models (Ho et al.) 2020 Austin et al. 202Tb, new approaches to molecular graph generation have emerged that overcome these limitations [Jo et al. [2022] Vignac et 2022 Jo et al., (003). These methods define a discrete diffusion process directly on the node and edge attributes and learn to reverse it to generate graphs all at once, i.c., not autoregressively. However, they still lack behind autoregressive methods, struggling to accurately estimate the joint distribution nodes and edges.
bo-de (Liu et al. 2018) or fragment-by.
"Equal contribution.
Our code is available at https///githhhccco//auaaaaCCCCC

---

Published as a conference paper at ICLR 2025
Paradkrr E
Foewaed Diliaion
Textc
K(U.))
Decoder S1
Revene Dillaeie
A
Zg
2r
Toining
Crach Space
Luser: Point Cloud
Sampling
Figure 1: Overview of EDM-SYCo. (Training) First, the autoencoder is trained to map between molecular graphs and latent Euclidean point clouds. Then, the diffusion model is trained on the fixed latent space. (Sampling) Starting with a Gaussian sample, the diffusion model denoises it for T steps to predict the clean point cloud, which is mapped to a molecular graph using the decoder.
In this work, develop method that combines the benefits of all-at-once generation with the benefits of continuous latent space while avoiding the information bottleneck of fixed-sized latent spaces, Inspired by recent developments in the adjacent field of 3D molecule generation, we propose to embed molecular graphs as latent 3D point clouds that implicitly encode the discrete graph structure in synthetic Euclidean coordinates and use point cloud generative models to learn their distribution. We call this framework Synthetic Coordinate Embedding (SYCo).
When combined with EDM (Hoogeboom et al. 2022), our framework gives rise to an atom-baseed all-at-once, and permutation-invariant generative model for molecular graphs, which we call EDM- SyCo, depicted Figure On the ZINC250K dataset, EDM-SYCo outperforms existing all-at-once diffusion-basee models by more than 26% distribution learning benchmarks and by up 15.6 times on conditional generation benchmarks, closing the gap to autoregressive models and highlighting the benefit of continuous latent space generation and optimization. In addition, similarity-consee constrained optimization procedure for diffusion models on point clouds without retraining specialized architectures, visualized in Figure Notably, our proposed algorithm enables the addition of new atoms during the reverse diffusion process, addressing a well-known limitation of prior 3D diffusion models, namely the fixed number of nodes.
propose new
RELATED WORK
Molecular Graph Generation Early work on molecule generation Bombarllo al. 2018 Segler et al., (018)) used language models to generate SMILES (Weininger] 1988), a text-based representation of molecules. However, this type of representation cannot capture the inherent structure of molecules well, which are more accurately depicted as graphs. For instance, small changes in the graph structure may correspond to large differences in the SMILES string (Jin et al., 2018).
Therefore, recent methods have focused graph-based approaches that can be broadly classified into two categories. All-at-once generation methods generate nodes and edges in parallel, often in multi-step process such diffusion models (Niu et al.) 2020 Jo et al. 2022 Vignac al. 2022) Autoregressive methods specify a generation order and generate molecules either atom-by-atom
al. 2018 Shi al. 2020) (Jin et al. 2018 2020, Maziarz et al. 2001 Kong et al.) 2022 Geng et al. (023). EDM-SYCo an all-at-once molecular graph generative model that operates the atom level.
soagmeen--b-ffaggeee
based
pre-computed fragment vocabulary
Molecule Generation in 3D Another line of work aims generate molecules in 3D) Euclidean space and solve a point cloud generation problem (Gebbuerr et al. 2019 Garcia Satorras et al. 2021
Ayadi et al., 2025). For instance, EDM (Hoogeboom et al. [0222 and GeoLDM (Xu et al. 2023
are diffusion-based models for 3D molecule generation that jointly generate atomic features and coordinates. While tangentially related, these methods solve different problem from molecular graph generation and require molecules with 3D information for training. Our proposed SYCo framework is an attempt to connect these two lines of work by enabling the training of 3D generative models on graph datasets through synthetic coordinates, formally introducing new graph generation methods. While we are the first to explore synthetic coordinates for generative models, Gasteiger et al. 2021) demonstrated that synthetic coordinates improve molecular property prediction tasks.
Latent Generative Models The idea of defining the generative model on a space different from the original data space is reminiscent of latent generative models DDai & Wipf 2019 Vahdat et al.
2

---

Published as a conference paper at ICLR 2025
Recently, this approach has become increasingly popular with latent diffusion models. Stable Diffusion models (Rombach et al.) (022) achieve impressive results on text- text-guided image generation, and GeoLDM (Xu et al.] 2023) extends this to molecular point cloud generative models. While EDM-S operates on similar latent space to GeoLDM, their fundamental difference lies in the original data space. Our model maps discrete molecular graphs to continuous point clouds and can be seen as a cross-modality latent generative model. In contrast, GeoLDM embeds point clouds as new point clouds with reduced feature dimension.
2021p.
3 SYNTHETIC COORDINATE EMBEDDING (SYCO) FRAMEWORK
Our goal is to model molecular graph distributions through distributions of Fuclidean point clouds that implicitly encode the graph structure into their coordinates. For this, we propose a novel autoencoder architecture that maps between molecular graphs and 3D point cloud representations. By training it using a reconstruction objective, we can reformulate the problem of generating discrete molecular graphs to an equivalent problem of generating 3D point clouds defined on the induced latent space.
Note on equivariance When dealing with physical objects like molecules, it is important to consider equivariance. Formally, function f A - B is equivariant to the action of group G iff f(TzA (z)) = T (f(z)) for all g E G, where Ty and T g in A and B respectively (Serre et al. 1977). As special case, it is invariant iff f(Tz(=)) In the context of generative modeling, we aim to learn distributions invariant to the symmetries of the data, namely graph permutations in our case. In practice, this has the benefit of not relying on any generation order and enabling efficient likelihood estimation since an invariant likelihood is identical for all permutations of the same graph. We leverage equivariant architectures such as EGNNs to learn such distributions, removing the need for data augmentation during training, further increasing efficiency. EGNNs (Satorras et al., (221]) are special type of graph neural networks that learn functions cquivariant to rotations, translations. reflections, and permutations of their input point cloud. More details this architecture are given in Appendix D Also note that due the reflection equivariance, our model is invariant to chirality.
g are the representations of the group element f()).
Notation We introduce helpful notation for the two molecule representations used in this work. Let N denote the number of atoms of a given molecule. On the one hand, the molecular graph g - (h, A) E G consists of atoms as nodes and chemical bonds as edges. Each atom has one of a atom types and one of с formal charges, while each bond has one of bond types. We represent atoms as stacked one-hot encodings h {0, 1) (x(+t)) and chemical bonds as one-hot vectors in an adjacency matrix A E {0. 1 Note that the molecular graph is sometimes called "2D molecule" as can be typically drawn as planar graph. On the other hand, the molecular point cloud P = (h, æ) E TP is a 3D point cloud, where æ E IR Nx3 represents the coordinates of atoms' nuclei in Euclidean space. The point cloud encodes the bond information implicitly in the atomic coordinates. We now present our autoencoder model, whose overview is shown in Figure
Encoding The encoder E G + (x++3) with parameters ó maps molecule's graph representation g -= (h. to its point cloud representation P = (h, x), then further compresses it into node-structured latent representation
Encoder
Confermer
h
EGNN
Generation
(2(h), 2(()), where (hh) Nxd and ((x) E R are continuous embeddings for h and æ with d < and E N xdd+3) is their concatenation.
lx]
MLPssds
R
MLPetge
In the first encoding step, the coordinates x are computed using the ETKDG algorithm (Riniker & Landrum (215)) which first computes a molecule's distance bounds ma- trix based on typical bond lengths between different atom types, then produces atomic coordinates that satisfy these bounds, which are further optimized using molecular force field. Crucially, this procedure allows to leverage chemical domain knowledge an inductive bias into the
Decoder Dr
Figure 2: Autoencoder architecture. The encoder maps a molecular graph point cloud, and the decoder learns the inverse. Both are trained jointly to mini- mize the reconstruction loss.
3

---

Published as a conference paper at ICLR 2025
model and get chemically meaningful 3D conformers. These coordinates encode the bond information of the molecular graph and ideally define an injective mapping from to P. Note that this step is not learned and, thus, not required to be differentiable. In practice, generate conformers for the whole training dataset in a preprocessing step and reuse them throughout training.
Then, apply EGNN that maps the intermediate representation P (h, x)) the latent representation (2 ), defining the space of the generative model. The main benefit of this is to incorporate the neighborhood information into each atom's 's embeddings and coordinates and mapping the discrete features h into a continuous and lower-dimensional embedding ((h)
Decoding The decoder D&= Nx(3(3) G with parameters & aims to approximate the inverse function 8 and maps the latent point cloud of molecule back to its graph representation G. It consists of an atom and a bond classifier. For each node the decoder applies 2-layer Multi-Layer Perceptron (MLP) on its latent embedding to predict the logits for the atom type and the formal charge as h, ML. (z( For each pair of nodes and j, the bond type is inferred by running a different 2-yayer MLP on their edge features to obtain the corresponding logits as follows:
(h) 2i
(K)
Au
MLP edge
2j
(h)
+ MLP MLPedge edge
where d, = 11 2/00 ll2 and we use the simple averaging trick to ensure that À is symmetric.
Autoencoder Training The encoder and decoder jointly optimized in variational autoen- coder (VAE) framework. We define probabilistic encoding and decoding processes as (0(= (
N(Ex(G), of) and (g(2)) I[ (g(h.122 [2) IT
respectively, where do RR con- trols the variance in the latent space. The VAE is trained by minimizing the loss function
(g(Aiylz) =)) P
Lv - (g(0) [pe [G]=]]
(1)
which, in practice, is computed as the cross-cntropp loss. The decoder performs three classification tasks to reconstruct molecular graph: atom types, formal charges, and bond types. The total loss is the unweighted sum of the corresponding three cross-entropy terms.
Before continuing with our approach, we present ablation study on our autoencoder architecture. The task consists of predicting all the bond types of a molecule from its 3D coordi- nates. We compare our EGNN-based approach with (i) an MLP al. (2022).
Table 1: Accuracy of predicting bond types from 3D coordinates on ZINC250K. Existing methods use rule-based algorithms, while we in- troduce an BGNN-based algorithm.
and (ii) rule-based system akin to Hoogeboom More details on the experimental setup are in Appendix G.] Results on ZINC250K are shown in Table [ and similar results are observed for GuacaMol. Our autoencoder model achieves an almost perfect reconstruction accuracy, highlighting the im- portance of the EGNN in our architecture. This experiment supports our assumptions (i) that the used mapping from C to P is injective and (ii that the decoder can accurately approximate its inverse.
METHOD
ACCURACY
EGNN MLP
99.93%
MLP
11.77%
RULE-BASED
11.78%
4 EDM-SYCo
Our SYCO framework lifts the graph generation problem to a point cloud generation problem in 3D. Concretely,
interested in learning the distribution of latent point clouds 9o (Z)
defined by the underlying molecular graph distribution and the trained encoder by approximating with parametric distribution Po- While this generation problem can, in principle, be approached by any 3D molecular generative method, this work adapts EDM under the SYCO framework due to its simplicity and empirical performance in 3D molecule generation tasks (Hoogeboom et al. 2022)
4.1 EQUIVARIANT DIFFUSION ON LATENT POINT CLOUDS
diffusion model generates samples by reversing a diffusion process, the process of adding noise to data (Sohl-Dickstein al. 2015 HH al. 2020). For ease of notation, flatten the latent repre-
4

---

Published as a conference paper at ICLR 2025
sentation of the point clouds 2, namely their latent coordinates and features, into -ie-dimensional vector denoted & IR with d' N(d + 3). The diffission or forward process is fixed Markov chain of Gaussian updates that transform 2n into latent variables 21,..., 2r of the same dimension as 20, defined
T
= II q(2x|2z-1), with q(21(z=1)) = N(z v 1 912
[=1
where 3, is typically chosen such that ((r)) converges M(zr; I)
By defining
Og = 1- Br and ag == -li-- 9i we can also map 2u directly to 2, through
=1
(<ze|zo) (N(z za,zo (1-a)))))
To reverse the above process, we start from a Gaussian noise sample z4 and iteratively sample from the posterior distributions q(zz-1 | which is also Gaussian if 8 is small (Sohl-Dickstein al. 2015). However, estimating them requires using the entire dataset. Therefore, diffusion models learn a reverse process po (zu:r) = p(=)) II (2 with p(zr) = (zr; 0, and
Po (z:-1 /2) = (2z po t). of I),
(4)
where the mean p0(2), t) E pdd is parametrized by a neural network.
We follow Ho et al. (2200) and, instead of directly predicting the mean, predict the Gaussian noise added to the latent point cloud representation during the diffusion process. The mean then becomes p5 (2 1)
1-0, co(ze. (z),t) where c0, t) IR is the output of the neural network. Following Hoogehoom al. (2022). we parametrize co via an EGNN (Satorras al., 2021). Additionally, the reverse process ensures equivariance to translations by defining the learned distribution po (zo) on the zero center of gravity subspace.
This network can then be trained by optimizing the L2 loss between the sampled Gaussian noise at each step and the network's output, LDM = Ez..e,t [||c cacc-o(zz) co(21, t)]
(5)
During sampling, we start from a Gaussian sample 2T ~ p(zr) and iteratively denoise it by sampling C ~ (N I) and transforming 2 22-1:
at co(zi,t)) +
v1 0:-1
21-1
(zt
B....
(6)
Va
VI az
v1 at
(0(
4.2 TRAINING AND SAMPLING FROM EDM-SYCO
After training the autoencoder as described in Section EDM is trained in a second stage by optimizing the loss function defined in Equation [ We describe the full training procedure of both parts in Algorithm 1 Consistent with previous work on latent diffusion models (Rombach et al.,
2022 Xu et al. 2023). found this two-stage training procedure to perform better than jointly training the autoencoder and the diffusion model.
For sampling, we need access the trained EDM and the decoder but not the encoder. First, starting with a sample øy (N I) representing a random 3D point cloud, we iteratively denoise it with the diffusion model (Equation 6 to obtain the clean sample zg. Then, we apply the decoder to get the corresponding molecular graph h, A = De( Zu). Algorithm 2 formally describes this procedure, further illustrates it. This sampling procedure defines a permutation invariinn molecular graph distribution. The following proposition, which we prove in Appendix C.2 formalizes this:
and Figure [
Proposition 4.1. The marginal distribution of molecular graphs po.c (9.:(9)
Epo(ze) (9)(G)z) [00]
defined by the EDM and the decoder, is an Sx-invariant distribution, i.e. for any molecular graph (h, A), po.g (h, A) -P.. (AP,PAP) for any permutation matrix P E Sx.
So far, we have assumed that the number of atoms N fixed across all molecules. In practice, to generate molecules with different sizes, we follow previous work (Hoogeboom et al. 2022) and estimate the empirical distribution of the number of atoms in the training set p(N) and use it sample different numbers of atoms before running the sampling procedure described above. In Appendix F2, we describe the use of conditional distributions to sample in a constrained generation optimization.
5

---

Published as a conference paper at ICLR 2025
5 CONTROLLABLE GENERATION AND SIMILARIY-CONSTRAINGE OPTIMIZATION
So far, we have introduced our approach to distribution learning molecular graphs. However, many tasks in drug discovery require generating molecules with specific conditions such as chemical properties, similarity constraints, or the presence of certain desirable substructures. In this section, we describe how we can adapt the sampling procedure of EDM-SYCO (or other models combined with SYCO) - trained in an unconditional setting to different conditioning modalities. While previous work on 3D molecule generation tackled conditional generation (Hoogeboom et al.) [002 Bao et al. 2022) and scaffolding (Schneuing et al., 2022), the similarity-constrainee optimization task (Jin et al., 2020) has not been addressed in the 3D molecule generation literature. We fill this gap by introducing a novel sampling algorithm for diffusion models to perform similarity-constraiie optimization.
Generation via Diffusion Guidance In conditional generation setting, want to generate a molecule with a specific property e by sampling from the conditional distribution ((2]c). To achieve this, we adapt the classifier guidance algorithm (Dhariwal & Nichol 2021), initially proposed to guide image diffusion models using a classifier. Instead, we use a property regressor guide the generation towards molecules with specific continuous properties similar to Bao et al. (2022). One can show that the denoising network's 's output relates to the score function of the data distribution via >x. log (=x) ~ VIーa. EB EZ(2t.)) (Dhariwal & Nichol 2021> and the
sampling procedure of diffusion models (Equation 66) is equivalent to running Langevin dynamics using the score function V z. (zz) (Welling & Teh 2011: Song & Ermon 2019). This provides
a way to sample from the distribution q only through its score function. For the case of g(z) (c), its score function can be written using Bayes' rule and approximated as:
V #e log g( (c) = Ve, log (Zt) + Vx, x, log (c|2z))
1
-1- at. (co(z), vI a Vz, log (c)2z))
Gaussian distribution approximating the probabil- ity of molecule zţ having property c, In is a property regressor trained to predict the property e given a noisy molecule zt. and s is a scale factor that controls the skewness of the distribution. With this, we can rewrite the conditional score function Vz, log g( c)) x VI-āØ, > (Eo(=z. f)
where Py, (c/2z) (xp(=s(9((2))e))
VI- a. (z, ))²). To sample from this distribution, we use the same sampling algorithm defined by Equation 6 and replace EB EZ(, t) by
E0 c0 f) VI a,sV ( c)2.
(7)
Intuitively, this can be seen as minimizing the loss (g, c)2. Appendix E.2 provides additional details and discussions about the equivariance of the regressor.
Unconstrained Property Optimization via Diffusion Guidance With the formulation introduced above, we can sample molecules with high property values by simply using the denoising step
(z,. t) - /1 a.sV (z))
(8)
Notice that replaced the loss with the negative gradient of the regressor.
Shaffold-Constrainnd Generation via Inpainting For tasks such as structure-based drug design, it
often desirable to generate molecules that contain specific substructure, usually called a scaffold (Schuffenhauer et al. 20077: Maziarz et al. 2021). This problem can be viewed as completion task and is reminiscent inpainting, which aims to complete missing parts of an image (Song et al., 2020)) Lugmayr et al.) 2022). Lugmayr et al. 2022 propose RePaint, simple modification to the sampling procedure of denoising diffusion models, enabling them to perform inpainting. This approach has already been successfully applied to molecule generation in 3D (Schneuing al. 2022).
The core idea of RePaint is to start the sampling procedure from a random sample 2T and, at each step, enforce the part that want to be present in the final sample. To illustrate this for the case of molecules, let &n denote the point cloud of a molecule containing the desired scaffold. Let m be a binary mask that indicates which nodes from 2n belong to the scaffold such that @ Eo denotes the

---

Published as a conference paper at ICLR 2025
A,4
Zi
Zt
Fnonder
odd
add nole: sumple modes
A'
Zn'
42'
Decoder
dervise with guidanze
dreuisc with iepaiting
Bi
Figure 3: Overview of our constrained optimization procedure. Based on a noising/denoising approach, the first steps of the reverse diffusion process using the inpainting algorithm to add new atoms and the remaining steps using the guidance algorithm to increase the target property. The depicted molecules have QED values of 0.79 (initial) and 0.91 (optimized), with 53% similarity.
scaffold point cloud and (1 - m) @ žy the unknown part. We aim to sample a new point cloud containing this scaffold while extending it to get a harmonized point cloud. Formally, we require that m @ = m @ žo. To achieve this, we iteratively apply the following modified sampling step starting from T:
2 unknown
scaffold
NN (1 a
(9)
(Mo (), T. (as in Equation 6
(10)
2-1
-t-1 m ④
2-11 scaffold (1 m) o t-1
unknown
(11)
We refer to Appendix E.3 for more implementation details.
CellaartyyConstrainee Optimization In this task, we aim to improve the target properties of given molecule while satisfying a similarity constraint. We propose novel approach to this task by combining the regressor guidance algorithm and the inpainting algorithm discussed above. The key idea is to add noise to the initial molecule and then denoise it with the regressor guidance algorithm to improve the target property. Formally, starting from the initial point cloud žg with N atoms, we sample noised 2, following Equation 3 for (0, T). t is hyperparameter that controls the optimization quality. On the one hand, increasing results in more reverse diffusion steps with the guidance algorithm, thus increasing the chance of improving the target property. On the other hand, it also results in losing information about the initial molecule due to noise. Intuitively, the best yields the highest property improvement while satisfying the similarity constraints.
However, since the diffusion model operates on N points, the denoised molecule also has N atoms. This limits the optimization of a molecule's property as prevents adding new atoms. To overcome this limitation, we sample a Gaussian point cloud œy with N' atoms, where N' > N, and run the inpainting algorithm from T to with 2, as the scaffold, meaning that z; will replace m O žµ in the inpainting algorithm described above. This results in new intermediate representation z!. whose first N points correspond to z,, and the rest are the newly added atoms, as desired. Finally, 2/ is denoised from / to 0 using the guidance algorithm with maximization objective (Equation X to get the point cloud z of the optimized molecule, which is then decoded to the graph using the decoder De. An overview of this procedure and example test molecule shown in Figure
LIMITATIONS
EDM-SyCo can be seen as an extension to EDM, which carries the limitations of diffusion models like high inference cost. To mitigate this issue in future work, we can incorporate more efficient sampling strategies (Karras et al. 2022 2023), or use more efficient models such as recent conditional flow matching approaches (Tong et al. 2023 Song et al. 2023). In addition, our sampling approach requires specifying the number of nodes priori. While this works well empirically, we leave tackling the fundamental issue of generating variable-see sized graphs or point clouds to future work. Lastly, our constrained optimization procedure cannot remove atoms during inference. Our experiments found that the decoder tends to predict disconnected subgraphs, which we treat as removing atoms, but a rigorous solution may improve performance.
7

---

Published as a conference paper at ICLR 2025
6 EXPERIMENTS
6.1 COMPARISON TO 3D MOLECULE GENERATIVE MODEL GEOLDM
EDM-SYCO leverages EDM to learn the distribution of 3D molecules in latent space, and combines it with an autoencoder that maps between 2D and 3D molecules. The overall architecture is therefore similar to GeoLDM in that both are EDM-base -asedd latent diffusion models for molecules. The main differences lie, however, in the design of the autoencoder and in the training and sampling procedure. In this section, we explain the need for such changes in order to adapt 3D molecule generative models like GeoLDM to the different modality of 2D molecules and validate the effect of these changes experimentally. In essence, there are two main challenges when it comes to applying GeoLDM or any other 3D molecule generative models to 2D molecule generation:
(i) GeoLDM requires datasets with 3D atomic coordinates for training. To enable training on 2D datasets like ZINC250K or GuacaMol, we leverage the conformer generation component introduced in Section 3] Crucially, this allows to inject chemical domain knowledge an inductive bias into the model by leveraging typical bond lengths and molecular force fields when computing 3D conformers.
(ū) GeoLDM uses simple rule-based algorithm to predict 2D molecules from 3D molecules, which fails medium-sized molecules. This algorithm predicts bond types from inter-atomic distances using lookup table. In our experiments, this algorithm can correctly predict all bond types of a molecule only on 11.78% of the ZINC250K dataset (See Table Tb. To overcome this limitation, we introduce our EGNN-based bond type predictor, which can leverage the richer neighborhood information of an atom and does not rely on hard-coded rules. To highlight the effect of this change on the quality of the generated 2D graphs, we compare GeoLDM and EDM-SyCo both trained on ZINC250K using the same synthetic coordinates and report the results in Table
Table 2: Comparison between GeoLDM and EDM-SYCO on ZINC250K. To enable the training and evaluation of GeoLDM, we use the same synthetic coordinates as EDM- YoCo and use GeoLDM's bond type predic- tion algorithm.
GEOLDM
EDM-SYCE
FCD (1)
0.17
0.85
KL (t)
0.79
0.96
NOVELTY (t)
1.0
1.0
UNIQUENESS (t)
1.0
1.0
VALIDITY (1)
0.12
0.88
From the above discussion, we can conclude that (i) applying 3D models off-the-shelf as 2D baselines is either not directly possible (for datasets without 3D information) or yields limited performance due to the inability of the used bond type prediction algorithm to produce large and valid molecules, and (ii) combining EDM or GeoLDM with our proposed components would yield exactly our model since EDM and GeoLDM both share the same diffusion framework, which we also use in EDM-SyCo.
6.2 COMPARISON TO 2D MOLECULE GENERATIVE MODELS
In the following set of experiments, we compare the performance of EDM-SYCo to several au- toregressive and to all-at-once diffusion-based baselines on de novo and conditional generation tasks as well as on a constrained optimization task. Specifically, comparing EDM-SYCO with the diffusion-basee baselines highlights the benefit of defining the diffusion process on the latent 3D space compared to directly defining it on the 2D space.. More experiments are discussed in Appendix G We use two datasets of different sizes and complexities: ZINC250K (Irwin et al., 2012) containing 250K molecules with up to atoms, and GuacaMol (Brown al. al.] 2019) containing R 1.5M drug-like molecules with up to 88 atoms. We train EDM-SY as described in Sections 3 and on these two datasets. Training details and hyperparameters are given in Appendix
De novo Generation We leverage the GuacaMol benchmark (Brown et al. 2019), an evaluation framework for de novo molecular graph generation. We consider two metrics that measure the similarity between the generated molecules and the training set. The Fréchet ChemNet Distance (FCD) compares the hidden representations of ChemNet, neural network capturing chemical and biological features of molecules (Preuer et al., 2018). and the KL divergence (KL) compares the probability distributions of variety of physiochemical descriptors. We also report novelty, uniqueness, and validity scores, which measure the fraction of generated molecules that are not the training set, that pairwise different, and that are chemically valid according to RDKit, respectively.
8

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation metrics on the ZINC250K dataset. Results for methods with * are taken from their original papers. Methods that do no report the KL metric are denoted with N.A.. We split the methods into autoregressive and all-at-once methods. The best scores within each category are in bold. EDM-SyCO outperforms all all-at-once baselines by more than 26% on the FCD metric. Standard deviations are across 3 different training runs of the models.
METHOD
FCD (t)
KL (t)
NOVELTY (1)
UNIQUENESS (1)
VALIDITY (t)
VA- 2018
0.75 + 0.01
0.93 + 0.00
1.00 0.00
1.00 + 0.00
1.00 : 0.00
ALL-AT-ONCE AUTOREGRESSIVG
GRAPHAF 2020
0.05 = 0.00
0.67 = 0.01
0.91 0.01
0.91 + 0.01
1.00 : 0.00
HIERVAE 2020
0,50 = 0.14
0,92 = 0.00
0.96 + 0.01
0.96 + 0.01
1.00 = 0.00
MOLER 202T
0.83 = 0.00
0.97 = 0.00
0.99 + 0.00
0.99 + 0.00
1.00 = 0.00
PS-VAE 2022
0.28 + 0.01
0.84 + 0.00
1.00 0.00
1.00 + 0.00
1.00 1 0.00
GRAPHARM 22023
0.04 0.00
N.A.
1.00 0.00
0.99 + 0.00
0.88 : 0.00
MICAM (2023)
0.63 0.02
0.94 = 0.00
0.98 0.00
0.98 + 0.00
1.00 = 0.00
MAGNET (2023
0.76 = 0.00
0.95 = 0.00
0.99 + 0.00
0.99 + 0.00
1.00 = 0.00
GDSS* 2022
0,10 0.01
N.A.
1.00 0.00
1.00 0.00
0.97 0.01
DIGRESS 2022
0.65 =
0.91 =
0,99 0.00
0,99 0.00
0.85 0.01
GROLDM 2023
0.17 :
0.79 =
1.00 0.00
1,00 0.00
0.12 0.00
GRUM 2023)
0.64 +
N.A.
1.00 0.00
1.00 0.00
0.99 0.00
SWINGNN (2023
0.67 =
N.A.
0,96 0.00
1.00 0.00
0.91
EDM-SYCo (OURS)
0.85
0.96
1.00 0.00
1.00 0.00
0.88 0.01
All scores, including FCD and KL, are normalized to lie between 0 and 1 and such that higher score corresponds better performance. More details on these metrics are in Appendix F.8 De novo generation is prerequisite for more complex tasks such as conditional generation and optimization.
Tables 3 and 4 show the performance of EDM-SYCo and the baselines on the ZINC250K and GuacaMol datasets, respec- tively. Notably, EDM-SYCO outperforms all all-at-once diffusion based models on the FCD metric by more than 26% ZINC250K and 16% on GuacaMol. This shows the ca- pacity of model to more accurately cap-
Table 4: FCD and KL scores on GuacaMol. means corresponding numbers are from (Maziarz et al. 2021). DiGress results are from Vignac et al. (2022). For our model, we additionally report standard devia- tion across 3 validations of the same model.
FCD (1)
KL (*)
AAO AUTOREG.
CGVAE
2018
0.26
N.A.
ture the joint distribution of nodes and edges encoded in the latent features and coordi- nates. EDM-SYCo sets the new state-of-the- art FCD score on ZINC250K, outperforming all autoregressive and fragment-buse base- lines, which incorporate more domain knowl- edge. Note that due to computational constraints, we use the same hyperparameters from ZINC250K to train our model on GuacaMol, while DiGress and MoLeR have been tuned for both datascts. Regarding validity, all autoregressive baselines achieve 100% validity scores due to valency checks after each intermediate step, which is usually not done for all-at-once methods. We show some sample molecules generated by our model in Figures 4 and 5 for the two datasets. We include an ablation study on the inductive bias our architecture mn Appendix G.10 compare to an EGNN-free encoder setting in Appendix G.II and provide runtime anaylsis in Appendix G..12
HIERVAE
2020
0.62
N.A.
JT V
2018
0.73
N.A.
MOLER
202T
0.81
N.A.
DIGRESS 2022
0.68
0.93
RDM-SYCO (OURS)
0.79
0.002
0.93
0.006
Condito-- Generation To evaluate the conditional generation performance of our approach, we fol- low Ninniri et al. (2023) and condi- tion the generation of 1000 molecules on LogP and molecular weight (MW) values sampled from ZINC250K. We report the mean absolute error (MAE) between the target and estimated values for the generated molecules (using RDKit) and the validity rate. More details on the experimental setup in Ap- pendix F.6) We run the regressor guidance algorithm, discussed Section [S] As baselines, we compare to DiGress, which uses discrete guidance scheme, and FreeGresss, which proposes
Table 5: Conditional generation results.
LoGP
MW
METHOD
MAE (L)
VALID (*)
MAE (L)
VALID (t)
DIGRESS
0.49 6.05
65%
60.30 + 6.10
73%
FREEGRESS
0.15 0.02
85 %
15.18 + 2.71
75%
EDM-SYCo
0.14
76%
3.86
88%

---

Published as a conference paper at ICLR 2025
Table 6: Cimilarity-constrain optimization results from (Jin al. 2020
We split all learning- based
baslines into translation and optimization approaches.
LOGP (SIM 0.4)
LOGP (SIM 2 0.6)
QED (SIM 0.4)
DRD2 (SIM 0.4)
METHOD
IMPROV.
DIV.
IMPROV.
DIV.
SUcc.
DIV.
SUCC.
DIV.
SEQ2SEQ
3.37 1 1.75
0.471
2.33 1: 1.17
0.331
58.5%
0.331
75.9%
0.176
OPTIM. TRANSL.
JTNN
3.55 =: 1.67
0.480
2.33
1: 1.24
0.333
59.9%
0.373
77.8%
0.156
ATOMG2G
3.98 = 1.34
0.563
2.41
4: 1.19
0.379
73.6%
0.421
75.8%
0.128
HIERG2G
3.98 = 1.46
0.564
2.49
=: 1.09
0.381
76.9%
0.477
85.9%
0.192
JT-VAE
1.03 1.39
0.28
8.8%
3,4%
CG- VAE
0.61
0.25 0.74
4.8%
2.3%
GCPN
2.49
0.79
9.4%
0.216
4.4%
0.152
EDM-SYCO
3.11 1.27
0.559
1.51 1.10
0.388
46.4% %
0.352
27.3%
0.303
regressor-free guidane mechanism Ninniri al. 2023) Table 5 shows the results. While EDM- SYCo achieves a comparable score to FreeGress on LogP, it outperforms by a factor of 3.9 on MW. Compared DiGress, improves LogP and MW by factors 3.5 and 15.6, respectively. This experiment highlights the benefit of our continuous latent space on conditional generation compared to operating directly on the discrete graph space. It also shows that graph-level properties are repre- sented sufficiently well in this space. We include additional related experiments in Appendix G.4 and Appendix G.5
Optimization We follow Jin et al. (2020) and evaluate our optimization procedure from Section 5 with EDM-SYCo their constrained optimization tasks: LogP, QED, and DRD2 on ZINC250K. These tasks consist of improving the properties of a set of 800 test molecules under the constraint that the Tanimoto similarity with Morgan fingerprints (Rogers & Hahn 2010) between the optimized and initial molecule is above a given threshold. We report the average improvement for the LogP tasks and success rates for the binary tasks of QED and DRD2 consisting of translating molecules from low range into higher range. Additionally, we report the average diversity among the optimized molecules. More details are given in Appendix F.7
We split the learning-basse based baselines from (Jin et al. 2020) into optimization and translation ap- proaches. Translation approaches are specifically designed for such tasks and are trained on pairs of molecules exhibiting the improvement and similarity constraints, giving them an advantage over optimization methods. Since we leverage the same generative model and regressor from previous experiments without further training on this task, optimization approaches are the main point of comparison. Tj that EDM-SYCo outperforms all optimization baselines by an average factor of 3.5 across all tasks while achieving comparable performance the translation approaches. These results further support the effectiveness of our SYCo framework and our novel constrained optimization algorithm. Figures 6 and illustrate some of EDM-SyCO's optimized molecules. Thanks to our atom- approach, the optimized molecules frequently only change in a few atoms from the starting molecule, which would be harder achieve for fragment-based models.
7 CONCLUSION
Despite the common practice of using fragment-based autoregressive models for molecular graph generation, we demonstrated with SYCo that a mapping between molecular graphs and latent Euclidean point clouds enables atom-based all-at-once approaches to competitive with special- purpose molecular graph generators. Further, we introduced similarity-constraiie procedure for 3D diffusion models based on guidance and inpainting, Based on this framework. we developed EDM-SYCO, which sets new state-of-the-art FCD score on ZINC250K. In our conditional generation and optimization experiments, we found our guidance- accurately guide the generation process, outperforming all basclines. With these results, conclude that latent Euclidean generative models hold significant promise for advancing molecular graph generation and accelerating drug discovery. We discuss work's broader impact in Appendix B
optimization
approach to
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This project is supported by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research. Further, it funded by the Federal Ministry of Education and Research (BMBF) and the Free State of Bavaria under the Excellence Strategy of the Federal Government and the Länder. This project is also supported by the Bavarian Ministry of Economic Affairs, Regional Development and Energy with funds from the Hightech Agenda Bayern.
REFERENCES
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:1 19911-1993,, 2021.
Sirine Ayadi, Leon Hetzel, Johanna Sommer, Fabian Theis, and Stephan Günnemann. Unified guid- ance for eeomeetry-condiioone molecular generation. Advances in Neural Information Processing Systems, 37:138891-13892, 2025.
Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy- guided sde for inverse molecular design. arXiv preprint arXiv:2209. 15408, 2022.
Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling. 59(3): 1096-1108, 2019.
Stefan Chmiela, Alexandre Tkatchenko, Huziel E Sauceda, Igor Poltavsky, Kristof T Schütt, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances, ((5)e160300)) 2017.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv:1903.05799 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Victor Garcia Satorras, Emiel Hoogeboom, Fabian Fuchs, Ingmar Posner, and Max Welling E (n) equivariant normalizing flows. Advances in Neural Information Processing Systems, 34:4181-4192, 2021.
Johannes Gasteiger, Chandan Yeshwanth, and Stephan Günnemann. Directional message passing on molecular graphs via synthetic coordinates. Advances in Neural Information Processing Systems, 34:15421-1543,, 2021.
Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapted generation 3d point sets for the targeted discovery of molecules. Advances in neural information processing systems, 32, 2019.
Zijie Geng, Shufang Xie, Yingce Xia, Lijun Wu, Tao Qin, Jie Wang, Yongdong Zhang, Feng Wu, and Tie-Yan Liu. De novo molecular generation via connection-aware motif mining. arXiv preprint arXiv:2302.01129, 2023.
Rafael Gómez-Bombarelli Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Sánchez-Lengeling
Benjamín Sánchez- Lengeling, Dennis Sheberla, Jorge Aguilcra-Ipane Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268-2) 2766, 2018.
Leon Hetzel, Johanna Sommer, Bastian Rieck, Fabian Theis, and Stephan Günnemann. Magnet: Motif-agnostic generation of molecules from shapes. arXiv preprint arXiv:2305. /9303, 2023.
Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. arXiv preprint arXiv:2207.12594 2022.
11

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-653 68111, 2020.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pp. 8867-8887.
PMLR, 2022.
John J Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan Coleman. Zinc: a free tool discover chemistry for biology. Journal of chemical information and modeling, 52(7): 1757-1768, 2012.
José Jiménez-Luna, Francesca Grisoni, and Gisbert Schneider. Drug discovery with explainable artificial intelligence. Nature Machine Intelligence. 2(10):573-584, 2020.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2323-2332. PMLR, 2018.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International conference on machine learning, pp. 18899-8442 PMLR, 2020.
Jaehyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. In International Conference Machine Learning, pp. 10362-10383. PMLR, 2022.
Jachyeong Jo, Dongki Kim, and Sung Ju Hwang. Graph generation with diffusion mixture. arXiv preprint arXiv:2302.035500 2023.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances Neural Information Processing Systems, 35:26565-26517 2022.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. arXiv preprint arX1v:2312.02666, 2023.
Hyunho Kim, Eunyoung Kim, Ingoo Lee, Bongsung Bae, Minsu Park, and Hojung Nam. Artificial intelligence in drug discovery: a comprehensive review of data-driven and machine learning approaches. Biotechnology and Bioprocess Engineering. 25:895-9300 2020.
Gary King and Langche Zeng. Logistic regression in rare events data. Political analysis, 9(2): 137-163, 2001.
Lingkai Kong, Jiaming Cui, Haotian Sun, Yuchen Zhuang, Aditya Prakash, and Chao Zhang. Autoregressive diffusion model for graph generation. In International conference on machine learning, pp. 17391-17408. PMLR, 2023.
Xiangzhe Kong, Wenbing Huang, Zhixing Tan, and Yang Liu. Molecule generation by principal subgraph mining and assembling. Advances Neural Information Processing Systems, 35: 2550-2563, 2022.
Greg Landrum al. Rdkit: Open-source cheminformatics, 2006.
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs. arXiv preprint arXiv:1803.03324. 2018.
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander Gaunt. Constrained graph variational autoencoders for molecule design. Advances in neural information processing systems, 31, 2018.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
12

---

Published as a conference paper at ICLR 2025
Krzysztof Maziarz, Henry Jackson-Flux, Pashmina Cameron, Finton Sirockin, Nadine Schneider, Nikolaus Stiefl, Marwin Segler, and Mare Brockschmidt. Learning to extend molecular scaffolds with structural motifs. arXiv preprint arXiv: 03864, 2021.
Konstantin Mishchenko and Aaron Defazio. Prodigy: An expeditiously adaptive parameter-free learner. arXiv preprini arXiv:2306.06100 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171, PMLR, 2021.
Matteo Ninniri. Marco Podda, and Davide Bacciu. Classifier-free graph diffusion for molecular property targeting, arXiv preprint arXiv:2312.1739) 2023.
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permu- tation invariant graph generation via score- based generative modeling. International Conference on Artificial Intelligence and Stafistics, pp. 4474-4484. PMLR, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, et al. Molecular sets (moses): benchmarking platform for molecular generation models. Frontiers in pharmacology. 11 11:566644, 2020.
Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Gunter Klambauer. Fréchet chemnet distance: metric for generative models for molecules in drug discovery. Journal of chemical information and modeling, 58(9):1736-117 1741, 2018
Sereina Riniker and Gregory Landrum. Better informed distance geometry: using what we know to improve conformation generation. Journal of chemical information and modeling. 55(12): 3562-2574 2015.
David Rogers and Mathew Hahn. Extended-conneectvvty fingerprints. Journal of chemical information and modeling, 50(5):742-754, 2010.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning, pp. 9323-9332. PMLR, 2021.
Nadine Schneider, Roger A Sayle, and Gregory Landrum. Get your atoms order an open-source implementation of a novel and robust molecular canonicalization algorithm. Journal of chemical information and modeling, 55(10):21 1-2120, 2015.
Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blundell, Pietro Lió, Carla Gomes, Max Welling, al. Structure-baset drug design with equivariant diffusion models. arXiv preprint 200.::2221.139999 2022.
Ansgar Schuffenhauer, Peter Ertl, Silvio Roggo, Stefan Wetzel, Marcus A Koch, and Herbert Waldmann. The scaffold tree- visualization of the scaffold universe by hierarchical scaffold classification. Journal of chemical information and modeling. 47(1):47-58, 2007.
Kristof Schütr,, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robal Müller, Schnet: continuous-file convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
Marwin HS Segler. Thierry Kogej, Christian Tyrchan, and Mark P Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS central science, 4(1): 120-131, 2018.
13

---

Published as a conference paper at ICLR 2025
Jean-Pierre Serre al. Linear representations of finite groups, volume 42. Springer, 1977.
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: flow-based autoregressive model for molecular graph generation. arXiv preprint arXiv:2001.09382 2020.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. pp. 2256 2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011..33456. 2020.
Yuxuan Song. Jingjing Gong, Minkai Xu, Ziyao Cao, Yanyan Lan, Stefano Ermon, Hao Zhou, and Wei-Ying Ma. Equivariant flow matching with hybrid probability transport for 3d molecule generation. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- -Boook, Kilian Fatras, Guy Wolf, and Yoshua Bengio. Conditional flow matching: Simulation-fiee dynamic optimal transport. arXiv preprint arXiv:2302.00482 2023.
A Vahdat, K Kreis, and J Kautz. Score-based generative modeling in latent space. arxiv, 2021.
Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint vrXhv:2209.14344 2022.
David Weininger. Smiles, chemical language and information system. 1. introduction methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1)331-36, 1988.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688, 2011.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923 2022.
Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovee. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning, pp. 38592-38610. PMLR, 2023.
Qi Yan, Zhengyang Liang, Yang Song, Renjie Liao, and Lele Wang. Swingnn: permutttion invariance in diffusion models for graph generation. arXiv preprint 27X1v:2307.01666, 2023.
14

---

Published as a conference paper at ICLR 2025
A VISUALIZATIONS FOR GENERATED AND OPTIMIZED MOLECULES BY EDM-SYCo
We visualize some of the molecules generated by EDM-SYCo. For de generation, show molecules of the model trained on ZINC250K in Figure 4 and of the one trained on GuacaMol in optimization task, we show some molecules found by our
Figure For the similartt-connsrraiee optimization approach in the LogP task in Figure 6 and in the QED task in Figure 7
peot
teog
ginon
Fard
gut
Sm
loxer
cad
dears
Figure 4: Sample molecules generated by EDM-SYCO trained on ZINC250K.
15

---

Published as a conference paper at ICLR 2025
1xp0% 10
oogg
Aooooo
Brueso
uyplactts
9:40
G-Rloggge
ta.
Kopaaiioo
%
Ocr
Oylo
A HPb
Tho 0
C-alli 0
800
Figure 5: Sample molecules generated by EDM-SYCo trained on GuacaMol.
16

---

Published as a conference paper at ICLR 2025
Plage=-6.59
imp=+8.31, sim==.50
imp=++743, sim.0.54
imp.#+7.42 sim=0.54
imp.=+6.88, sim.=0.48
OH
61.1Pe-6.16
imp.*+7.1,, sim.0.48
imp.*+3.22. sim-0.44
imp.*+334, sim=0.47
imp.--191, sim.=0.40
PoogP=66.00
imp=++.65,, sim.=0.52
imp.=+6.57, sim.-051
Imp.=+5.13, sim.=0.47
imp.*+.166 smm=0.10
PlogP=-6.30
imp.*+6.64,
imp.=+6.07,
imp.=+5.67, imp. sim.0.422
PlogPe-4.68
16p.-1661, sim.-055
imp.++5.33, sim.-0.88
imp.-13866 sim-0.56
imp.++3.88, sim.-0533
PlogP=-4.50
imp.=+6.46, sim.=0.43
imp.=+6.41, sim.=0.46
imp.=+6.40, sim.=0.66
Imp=++.14, sim.=0.43
PlagP=-6.18
imp.=46.45, sim.0.43
imp.=+4.1,, sim.0.54
imp.++4.3, smm=0.43 imp.=+4.06, sim:=0.42
Figure 6: Molecules with the highest improvement in the LogP-constrained optimization task. Each row corresponds test molecule and successful optimization results. We show the LogP value of the initial molecules, and for the optimized molecules, the achieved improvement (imp.) and the Tanimoto similarity the initial molecule (sim.).
17

---

Published as a conference paper at ICLR 2025
HO
QED=0.703
imp.=+0.243, sim.=0.44
imp.==0.243, s0m=0.40
imp=+0.243, sim.=0.42
imp=+0.245 243, sim=0040
QED+0.704
imp=++0.42, sim-0.47
imp.-10.242, sim.00477
imp.=10.242, sim.=0448
imp.-10,242, sim.-0.51
HB
QED-0.712
imp.=+0.230 sim.=045
imp.=10.236. sm-00.48
imp.=+0.236, sim.=0445
Imp.=+0.236, sim.=0.51
QED=0.715
imp.=-0.234,
imp=+0.233,
imp=+0.232, sim=0.44
imp.=+0.229, sim.00.22
QED=0,702
imp.-10.230, sim-0.41
imp.+00227. s.5-0.52
imp.=+0.221, sim-0.42
imp.-10,220, s.5-0.53
QED=0.705
imp.=40,229, sim.0.65
imp.=+0.216, sim.00.33
imp=+00.08. sim==0.33
imp==+0.206 2020, s.m=0666
QED=0.706
imp.=+0.225, sim.=0.46
imp.=+0,225, sim=0.43
imp.++0.224, sim.=0.43 imp=+0 224, sim:0045
Figure 7: Molecules with highest improvement in the QED constrained optimization task. Each row corresponds to test molecule and 4 successful optimization results. We show the QED value of the initial molecules, and for the optimized molecules, the achieved improvement (imp.) and the Tanimoto similarity the initial molecule (sim.).
18

---

Published as a conference paper at ICLR 2025
B BROADER IMPACT
This paper advances the field of computational drug discovery by proposing efficient methods for generative models. While there may be unintended consequences, such as the misuse of chemical weapons, we firmly believe that their benefits significantly outweigh the minuscule chance of misuse.
C INVARIANCE AND EQUIVARIANCE
In this work, we dealing with two types of symmetries arising from how we represent the data and/or physical symmetries. Specifically, graphs are invariant to permuting their nodes, meaning that single graph of N nodes can be represented by N! different representations corresponding N! different orderings of its nodes. We refer to this transformation as the elements of the symmetric group Sy. Similarly, a point cloud representing molecule and/or translated and still refer to the same molecule. This corresponds to an infinite number of different representations for the same object. We refer to this transformation as the action of the Euclidean group E(3).
can be arbitrarily rotated, reflected,
C.I SN- AND E INVARIANNT TRAINING PROCESS
To efficiently learn from such data, we need to develop training methods invariant to these symmetries without needing data augmentation techniques. Concretely, we must ensure that the gradient updates used to train our model do not change if the molecular graph is permuted and/or the latent point cloud representation is rotated/refecnted/raasaeeee equivariant architecture and an invariant loss (Vignac et al. 2022). Based on the model architecture and the loss function described in the main text, we can show that EDM-SYCo satisfies both requirements and that its training process is, thus, invariant to the action of the permutation group Sw and the Euclidean group E33)
We require two ingredients to achieve this: an
As our ultimate goal is generate molecular graphs, we provide more details on the permutation invariance properties of the distribution of molecular graphs learned by EDM-SYCo the next section.
C.2 PROOF OF PROPOSITION 4.1 (Sx- INVARIANT MARGINAL DISTRIBUTION]
In this section, we provide a formal proof for the statement from the main text that the molecular graph distribution defined by EDM-SYCo invariant permutations. For convenience, we provide the proposition again:
Proposition C.1. The marginal distribution of molecular graphs (G) = Epp(ze) lyr =0)]
defined by the EDM and the decoder, is an S -i-invariant distribution, i.e. for any molecular graph G = A). Po.g (h, A) (Po, PAP) for any permutation matrix P Sx-
This invariance property is required for efficient likelihood computation, as the likelihood of graph is the sum of the likelihoods of its N! permutations, which is intractable to compute. However, when ensuring that all these permutations are assigned equal likelihood, suffices to compute the likelihood of an arbitrary permutation.
Note:
Proof. The idea of the proof is when the initial distribution of the diffusion model pa (2T) invariant and the transition distributions po(z1-1(2,) diffusion time steps po (z1) are invariant, including (2())) [u et al. 2022 2023 Hoogeboom et al.) 2022). With the same logic and with the equivariant decoder taking the place of the transition distribution, the graph distribution (9) will be invariant.
equivariant, then all marginal distributions at all
We prove this result by induction:
Base case: (0(z)) = =A -2(22; I) is permutation-invariae i.e. (0 = Po (P2r) for all permuta- tion matrices P E {0, 1] N N acting on ž7 E R (d4+3) by permuting its rows. This holds because all i.i.d., as po (2r) has diagonal covariance matrix.
19

---

Published as a conference paper at ICLR 2025
Induction step: Assume po (=())) is permutation-invarian We show that if p=(=(11 is permutation- equivariant, i.e. po(2t-1(2:) = po Po 1 PPz) (which holds when using permutation-equuiarlaatt architecture E6), then po 12 will be permutation-invarian.
pe (
Po Po (0(P2-112))00(2))
(Chain rule of probability)
Po P0(121111 PP Po (PP-1
(PP-1=1)
po (/22--11-- PP-1 21) (p- (P (P-1 (2)
(Equivariance and invariance)
/ pe pe(=1-11 u ()) det P
(Change variables u p-1 2, and P is orhtogonal, so P =1)
=po(=(-1)
By induction, (o(z)). ... Po (z)) are all commuttaioo-invariin
Finally, since the decoder Pş (9) zu) is commutatioo-equivarntn (as it applies the same operation on each node and each pair of nodes, thus not depending on the nodes ordering). with the same derivation, we also conclude that the final induced distribution po.c permutation-invariee
Note: similar derivation can also be applied show the invariance of p=() to rotations and reflections of the coordinates, as the only requirement on P for the derivation to hold is to be orthogonal, which is the case for rotation and reflection matrices R.
D E(n) EQUIVARIANT GRAPH NEURAL NETWORKS (EGNNS)
To perform equivariant updates our latent space, we use the E(n) Equivariant Graph Neural Network (EGNN) architecture (Satorras et al. 2021).
An EGNN operates on a point cloud of size N, where each point has features h2 E Rd and coordinates I₁ 3 associated with it. Let h R. and æ e Nx3 be the stacked features and coordinates, respectively, of all points. An EGNN composed of Equivariant Graph Convolutional Layers (EGCLs) h+1 o EGCL(h!, a.)). A single EGCL layer is defined
my = de (l1, hly, di., a.)
(12)
h+1 oi (h1, Σ E.ymiy))
(13)
jf1
a 2 +
2z (ti M/y, a.y,, aiy),
(14)
dij
+1
where 1 denotes the layer index, d, = |o 21112 is the Euclidean distance between points and j, and asj are optional edge attributes which we set to ||^ - 29112. diy serves as attention mechanism that infers soft estimation of the edges éy = Øins (my). Note that to update each point's features and coordinates, the EGCL layers consider all the other nodes, effectively treating the point cloud as fully connected graph. All learnable components (de, Φr. &z. and Φinf) are fully connected neural networks. An EGNN architecture is then composed of L EGCL layers, denoted as h -t EGNN(hO, x² with hd h and x æ. The main hyperparameters of the EGNN architectures are the number of layers L and feature dimension n used to control the width the fully connected neural networks.
E ADDITIONAL METHOD DETAILS
In this section, we provide additional details for method introduced in Sections
4 and
20

---

Published as a conference paper at ICLR 2025
E. TRAINING AND SAMPLING
We start by describing in more detail the training and sampling algorithms for EDM-SYCo in Algorithms 2
Algorithm Training Algorithm
Input: a dataset of molecular graphs 9 (h, A) Initial: Encoder network &g. Decoder network D,, denoising network co
First Stage:
Autoencoder Training
repeat
2 + (h, A) (Encoder)
G, ~ NN,, I) and subtract center mass from cz
(at
2 4 P2 + O0Ez
h, Å De(z)
(AA
h A
AE E) + cross-entropy
Take an optimizer step on ( until & and & have converged Second Stage: Diffusion Model Training Fix Autoencoder parameters & and f repeat
20 qá (= A)
62 ~ (0, I) and subtract center of mass from 6z
(0, 1...., T)
2: + Va + VI o₁Ez
(D) - ||Ez E0(ze, 1)
Take an optimizer step on LDM (0) until 0 has converged
return Eg. De, Eg
Algorithm 2 Sampling Algorithm
Input: Decoder network Dg. denoising network co
2T (NO I) and subtract center of mass from =T (m) for t = T downto do
6 (0, I) and subtract center of mass from (wr
2,-1 Var 22z
VI-a. co(ze:t) 1)) + 0,62
1-a,
end for h A
D2()
return 2D molecule 9 = (h, A)
E.2 CONDITIONAL GENERATION
Here, we detail the regressor guidance algorithm we developed in Section 5 text that we can sample from the conditional distribution ((2/ /c) by replacing the predicted noise of the denoising neural network by Equation 7 By putting this back into the sampling equation, Equation 6 we get the following modified denoising step
We showed in the main
2 tz)(
SV (0,(2x) c)² + occ.-
(15)
Va
A more general formulation for the new guidance term is given by st) 2z. (2,, c). where s(t) is time-dependent scaling function and can be any invariant loss function. the specific case that we derived, s(t) 1-as and ((z, c) = (z,) c)2. can be replaced by any other loss function that /az want to optimize, or in the of property maximization, it can be the prediction of the regressor
21

---

Published as a conference paper at ICLR 2025
directly (Equation 8). In experiments, found simple linear schedule s(t) at to perform well, where a E IR is hyperparameter that we tune for each task.
To ensure the equivariance of the distribution even with guidance, the new term s(t) V (x,, c) needs to be [(3)-qqiiaarian.. This is done by subtracting its center of mass at each sampling step and by choosing invariant loss function 1, as the differential operator of an invariant function yields equivariant function (Chmiela et al. 2017 Schütt et al. 2017).
E.3 SCAFFOLD-CONSTRAINEE GENERATION
Here, we provide additional details for the inpainting algorithm described Section [ where we described an approach that allows to generate point cloud that contains chosen scaffold.
Lugmayr et al. (2022) observed that the version of their algorithm described in the main text leads to inconsistent completions and proposes to each sampling step multiple times allow the model to harmonize its generations. Concretely, each denoising step is repeated r times to improve the generations' quality and allow the model to harmonize the unknown part the scaffold. This is achieved by applying the forward diffusion model (Equation?) 2 after each denoising step. Lugmayr al. (2022) also propose to go more than one forward step with the diffusion model to allow the model to better harmonize its generations. This choice is fixed by the jump length j. Note that this procedure increases the computation cost of running the model times, which might be limiting factor. For this reason, we combine this approach with the approach described in Appendix that allows to reduce the number of reverse diffusion steps to run.
The procedure described in the main text gives way to condition the diffusion model, which operates on the latent point cloud space. However, we are ultimately interested in generating molecular graph containing a scaffold defined by subset of atoms and the bonds between them. We map the molecular scaffold to its point cloud representation using our encoder to get m @ Zg- This enables running the inpainting algorithm described above. Then, decode the sampled 3D point cloud to a molecular graph. Since our decoder model operates on each node and edge separately, the final molecular graph is guaranteed to contain the original scaffold as long as the scaffold in isolation can be correctly reconstructed with our autoencoder.
F IMPLEMENTATION DETAILS
F.1 LICENSES FOR USED CODE, MODELS, AND DATASETS
We start by listing the assets we used in our work and their respective licenses. We use the following tools, codebases, and datasets:
* RDKit [Landrum et al.
2006
(BSD 3-Clause License)
PyTorch PPaszke et al.
2019)
(BSD 3-Clause License)
EDM (Hoogeboom et al.
2022)
(MIT license)
GeoL.DM (Xu et al.
2023 (MIT license)
ZINC250K dataset [rwin et al.
2012
(Custom license)
GuacaMol dataset (Brown et al.
2019 (MIT license)
F.2 ARCHITECTURE DETAILS
We train two sets of models with the same architectures on the two used datasets, ZINC250K and GuacaMol. The EDM, together with the autoencoder, has 9.2M total parameters, while the regressor has 4.2M parameters.
Encoder. The first part of the encoder E, introduced in Section 3 is the conformer generation method, while the second part is an EGNN with layer and 128 hidden features (See Appendix D). We found that having least I EGNN layer is crucial to achieve high reconstruction accuracy for the autoencoder, and as this allows us to reach more than 99% accuracy, we did not use more layers. This EGNN takes input a point cloud with the one-hot encodings of the atom types and formal charges
22

---

Published as a conference paper at ICLR 2025
as features h and the synthetic coordinates computed by the conformer generation part coordinates x. The output is a processed point cloud with continuous node embeddings of dimension d = 2 and updated coordinates.
Decoder. The decoder D, also introduced in Section 3 consists of 2 fully connected networks MLP and MLP edges both having 256 hidden dimensions.
EDM. The denoising network of EDM is parametrized with an EGNN with 9 layers and 256 hidden features. The EDM has T = 1000 diffusion steps.
Regressor. The regressor used for the conditional generation and optimization experiments (sce Section (.2) is also EGNN with 4 layers and 256 hidden features, followed by sum pooling operation over the node features and then a fully connected network with layers and a hidden dimension of 256 that maps the pooled graph embedding to the target values of the properties. p
F.3 TRAINING DETAILS
We use the original train/validation/test splits of the used datasets (Irwin et al., 2012 Brown et al. 2019). The autoencoder is trained in the first stage to minimize the cross-entropy loss between the ground truth and predicted graphs for a maximum of 100 epochs, with early stopping the validation accuracy does not improve for 10 epochs. To deal with the class imbalance problem caused by the sparsity of the graph adjacency matrices and the dominance of some atom types and formal charge values, we scale each term of the cross-entropy loss with a class-specific value based the statistics of the training set (King & Zeng 2001). In the second training stage, the EDM model is trained for 1000 epochs on ZINC250 and approximately 300 epochs on GuacaMol.
The regressor is trained with the L1 loss to predict the molecular properties on the noisy latent point cloud representations of the molecules by applying the same noise schedule of the EDM. The regressor is trained for 500 epochs, with early stopping if the MAE on the validation set does not improve after 50 epochs.
All models are trained with a batch size of 64 and using the recent Prodigy optimizer (Mishchenko) & Defazio 2023) with dccef 0.1, which we found be a very important hyperparameter for the stability of training.
We train all models on ZINC250K on a single Nvidia A100 GPU, and on GuacaMol, we use multi-GPU training on 4 Nvidia A100 GPUs.
F.4 DATASET DETAILS
We use two datasets in our experiments: ZINC250K (Irwin et al. 2012) and GuacaMol (Brown et al. 2019). For both datasets, we represent a molecule as a graph. The nodes are atoms with the one-hot encoding of their atom type and their formal charge features. In addition to modeling all heavy atoms, we also model explicit H atoms, which we found to increase the validity of the generated molecules it reduces some kekulization errors produced by rdkit The edges are chemical bonds with the one-hot encoding of the bond type as features. We model the following bond types: no-bond, single, double, triple, and aromatic. We model the absence of bond as a separate bond type to allow the autoencoder to reconstruct the full bond information. These graphs present very unbalanced scales for the atom types, formal charges, and bond types. We use class-specific weights to train the autoencoder, as outlined in Section 3
ZINC250K has total of 250K molecules and 220,011 training molecules. It has 10 atom types (including H) and 3 formal charge values (-1, 0, 1), and the largest molecule has 40 atoms.
GuacaMol has a total of 1.5M molecules, of which we use 1,273, 104 for training. It has 13 atom types (including H) and 5 formal charge values (-1, 0, 2, and the largest molecule has 88 atoms.
We compute the synthetic coordinates needed for our model in preprocessing step for both datasets and use the computed coordinates throughout the training.
23

---

Published as a conference paper at ICLR 2025
F.5 SAMPLING THE NUMBER OF ATOMS
In this work, the diffusion generative model operates fixed-size point cloud N. During sampling, the first step is to sample this number N. To sample molecules of similar sizes to those in the training set, approximate the distribution of the number of nodes using categorical distribution.
To this end, we count the number of occurrences of each molecule size in the training set and use the categorical distribution parameterized by the normalized counts sample the number of atoms of generated molecules. This simple procedure works well for the task of unconditional generation. However, in the property-conditioned generation, we must consider that some properties correlated to molecule size, such as molecular weight (MW). Therefore, for this task. we start by dividing the range of property values from the training set into 100 equal-sized bins. We define a different categorical distribution for each bin that counts the sizes of molecules falling in that bin. Given the conditioning value, we first specify the bin it falls into and use the corresponding categorical distribution to sample the number of nodes. Finally, for molecule optimization, we curate a dataset of molecule pairs that exhibit the desired property improvement and similarity constraints and create different conditional categorical distributions, one for each size of the initial molecule.
highly
Note that these different procedures are needed due to the inherent limitation of diffusion models on graphs that require the number of nodes to be known and fixed in advance. Fundamentally, solving this issue would be a valuable contribution to the field, and we leave it for future work.
F.6 PROPERTY-CONDITIONAL GENERATION EXPERIMENT
Following et al. 2023), for each task, we sample 100 molecules from the test set and condition on their property values generate 10 valid molecules each and report the mean absolute error between the target and the estimated values for the generated molecules measured by RDKit. We experimented with different values of the guidance scale and with using the and l2 losses to compute the guidance loss but found the l2 loss with scale values = 2.5 and = 1.0 for LogP and MW, respectively, to work best.
F.7 CONSTRAINED MOLECULE OPTIMIZATION
Here, we provide more details on the constrained optimization tasks described in Section 6 Given a molecule G, these tasks aim to find different molecule G' with higher property values satisfying the similarity constraint simi( (G, G') > 0.4 (or 0.6), where sim computes the Tanimoto similarity with Morgan fingerprints (Rogers& & Hahn 2010). For the LogP task, the new molecule G' needs to have higher LogP value and we report the average improvement over 800 molecules from the test set with the lowest LogP values. For the QED task, the goal is optimize another 800 molecules from the test with QED values in the low range [0.7, 0.8] into the high range (0.9, 1.0). and we report the success rate over these molecules. For DRD2, the goal is to translate inactive compounds (p S 0.05) into active ones (p Z 0.5), and we similarly report the success rate over these molecules. In addition, for all tasks, we report the diversity as the average pairwise distance between the successfully optimized molecules for each initial test molecule. The distance is defined as dist(G, G') = 1 - sG g').
Choosing As described in Section 5 the intermediate time step t controls the trade-off between the similarity to the initial molecule and the capacity to improve its target properties. In principle, for each test molecule, there an optimal yielding the highest improvement under the similarity constraints. However, finding the optimal & for each single molecule is computationally expensive. Therefore, we use a subset of 20 molecules and run our optimization algorithm for different values of on these molecules to find the value of i that yields the highest average improvement under the considered similarity constraints. We use this value for all test molecules. Our experiments found that value of t in the range [500,700 usually works best with the total T 1000. Figure 8 shows the effect of the time step t on the similarity and property value for one molecule from the test set. Table lists the exact value of l used for each task.
24

---

Published as a conference paper at ICLR 2025
c
0.6
Tanimoto Similarity
Improvement PLog?
Tanimoto Similarity
Improvement in PLogP
0.4
0.5
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
200
400
600
800
1000
200
400
600
800
1000
Diffusion Time
Diffusion Time
0.6
Tanimoto Similarity
M
00
improvement in PLogP
Tanimoto Similarity
Improvement in PLogP
0.6
0.4
0.4
0.3
0.2
0.2
0.1
200
400
600
800
1000
200
400
600
800
1000
Diffusion Time
Diffusion Time
Figure Effect using different time steps on the similarity and improvement values in the simillrity.constraiedd optimization task of PLogP. We visualize 4 different molecules from the test set.
Table Values of the time step used in optimization tasks.
TIME STEP
PLoGP (SIM 2 0.4)
600
PLogP (SIM 2 0.6)
500
QED
650
DRD2
650
F.8 EVALUATION DETAILS
We provide additional details about the computation of the FCD and KL metrics (Brown al. 2019) and additional insights into the novelty and uniqueness metrics. Additionally, we provide a detailed anaylsis of the validity rate of our model.
KL: The following descriptors are computed for the generated and reference molecules: BertzCT, MolLogP, MolWt, TPSA. NumHAAcceptors, NumHDonors, NumRotatable NumAliphatticRing, NumAromatiicings, and similarity to nearest neighbor with ECFP4 fingerprints. The KL divergence D is computed for each descriptor between the two sets of molecules and aggregated to final normalized score via 1 - exp(-DxL.s).
FCD: The FCD is computed based on the hidden representations of molecules ChemNet, trained for predicting biological activities, similarly to the FID usually applied to image generative models. Concretely, the means and covariances of the last hidden activations of ChemNet computed for the reference and generated molecules and the Frechet distance between them is computed. This distance is then normalized via exp( -0.2 FCD) to lie between 0 and 1.
25

---

Published as a conference paper at ICLR 2025
Interestingly, most of the methods used in this work achieve novelty and uniqueness scores close to 100% on the two datasets ZINC250K and GuacaMol. These high scores are mainly due to the larger molecular sizes in datasets compared to smaller datasets such as QM9. The number of possible molecules grows exponentially with the molecule size, making it increasingly unlikely to sample exact replica of the training molecules or sample the exact molecule twice (note than even if a single atom or bond type is different, the molecules are considered different).
Novelty & Uniqueness:
Validity: We analyze the invalid molecules generated by our model and find the following common errors raised by RDKit: Kekulization errors ( (55%) Valency errors ( (33% 0 non-ring atom is marked aromatic 12%)..
All these errois are due inconsistent bond type predictions for a given molecule. Because the autoencoder achieves a near-perfee reconstruction accuracy on the training and test sets, this inconsistency is likely due a mismatch between the training distribution and the generated 3D molecules distribution. This could be improved by using more powerful 3D generative models, or one could try to jointly train the autoencoder and the latent diffusion model to adapt the decoder to the distribution of generated molecules.
G ADDITIONAL EXPERIMENTS
In this section, we provide additional experiments to showcase the performance of our model and analyze some of its properties.
G.1 BOND TYPE PREDICTION FROM SYNTHETIC COORDINATES
A crucial part of successfully applying our method is to be able to accurately reconstruct the molecular graph from the Euclidean point cloud. This section provides an ablation study to compare approach to other methods.
Formally, we are given point cloud with features h E {0, 1 containing the atom type and formal charge of each atom and coordinates æ E RR NS x5 computed using the conformer generation algorithm ETKDG (Riniker& & Landrum 2015). The goal is to reconstruct the bond types stored in the adjacency matrix E {0, 1) I X We test the following approaches:
Rule-based: This approach is widely used by generative models for molecules in 3D (Hoogeboom et al.] [2222] Xu et al., (023) and consists of using typical bond type lengths to infer the bond types between two atoms based on their features and the inter-atomic distances between them It uses set of if-else statements and predicts a specific bond type the length falls within a specific range that takes the features of the two atoms into account.
MLP: We develop this baseline to generalize the lookup table approach and replace the set of if-else statements by an MLP. Formally, we construct edge features between atoms and j as [hi,, hy, day]. where dy is the Euclidean distance between the two atoms. This is also the same information that the lookup table method above uses to infer the bond types. Instead, we train an MLP to predict the bond type from this edge feature in a classification setting. Note that this MLP is defined on the edge level, so a molecule with N atoms provides N2 training examples for the MLP to learn from. Also note that we do not need to have an extra MLP for the atom type as they are directly available this case since we do not run an EGNN in the first step.
The major limitation of the above approaches is that to predict the bond type between two atoms they only consider the distance between them and their respective features, without considering other atoms. However, not all molecules exhibit typical bond lengths between their atoms (Hoogeboom al. 2022). To overcome this limitation, we combine the MLP approach with an EGNN that first computes updated coordinates and
EGNN + MLP:
http://wwwww.nccc..o.oooomm//m/a permitted
26

---

Published as a conference paper at ICLR 2025
atom features considering the whole graph before running the local MLP to predict the bond type. Here, we add an extra MLP to predict the node labels (type and formal charge). This is the same architecture presented in the main text.
We evaluate these three approaches on ZINC250K using the synthetic coordinates computed as outlined in Section[] We report the molecule reconstruction accuracy on the validation set, where a molecule considered correctly reconstructed if all of its atom features (atom types and formal charges) and bond types are correctly reconstructed. Table TableTshowe shows that the EGNN approach achieves an almost perfect reconstruction accuracy. This shows that (i) the synthetic coordinates contain all the needed information to reconstruct the bond types, which might not be directly clear if limited to the other methods, and (ii) incorporating the global graph information into the prediction task crucial accurately reconstruct the bonds. We adopt the EGNN approach into model.
G.2 CONFORMER GENERATION COMPARISON
To analyze the effect of different conformer generation algorithms on the performance of our model, perform ablation study using different algorithms/packages from RDKit to compute the synthetic coordinates: the ETKDG algorithm and the Pharm3D package. We perform two training runs using the same setup except for the conformer generation method. For these two runs, smaller models (we halve the number of layers) than those reported in the main text and train for 500 epochs, they are not directly comparable. Results shown Table 8
Table 8: Model performance ZINC using different conformer generation methods from RDKit.
ETKDG
PHARM3D
VAE RECONSTRUCTION ACCURACY
99.92%
99.90%
KL
0.94
0.93
FCD
0.75
0.70
G.3 COMPLETE BENCHMARK RESULTS
We provide complete benchmark results for the GuacaMol benchmark (Brown et al., 2019) and the MOSES benchmark (Polykovskiy et al. 2020 in Tables andd 10 respectively. In addition to the baselines reported in the main text, we add two SMILES-basee baselines and we add different flavours of MoLeR with different vocabulary sizes, denoted MoLeR-V, where V denotes the vocabulary size.
Table 9: GuacaMol benchmark metrics on the ZINC250K dataset. We report mean and standard deviations across 3 random training runs for all methods and metrics.
METRIC
FCD (1)
KI. (t)
NOVELTY (t)
UNIQUENESS (t)
VALIDITY (1)
CHARVAE
0.17 +: 0.08
0.78 + 0.04
0.99 + 0.00
0.99 + 0.00
0.09 + 0.01
SMILES-LSTM
0.93 + 0.00
1.00 + 0.00
0.98 + 0.00
1.00 0.00
0.96 + 0.01
GRAPHAF
0.05 + 0.00
0.67 + 0.01
0.91 0.01
0.91 0.01
1.00 + 0.00
HIERVAE
0.50 0.14
0.92 0.00
0.96 + 0.01
0.96 0.01
1.00 1: 0.00
MICAM
0.63 0.002
0.00
0.98 0.00
0.98 0.00
1.00 0.00
UTVAE
0.001
0.00
1.00 1
1.00 1 0.00
1.00 1. 0.00 0.000
PSVAE
0.28
0.00
1.00 0.00
1.00 = 0.00
1.00 + 0.00 0.00
MAGNET
0.00
0.00 0.00
0.99 0.00
1.00 - 0.00
MOLER-5
0.00
0.93
0.00
0.97 0.00
1.00 0.00
MOLER
0.00
0.98
1.00 0.00 0.00
1.00 0.00
0.00
MOLER-2000
4.33
0.97 0.00
0.99 + 0.00
0.99 0.00
1.00 + 0.00
DIGRESS
0.65 0.00
0.91 0.00
0.99 + 0.00
0.99 0.00
0.85 0.01
EDM-SYCO
0.85 0.01
0.96 - 0.00
1.00 - 0.00
1.00 = 0.00
0.88 - 0.01
27

---

Published as a conference paper at ICLR 2025
Table 10: MOSES benchmark metries on the ZINC250K dataset. We report mean and standard deviations across 3 random training runs for all methods and metrics.
METRIC
FILTERS (J)
INTDIV (1)
INTDIV2 (t)
QED (U)
SA (L)
LOGP (4)
WEIGHT (4)
CHARVAR
0.43 + 0.02
0.88 + 0.01
0.87 + 0.01
0.06 + 0.03
0.48 + 0.13
0.87 + 0.14
34,23 + 07.05
SMILES-LSTM
0.59
0.87 0.00
0.86 0.00
0.00 + 0.00
0.04 0.02
0.12
02.62 + 00.16
GRAPHAF
0.47 0.03
0.93 = 0.00
0.91 = 0.00
0.22 = 0.01
0.88 + 0,10
0.41 0.02
96.93 + U4.99
HERVAA
0.58 0.05
0.87 0.01
0.86 0.01
0.03 1. 0.00
1.19.10.1
0.35 0.21
18.97 1. 10.43
MICAM
0.54
0.87 = 0.00
0.87 2 0.00
0,08 = 0.00
0.51
0.20 + 0.05
51.19 + U6.58
JTVAE
0.57 1.000
0.86 1 0.00
0.86 1. 0.00
0.01 1. 0.00
0.35 0.01
0.29 1 0.01
02.93
00.07
PSVAE
0.85 0.00
0.89
0.88 = 0.00
0,05 0.00
0.06
0.34 + 0.01
38.26
+ U2.74
MAGNET
1..00
0.87 0.00
0.87 0.00
0.01 1. 0,00
0.23 0.00
14,52
00.40
MoLER-5
0.001
0.85 0.00
0.01
0.01
0.17
10.67
04.13
MOLER 350
0.56 1.500
0.87 0.00
0.86 1 0.00
0.01 0.001
0.00
0.13
05.96
01.16
MoL:R-2000
0.55 0.01
0,86 0.00
0.86 0.00
0.02 0.00
0.08
0.11
07.62
01.80
DIGRESS
0.77 0.01
0.86 = 0.00
0.86 = 0.00
0.05 = 0.00
0.01
0.61 0.02
32.51 = 00.20
EDM-SYCO
0.59 = 0.00
0.87 + 0.00
0.87 = 0.00
0.003 1 0.00
0.19
0.06 4 0.02
04.49 1 00.22
G.4 CONDITIONAL UNCONDITIONAL GENERATION
In addition to the conditional generation experiment from the main text, we perform an extra experiment to evaluate our approach and compare it to the unconditional model, which does not consider the conditioning value. We consider four properties: (1) the penalized LogP score (PLogP), a compound's solubility and synthetic accessibility; (2) QED, a compound's d's drug-likeness: (3) DRD2, estimate of the biological activity against dopamine type receptors; (4) TPSA, drug's ability to permeate cells.
We generate 10,000 valid molecules for each property and report the mean absolute error between the targets and the estimated values for the generated molecules measured by RDKit. The results are shown in Table IT Our guidance mechanism achieves an average improvement factor of more than 5.5 times compared to the unconditional model across all properties.
Table 11: Mean absolute error between the targets and the property values of 10,000 generated molecules using regressor guidance, and the improvement factor compared to unconditional genera- tion. We report standard deviation across 3 runs with different random seeds.
PLOGP
QED
DRD2
TPSA
UNCONDITIONAL
2.2440 + 0.0018
0.1530 = 0.0002
0.0148 + 0.0000
25.9017 + 0.0206
GUIDANCE
0.3557 +: 0.0042
0.0350
1: 0.0004
0.0051
0.0001
03.0382 1: 0.0164
IMP. FACTOR (1)
6.3087
4.3714
2.9020
08.5253
G.5 CONDITIONAL GENERATION WITH CONDITIONAL DIFFUSION MODELS AND EEGRESSOR-FREE GUIDANCE
In this section, we benchmark additional approaches for conditional generation presented in Sec- tion 6.2. In the main text, we reported results with the regressor guidance algorithm on top of an unconditional diffusion model inspired by the classifier guidance algorithm (Dhariwal & Nichol 2021). Here, we additionally test a conditional diffusion model, which is simple extension the unconditional model, obtained by adding the target property to the node features and training a new model (Hoogeboom al., (2022), referred to Conditional Diffusion Table [2] Inspired by the classifier- free guidance algorithm (Ho & Salimans| (022). we train another variant of the conditional model by dropping the target property 15% of the time during training. This allows us to sample from the same model in a conditional unconditional way, and by combining the two modes, we can guide the generation process towards the target properties, similar to the regressor-guidand
case. We refer to this case as Regressor-Free Guidance in Table 12 We additionally test guiding the conditional model with an external regressor model (Regressor Guidance Conditional). All results for these models are shown in Table I2 Note that experimented with smaller models than the one reported in the previous experiment due to the computational costs associated with training and running these models. Therefore, these numbers are not directly comparable the numbers from the previous
28

---

Published as a conference paper at ICLR 2025
experiment. However, note that the best model presented here, the Regressor Guidance Conditional, has an MAE 0,36, which very close to the number from the previous experiment (0.358 1), even though the model here is smaller. We expect that guiding the conditional model performs better if compared properly. However, it has the drawback of training a new diffusion model for each target property, compared to using the same diffusion model and one regressor that predicts all properties of interest.
Table 12: MAE between condition value e and the property value computed on the generated molecules. The considered property in PLogP. $ vI a dqs.
METHOD
SCORE FUNCTION
MAE
UNCONDITIONAL CONDITIONAL DIFFUSION
2.24
c0(Z(,(,)) (s + 1)00,, s29(2),
0.60
REGRESSOR- GUIDANCE
0.48
REGRESSOR GUIDANCE CONDITIONAL
En c, &V (2) ce)²
0.36
G.6 UNCONSTRAINED MOLECULE OPTIMIZATION
This task aims to generate molecules from scratch with as high property values as possible without any other constraints. We this task demonstrate the performance of the proposed guidance mechanism for property maximization, described Equation g Similarly to the conditional gen- eration task, start from Gaussian noise and the modified sampling algorithm to generate molecules with high properties. Table 13 shows the top-3 scores of our model's generated molecules and for all baselines from (Kong al. (022). Note that we achieved the best scores compared to all these baselines, even though we only generated 1,000 molecules compared to 10,000 molecules generated by these baselines. This illustrates the high performance of our method and motivates its use for the harder task of constrained optimization.
Table 13: Top-3 property values for molecules generated in the unconstrained optimization task. Results for all baselines are taken from (Kong et al. 2022). All baselines generate 10,000 molecules, while we only generate 1, 1,000 for this task.
PLOGP
QED
МETHOD
2ND
3RD
1ST
2ND
3RD
JT-VAE
5.30
4.93
4.49
0.925
0,911
0,910
GCPN
7.98
7.85
7.80
0.948
0.947
0.946
MRNN
8.63
6.08
4.73
0.844
0.796
0.736
GRAPHAF
12.23
11.29
11.05
0.948
0.948
0.947
GRAPHDF
13.70
13.18
13.17
0.948
0.948
0,948
GA
12.25
12.22
12.20
0.946
0.944
0,932
MARS
7.24
6.44
6.43
0.944
0.943
0.942
FREED
6.74
6.65
6.42
0.920
0.919
0.908
H-VAE
11.41
9.67
9.31
0.947
0.946
0.946
F-VAE
13.50
12.62
12.40
0.948
0.948
0.947
PS- VAE
13.95
13.83
13.65
0.948
0.948
0.948
EDM-SYCO
14.73
14.31
14.21
0.948
0.948
0.948
G.7 FRAGMENT OCCURENCES
In this section, we test our method's 's ability to generate molecules containing fragments of different occurrence frequencies in the training set and compare it with several baselines. Table shows the results of this experiment. Even though our model is not motif-based can match the distribution of fragments very well.
29

---

Published as a conference paper at ICLR 2025
Table 14: Frequency occurrence of different fragments computed the generated molecules from different methods. Methods are split into SMILES- based, graph-based base- motif-level, and graph-based atom-level. The method that achieves the closest percentage to that in the training set is highlighted. SYCO can successfully reproduce the percentages of different fragments in its generated molecules without relying on motifs.
HN
FRAGMENT
TRAINING
76.47%
14.15%
9.37%
1.19%
1.09 %
0.97%
0.39%
0.27%
CHARVAE
$2.98%
4.06%
1.09%
0.13%
0.40%
2.31%
1.21%
0.00%
SM-LSTM
75.73%
14.02%
9.73%
1.01%
1.09 %
1.10 %
0.34%
0.24%
PS-VAE
71.31%
13.08%
5,58%
2.84%
%
2.61%
0.27%
0.12%
HIERVAE
75.78%
12.80%
7.92%
0.71%
0.45%
0.56%
0.03%
0.00%
MICAM
79.47%
11.37%
5.74%
0.79%
0.94%
2.16%
0.37% %
0.06%
JT-VAE
80.15%
13.84%
9.70%
1.99%
%
1.48%
0.60%
MAGNET
69.01% S
11,54%
7.58%
0.99%
1.15%
0.87%
0.23%
0.191
MOLER
78.47%
15.12%
7.85%
1.26%
0.64%
1.24%
0.25%
0.21 %
GCPN
53.37%
2.03%
0.33%
0,44%
3.53%
0.38%
0.89%
0.13%
GRAPHAF
34.16%
1.27%
2.29%
0.11%
0.29%
0.25%
0.03%
0.00%
DIGRESS
83.96%
11.63%
8.03%
1.18%
0.52%
1.49 %
0.38%
0.10%
EDM-SyCo
75.57%
12.84 %
9.17%
1.02%
0.56 %
1.89%
0.33%
0.16 %
G.8 MODEL RUNTIME
To understand the computational cost associated with our model, we report the average runtime for training and inference Table 13 the ZINC250K dataset, our model is trained for 1000 epochs, which takes approximately 10 days (considering intermediate evaluations every 10 epochs), with an average of 740 seconds for training per epoch. During sampling, our model greatly benefits from rumning batch-wise and takes, average, 86 seconds generate a batch of 100 molecules, which corresponds to an amortized runtime of 0.86 seconds per single molecule. Generating molecules conditioned target values further increases the runtime because requires running one forward and one backward pass of the regressor model at each diffusion step, increasing the average runtime 1.8 seconds per molecule. All numbers computed using single Nvidia A 100 GPU.
Table 15: Average runtime of EDM-SYCo. Numbers are on the ZINC250K dataset, the training set contains 220K molecules, and the sampling times are obtained using batch size of 100.
RUNTIME
TRAINING (EpOCH)
740 SECOND / EPOCH
TRAINING (TOTAL)
10 DAYS
UNCONDITIONAL SAMPLING
0.86 SECOND MOLECULE
CONDITIONAL SAMPLING (GUIDANCE)
1.8 SECOND / MOLECULE
G.9 SAMPLING SPEED VERSUS GENERATION QUALITY TRADE-OFF
Our diffusion model is trained with T = 1000 diffusion steps, which can be a limiting factor in some practical applications due the resulting slow sampling process. One simple way to accelerate this process is to run a strided sampling schedule NNchol & Dhariwal (2021). The key idea is that to reduce the sampling steps from 7' to S, we run the denoising network (Equation (6) every [T/S] steps and update the noise schedule accordingly Nichol & Dhariwal (2021). In this experiment, evaluate the quality of the molecules generated by running the model, initially trained with = 1000 steps, using only 500, 250, 200, and 100 steps evenly spaced between and T (without any additional training). Figure shoows that the model performs well even with only 200 steps, which offers 5x speedup (if we neglect the cost of running the decoder, which is less than I step of reverse diffusion).
30

---

Published as a conference paper at ICLR 2025
0.95
0.90
0.85
GuacaMol
0.80
0.75
FCD
KL
0.70
1000
800
600
200
Number diffusion steps
Figure 9: Effect of varying the number of reverse diffusion steps on the performance, using the same model trained for 1000 steps.
G.10 ANALYSIS 3D INDUCTIVE BIAS
We analyze the inductive bias of our architecture in two steps and perform an ablation study to highlight it:
Chemical bias of RDKKit-genrrater coordinates; train new variant of EDM-SyC So on 3D coordinates produced by the Buuchterman-Reingeee force-directed algorithm generates 3D positions solely based the graph structure without any chemical inductive bias. We refer to it as
of which
Representation bias of the 3D latent embedding: we highlight baselines that operate directly on the graphs. through discrete diffusion (DiGress) continuous diffusion by adding Gaussian noise to the adjacency matrix (GDSS).
Results are presented in Table 16
We see that our base model performs better than EDM-SyCo- graph-layout across all metrics, which in turn outperforms GDSS and Digress on FCD and KL.
Table 16: Performance comparison of different model classes. Higher values indicate better perfor- mance for all metries.
(1)
(f)
Novelty (1)
Uniqueness (1)
Validity (f)
3D diffusion RDKit-generated cocrdinates (EDM-SyCo-RDDE)
0.85
0.96
1.00
1.00
0.88
diffusion on graph-based coordinates (EDM--yCCogrnph-layan) Contimous diffusion on graph (GDSS)
0.73
0.94
1,00
1.00
0.54
0.10
N.A.
1.00
1.00
0.97
Discrete (DiGress)
G.11 MOVING THE EGNN FROM THE ENCODER TO THE DECODER
To understand the effect of training the diffusion model on the latent space defined by the encoder's EGNN versus on the raw synthetic coordinates, we train new diffusion model directly the -egeeatgeeerteed coordinates and move the EGNN to the decoder. We note that also this setup, training the diffusion model is necessary because, for the used datasets, there is no public ground- truth coordinates and thus no pretrained 3D diffusion models. Although both model variants achieve similar unconditional generation performance, benefit of having MLP-only decoder is in
31

---

Published as a conference paper at ICLR 2025
generation setting: Given scaffold that can be reconstructed by the autoencoder, we perform inpainting in the latent 3D space and generate a completion for this scaffold. The issue with having the EGNN in the decoder is that the bond prediction between the atoms in the scaffold will depend on the newly added atoms, and the final decoded graph is no longer guaranteed to contain ML.P-onl decoder. We validate this argument by
the original 2D scaffold, in contrast to having generating 1000 valid completions for 6-Carbon-ring (smiles string: CICCCCCI, present in 7% of the training molecules) and computing how often the decoded molecules actually contain the scaffold (Scaffolding success rate). Results are shown in Table 17
Table 17: Performance comparison of different models on FCD, KL divergence, and Scaffolding Success Rate. Higher values indicate better performance.
Model
FCD
KL
Scaffolding Success Rate (%)
Diffusion model on HTKDG output + HGNN-MI Diffusion model on EGNN output MLP-onll-decee
0.95
94.2
0.85
0.96
100.0
G.12 MODEL SCALING WITH MOLECULE SIZE
To check the scalibility of our model with respect the molecule size compared previous diffusion models, we compare the average runtime required to generate a single molecule using DiGress and model. DiGress is discrete diffusion model that operates 2D graphs. For both models, generate a batch of 100 molecules of size N, for an increasing N, and report the average runtime to generate one molecule. Results are shown Table 18
Table 18: Comparison of generation times (in seconds) for DiGress and EDM-SyCo across different molecule sizes.
# Atoms
10
20
30
40
50
60
DiGress (time in seconds)
4.05
12.42
27.04
46.87
70.73
114.14
EDM-SyCo (time in seconds)
0.15
0.49
1.02
1.75
2.69
3.66
G.13 MOLECULE SIZE EXTRAPOLATION
To evaluate generalization, we generate molecules larger than every other molecule from the training set and report the validity rate in Table 19. The better performance on GuacaMol compared to ZINC250K can be explained by the higher diversity of training molecule sizes in GuacaMol. We outline some ideas that can improve the generalization capability:
instead of modeling point clouds as fully-connected graphs for the diffusion model's EGNN, use distance cutoffs to make the number of neighbors an atom roughly independent the total number of atoms, as e.g. done in [3].
modify the forward diffusion process to avoid that all atoms cluster around the CoM, e.g. by increasing the variance of the transition kernels.
Table 19: Generalization performance of models on molecules larger than the training set, with results reported validity rates (%).
Molecule Size
Nmaax + 5
Nmmax +
10
n.max +
20
ZINC250K (Nmean 23.66, nstdd 4.4, Amar 40)
45.3
24.5
21.1
GuacaMol (Amean 28.0, N.td = 7.9, Tmmaz = 88)
70.1
55.4
15.9
32


Reviews:
[{"review_id": "uG811jE6oO", "reviewer": "Reviewer_Kh9X", "summary": "The paper introduces a novel 3D latent embedding scheme for 2D molecular graph generation. It presents a set of experiments which try to show that this inductive bias is helpful for a set of 2D molecule generation tasks.", "questions": "What is the KL divergence references at the end of page 8? The description \"a variety of physiochemical descriptors\" is too vague. Sorry if I missed a more precise definition elsewhere.\nIs there a reason the results for \"Similarity-Constrained Optimization\" are not presented in more detail in the main paper? This is by far the most real-world useful application, and the one which has the highest probability of showing signal on your hypothesis (in my opinion).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The model and experiments are clearly described. The paper acknowledges the importance of generation tasks beyond vanilla distribution matching on the space of 2D molecular graphs. The experiments are chosen to attempt to shine light on the effectiveness of the 3D latent inductive bias hypothesis.\n\nFor guided generation, the comparable results with \"Translation\" methods are impressive.", "weaknesses": "The experiments are ambiguous on whether this inductive bias is, indeed, helpful. \n\nIt appears from table 3 that results on the ZINC250K dataset are saturated. The FCD and KL values reported in Table 4 are interesting, but presented without much interpretation. Without prior expertise it is difficult to evaluate FCD scores. Furthermore, de novo generation is not a terribly useful task in practice.\n\nThe most compelling experiment presented in the paper is \"Similarity-Constrained Optimization\". I could see how a 3D latent would improve performance in this area, but the data presented for this section is scant.\n\nOverall, none of the experiments prove or disprove the inductive bias hypothesis of the architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IKKPBFzDix", "reviewer": "Reviewer_8xjK", "summary": "This paper proposes a new method for 2D molecular graph generation using 3D molecular diffusion models, using the following steps:\n- Train an autoencoder on paired (2D molecular graph, latent 3D molecular graphs).\n  - The encoder consists of two parts:\n       - First, 3D coordinates for all atoms are obtained using ETKDG, an well-known 3D conformation generation algorithm. No learning is required here.\n       - Then, an EGNN is used to map the 3D molecular graph to a latent 3D graph, with updated node features and coordinates. The idea here is to incorporate neighborhood information for easier decoding of atom and bond types (as seen in Appendix G.1).\n   - The decoder consists of two MLPs: one to predict atom types, and another to predict the bonding between the atoms.\nThe encoder and decoder are trained in the standard variational (VAE) framework.\n* Train a diffusion model to generate latent 3D molecular graphs.\nHere, the authors use the well-known EDM model (Hoogeboom, et al. 2022).\nWhile the task is similar to that of standard 3D molecular graph generation, here the latent 3D molecular graphs do not necessarily correspond to anything physical)\n* To sample:\n    - Run the diffusion model, starting from noise, to generate latent 3D molecular graphs.\n    - Run the decoder on the latent 3D molecular graphs to get 2D molecular graphs.\n\nFirst, the authors compare their framework (EDM-SyCO) to GeoLDM, a 3D molecular graph generation algorithm combined with a simple bond prediction algorithm to create 2D molecular graphs. They show that the performance drops significantly. However, it is unclear whether the performance drop is due to the inaccurate bond prediction, or the use of standard 3D molecular graphs instead of the author’s latent 3D molecular graphs.\n\nNext, the authors compare their model to autoregressive baselines on the well-established ZINC250K and GuacaMol datasets. They find that their model outperforms existing all-at-once 2D graph generative models. However, the performance improvement over state-of-the art autoregressive models is unclear, especially on the larger GuacaMol dataset.\n\nOverall though, this is an interesting idea for 2D molecular graph generation. My only real concern is that the utility of the EGNN latent space is not conclusively shown; I have mentioned two experiments in the first bullet point of my Weaknesses section that I would love to see performed.\n\nEDIT: I am happy with the author's response and additional experiments, and have increased my score in response.", "questions": "- How does your model handle chirality, since the diffusion model is E(3)-equivariant? This could motivate the use of SE(3)-equivariant GNNs instead.\n- Section 6.2: Please report how the normalization of the FCD and KL divergence metrics was performed for clarity.\n- How is the novelty metric so high? This is very different from the 3D molecular graph generation setting (Appendix C of https://arxiv.org/abs/2203.17003) where the novelty usually reduces as training progresses. (This might be because the datasets for 3D molecular generation such as QM9 are the exact enumeration of all compounds with a certain property. However, it is surprising to see a novelty score of 100% in any case, indicating that none of the training molecules were sampled.)\n- Some experiments on generalization would be very nice to see; can you train only on smaller graphs but generate much larger ones?\n- Line 503 has a typo: “... factors 2.7 ad 15.6”\n- Analysis of invalid molecules would be good, especially to motivate future work.\n- Comparison of training and sampling time to other models would be nice, to understand the tradeoff between compute and performance better.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well-written and full of details which are hard to find in most papers on this topic.\n- The overall idea is definitely interesting and novel; it matches up with geometric intuition that molecules naturally live in 3D space.\n- The authors have definitely improved results for non-autoregressive models on ZINC250K.\n- Experiments on inpainting and conditional generation are creative, and quite comprehensive.", "weaknesses": "- If the motivation for having an EGNN in the encoder is to incorporate neighborhood information for predicting atom and bond types for decoding, I would have liked a comparison to a setup where the EGNN is used as a decoder (over the simple MLPs). To summarize this scheme:\n    - The encoder would just output the ETKDG-generated coordinates.\n    - The decoder would be an EGNN + MLP to predict atom and bond types.\n    - A diffusion model could then generate (as standard) 3D molecular graphs. In fact, any pretrained diffusion model could be used, in contrast to the author’s scheme here where the diffusion model must target the latent space of the EGNN encoder.\n\n- In Section 6.1, I would have liked to see GeoLDM paired with a more sophisticated bond-determining algorithm such as RDKit's rdDetermineBonds (an implementation of xyz2mol) or OpenBabel. For example, as shown in Table 1 of https://openreview.net/pdf?id=MIEnYtlGyv, there can be significant variation in the results from each of these bond-determining algorithms. This could help prove the utility of the EGNN latent space for the diffusion model to target.\n\nI would be willing to increase my score if the above experiments are performed convincingly.\n\n- Training time seems high (~10 days on a single A100 GPU) relative to other methods; this is probably related to why the authors did not tune hyperparameters on the GuacaMol dataset. I would like a confirmation from the authors on this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tIZHOc6F16", "reviewer": "Reviewer_j8BU", "summary": "This work presents a 2D molecule graph generation method that uses 3D molecule generative models by mapping 2D molecule graphs to 3D point clouds via synthetic coordinates and learning the inverse mapping using EGNN. Based on EDM, the proposed method shows superior generation quality.", "questions": "Please address the weakness mentioned above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- As molecules have 3D structures, using the 3D information to generate 2D molecular graphs seems intuitive and reasonable.\n\n- The reconstruction accuracy ablation study on autoencoder shows the justification of using EGNN architecture.\n\n- The framework can also be used for conditional generation/optimization of molecules in diverse settings, such as scaffold-constrained generation and similarity-constrained optimization.\n\n- The experimental results show the effectiveness of this approach.", "weaknesses": "Although using latent Euclidean point clouds instead of a simple 2D graph structure seems to be a reasonable and effective way of modeling molecular graphs, it is likely that this approach is not scalable with respect to the molecule size. Working with latent point clouds would likely require far higher computational costs and memory compared to previous diffusion models. Especially, it seems that learning the 3D information of large molecule structures would be challenging compared to learning the 2D graph distribution. Is there any experiments or explanations on this issue?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wsfc1GpWe3", "reviewer": "Reviewer_HWqN", "summary": "**Summary:** The authors present SyCO as a straightforward method for generating 2D graphs using 3D generative models.\n\n**Recommendation:** Given the notable strides yet remaining work to be clarified, discussed, or explored in this manuscript, I am currently recommending a weak reject.\n\n**Rationale behind Recommendation:** If the authors can discuss some of the empirical benchmarking shortcomings of their model (e.g., what future steps might look like) and also include a small sample of experiments using a geometric neural network architecture other than EGNNs (to verify that the authors' ideas generalize beyond this architecture), I will consider raising my score.", "questions": "- The authors should clarify what the error bars in Table 3 represent. Standard deviations across 3 runs of a model, as in Table 4?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- SyCO's simplicity and versatility should encourage more development in the intersection of 2D and 3D graph generation models.\n- SyCO's autoencoding reconstruction performance for bond type prediction is compelling.\n- The authors provide a succinct and well-motivated case study of how to add atoms to a molecule during molecular optimization via diffusion generation.\n- The authors' classifier guidance is well explained and motivated for property-specific molecule generation and optimization.\n- The authors' code is well documented.", "weaknesses": "- SyCO borrows most of its diffusion methodology from previous works [1, 2], to the best of my knowledge with adding nodes during inpainting being the only notable example to the contrary.\n- Although it is simple to start with, the EDM architecture is no longer state-of-the-art for 3D molecule generation [3], suggesting the authors' results could be improved with a different choice of e.g., geometric GNN architecture. The authors seem to have a taken a wide approach to benchmarking rather than a deeper approach to seeing which geometric GNNs can best encode 2D graphs as latent 3D point clouds. This latter area of investigation could strengthen this work notably.\n- The authors' conditional generation results for LogP in Table 5 and diversity throughout Table 6 seem a bit underwhelming. Besides attributing this as a flaw of molecule denoising networks, it would be helpful for the authors to expound how they think these results might be improved over time.\n- The authors should discuss the pros and cons of relying on RDKit for conformer generation. For example, are there inherent biases in its distance-based coordinates estimation algorithm that might limit or impact the behavior of molecule generative models trained on its coordinate embeddings?\n\n**References:**\n1. Hoogeboom, Emiel, et al. \"Equivariant diffusion for molecule generation in 3d.\" International conference on machine learning. PMLR, 2022.\n2. Du, Yuanqi, et al. \"Machine learning-aided generative molecular design.\" Nature Machine Intelligence (2024): 1-16.\n3. Morehead, Alex, and Jianlin Cheng. \"Geometry-complete diffusion for 3D molecule generation and optimization.\" Communications Chemistry 7.1 (2024): 150.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)