Chemistry-Inspired Diffusion with Non-Differentiable Guidance
Authors: Yuchen Shen
Chenhao Zhang
Sijie Fu
Chenghui Zhou
Newell Washburn
Barnabas Poczos
Venue: ICLR 2025 Poster (2025)
Keywords: guided diffusion
ai4science
molecule generation
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Recent advances in diffusion models have shown remarkable potential in the conditional generation of novel molecules. These models can be guided in two ways: (i) explicitly, through additional features representing the condition, or (ii) implicitly, using a property predictor. However, training property predictors or conditional diffusion models requires an abundance of labeled data and is inherently challenging in real-world applications. We propose a novel approach that attenuates the limitations of acquiring large labeled datasets by leveraging domain knowledge from quantum chemistry as a non-differentiable oracle to guide an unconditional diffusion model. Instead of relying on neural networks, the oracle provides accurate guidance in the form of estimated gradients, allowing the diffusion process to sample from a conditional distribution specified by quantum chemistry. We show that this results in more precise conditional generation of novel and stable molecular structures. Our experiments demonstrate that our method: (1) significantly reduces atomic forces, enhancing the validity of generated molecules when used for stability optimization; (2) is compatible with both explicit and implicit guidance in diffusion models, enabling joint optimization of molecular properties and stability; and (3) generalizes effectively to molecular optimization tasks beyond stability optimization. Our implementation is available at https://github.com/A-Chicharito-S/ChemGuide.

Published as a conference paper at ICLR 2025
CHEMISTRY-INSPIREE DIFFUSION WITH NON-DIFFE- RENTIABLE GUIDANCE
Yuchen Shen', Chenhao Zhang', Sijie Fu", Chenghui Zhou, Newell Washburn', Baarnabs Póczos Carnegie Mellon University Piusburgh, PA 15213, USA
{chenhao4, sijief, chenghuz, washburn] @andrew. com.ddrwwwc........ edu {yuchens3, bapoczos}åcs. cmu. edu
ABSTRACT
Recent advances diffusion models have shown remarkable potential in the con- ditional generation of novel molecules. These models can be guided in two ways: (i) explicitly, through additional features representing the condition, or (ii) implic- itly, using a property predictor. However, training property predictors tional diffusion models requires an abundance labeled data and is inherently challenging in real-world applications. We propose novel approach that at- tenuates the limitations of acquiring large labeled datasets by leveraging domain knowledge from quantum chemistry conditional diffusion model. Instead of relying on neural networks, the oracle pro- vides accurate guidance in the form of estimated gradients, allowing the diffusion process to sample from conditional distribution specified by quantum chemistry. We show that this results in more precise conditional generation of novel and stable molecular structures. Our experiments demonstrate that our method: (I) significantly reduces atomic forces, enhancing the validity generated molecules when used for stability optimization; (2) is compatible with both explicit and im- plicit guidance in diffusion models, enabling joint optimization of molecular prop- erties and stability; and (3) generalizes effectively to molecular optimization tasks beyond stability optimization.
condi-
non-differentiable oracle to guide an
https://gitub..omm// http:///thbbbooooAAACCCcchatttCe Chicharito-S/ChemGuii
INTRODUCTION
Diffusion models have received increasing attention in molecular design. Their ability to generate novel molecules with desired properties (i.e.. guided diffusion) has fostered advances in material
science (Manica et al.) 2023) Yang et al. 2023), chemistry (Anstine & Isayev] [023), protein de-
sign (Watson et al. [dl3|[222]/rraaa Abramssn et al. 2024), ) etc. To achieve guided diffusion, one can explic-
itly condition the diffusion process specific properties in training (Hoogeboom et al. 2022: Xu ict al.) (202),, such that the model is naturally a conditional generator during inference. Alternatively, an unconditional model can be trained without labels, and the diffusion process is guided implicitly at inference time (Dhariwal & Nichol, 2021 Vignac et al. 2022) using a property predictor that provides guidance gradients steer the model toward sampling from the conditional distribution.
Both explicitly and implicitly guided diffusion require a labeled dataset to train the model or the be expensive, time-consuming, and of-
property predictor. The acquisition of labels, however, ten impracticable[] When only small labeled datasets are available (Power et al. 2022). ). the model and the property predictor can potentially struggle generalize beyond seen structures (see Ap- pendix [..)). degrading the performance of the diffusion process when generating novel molecules. Implicitly guided diffusion can help address such challenges. This approach (i) requires no labels to train the model and (ii) can replace the property predictor with domain knowledge from quantum chemistry, fulfilled by quantum chemistry software such as xTB (Bannwarth et al.) 20199 (02T)) and Gaussian (Frisch et al., 2016). Such software can act as an expert oracle to create labeled datasets; thereby, avoiding the extrapolation shortcomings of neural networks.
Despite following certain computation procedures, quantum chemistry is by nature a non- differentiable oracle that can not be backpropagated with neural networks, as those procedures
Equal contribution. "Correspondence to: Barnabás Póczos Newell Washhurn
For example, I. (Erhirhie et al. 20188) measures the lethal dose of a test substance, which is difficult to calculate from the molecular structure of the substance alone; thus, requiring animal testing.

---

Published as a conference paper at ICLR 2025
CH2
Force
0.016
0.011
0.090
Diffusion Model
Training
Optimize
CHŞO
Generation
Quantum Chemistry
Molecules
CH
Force 0.007
0.006
0.023
Figure 1: The overview of CHEMGUIDE. On the left, we present the space of all molecules (roughly) as a unimodal distribution, where red/blue region indicates molecules for training/novel molecules generated by the diffusion model. In the middle, CHEMGUIDE derives non-differentiable guidance from quantum chemistry to steer the diffusion process towards a conditional distribution (e.g.. min- imized forces). On the right, present the average forces of sets of molecules generated by GeoLDM (Xu et al. 2023) trained on QM9 (Ramakrishnan et al.) (left two) and GEOM (Axelrod & Gömez-Bommarellij (rightmost one) without (above) and with (below) CHEMGUIDE.
usually involve proprietary algorithms or extremely computationally expensive to perform, which prevents the integration of quantum chemistry to guide the diffusion models. In this work, we aim to implicitly guide the diffusion process with a non-differentiable oracle, and employ zeroth-order optimization methods (Nesterov & Spokoiny 2017] Malladi et al. 2023) to estimate the necessary guidance gradients. We refer to our method as CHEMGUIDE, a method that leverages quantum chemistry to provide more accurate diffusion guidance. Specifically, given the oracle xTB with the GFN2-XTB method (Bannwarth et al., 2019) that conducts quantum chemistry calculation of potential energies and atomic forces, we first CHEMGUIDE improve the stability of the gen- erated molecular geometry (see Fig. []. by guiding the diffusion process towards sampling from distribution where the net force on the atoms zero. Further, we formulate diffusion guidance
on both molecular property and stability as a bilevel optimization (Colson et al. 2007) problem, to demonstrate that CHEMGUIDE is compatible with both explicitly and implicitly guided diffusion. We summarize our contributions as follows:
We propose CHEMGUIDE, which leverages zeroth-order optimization techniques to intro- duce non-differentiable chemistry oracle diffusion guidance (Section 3).
We show in a bilevel optimization framework (Algorithm 2} that CHEMGUIDE can be utilized in conjunction with existing guided diffusion methods.
Experiments and analyses (Section and 5) demonstrate that CHEMGUIDE (i) improves the stability of generated molecules when guided with domain knowledge from the semi- empirical quantum chemical method, GFN2-XTB (Section 4.2); (ii) generalizes to molecu- lar optimization other than stability (Section 4.5 (iii) effectively improves both molecular property and stability when used with other guided diffusion methods (Section 4.4
2 PRELIMINARY
In this section, we introduce the key concepts of diffusion models and explore the architecture of a specific diffusion model designed for 3D molecule generation. We will explain how achieve equivariance in the generated molecules and how the semi-empirical quantum chemical method GFN2-xTB can be utilized for guidance. Additionally, we will also outline our motivation for inte- grating CHEMGUIDE with 3D diffusion models to optimize molecular stability.
Diffusion Models In general, a diffusion model (Sohl-Dickstein et al., 2015 Ho et al. 2020: Song et al.] (2200) consists of forward diffusion process and a reverse denoising process. The diffusion process is a Markov chain that gradually adds Gaussian noises with predefined variance schedule 31.r from timestep to T to the original datapoint ano which chosen such that æ* ~V(0, I). The forward diffusion process g is usually defined as a fixed schedule by the following:
T
q( xo)
II gxx xz-1) t=1
q(x, xT-1) = Ni(x; VI 3,0;-11 B,1)
(I)
2

---

Published as a conference paper at ICLR 2025
The denoising process starts with øy and recovers the original datapoint Zo by predicting the mean of 1t-1 given Dy using a neural network denoted as (1, ., t), where is the parameter.
(0 p(TT) II po (x xx)) 1=1
po( xx) (2 t), (2 t))
(2)
In practice, Eg = I with oa
1- T-of
Br. 0r = VII (1 Bi) for all t. For simplicity, Ho
et al. (2020) reparametrize Eq. such that 2; 0, v1 afc, where ((,, I). Hence, instead of predicting Hig (2 t), we now predict the noise co (z,, t), where ue (2(, t) is parameterized co t)). Consequently, we have
Be
as V1-
3r VI- co (z,, )),
H1-1 N( VT
Latent Diffuuson Architecture for 3D Molecule Generation An N-atom molecule can be repre- sented as point cloud G = [x, hj E (3+d), with x E Nx3 being the atom's 3D coordinates and h R the atom features indicating atom types. A latent diffusion architecture (Rom- bach et al. 2022 Xu et al., 2023) consists of a Variational Autoencoder (VAE) and a diffusion model trained consecutively. Particularly, the geometric latent diffusion model (Xu al. 2023) uses the encoder of the VAE to project molecules to a continuous latent space, on which the difiu- sion model is then trained. Denote the encoder as & and the latent variable by z E IR (3+d.)), then [xx.0; Zh,o] = {(x, hJ), with 24,0 E R.Nxd= and d2 < d. Let 2, [xx,,, Zh,t]. the latent forward diffusion process and reverse denoising process are defined by replacing æ with x in Eq. [ and 2 We denote the decoder of the VAE as D, which maps Zo back to the original molecular space, such that D[[2x.0 2), [x, h] (3++dd) The encoder & and the decoder D are parameterized with an equivariant graph neural network (EGNN) (Satorras et al.) 2021) to translate between dis- crete molecular data and latent variables, such that the atom types are invariant and the positions are equivariant to transformations as the following:
(4) for any rotation matrix R and translation matrix T. where Zx, E RN Nx3 are required satisfy zero center gravity and have zero-mean over N atoms for each position. In addition, the latent diffusion model is also parameterized by EGNN such that transitions between each timestep in the denoising process also respect the same characteristics.
Rzx.: + T, Zh, = {Rix + T, h₂) Rx. + T, h, = D(Rzx,: + T, Zh.t)
GFN2-XTB Method Matter such as electrons, atoms, and molecules, interacts with matter inher- ently to reach configurations with lower potential energies, i.e stability. To formulate this as a molecular geometry optimization problem, let h1, .yy be the N atoms of given molecule and X1,' XN E I be their coordinates in the 3D space. Each atom h, is subject to a force:
(g)) PEp (x1, XN h,,-.. hN) Vi [N)
(5)
0x;
where E.p represents the potential energy of the conformation. The force here manifests valid phys- ical interpretations:
high level, an atom is pushed accordingly by the exerted force until the force reduces to zero and an equilibrium is achieved. The two necessary (but not sufficient) condi- tions for a stable molecular geometry are: (i) all forces on the atoms should be (close to) zero, i.e., Vi E [N], (G) = 0; and (ii) the Hessian matrix, i.e. the second derivative of the potential energy, must be non-negative.
However, the exact mathematical potential energy evaluation, i.e., the solution to the Schrödinger equation, is still a "black box" (Cao et al.). Over the years, different levels of thcories and methods have been developed to evaluate potential energies, such force field (FF), semi-empirical methods (e.g., xTB), and density functional theory (DFT) methods (e.g., SELYP/6-31G(2d2) order of increased accuracy and cost. After trading-off between accuracy and efficiency within a feasible computation cost, we selected GFN2-xTB, a more recent and advanced semi-empirical method (Bannwarth et al. 2019), to calculate the forces of the generated molecular geometry in the diffusion process. More details about GFN2-XTB can be found in Appendix [
listed in an
Motivation for Stability Optimization with 3D Diffusion Stable molecular geometry is favored for molecule generation. novel but unstable molecular geometry may not be physically achieved at all and thus carries substantial meaning. To optimize stability, one minimize the potential energy of the molecule by iteratively minimizing the forces on the atoms (Eq, 5). motivating our choice of 3D diffusion model. More details and discussions are provided in Appendix E
3

---

Published as a conference paper at ICLR 2025
Noisy Guidance The goal of neural guidance is to direct the denoising process towards target property value y. Dhariwal & Nichol (2021) propose to modify the denoising process to achieve conditional generation with unconditional model and using scalar that controls the guidance strength:
B
21-1 N
3 2
Eg (zt: aj
()) sae V.. log (o 1f
(6)
VI
Here (v(()) is parameterized by a classifier, and is categorical label, such that the modification fo? V., mi log Po (z.)) shifts the mean of the sampling distribution to provide guidance. Let E IR
and fn :9 R be the neural regressor for the molecular property of interest (Vignac al. 2022).
Now, assuming y|a, ~ (f T, and of 1, we have:
Væ, log Ps xx) x - Va, lly - fn (3 = V S (xx))
(7)
where Cly, (xx)) the Mean Square Error (MSE) between the target and the prediction of f,(()
However, in the early stage of the denoising process, I; = 0,20, + -1--aa ofe might not be infor- mative enough predict it consists mostly of Gaussian noise. For effective prediction during the denoising process, we estimate the denoised version of e; as Kawar et al. (2022) Song et al.
2020
to(x;) == 20 Dt V1- ee
(8)
ax
where (zo) is used in place of (x) as the predicted molecular property.
Clean Guidance Besides applying the gradient as guidance on noisy æ;, we build insight from Bansal et al. (2023) and derive guidance in the clean (=noise free) space dy as:
A arg min Cly, ( + A))
(9)
where Axp is approximated using K steps of gradient descent starting from = 0. Note that Awo is in clean data space, need to translate it back to the noisy space while recovering æy:
H = at + Awo) + / afe
(10)
where ē is the augmented noise used to to sample #;-1 and is thus given by:
02
غ = c0(
Auo
(11)
V at
METHODOLOGY
3.1 GUIDANCE FROM A NON-DIFFERENTIABLE ORACLE
We aim to tackle challenging problem where the guidance is specified by non-differentiable oracle (c.g., the GFN2-xTB method). We change our notation from æ to Z₁ at each time step to indicate diffusion in the latent space. With the guidance target y being O, our non-differentiable function (Eq. for force guidance defined as follows.
1 N
N Σ 1 (())
f(l)
(12)
where the molecular graph is decoded as aim to achieve f(G)) < f(Go). Here G Go provided by CHEMGUIDE and an unguided diffusion model. As f is non-differentiable across different molecules, we can not directly add guidance using Eq. 7 Instead, we estimate the gradient analytically. Recall that (()) is the MSE loss. f g + R is the non-difierentiable oracle, and D is the decoder. Let F. be the composition f D lo. and we estimate the gradient as:
with En = to(x;) estimated by Eq. 8 and
V., log (v) x - -V(,,) F(z:)) F(z)) F([Ex, + Ç1Nx3, #h,e) F([xx,t - (1Nx3, zh.1)
(13)
-V))).))(1) F F(z))) lim 5+0
2
(14)
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Bilevel guided diffusion sampling with noisy neural guidance
Input: a latent diffusion model Eø, VAE decoder D, a composition function F, target property
score y, oracle guidance scale 8,, regressor guidance scale 8,, SPSA perturbation Ç.
Output: optimized molecule [x, h] 2T t (O)
for all from T to 1 do
14-1, I
vT 22
3. co(2t, t)),
(Eq.
v of
91 9t-1 i.gresrr x
V Vx, 11y - (D ( (x))) )
(Eq.
2 + (N4--11 + S, -1-1-119--1.grrsee
1
(Eq.
zi a- --=1111 / 0i-1
o² e', such that é' ~ N00,1)
|/ project from t - back to é
Ut (
---1 x - -VF(z;) [[())|]
(F(Z2, + (U, =h,1) - F( - CU, 2h U
(Bq. 15
z
11-1, E11 F - (2t
Vi ( o9
&
(Eq. 3
vI-
2:-1 + (N + s. -,,
(Eq. 6
end for
xx. h] + D(z))
return [x, h]
This approximation is possible because 2, is continuous and it eliminates the need to train neural regressors for properties such as forces, which comes with approximation error (Gasteiger et al, 2020). In addition, these regressors are often trained on ground state geometries (optimized stable geometries) instead of non-optimal geometries, which are the most common geometries during the denoising process, making them dubious options for property guidance. However, directly adding (or subtracting) (1 xI 2x,t would break the equivariance requirement in Eq. on the latent variables, as it shifts the mean of the coordinates by Ç. To maintain zero central gravity (Xu et al.) 2023] of input to F(.), we construct a perturbation matrix U E Rxx3
(O 1). and apply
Simultaneous Perturbation Stochastic Approximation (SPSA) (Spall 1992 Nesterov & Spokoiny 2017) to estimate the gradient, then Eq. becomes:
V s. log Po 21) x Fly F(z)) F([zx,t + (U,, 2h,1) F([zx,t CU, 21.) (15)
2
where Ç is small perturbation scale (e.g.. 10-9). The perturbed representations [zx,t + (U, zh obey zero central gravity required by 4o. as R[[U]] = O. Note that, unlike neural regressor guidance, we only add guidance to the positions (i.c. zx.1) and apply no gradient to the atom types (i.c. æh.:).
We do this because the force definition (Eq. 5 is only physically grounded when the set of atoms stays constant (i.e. no matter/mass created from or reduced to void). In other words, since energy is relative concept, any gradient that arises from atom-type change has physical interpretation. Further discussions on the choice of SPSA for gradient estimation can be found in Appendix FF
3.2 COMBINE GUIDANCE FROM NEURAL NETWORK AND NON-DIFFERENTIADEE ORACLE
Apart from guided generation conditioned on one single property (e.g., the norm of static polariz- ability a) with a regressor or a non-differentiable oracle (e.g.. GFN2-xTB), we can perform joint op- timization with the guidance specified by both the differentiable regressor and the non-differentiable oracle, which is encapsulated under the bilevel optimization framework. Formally, recall that f, represents the regressor and J f D is the composition of the non-differentiable oracle the VAE decoder D, the latent diffusion model, respectively, the joint optimization is formulated as:
min Zx,1
F([Zz,11 (f..)
(16)
s.t.
[ el = arg min [l, 2h,,)) y/
(17)
[=2,),P..1]
where is the target property value (e.g., we hope the static polarizability -+ y after optimization). As detailed in Algorithm 2 we first optimize molecular property with fa, and obtain the optimal atom positions = and features 2//, (Eq. [7]; we then optimize stability with F by fixing z and
initializing z,,t (Eq. 16 as --,,- The chemistry intuition of our design is detailed in Appendix G
5

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
4.1 EXPERIMENT SETTING
Dataset The models in our experiment are trained on the QM9 dataset (Ramakrishnan et al.) and the GEOM dataset (AxeIrod & (oanez-Bmmbareli). The QM9 dataset is catalog with 134K small drug-like molecules consisting of up to nine heavy (non-hydrogen) atoms. The Geometric Ensemble Of Molecules (GEOM) dataset includes 450K molecules with up to 91 heavy atoms (on average, 24.9), where 37 million molecular conformations are generated and reported with their geometries, energies, and statistical weight.
Guidance Property We study guided generation for force and thus energy optimization on QM9 and GEOM. For neural and combined guidance, we evaluate on QM9 as there are no labels in GEOM to train the regressors. We consider the following 6 properties reported in QM9: the norm of static polarizability (a, Bohr"), the norm of dipole moment (1,, Debye), heat capacity at room temperature (C,,, cal/(mol-K) K)), the energy of the electron in the highest occupied molecular orbital (сhомO. eV), the energy of the electron in the lowest unoccupied molecular orbital (ELUMO. eV), and the HOMO-LUMO energy gap (Ac, eV). We choose 8 (Eq. 6) from (1, 10-10- 10-2, 10-3, 10-1) for all experiments, and additionally (2, 10, 20, 25, 30, 40, 50} for the 6 properties.
Model We integrate CHEMGUIDE with GeoLDM (Xu et al., 2023), an improvement from EDM (Hoogeboom et al. 2022). For non-differentiable guidance on force, we compare our method to unconditional EDM and GeoLDM since there is no available ground-truth foree to explicitly train a conditional model. For noisy neural guidance and combined guidance, we choose condition- ally trained EDM (C-EDM) and GeoL.MM (C-GeoLDDM) as the baselines.
Evaluation Metric For non-differentiable guidance on the force, we use the Root Mean Square (RMS) of the forces calculated at GFN2-xTB as the evaluation metric. We also report validity, uniqueness, atom stability, molecule stability, and energy above the ground state. For differentiable neural guidance on the 6 properties (a, μu, Ce., EHOMO: ELUMO, Ae), we follow Xu et al. (2023) and use the Mean Absolute Error (MAE) between the target property y and the predicted value 9 from the regressor of the generated molecule as the evaluation metric.
We add guidance to the last 400 of the 1000 diffusion steps (Han et al. 2024). and calculate the change percentage between our method and the DioolDDC/CGGoaaaa. baseline. The implementa- tion details are provided in Appendix B
Table 1: Metrics of 500 generated molecules from QM9 using GeoLDM with non-differentiable oracle (i.e. xTB) guidance. and bold denote the overall best result and our best result, respectively. Percentage changes between our results and GeoLDM are shown in parentheses.
Guidance Scale
Baseline
Metric
0.0001
0
0.01
0.1
1.0
BDM
GooL.DM
Force RMS (Eh/Bohh)
0,0104 (-6.76%-)
0.0104
0.0107
0.0108
0,0125
0.0114
0.0111
Validity
91.40% (1 (.60%)))
91.20%
91.20%
90.00%
89.40%
86.60%
89.80%
Uniqueness
100.00%
100.00%
100.00%
100.00%*
100.00 %
100.00%*
100.00%*
Atom Stability
99.02%
98.97%
98.98%
99.03%*
(0.10
98.67%
98.53%
98.93%
Molecule Stubility
90.60%
90.40%
90.20%
91.00% % (2.20%
87.80%
83.60%
88.80%
Energy above Ground State (Eh)
0.0042 (-15.78%)
0,0042
0.0045
0,0050
0.0061
0.0072
0.0050
4.2 NON-DIFFERENTIABLE ORACLE GUIDANCE ON FORCE
We adopt various guidance scales on QM9 and show numeric results in Table 1 It is observed that over different scales, our non-differentiable oracle guidance generates molecules with up to 6.31% lower force, 1.60% and 2.20% higher validity and molecule stability, respectively. Despite optimizing molecular stability via force instead of directly with energy, CHEMGUIDE exhibits a smaller energy change than GeoLDM with 15. 15.78% decrease. These results show that our method generates molecules closer to the ground state, i.e. the most stable geometry, suggesting that our methods can generate molecules with higher validity and stability. Further, we explore CHEMGUIDE on the GEOM dataset with larger molecules and report the results in Table where a similar trend is observed in our favor: method generates valid molecules closer to the ground state. Surprisingly, CHEMGUIDE performs better on larger molecules than smaller ones (c.g., a higher decrease in force RMS), which brings potential applications in large-molecule production, e.g., protein. This is partly because larger molecules have more intricate structures and conformational space; thus, are likely to suffer more from distorted geometries than smaller ones, yielding more space for improvement.
FEDM and Geol. .DM in short of the unconditional version of the models when the context is clear.

---

Published as a conference paper at ICLR 2025
0.004 0.007 0.010 0.013 0.016
0.01
0.105
0.0
0.12
0.24
0.96
0.48
0.60
140
'20
70
GeoLDM
100
60
50
40
80
30
20
20
0.02 0:05 0.05 0:11
8
3.00
(a) QM9 Force RMS
(b) QM9 Energy Change
(c) GEOM Force RMS
(d) GEOM Energy Change
Figure 2: Histograms and distributions of force RMS and energy change of 500 generated molecules
from QM9 and GEOM using GeoLDM with CHEMGUIDE under scalc=0.0001. Energy change refers to the energy of our generated molecule above its ground state energy.
Table 2: Metrics of 500 generated molecules from GEOM using unconditional GeoLDM with non- differentiable oracle (i.e. GFN2-xTB) guidance. and bold denote the overall best result and our best result. Percentage changes between our results and Ge are shown in parentheses.
Metric
Guidaacee Scale
Baseline
0.001
EDM
GosLDM
Force RMS (Eh//ohhr
0.0445
0.0447
0.0434
0,0411" (-14.16%)
0,0513
0.0742
0.0478
Validity
47.00%
45.20%
51.60% % (2.609;))
S0.40%
49.40%
46.40%
49.00%
Uniqueness
100.00%
100.00%
100,00%
100.00%
100.00% %
100.00%*
100.00%
Atom Stability
84.47%
84.49%
84.68 % (0.12%)1)
84.36%
83.45%
81.22%
84.53%
Energy aboue Ground State (Fh)
0,2085
(-13.92%)))
0.3742
0.2248
Fig. shows the plot of the force RMS and energy change histogram and distribution of the 500 generated molecules from EDM, GeoLDM, and CHEMGUIDE on QM9 and GEOM. Our method more concentrated around 0 and less spread to higher regions. Also, note that both EDM and GeoLDM generate outlier molecules with significantly large force and energy change, while no outlier is generated by CHEMGUIDE. For example, in Fig. a). an outlier with force RMS 0.155 is generated by GeoLDM, which is 15 times higher than the force RMS where the molecules are concentrated. It demonstrates that method performs better both on average and molecule-wise, and we provide detailed plots of various scales in Fig. I in Appendix K
To further validate the effectiveness of CHEMGUIDE, we compute stability metrics at the DFF78B11YY//-331(((2dd).)) level of theory with Gaussian/6 (Frisch et al., 2016) for more precise and strict benchmarking. As DFT is computationally demanding, we only present the results of CHEMGUIDE with the best scale, GeoLDM, and EDM on QM9 in Table 3 Despite applying a less accurate but computationally feasible oracle (i.e., GFN2-xTB), CHEMGUIDE still improves validity and stability (i.e., valid molecular geometry close to optimized ground state geometry), when mea- sured against higher standards (i.e., DFT). On the other hand, applying DFT as the non-differentiable oracle guide the generation process is computationally expensive and impracticable.
Table 3: Metrics of 500 generated molecules from QM9 using GeoLDM with CHEMGUIDE using scale=0.0001, calculated at DFT7B33YY//-331 computationally expensive than GFN2-XTB. Strict validity is defined as generated geometries within 50 kcal/mol (0.07968
Nony Guitlence Guidance
level of theory, more precise but
GeoLDM
C-Geo.DN DM
MAE of of
Eh) of the optimized geometries. and bold denote the overall best result and our best result. Percentage changes between our results and GeoLDM are shown in parentheses.
Metrie
Ours
EDM
GeoLDM
170
150 200 230
300
350
Validity
96.2% (2.6%*)
89.0%
93.6%
Scops
Strict Validity
95.6% " (2.4%))
88.4%
93.2%
Figure 3: MAE trajectories of Ge- oLDM, with noisy/clean guidance, and C-GeoLDM on C..
Energy above Ground State (Eh)
0.0040511 (9.6%))
0.006314
0.004480
Energy above GS (kcal/mol)
2.542 (-9.6%))
3,962
2.811
4.3
NEURAL NETWORK GUIDANCE ON PROPERTIES
4.3.1
UNCONDITIONAL DIFFUSION WITH NOISY NEURAL GUIDANCE
We present the results of noisy guidance (Section 2 in Table provide further details in Table 16 in Appendix L For all the properties a, μ. C.,, CHOMO. LLUMO and Ae, noisy neural guidance outperforms unconditional GeoL.DM by 57.92%, 37. 24% 77.09% 23.18%, 36.87%, and 37.88%

---

Published as a conference paper at ICLR 2025
Table 4: MAE of 500 generated molecules from QM9 using GeoLDM with noisy/clean neural guidance, where the result from the best configuration is reported. We use C-EDM, C-GeoLDM, and GeoLDM as basclines. * and bold denote the overall best and the best within different scales, respectively. Percentage changes between our results and D- are shown in parentheses.
Property
C,
FHOMO
+LUMO
A
Units
Bohr
D
cal
K
meV
meV
meV
mat
Conditional EDM
2.6308
1.1257
1.0804
0.3207
0,5940
0.6301
Conditional GM
2,5551
1.1084
0.9703
0.3327
0.5518
0,5878
GeoLDM
5.6732
1,6461
3,1046
0.6151
1.1778
1.2022
2.3870
GeoLDDM w/ Noisy Guidance
(-6.58%
1.0331
0.7112
0.4725
0.7436
0.7468
0.2031
0.3151
0.3109
0.3830
GeoLDM w/ Clean Guidance
5,4798
0.5688
(-81.68%L)
(-67.53%L)
(-3.06%%)
(-34.84%.)
respectively, which verifies its effectiveness. Furthermore, GeoLDM with noisy neural guidance yields comparable performance with C-GeoLDM and C-EDM, provided that conditional models are explicitly trained for property optimization; surprisingly, we use unconditional GeoLDM as the backbone and outperforms C-GeoLDM in a, Jl, and C.. To understand the details of guided generation, we plot the MAE of C. on sampled molecules at each guidance step in Fig.Bi Compared GeoLDM which oscillates in the first 200 steps and converges slowly high MAE, the trajectory of noisy neural guidance drops fast in the first 50 steps and the amplitude of the oscillation is less intense, which suggests the guidance smooths the MAE change and quickly steers the model toward better properties. The property MAE trajectories are detailed in Fig. 12 in Appendix
4.3.2 UNCONDITIONAL DIFFUSION WITH CLEAN NEURAL GUIDANCE
As presented in Table 4 clean neural guidance outperforms unconditional GeoLDM on all prop- erties and beats D- in μs, C., CHоMO, and Ae by 81.68%, 67.53%. 3.06%, and 34.84%. Interestingly, it seems that noisy and clean guidance complement each other on different proper- ties (e.g., Or Ac). Fig. 3 clean neural guidance shows a similar oscillation trend in the first 200 guidance steps compared to GeoL.DM, then exhibits a sharp drop in the last 50 steps and achieves the lowest MAE, which is also observed on other properties shown in Fig. 12 in Appendix [L] We hypothesize this is due to that molecules are less noisy denoising steps accumulate; thus, guid- ance optimization in the clean space manifests its strongest power when z is less dependent on the one-step estimation function t.):) in later steps, where it is closer and more similar to zu (Eq. g
4.4 COMBINED GUIDANCE ON FORCE AND PROPERTIES
We present the results for the bilevel optimization on molecular property and stability, with MAE shown in Table 5 and the force RMS and energy change in Fig. We sample 200 molecules for combined guidance, as there are properties to optimize, and force optimization with xTB only supports CPU computation. We choose conditional GeoLDM (C-GeoLDM) the baseline.
Table 5: MAE of 200 generated molecules using DM with CHEMGUIDE and GeoLDM with noisy neural guidance, where the result from the best configuration is reported. We use C-EDM and C-GeoLDM baselines. and bold denote the overall best and the best within different scales, respectively. Percentage changes between our results and C-Geol.DM are shown in parentheses.
Property
C₂
CHOMO
GLUMO meV
Ae
Units
Bohr
D
cal K
meV
mcV
mol
C-EDM
2.5089
1.0571
1.0624
0.3304
0.5046
0.6191
C-GcoLDM
2.5593
1.1582
0.9646
0.3407
0.5927
0.4982
Bilevel Optimization
2.4430*
0.3404
0.5077*
0.5225
1.0483
0.9393
w/ Explicit Guidance
(-4.54%)1
(-0.09%1)
(-14.34%..
(4.88%1)
Bilevel Optimization
0.7246
0.9369
2.4860
0.4445
0.8099
0.7041
w/ Noisy Guidance
(-37.43%L)
(-2.87%%)
4.4.1 EXPLICITLY GUIDED DIFFUSION WITH CHEMGUIDE
We employ C-GeoLDM with CHEMGUIDE, which resembles the bilevel optimization framework as C-GeoLDM automatically optimizes molecular property during the denoising process. Compared
to C-GeoLDD DM alone, expect method to reduce the force while not hurting MAE. We present
8

---

Published as a conference paper at ICLR 2025
10
Explict Guidance Implit: Guidonce
Bitevel Opt. wl Konsy Guidence
1.0
C-GeoLDM
GOOLOM
treag thange
E-Geeti DM
0.
MAE of
06
1.0
0.
C
Nomo
010h13
C4
NOmT
100
Guidanice Staps
356
250 300
355
430
Figure 4: Force RMS and energy change of 200 generated
Figure 5: MAE trajectories of GeoL.DM, C-GeoLDDM, and our bilevel optimization on C..
molecules using explicit bilevel optimization, implicit bilevel
optimization with noisy neural guidance, C-EDM, and C- GeoLDM. Energy change refers to energy above ground state.
the results in Fig. 4 and Table (details are provided in Table 17 and Table 18 in Appendix MM. For the 6 properties (a, μ. C.. ÉHоMO- ELUMO, Δe), our method reduces the force RMS by 9.21 %, 3.33%. 8.75% 6.69%, 1.27%, and 10.19%, and for the first 5, MAEs decrease 4.54%, 9.49%, 2.62%, 0.09% and 14.34%. The results suggest feasible and effective to combine explicitly guided diffusion (e.g., C-GeoLDM) with CHEMGUIDE, which goes beyond our expectation as both force RMS and property MAE are reduced.
4.4.2 IMPLICITLY GUIDED DIFFUSION WITH CHEMGUIDE
We apply two aforementioned neural network guidance approaches (noisy/clean guidance, Sec- tion 2 for the property optimization step in our bilevel optimization framework (Section (3.2). We expect that both the property MAE and the force metrics are better than conditional GeoI.DM.
Bilevel optimization with noisy guidance As shown in Table 5, our method outperforms C- GeoLDM in a, 14, and C, by 2.86%, 37.43%, and 2.87% in MAE. is also observed in Fig. that, for the six properties, our method achieves lower force RMS than C-GeoLDM by 1.94%, 4.57%, 8.97%, 10.66%. 16.26%, and 7.18%, with smaller energy change compared to baselines. Details of MAE and force metrics are provided in Table. 18] and Table. [99 in Appendix M Combining prop- erty optimization with force optimization, noisy guidance with CHEMGUIDE reduces MAE and improves the stability of generated molecules, which suggests that the combined guidance prompts the diffusion model towards generating stabler molecules with better properties.
To further understand how our bilevel framework optimizes property, we provide the MAE trajectory of C. in Fig. [ The trajectories our noisy neural guidance method (Fig and bilevel optimization behave similarly, such that the MAE drops fast in the first 50 steps and oscillates less compared to GeoLDM. However, bilevel optimization is slightly worse than the noisy neural guidance method (i.e. it converges closer to C-GeoLDM than noisy guidance). We suspect it is because guided by gradients from the neural regressor and then CHEMGUIDE, the generation process Icans less toward lower MAE, it keeps a balance between property and stability optimization. Details and explanations of the MAE trajectories of each property can be found in Fig. 13 in Appendix M
Bilevel optimization with clean guidance
We present the details of this section in Appendix 0
4.5 CHEMGUIDE BEYOND STABILITY
In this section, we explore the potential of using CHEMGUIDE to optimize molecular proper- ties other than force and energy. We sample 200 molecules using unconditional GeoLDM with CHEMGUIDE o, J4, CHOMO. CLUMO. and Ac, which can also be calculated at GFN2-xTTH and the results are shown in Table [ Specifically, since the property label of a molecule from QM9 (calculated at DFT) may be different from the xTB computation, we obtain property scores of all QM9 molecules with xTB as distribution where we sample the guidance target, which is the reason that GeoLDM has different MAE than Table 3 and GEOM is processed similarly. It is observed that CHEMGUIDE on properties improves the baseline in μ. ML and Ac. Our method performs less satisfyingly on a and еHоMO because they are inherently more challenging to optimize, as observed in Table We can see that CHEMGUIDE is general and applicable to broad range of molecular optimization tasks other than stability across different datasets. Full results can be found in Table [4] and 15 in Appendix K and our discussion the difference between CHEMGUIDE and neural guidance for properties is presented in Appendix H.4
*Co is skipped its calculation requires the Hessian matrix, which expensive compute.

---

Published as a conference paper at ICLR 2025
Table 6: CHEMGUIDE for properties (o, μ. cHоMO. ELUMO. Ac) with 200 generated molecules on QM9 and GEOM, where the result from the best configuration is reported. We use GeoLDM as the baseline. * and bold denote the overall best and the best within different scales, respectively. Percentage changes between results and GeoLDM are shown in parentheses.
Property
a
CHOMO
CLUMO
A
Units
Bohr
meV
meV
meV
GcoLDM
4.7555"
1.5490
0.6264"
2.1790
2.1568
QM9
4.8781
1.5295'
0.6392
2.0091*
1.9318
GeoLDM w/ CHEMGUIDE
(2.58%1)
(-1.26%.4)
(2.04%1)
(-7.80%%)
(-10.43%%L)
GcoLDM
24.3424
3.3676
1.1922"
5,0903
1.9956
GEOM
24.2168'
3.3412
1.2939
5.0802
1.9264'
GeoLDM w/ CHEMGUIDE
(-0.52%.)
(-0.78%
(8.53%1)
(-0.20%))
(-3.47%%)
5 ANALYSES
In Appendix H. we discuss acceleration methods for CHEMGUIDE (Section H. different guidance
steps (Section H.2 neural guidance for stability optimization (Section H..3 generalization analyses of property regressor (Section HH.4) and conditional diffusion model (Section H.5); the effect of
guidance scales for noisy/clean guidance (Section H.6 the effect of optimization steps for clean guidance (Section
6 RELATED WORK
Due to space limits, we present the discussion of related works in Appendix C
7 CONCLUSION & FUTURE WORK
In this work, we study conditional generation for 3D diffusion models molecules. We pro- pose CHEMGUIDE, which guides the generation process of an unconditionally trained diffusion model using an external, typically non-differentiable, chemistry oracle. By analytically estimating the guidance gradients using zeroth-order optimization methods, while ensuring the equivariance and invariance requirements for 3D diffusion models, we aim enhance the stability of generated molecules without relying on labeled data to train an additional property predictor. We observe re- duced atomic forces and improved molecular validity across both small and large molecule datasets when the model is equipped with CHEMGUIDE, highlighting the effectiveness of our approach. Fur- ther experiments incorporating both CHEMGUIDE and neural guidance within bilevel optimization framework demonstrate improvements in both molecular properties and stability. This suggests that our non-diffcrenniable guidance compatible with existing methods. Moreover, when CHEMGUIDE is used for property optimization alone, it shows strong generalization beyond force minimization.
However, since xTB runs on CPU and neural guidance requires gradients that heavily utilize GPU resources, method may be constrained by limited insufficient computational resources. Fu- ture work could explore ways to accelerate the guidance process while reducing computational costs without compromising performance. Additionally, we observe that the guidance scale plays crit- ical role in the generative process of guided diffusion for molecules, but it is generally difficult determine in advance. This opens the possibility for research into more effective methods for selecting guidance scales, including automated scale scheduling that optimizes guidance strength.
In summary, we propose CHEMGUIDE, which introduces guidance signals for the diffusion process from non-differentiable quantum chemical oracle. We hope this work sparks further exploration into how domain knowledge from physics and chemistry can be more effectively integrated into neural networks to enhance molecular design.
ACKNOWLEDGMENTS
We are grateful to our anonymous ICLR reviewers, with whose help this work has been greatly improved. The work was partially supported by the NSF Award 2307698.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger. Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Dylan Anstine, Roman Zubatyuk, and Olexandr Isayev. Aimnet2: neural network potential to meet your neutral, charged, organic, and geemental-organie needs. 2024.
Dylan M Anstine and Olexandr Isayev. Generative models as an emerging paradigm in the chemical sciences. Journal of the American Chemical Sociery, (5(16))8736-8755) 2023.
Simon Axelrod and Rafael Bonbaz--o GEOM, energy-annotate molecular con-
formations for property prediction and molecular generation. 9(1):185. ISSN 2052- 4463. doi: 10. 141-0033/841599--2202222888 841597-022-01288-
hestps::/www.rret wwm.surrreccommeoooooommmm.mo.oe aariiittcclsss...
Christoph Bannwarth, Sebastian Ehlert, and Stefan Grimme. Gfn2-xtb-an accurate and broadly parametrized coff-soostteetttte hight-binding quantum chemical method with multipole electrostatics and
dispersion contributions. Journal of chemical theory and computation, 15 13):1622-167,, 2019.
Christoph Bannwarth, Eike Caldeweyher, Sebastian Ehlert, Andreas Hansen, Philipp Pracht, Jakob Seibert, Sehastian Spicher, and Stefan Grimme. Extended tight-binding quantum chemistry methods. WIREs Computational Molecular Science, 11(2):e1493, 2021. doi: https://doi.oo org// 10.1002/wems. 1493. 10. 1002 wcms. 1493
wetaaar..willee. wiley. cow ht///ooi//aa///
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 843 852-852,20 2023.
Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy- guided sde for inverse molecular design. In The eleventh international conference learning representations, 2022.
Yudong Cao, Jonathan Romero, Jonathan Olson, Matthias Degroote. Peter D. Johnson, Mária Kieferová, Ian D. Kivlichan, Tim Menke, Borja Peropadre, Nicolas P. D. Sawaya, Sukin Sim, Libor Veis, and Alán Aspuru-Guzik, Quantum chemistry in the age quantum computing. 119 (19):10856-10915 ISSN 0009-2665, 1520-6890.
doi: 10..002/accscceeee chemrev.8000003 URL https://pubs. acs. org /tiil/ii11111011 1021/002/cccc chemrev. 8b00803
Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research, 153:235-256, 2007.
Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. arXiv preprint arXiv: 1805. 2018.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021.
Earnest Oghenesuvwe Erhirhie, Chibueze Peter Ihekwereme, and Emmanuel Emeka Ilodigwe. Ad- vances in acute toxicity testing: strengths, weaknesses and regulatory acceptance. Interdisci- plinary toxicology, 1 11(1):5-122 12, 2018.
M. J. Frisch, G. W. Trucks, H. B. Schlegel, G. E. Scuseria, M. A. Robb, J. R. Cheeseman, G. Scal- mani, V. Barone, G. A. Petersson, H. Nakatsuji, X. Li. M. Caricato, A. V. Marenich, Bloino, B. G. Janesko, R. Gomperts, B. Mennucci, H. P. Hratchian, J. V. Ortiz, A. F. Izmaylov, J. L. Sonnenberg, D. Williams- Mouiiamm-Young,,
Ding, F. Lipparini, F. Egidi, Goings, B. Peng, A. Petrone, T. Henderson, D. Ranasinghe, V. G. Zakrzewski, J. Gao, N. Rega, G. Zheng, W. Liang, M. Hada, Hasegawa, M. Ishida, T. Nakajima, Y. Honda, O. Kitao, H. Nakai, T. Vreven, K. Throssell, J. A. Montgomery, Jr., J. E. Peralta, F. Ogliaro, M. J. Bearpark, J. Heyd, E. N. Brothers, K. N. Kudin, N. Staroverov, T. A. Keith, R. Kobayashi, Normand, K. Raghavachari, A. P. Rendell, J. C. Burant, S. S. Iyengar. J. Tomasi, M. Cossi, J. M. Millam, M. Klene, C. Adamo, R. Cammi, J. W. Ochterski, R. L. Martin, K. Morokuma, O. Farkas, J. B. Foresman, and D. J. Fox. Gaussian"T 16 Revision C.01, 2016. Gaussian Inc. Wallingford CT.
M. Ehara, K. Toyota, R. Fukuda,

---

Published as a conference paper at ICLR 2025
Johannes Gasteiger, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann. Fast and uncertainty- aware directional message passing for non-equilibrium molecules. In Machine Learn- ing for Molecules Workshop, NeurIPS, 2020.
Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. Advances in neural information processing systems, 32, 2019.
Nate Gruver, Samuel Don Stanton, Nathan C. Frey, Tim G. Rudner, Isidro Hotzel, Julien Lafrance- Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew Gordon Wilson. Protein design with guided discrete diffusion. In Thirty-seventh Conference on Neural Information Processing Sys- lems, 2023. URL hnttps://openrevvtt/ net htwp:/wwwwwweeeeeoeonnnn.. www.forrmooooooooo werrumiid-MM.. KK
Rafael BombrrBo Jennifer N. Wei, David Duvenaud, José Miguel Hernández-Loband Benjamín Sánchez-Lengelingg, Dennis Sheberla, Jorge Aguilera-Iparraguirre. Timothy D. Hirzel, Ryan P. Adams, and Alán Aspuru-Guzik, Automatic chemical design using data-driven con- tinuous representation of molecules. ACS Central Science, 4(2):268 276, 2018. doi: 10. 2t21/acscentsci.7b00553 PMID: 29532027.
URL /tt
org. wtt.:001111aacccccccccoo 7b00572
Xu Han, Caihua Shan, Yifei Shen, Can Xu, Han Yang. Xiang Li, and Dongsheng Li. Training- free multi-objective diffusion model for 3d molecule generation. In The Twelfth International Conference on Learning Representations, 2024. URLLhttps://openeette
net forum?
id=x41 c4444k4
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Emiel Hoogeboom, Victor Gareia Satorras, Clément Vignac, and Max Welling. Equivariant diffu- sion for molecule generation in 3d. In International conference machine learning. pp. 8867- 8887. PMLR, 2022.
Yujia Huang, Adishree Ghatare, Yuanzhe Liu, Ziniu Hu, Qinsheng Zhang, Chandramouli S Sas- try. Siddharth Gururani, Sageev Oore, and Yisong Yue. Symbolic music generation with non- differentiable rule guided diffusion. arXiv preprint arXiv:2402.14285 2024.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference machine learning, pp. 2323-2332. PMLR, 2018.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hicrarchical graph-to-graph translation for molecules, arXiv preprint 1xXX::1907.11220 2019.
Jachyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. In International conference on machine learning, pp. 10362-10383. PMLR, 2022.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593-23606 2022.
Leo Klarner, Tim Rudner, Garrett Morris, Charlotte M Deane, and Yee Whye Teh. Context-guided diffusion for nut-of-distribution molecular and protein design. arXiv preprint arXiv:2407.11942 2024.
Lingkai Kong, Jiaming Cui, Haotian Sun, Yuchen Zhuang, B Aditya Prakash, and Chao Zhang Autoregressive diffusion model for graph generation. In International conference machine learning, pp. 17391-17408. PMLR, 2023.
Matt J Kusner, Brooks Paige, and José Miguel Hernández-Lobato Grammar variational autoen- coder. In International conference on machine learning, pp. 1955-1954, PMLR, 2017.
Seul Lee. Jaehyeong Jo, and Sung Ju Hwang. Exploring chemical space with score-based out-of- distribution generation. In International Conference on Machine Learning, pp. 18872-18892. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Yibo Li, Liangren Zhang, and Zhenming Liu. Multi-objective de novo drug design with conditional graph generative model. Journal of cheminformatics, 10:1-24, 2018.
Youzhi Luo and Shuiwang Ji. An autoregressive flow model for 3d molecular geometry generation from scratch. In International Conference on Learning Representations (ICLR), 2022.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alexandru Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. ArXiv, abs/2305. 173333
2023. btLpstos:: api www.::sstaannccchhheoeeeeccccm wtt.://oww.....cooooora.aaaaaaaa
Matteo Manica, Jannis Born, Joris Cadow, Dimitrios Christofidellis, Ashish Dave, Dean Clarke, Yves Gaetan Nana Teukam, Giorgio Giannone, Samuel C Hoffman, Matthew Buchan, et al. Ac- celerating material design with the generative toolkit for scientific discovery. npj Computational Materials, 91):69, 2023.
Yurii Nesterov and Vladimir Spokoiny. Random gradient- free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527-506 2017.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.
Oleksii Prykhodko, Simon Viet Johansson, Panagiotis-Chrisce Chriistos Kotsias, Josep Arús-Pous, Es- ben Jannik Bjerrum, Ola Engkvist, and Hongming Chen. A de novo molecular generation method using latent vector based generative adversarial network. Journal of Cheminformatics, 11(1):1- 13, 2019.
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. 1(1):140022. ISSN 2052-4463. doi: 10. 1...088/data.201.2222 URL https: /tt:://wwwww /www nature www. hatm/articles/// adat ad11
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp.
2022.
Victor Garcia Satorras, Emiel Hoogeboom, Fabian B Fuchs, Ingmar Posner, and Max Welling. (n) equivariant normalizing flows. arXiv preprint arXiv:2 2021.
Arne Schneuing, Charles Harris, Yuanqi Du, Kieran Didi, Arian Jamasb, Ilia Igashov, Weitao Du, Carla Gomes, Tom Blundell, Pietro Lio, al. Structure-bass drug design with equivariant diffusion models. arXiv preprint arXiv: 10..:22200.33.... 13695, 2022.
Marwin H. S. Segler, Thierry Kogej. Christian Tyrchan, and Mark P. Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS Central Science, (1):120-131, 2018. doi: 10. hta////cetsci.7000S... URL
acscentsci. 7b00512
PMID: 29392184.
Martin Simonovsky and Nikos Komodakis. Graphvae:
Towards generation of small graphs using variational autoencoders. In International conference on artificial neural networks, pp. 412-422. Springer, 2018.
Jascha Sohl- Dick-iisst,,,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
James C. Spall. Multivariate stochastic approximation using simultancous perturbation gradient approximation. IEEE Transactions on Automatic Control, 37:332-341, 341, 1992. URL https: 7/api. semant cacholaaaaao wtwpcholar..org/oorpussssssssssa
13

---

Published as a conference paper at ICLR 2025
Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pas- cal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14733 2022.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eise- nach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of 2023.
protein structure and function with rfdiffusion. Nature, 620(7976):1089-11
Tomer Weiss, Eduardo Mayo Yanes, Sabyasachi Chakraborty, Luca Cosmo, Alex M Bronstein, and Renana Gershoni-Poranne. Guided diffusion for inverse molecular design. Nature Computational Science, 3(10))873-882 -82, 2023.
Luhuan Wu, Brian L. Trippe, Christian A Naesseth, John Patrick Cunningham, and David Blei. Practical and asymptotically exact conditional sampling in diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. hett/forumiid=eNNKeee
Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning, pp. 38592-38610. PMLR, 2023.
Sherry Yang. KwangHwan Cho, Amil Merchant, Picter Abbeel, Dale Schuurmans, Igor Mordatch, and Ekin Cubuk. Scalable diffusion for materials generation. In NeurlPS 2023 Al for Science Workshop, 2023. URL /aattps:/cpperretttt
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1836-3847, 2023.
Lingxiao Zhao, Xueying Ding, and Leman Akoglu. Pard: Permuration-invariint autoregressive diffusion for graph generation, 2024a. URL/attps://arkkt org/ attpg//bb/244022 03687
Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Grosse. Probabilistic inference in language models via twisted sequential monte carlo. arXiv preprint 1755: 2024b. 2024h.
14

---

Published as a conference paper at ICLR 2025
A LIST OF CONTENTS
We detail the contents in the appendix below.
Implementation Details (Section B) includes the hardware and running time, the models,
the evaluation metrics for our experiments, and the link to the code repository.
includes related literature on molecule generation and guided Related Work (Section C diffusion.
GFN2-xTB (Section D introduces the method used by CHEMGUIDE to perform quantum chemistry calculation.
Motivation for Stability Optimization (Section E) explains the importance and motiva-
tion of stability optimization for molecule generation.
Motivation for Gradient Estimation with SPSA (Section explains the motivation of choosing SPSA as the zeroth-order optimization method to estimate gradients.
Chemistry Reasoning behind Bilevel Property Optimization (Section G explains the
chemistry intuition for the bilevel optimization framework on molecular property and sta-
bility.
Analyses (Section Hp discusses acceleration methods for CHEMGUIDE (Section H.
different guidance steps (Section H.2 neural guidance for stability optimization (Sec- tion HL33: ; generalization analyses of property regressor (Section HH44) and conditional dif-
fusion model (Section H.5)): the effect of guidance scales for noisy/clean guidance (Sec-
tion H. the effect optimization steps for clean guidance (Section H.7).
Other Optimization Algorithms (Section D discusses the possibility of conditional gen- eration and stability optimization with evolutionary algorithm, and how it can be used to- gether with CHEMGUIDE.
J discusses using clean guidance and CHEMGUIdE in the bilevel optimization framework.
Bilevel Optimization with Clean Guidance (Section
Comprehensive Results on Non-differenniiable Oracle Guidance (Section presents detailed results of CHEMGUIDE for stability and molecular property optimization.
Comprehensive Results Neural Guidance (Section noisy and clean guidance for molecular property optimization.
L presents detailed results of
Comprehensive Results on Bilevel Optimization (Section for molecular property and stability optimization.
M presents detailed results
Molecule Visualization (Section N
visualizes the generated molecules.
B IMPLEMENTATION DETAILS
Hardware & Time We use 48 GiB A6000 GPU with AMD EPYC 7513 32-Core Processors for our experiments. For molecular optimization conducted with xTB (e.g.. force, a), it takes around and 18 hours to sample 100 molecules the QM9 and GEOM datasets, respectively.
Model For unconditional EDM (Hoogehoom al. 2022 and GeoLDM (Xu al., 2023), we the checkpoints available in the official implementations. For the conditional models (i.c., C- EDM, C-GeoLDM) and the property regressors, we follow the instructions and train them with the hyperparameters specified by the authors.
Evaluation Metric We present the metrics to evaluate molecular stability and property our experiment as follows.
For stability optimization, we report the following metrics:
force RMS (Eh/Bohr): we compute the root mean square (RMS) of the forces the atoms of a molecule; it is the lower the better.
validity: it measures how many percent the generated molecules are valid, and is caleu- lated by either xTB or DFT and specified in the context; it is the higher the better.
15

---

Published as a conference paper at ICLR 2025
uniqueness: it is the percentage of unique molecules among the generated molecules (Xu ct al. 2023); it is the higher the better.
atom stability: is the percentage of atoms that have the correct valency (Xu et al. 2023 it is the higher the better.
molecule stability: it is the percentage of generated molecules whose atoms are all sta- ble (Xu et al., 2023); is the higher the better.
energy above ground state (Eh): it is the difference between the energy before and after using xTB or DFT (specified in the context) to optimize the geometry of the generated molecules, and a smaller energy above ground state indicates the geometry closer the stable ground state of the molecule; it is the lower the better.
strict validity: it is defined as generated geometries within 50 kcal/mol (0.07968 Eh) of the optimized geometries; it is the higher the better.
For property optimization (eV), we report the following metrics:
(Bohr"), μ (Debye), C, cal/(mol-K), K), eHOMO (eV), ELUMO (eV), Ae
MAE: we compute the Mean Absolute Error (MAE) between the target property y and the predicted value by either the regressor or xTB (specified in the context) of the generated molecules; the lower the better.
We use one step of guidance/optimization for our experiments (e.g., clean guidance in Section 2 by default. Our implementation is available at httpss/ggffuubcooomm----ccchhaao
C RELATED WORK
Molecule Generation Previous works focus on generating molecules as SMILES strings (Kusner
al.
2017:
GombarBoo al. 2018 Segler et al. [2188). and 2D graphs (Li al. 20188Jo
et al.
2022
where various models, such as VAE (Simonovsky & Komodakis, 2018 et al.
2018
2019),
GAN (De Cao & Kipf 2018 Prykhodko et al. 2019), diffusion models (Vignac et al.
2022
Kong al.
2023] Zhao et al., 2024a). have been proposed to serve the generative backbone. To
model the spatial structure of molecules, G-Schnet (Gebauer et al.) 2019) and G-SphereNet (Luo)
& Jil 2022) use autoregressive models to generate the 3D coordinates of the molecules, however, they are less powerful compared with diffusion models. |Hoogeboom et al. (2022) propose to use the equivariant graph
equivariant diffusion model (EDM) for 3D molecule generation, which utilizes
neural network (Satorras et al. (02T) to model the molecules as a 3D graph, with coordinates and
atom types its node features. GeoLDM (Xu et al. (223) further extends EDM to latent diffusion, and has shown improvements in stability and validity of the generated molecules. However, to achieve conditional generation, both EDM and GeoLDM need be re-trained with labels, where the target property value is appended to the feature space auxiliary information, which guides the generation toward fulfilling certain molecular property requirements
Guided Diffusion Guided diffusion has shown promising results in vision (Bansal et al.) 2023
where the guidance can be provided by texts (Rombach et al. 2022), poses and edges (Zhang et al. adapted for diffusion-
2023). classifiers (Dhariwal & Nichol, 2021). etc. Similar approaches
based molecule generation, where explicitly guided diffusson (Hoogeboom et al., 2022 Xu et al.
2023 trains a conditional model with labels, and implicitly guided diffusion (Vignac et al. 2022
Bao et al., 2022) combines an unconditional model with a conditional property network, which
provides guidance as gradients to the diffusion process inference. Gruver al. (2023 propose
guided discrete diffusion for multi-objective protein sequence optimization, and Han et al. (2024)
achieve multi-property guidance by modeling property relations as probabilistic graphs. Weiss et al.)
(023) investigate guided diffusion and demonstrate the capability to design molecules beyond the seen property distribution, while MOOD (Lee et al. 2023) and GCD (Klarner et al., 2024) incorpo- rate out-of-distribution control to improve generalization of the property predictor. However, unlike CHEMGUIDE, all of the above works require a labeled dataset to train a property network; thus, inevitably constrained by the generalization ability of the neural networks. Beyond molecule opti- mization, diffusion guidance has been discussed in the broader literature, such as protein design (Wu)
et al. 2023). large language models (Zhao al. 2024b), and music generation (Huang al. 2024)
16

---

Published as a conference paper at ICLR 2025
D GFN2-XTB
GFN stands for, respectively, geometry optimization, vibrational frequencies, and non-covalent in- teractions. xTB refers extended tight binding, and 2 refers to the version. In the GFN2-xTB method, the total energy expression is given by Bannwarth et al. [221 2019)
ETB XTB
Frep + Edisop FEIT + FIES+IX + EAES + Exxc + Gremi
(18)
where Erep is the repulsive energy contribution from short-range interactions, Edisp is the disper- sion energy contribution from long-range interactions, ΕETT is the energy contribution from the extended Hückel theory (EHT), EIES+IXC is the isotropic electrostatic (IES) energy contribution and the isotropic exchange-correlatee (AES) energy contribution, Eaxc is the anisotropic exchange-corrclation (AXC) energy contribu- tion, and Geemi is the entropic contribution of an electronic free energy at finite electronic tempera- ture Ta due to Fermi smearing.
(IXC) energy contribution, ÉAES the anisotropic electrostatic
Its accuracy and efficiency come strictly from the element-specife and global parameters for all ele- ments up to radon (Z. 86) (Bannwarth al. (2019), hence the semi-empiricism. The pre-computed hight-binding parameters and empirical corrections are utilized to approximate the electronic struc- ture and calculate energy contributions efficiently.
E MOTIVATION FOR STABILITY OPTIMIZATION
The geometries of the generated molecules have domain-specific implications molecule's sta- bility and properties depend significantly its preferred quantum geometric states, i.e., atomic and molecular geometries. For example, polarity is closely related to molecular geometry. A wa- ter molecule H₂O has stable V-shaped H-Q-H geometry of 104.5 degrees and is thus polar. A generated H₃O molecule with linear H-O-H geometry would instead be nonpolar but unstable. Therefore, when we discuss molecules and their properties, it is essential to start from their pre- ferred stable geometries (ground states), which motivates our work to improve the stability of the generated molecular geometries the diffusion process.
F MOTIVATION FOR GRADIENT ESTIMATION WITH SPSA
To achieve equivariance to transformations (Eq. the 3D diffusion models must satisfy the zero center gravity requirement that Zx,1 N33 should have zero-mean over N atoms on the x, y, axis. To derive guidance, one can estimate F(z)) in the gradient using finite-different approxi- mation (from Eq. 14p as:
F([Ex, + (1Nx3, Fh.)))
F([Ex.1 - -(1--11140 SINxx, Fn.e))
V2, s lim
(19)
5-0
2
which however violates zero center gravity as the perturbed variable 2x,1 Ç1Nx3 is shifted by NxINx3 over the zero-mean of 2x,1.
To obey the zero-mean requirements, one can employ SPSA (Simultancous Perturbation Stochastic Approximation) (Spall 1992) by perturbing 2x,: with a zero-mean random variable A, as follows.
F([Ex,1 + (A,, =n.)) F([Ex. <Ae, #h.e))
lim 5-0
(20)
2ÇA.
where abuse the notation by using dividing A denote the element- operation. Spall (1992) assumes zx,t to have bounded inverse moments, which excludes A, being Gaussian. However, the Gaussian perturbation [=x,: + (U, 2h.]] with E Nx3 ~ (O, 1) would be favored, since it is one of the commonly used distributions that obey zero central gravity with E[(U) = while only introduce small computational overhead compared to diffe-ddiferrnt methods, as sampling from (O, 1) and matrix multiplication are fast in modern machine learning practices on GPUs.
To achieve so, one can adopt the random gradient-free oracles (Nsttrovv Spokoiny 2017) to estimate the gradient for differentiable f - R. Given the random vector F. being Gaussian
17

---

Published as a conference paper at ICLR 2025
with correlation operator B (e.g., B I) and j the gradient at x E is estimated as follows.
(a(x) [xx + pm)) - f(x - pm) Bu.
(21)
2/1
which leads to our estimation for the guidance gradient in Eq. 15
G CHEMISTRY REASONING BEHIND BILEVEL PROPERTY OPTIMIZATION
The task of generating molecules with target properties is closely related to the field of rational chemical design such as drug design. Domain knowledge widely used guide the design process. For example, benzene rings that contain oxygen (e.g., dioxins) and sulfur (c.g.. thiophenes) are typically known to be toxic and carcinogenic. Therefore, when designing drugs, typical to stay away from such species prior to performing further property analysis such as bind-affinity to specific protein DNA. In such design processes, the overall structures are considered by experts before further target property analysis.
The conflict behind generating stable molecules with target properties is that properties are closely related to both atom types h and atom coordinates x. We are mostly concerned with the properties of a stable molecule with an optimized geometry but is not known a priori given the atom types. In the diffusion process to generate molecules, the atom types generated and stabilized during earlier steps, and their coordinates are further optimized in later steps. Since our target to generate stable molecules with desired properties, there is a conflict between stability-fin firsstt and property first in the generation process.
The bilevel property and stability optimization proposed here takes inspiration from the rational design process: higher-level overall structure before lower-level property detail analysis. An overall molecular structure generated with desired properties in earlier steps, where the diffusion model is considered as chemistry/biology/eec experts performing the initial structure-property analysis. In the later steps, the geometries are further optimized.
H ANALYSES
H.1 ACCELERATION METHODS FOR CHEMGUIDE
GPUs can significantly accelerate computation for neural network inference and training, but xTB operates on the CPU, which lowers the running speed of the inference. Hence, we propose skip- step acceleration method for xTB- xTB-guided optimization. Specifically, suppose set the skip-step to be k. then we only calculate gradients from xTB for every & step. We use the best scale (=0.0001) found in Table [ for the skip-step acceleration experiment, and the results shown in Table 7 As we skip more steps. the performance becomes worse and approaches unconditional GeoLDM without any guidance. Hence, there trade-off between time and performance: the more guidance from x xTB we add to the generation, the better performance we can end with. This also demonstrates the effectiveness and necessity of the xTB guidance: more guidance almost monotonically enhances performance.
Table 7: Metrics of 500 generated molecules from QM9 using GeoLDM with non-differentiable oracle (i.e. xTB) guidance. and bold denote the overall best result and our best result, respectively. Percentage changes between our results and GeoLDM are shown in parentheses.
Every step (s)
Baseline
Mctric
3
5
EDM
GeoLDM
Force RMS (Eh/Bohr)
0.0104 (-6.76%))
0.0106
0.0011
0.01
0.0111
Validity
91.40% (1.60%f)
91.40%
89.40%
86.60%
89.80%
Uniqueness
100.00 %
99.79%
100.00%
100.00%
100.00%
Atom Stability
99.02%
99.01 %
98.63%
98.53%
98.93%
Molecule Stability
90.60%
90.60%
87.20%
83.60%
88.80%
Energy above ground state (Eh)
0.0042 (-15.78%()
0.0045
0.0055
0.0072
0.0050
18

---

Published as a conference paper at ICLR 2025
Table 8: Metrics of 500 generated molecules from QM9 using GeoLDM with non-differentiable oracle (i.e. xTB) guidance with various guidance steps (100, 200, 300, 400) and guidance scales (0.0001, 0.001, 0.01, 0.1, 1.0). * and bold denote the overall best result and our best result, respec-
tively. Percentage changes between our results and GeoLDM are shown in parentheses.
Marri
Chidanse (oaee100 Guidanee Stepe
Basolinc
0.000]
0,00]
0,01
0.1
10
EDM
GeoLDDJ
RMS (El/Rube)
0.0110 (-03866))
0.0110 (-08866))
0.01
0.013
0.01.0
0.0114
0.0111
Validiy
90.40%
S0L40'
90.40%
90,80%
(1
90.60S
96,609
59,505
Uniquencse
100,00 %2
100,08
100.00%
100,09%
100,00%
100,00%
10IL009
Stability
98.941% (0.015
(00011113
98.915
10.0USE
Vellocele Stability
89 40%-
89 40.
89.40%
R9.00%
(0.8081)
89.005 (0. (0.80R1)
83,000
58, 50S
Energy showe ground
0.0081
8,0051' (0.8991))
0,005
0.0085
0,0057
0.0072
0.0050
Metric
Cuickance Stale (200 Gridunee Stepei
Reseline
0,0000
0,003
0,01
0.1
1.0
EDM
GeoLD5I
(888)
6.0114
Vilidity
90.0005
91.00%
(1.33 (1.20981)
90.00%
90.60'S
90.606
86, 60.6
50.50 505
Uniqueness
100.09%*
100.01%
100,00%
100.08%
100,005
100,00%
100,00%
Alsmn Stabiiity
98.87%
98.72%
98.37%
98.90%
(Q00038L)
95.86%
59.53%
98.93%
Molecnle Stablitt
39.00%
88.40%
10.00%
89, 19.,0% (0L600E 13
39.00%
83,00%6
58, 5016
Energy shove ssound
0.0054
0,0045"
(-9.32%))
0.0054
0,0049
0,0050
DL0050
Matric
Guidance Scale (300 Guiddnce Steps)
Baseline
0.0001
0,001
0.01
0.1
1.0
EDM
GeoLDMI
KMS
0.0101
0.0109
0.0107 (.98%L))
0,0105
S00111
Vilidity
92,99/% (3.000%))
92.891%
(3 13
92.605
92 92.20'6
88.808
86.609
59 30.9
Uaiqueness
99.57%
100.0011"
.9.5%%
99.36%
100,0015
100,009
100,00055
Alum Stability
98-9316
99.09%
(0:15599))
99.00%
98.99%
08.88%
58.5
Moleccle Stablitty
89 8008
(2,00013
89 60%
89 606
86.00%
63,004
58 601%
Enery shove yound
0.0049
0,0052
0,0045
0,0046
0.0086
Maric
Guilance Scale (400 Guidancce Steps)
Baseline
0,0001
0.001
0.01
0.1
1.0
EDM
Geal FM
Foece RMS (EbrBoba)
0.0104" (-6.766))
0.0104
0.0107
0.0105
0.0125
0.0111
Vilidit
01.40%5
(1.60%11)
91 117099
91 709
90.00%
89.40%
86, 6046
50 5054
Uniqueazs
100.08%
100,009
100.00%%
100,0095
100,0052
Atnm Stahility
99 02%
98.95%
99%
99.03%
vOL 10R 1)
98.67%
98.511
58 9911
Molecule Stablitt
90.60%
90.40%
90.20%
91.00% (2.209E
87.80%
83,001
53,5016
Easrgy
Ground State (Eb)
0,09422 (-15.7898
0,0042
0.0045
0.0050
0.0050
H.2 EFFECT OF GUIDANCE STEPS
As mentioned in Appendix B. adding guidance to the last 400 of the 1000 diffusion steps is time- consuming, so explore relaxations in this section, where we add guidance to later steps such as the last 300, 200, and 100 steps. The results are presented in Table S for QM9 and Table g for GEOM. It is observed that more guidance steps yield better results in terms of force RMS and energy above ground state while the effect on validity and stability does not seem to be related to the number of guidance steps. This is reasonable because we optimizing the force during the diffusion process, so adding more guidance steps provides more guidance towards lower force and smaller energy above ground state.
H.3 CHEMGUIDE VS. NEURAL GUIDANCE ON FORCE
In this section, replace the non-differentiable oracle (i.e. xTB) guidance with neural net- work that predicts energy and force. We employ a machine learning interatomic potential (MLIP), AIMNet2 (Anstine et al. 2024), a recent neural network model trained on 20 million hybrid quan- tum chemical calculations to efficiently predict energy, force, and properties. By combining ML- parameterized short-range and physics-based long-range terms, AIMNet2 achieves better general- ization across diverse molecules. We sample 500 molecules from QM9 and 50 molecules from GEOM, and present the results in Table TO for QM9 and Table IT] for GEOM, for reference and comparison, we also present the metrics of CHEMGUIDE with the best scales.
Using AIMNet2 as guidance doesn't 't improve over xTB terms of time, because it introduces a prohibitive memory constraint that makes sampling almost linear. Recall from Section 2] the molecule position is represented as [N, 3): thus, the force is [N,3), and the derivative (i.e. Hessian) of the force on the position is [N, 3, 3). which is required when calculating the guidance gradient via backpropagation. It requires a large amount of GPU memory and is problematic when sampling large molecules (e.g., GEOM). In CHEMGUIDE, only force is required to estimate gradient, and 100 molecules are sampled at a time on a 48GiB GPU, however, with AIMNet2 guidance only molecules can be sampled at time on the same machine. This is because calculating Hessian scales up the memory by N x 3, which is a major drawback of using neural networks on force guidance as it requires more memory.
19

---

Published as a conference paper at ICLR 2025
Table 9: Metrics of 500 generated molecules from GEOM using GeoLDM with non-differentiable oracle (i.e. xTB) guidance with various guidance steps (100, 200, 300, 400) and guidance scales (0.0001, 0.001, 0.01, 0.1, 1.0). * and bold denote the overall best result and our best result, respec-
tively. Percentage changes between our results and GeoLDM are shown in parentheses.
Guidance Scale /000 Cuidamce Stepe)
Baseline
Manic
0,0011
0:001
0.01
1,0
EDM
GcoLDM
Force RMS (BlrBokr)
8,034
0,0434
0.04332
0JB430
0,0414
(-13.368))
0.0742
0.0478
Validity
49.20%
49,40%
50.00% (1 (1,00961)
48.00%
46.40%
49.00%
Unigueness
100.00%
100.00%
100,00%%
100.00%
100.00%
100L009
100.009
Atomn Stability
54.70%
$4.70%
17.17)
84.67%
84.67%
$1.225
84.53%
Enegy ahove Ground State (bbb)
0.2043
0,21139
0.20311
(-9.66%))
0.2049
0.2033
0.3742
0.2248
Guiddance
Stepe)
Roseline
0.0001
0
0.01
0.1
1.0
EDM
GeoLDM
RMS (Ci/Bohrr
0,0433
13.1))))
0.0477
0.0042
0.0478
Validity
48.801
48.40%
501,00%
47. 8058
51.00%
12,00961)
46.4015
49,0075
Uniqueness
100.00%
100,00%
100.00%*
100,00%
100,0015
100.00%5
Atom Stability
$4,31%
N4.285
84.28%
84.45% (-0.08961)
X3.9195
S1.22%6
84,53%
Energy above Ground State (Ch)
0,2183
0,2254
(4.72%%
0.3942
0.2248
Guidance Scall (300 Cuiddance Stepe)
Baseline
Matrcc
0.0001
0.001
0:31
FDM
Gool DM
Force RMS (BiBoorr)
0.0462
0.047)
0,0468
0,0440 (-7 (-.96%
0.0478
0.0742
0.0478
Validity
47.20%
48.50%
48.50%
$1.80%* (2 (088))
50,00%
46.90%
49,00%
Unigaeness
100.00%
100.00%
100.00%
100.00%
100.00%
1000009
100,0011
Atom Stability
53.73%
83.59%
83.75%
83.80%
(.0.3%%
83.38%
$1.22%
84.53%
Energy above Ground State (b2))
0.2190
0,2248
0,2223
0,2071 (37.89%))
0,2520
0.3742
Metric
Guiddnee Scale (400 Guidanee Stepe)
Beseline
00001
0J001
EDM
GeoLDM
Force RMS (Ua//ehr))
0.0165
0.0:447
0,0434
0.0411 (-14,16%
0.0513
0.0742
0.0478
Validity
47.00%
45.209%
51.60% 1.600)))
49,40%
46 40%
49.00%
Uniqueness
100.00%
100.00%
100.00%
100.00%
100.00%
100.00%
100.00%
Atom Stahility
84.47%
84,49%
84.65% 12561)
83.45%
$1.225
84.53%
Energy above Ground State (Ch)
0.2144
0,2085
0,2104
0,1935
(-13.92%
0.2650
0.3742
0.2248
For QM9, we sample 500 molecules, and using AIMNer2 as guidance yields much worse results in all metrics than CHEMGUIDE. For GEOM, we sample 50 molecules due to the memory constraint, and AIMNet2 achieves compatible results as CHEMGUIDE for guidance, where the force RMS, validity, and stability are close.
However, on GEOM, CHEMMUIDE achieves much lower energy above ground state compared to AIMNet2 guidance, indicating that CHEMGUIDE generated molecules closer to their corresponding ground states. Recall that in the energy curve of a molecule, the x-axis and the y-value refer to the molecule configuration and its energy, where the derivative of energy gives the force, and we aim to optimize the molecule towards its ground state (i.e. the global minimum of the energy curve). Hence, although AIMNet2 guidance reduces the force similarly to CHEMGUIDE, it is farther from the global energy minimum, which demonstrates that guidance directly from quantum chemistry provides more intrinsic optimization than neural networks. Additionally, we sample 500 molecules in Table 2 while we only sample 50 molecules for AIMNet, which might not be comparable as smaller populations are more likely to be biased.
To better understand the difference, we plot the co- sine similarities of the gradients from AIMNet2 and CHEMGUIdE over 400 guidance steps on 10 molecules
mean
mesn std
from QM9, as shown in Fig The results show that the cosine similarities are small and concentrated around zero, suggesting that the gradients from AIMNet2 and CHEMGUIDE are nearly orthogonal. This observation helps explain the significant performance differences re- ported in Table TO CHEMGUIDE achieves much bet- ter results than AIMNet2-guided optimization on QM9 because CHEMGUIDE provides more effective and sta- ble guidance, whereas the guidance from AIMNet2 neg- atively impacts stability.
coshee dellamy
0.0
-0.1
-0.2
Figure 6: Cosine similarity between the guidance gradients of AIMNet and CHEMGUIDE of 10 molecules sampled on QM9 for the last 400 steps.
20

---

Published as a conference paper at ICLR 2025
Table 10: Metrics of 500 generated molecules from QM9 using GeoLDM with neural network guidance (i.e. AIMNet2) on force. and bold denote the overall best result and our best result, respectively. Percentage changes between our results and GeoLDM are shown in parentheses. For ease of comparison, we report the results using the best scale of CHEMGUIDE from Table |
Cuidarce Scale
CHEMGRIDE
Bascline
Metric
001
BU0001
EDM
GeoLDM
Force RMS (Eh/Botr)
0.0113
0.0113
0.0112
0.0114
O.OLIO L.00SL)
0.0004' (6.70%)
0.0114
0,0111
Vilidit
59,209 (-0.60921)
89 20% (-0,60%L))
89.0375
88.605
$9,00%
91.40%
(1.6015)
86.60%
89,80%
Uniqueness
100,00%
Alozs Siubility
98.72'6
98.72%
95.71%
98.74-5
98.75% (40. 181. 11
99.02%
(0.095))
98.5568
98.935
Moleccule Subiiity
87.8015
$7.80%
87.60%
87.80%
88.00% (.0,006..)
99.0015 (1.8015))
83.60%
85,80%
Enegy alove giana viatc (E3l)
0U0055
0.0057
0.0056
0,0099
0.0055 (9.30 (0.30))))
0,0042 (-15.78%.
0.0072
0,0000
Table 11: Metrics of 50 generated molecules from GEOM using GeoLDM with neural network guidance (i.e. AlMNet2) on force. * and bold denote the overall best result and our best result, respectively. Percentage changes between our results and GeoLDM are shown in parentheses. For ease of comparison, we report the results using the best scale of CHEMGUIDE from Table
Meuicc
Guadance Scale
CHRNGouee
Bascline
0:000
00001
0.01
0.1
1.0
scale =01
EDM
GeoLDM
(ElVBolrr
033488
0.0188
0UD418
0.0411
9:411 ()
DA742
U.047K
Vilidity (xTR)
50.00% (1.00%f)
50,005
50,00%
50,00%
50,009
50.40% (1AR)))
46.4058
40,00%
Uniqueness
100.00%
100,00%
100.00%
100.015
100,00%
100,00%
100,002
Atoo Stahility
85.01%
85.01%
85.01%
85.15%
85.26% (0.7398))
84. 14.3 (40 (-0117%))
81.2258
$4.53
Energy shove
0,22
012392
012392
0.23%6
(2.2098))
0,1935' (-13999)))))
0.2248
H.4 GENERALIZATION ANALYSIS OF NEURAL REGRESSORS
As observed in Table 16 Appendix [ and Table 18 in Appendix M our results are not always stable and some- times collapse (see the collapse analysis Section H. In this section, work used for gradient calculation, with the results shown in Fig. 7 We sampled 1,000 molecules from QM9 and added Gaussian noise with varying variances (Ie-4, 1e-3,
200
alpha
XTB
provide amalysis of the neural net-
400
mu
200
Percentage Change
1e-2, 1.0, 2.0, 3.0, 4.0) to their positions. We then eval- uated the neural regressor and xTB, calculating the per- centage change predicted values from those predicted when no noise was added. For visualization purposes, we clipped the maximum percentage change be less than 500%. The line represents the mean of the percent- age changes, while the shaded area (magnitude) repre- sents the range from maximum to minimum. As the noise scale increases, the regressors become increasingly unsta- ble, with percentage changes reaching as high 100%. This instability can explain the fluctuating MAE in our results, as ation process, where the molecules are noisier and less optimized than those in the QM9 dataset. Consequently, the regressor may provide suboptimal or even incorrect guidance during the early stages of the generation.
0
100
homo
400
lumo
200
100
gap
the neural regressor during the gener-
-100
1e-4 1e:3 1e.2 0.1
2
Noise Scale
Figure 7: a, Hr, CHOMO, ELUMO, A. predicted by the neural regressor when adding various noise.
Percentage change of
By comparing the performance of the regressor and xTB, is clear that xTB provides more stable predictions, as both the mean and the magnitude of instability remain lower than those of the regressor. This helps explain the performance differences in Table 4 and Table 6 where CHEMGUIDE performs worse than the regressor produces more stable results in a. However, CHEMGUIDE performs better in Ae because the regressor becomes more stable at larger noise scales, and xTB, average, begins to "prune" invalid molecules when the noise scale is large, ceasing to provide guidance (i.c., gradients) to these invalid molecules.
21

---

Published as a conference paper at ICLR 2025
This behavior is evident from the "peak" in xTB prediction magnitude around noise scale 1.0, after which the magnitude quickly decays and converges to the mean, while the magnitude of the regressor's performance continues to increase. This trend is more obvious in Σc, where a -100% change is observed, indicating that xTB either throws an error on invalid molecules, in which case the gradient is O, or returns negligible values for distorted molecules, and its guidance diminishes and stops further changes to molecular structures.
This validates our motivation for using a non-differentiable chemistry oracle to guide generation and optimization. A chemistry oracle, based on deterministic chemistry-rule calculations, can better capture the properties of molecules, even for previously unseen ones, and will not return high scores on invalid molecules, which further distort the molecular structures.
H.5 GENERALIZATION ANALYSIS OF CONDITIONAL. GEOLDM
As discussed in the introduction (Section D. explicit guidance re- quires training a conditional model with labels; thus, the model is likely to be not robust and does not generalize well to distribution shifts. It performs poorly when conditioned on property scores that were not encountered during training. To demonstrate this, we per- turb the sampled context from the property distribution on which C-GeoLDDM is conditioned and generate 200 molecules. Since the sampled property scores are standardized with z-score normaliza- tion, we shift the context by 22, #3 (which means we look at the +1 +2 σ, and +3 regions of the property score distribu- tion). We then plot the percentage change in MAE compared to the case with no distribution shift (i.e., perturb scale=0) in Fig. 8 For percentage changes greater than 1000%, we clip the values (for ex- ample, in the case of μr, the percentage change exceeds 1000%, it is not fully displayed). The percentage change in MAE can be as high as 50% even within the standard deviation of +1 ar, and in- creases significantly when the distribution is shifted by =20., which suggests C-GeoLDM is prone to be affected by distribution shifts; thus, unlikely generalize well when conditioned on scores beyond the property distribution seen during training.
1000
alpha
Percentage Change (9)
mu
800
Cv
homo
lumo
gap
400-
200
Perturb Scales
Figure 8: Percentage changes of MAE 200 generated
molecules using C-GeoLDM
with various perturbed scales.
H.6 ANALYSIS OF GRADIENT COLLAPSE
It's observed in Table. 16 Appendix D that some gradi- ents of noisy neural guidance and clean neural guidance collapse for some scales. For example, scale 5, 20, 25, 30 when optimizing a, so we explain why such collapses happen in this section via case study of optimizing a with scale 10 (optimal scale with no collapse), 30, 40, and
14000
Optimal
scale=30
12000
scale-40
scale=30
MAE of alpha
30000
C GooLDM
8000
6000
50. As we increase the scale, the MAE grows quickly in the first 50 steps and the molecules collapse because the added gradient is so high that it pulls the atom apart and distorts the molecule structure. Hence, the guidance scale for each property must be carefully searched and selected so that it provides strong enough guidance while main- taining a valid molecule structure.
4000
2000
50
100
150
200
250
300
350
400
Guidance Stegs
Figure 9: Gradient collapse of a using GeoLDM with noisy neural guidance.
H.7 MORE CLEAN GUIDANCE OPTIMIZATION STEPS
Recall that the clean guidance is provided in Eq 10 where we use steps of gradient descent (GD). In this section, experiment with more GD steps. For each property, we select the scale that results in the lowest MAE without collapse, as shown Tablc For example, we choose a scale of 0.1 for The results are presented in Fig. T0
To investigate the effect of additional recurrent steps and to prevent a single collapse from stopping the entire generation process, divide the molecules into groups of 50 and calculate the MAE on group-wise basis. In the figure, "High" indicates that the MAE is too large to be displayed properly, and circle (o) means that the MAE is either too low or too high; thus, it is classified as
22

---

Published as a conference paper at ICLR 2025
outlier. For instance, in the case of μ., most MAEs are high when GD step=5, except for one group with an MAE of 0.12. In terms of the lowest MAE a group can achieve, using more GD steps tends to decrease the MAE, as observed for and Ac, where the lowest MAE is reduced by approximately 30% for a and 55% for (even though the lowest MAE for μ4 is already small, can still be significantly reduced from 0.20 with GD) step=1 to 0.09 with GD steps=3). However, in terms of average MAE, since clean guidance inherently has higher instability during generation (with details and analysis of this instability available in Fig Section D adding more GD steps amplifies this effect. This is most evident in the case of C.. where all groups with GD steps=3 and 10 collapses, and the MAE becomes exceptionally high when GD=5. Moreover, adding more steps does not guarantee a monotonically better MAE, as seen in MπиMO, where 3 steps yield the optimal result while steps=5 increase the variance and mean. Therefore, selecting the number of GD steps and guidance scales carefully for each property is essential, which we leave for future work.
6.0
alpha
High
High
High
High
MAE of alpha
MAE of mu
2.5
Collapse
Collapse
1.5
Outlier
0.5
Gradient Descent Stepís)
Gradient Descent Step(s)
Gradient Descent Stepls)
homo
lumo
gap
0.5
MAE of homo
1.25
2.0
MAE of gap
1.00
Outlier
Outlier
1.5
0.75
0.50
0.2
0.1
Descent Stepís)
Gradient Descent Stepls)
Gradient Descent Stepts)
Figure 10: Box plots of MAEs of 500 generated molecules using clean guidance with more GD steps to optimize all 6 properties. The first, second (median), and third quantiles are plotted.
OTHER OPTIMIZATION ALGORITHMS
In this section, we compare our proposed CHEMMUIDE with alternative class of gradient-free methods that incorporate non-differentiable oracles into the diffusion process: evolutionary algo- rithms (Schneuing al. 2022 Huang al. 2024). The evolutionary algorithm, detailed in Algo- rithm has two main parameters: the variant size (k) and the evolution interval (E). During the diffusion process, the population preserved and evolution is performed fixed intervals. At each evolution step, le 1 noise perturbations are added to the population, resulting variants for each molecule, where the unperturbed molecule is treated the new population based on evaluations from the non-differentiable oracle (i.e. xTB). Specifically, the oracle calculates the force RMS, and the variant with the lowest force RMS is selected.
variant. The best variant then selected as
We explored several combinations of variant size and evolution interval on QM9, and the results are summarized in Table 12 For ease of comparison. the results using the best scales CHEMGUIDE are also provided in Table We can observe CHEMGUIDE is on par with the evolutionary algorithm in validity and stability, and significantly outperforms it in terms of force RMS and energy above the ground state. This advantage can be attributed to the fact that gradients provide informative guidance by indicating the correct direction for optimizing stability. In contrast, the evolutionary algorithm operates without relying on gradients, which results in lack of directional guidance. Consequently, our approach enables controllable optimization with this gradient- gradient-based guidance. While the evolutionary algorithm selects the best variant each evolutionary step, its optimization direction is essentially random, making the process less controllable. Additionally, the gradient in- herently adjusts the direction of the generative process, whereas the evolutionary algorithm does
23

---

Published as a conference paper at ICLR 2025
Algorithm 2 CHEMGUIDE diffusion sampling with evolutionary algorithm
Input: a latent diffusion model εø, VAE decoder D, a composition function F, target property score y, variant size k, variant scale 8., evolution interval E. Output: optimized molecule [x. h]
2T (0,1))
population < [=c]
for all from 7 to 1 do
I/ Add variants
if %E = then
population.extend((z + 8y E1, 2t 8y .2. , 8r -E--I)) where e.,.. , , I),
end if
for all Zt,i population do
14-18 : (2 VI-o? Vi co(z,, 0f
Be
(Eq. 3 (Eq.
21-11 + (N E1-1,i
Replace ž₁,i in population by 1,- end for _ t arg = 11]
// Select the best variant
population [27-1] end for
x. h] D(2)) return [x, h]
Table 12: Metrics of 500 generated molecules from QM9 using GeoLDM with evolutionary algo-
rithm (Algorithm 2 k; E denote the variant size and evolution interval. and bold denote the
overall best result and best result, respectively. Percentage changes between our results and Ge-
oLDM are shown in parentheses. For ease of comparison, we report the results using the best scale of CHEMGUIDE from Table
Metric
(voriint sine, evolution interval)
CHEMGUIDE
Boseline
3,E.
200
(1-ma
50)
5,
20)
(k- A. 50)
scale - 0,001
FDM
Force RMS (LbiBehr
0,0112
0.0111
0.0100
0,0109 (-1.7598))
0.0104" (16.7618
00114
0.0111
Validity
91.60%
87 80.
90.00%
(2.60% () 92.40%
91.40% (1.60%))
$6.50%
89.80%
Uniquesess
100.00%
100.00%
99.78%
100.00%
100.00%
100,0015
100,00%
Atum Stabilitt
98.95%
98.75%
98.66%
99.14%
(0.215% ()
99 02% (0.09%)
98.53%
98.93%
Molecule Siability
90.00%
$7.60%
38.00%
91.201
(2.40%81)
90.60% (1.80%)1)
83.60%
88, 80%8
Enegy ahove ground state (EN)
0UD0R8 (-4 (-4-4598L)
0.0053
0.0056
0L0054
0.0042" (-15.78%6.)
0.0072
0.0050
dynamically adapt its direction. Therefore, our CHEMGUIDE offers more generic and inherently guided optimization process.
J BILEVEL OPTIMIZATION WITH CLEAN GUIDANCE
As shown Section 4.3.2 our clean guidance method can achieve significantly lower MAE than the baseline. However, its ability to drastically lower MAE also contributes unstable molecular geometries. To illustrate this, we pick two properties as a case study from Table which are easiest (n) and hardest (a) optimize using clean guidance, and the results shown Table [3 Com- paring the MAE and force RMS across two experiments in each property optimization, we observe that the bilevel optimization framework helps reduce force RMS and MAE the same time. indicat- ing that optimizing force/property yields better results on the other. However, we can observe that the force RMS of bilevel optimization is still higher than the baseline (right block of Table 13] with 200 molecules generated), but this is because the force RMS of clean guidance is much higher than bilevel the baseline (left block of Table [3 with 500 molecules generated); thus, CHEMMUIDE optimization can't 't minimize the force significantly to beat the baseline, which is the natural hard- ness of optimizing force after clean guidance as the clean guidance so strong that it significantly changes the molecular structures, making it challenging for xTB to optimize. We notice trade-off between force and property optimization when comparing the performance across a and μ in bilevel optimization: lower force yields higher MAE while lower MAE brings higher force. Hence, given
24

---

Published as a conference paper at ICLR 2025
2.000 0.004
6.012
0.00E
0.004
0.065
2.08
0.22
0.20
1.300
6,005
0.2034 a 8.000
5.002
0.822 9.0500
0.44
0.12
124
6.36
0.41
0.6
70-
120
scale-2.00 022
scalem2.
40
10
6510 EIGA REMEE 03146 0.000 8120
0446
3540
dess 0.14
1297 8343
012
1.28
(a) QM9 Force RMS
(b) GEOM Force RMS
(c) QM9 Energy Change
(d) GEOM Energy Change
Figure 11: Histograms and distributions of force RMS and energy above the ground state of 500 generated molecules from QM9 dataset using unconditional GeoLDM with CHEMGUIDE using various scales. Energy change refers to energy above ground state.
the difficulty of optimizing force and MAE at the same time to beat the baseline, it's important and tricky to pick the best scales or schedules for force and property optimization, which we leave to future work. Similarly, we report full numerical results of the case study a and μ optimization to Appendix M
Table 20
Table 13: Comparisons of force RMS and MAE of generated molecules from QM9 using baselines, unconditional GeoLDM with clean neural guidance, bilevel optimization with clean neural guidance.
Clean Neural Guidance
Bilevel w/ Clean Neural Guidance
Models
(Bohr")
je (D)
(Bohr")
(D)
Force RMS
Force RMS
Force RMS (Ch/Bohr)
Force RMS
(Eh/Bohr)
MAE
(Eh/Bohr)
MAR
MAE
(Gh/Bohrr)
MAE
C-EDM
0.0118
2.6308
0.0122
1.1257
0.0125
2,5089
0.0123
1.0571
Basclines
C-GeoLDM
0,0117
2.5551
0.0120
1.1084
0.0135
2.5593
0,0123
1.1582
Ours
0.0326
4.9992
0.3962
0,4025
0.0274
4.2058
0.1348
0.3150
K
COMPREHENSIVE RESULTS ON DON-DIFFERENTIABLE ORACLE GUIDANCE
We report the distributions force RMS and energy above ground state 500 generated molecules from QM9 dataset and GEOM dataset using unconditional GeoLDM with CHEMGUIDE using var- ious scales (0.0001, 0.001, 0.01, 0.1, and 1.0) in Fig. II numerical results are shown in Table Tand Table in Section 4.2 We observe that smaller scales produce better results while larger scales (i.e. 1.0) produce worse results than the baselines. This is because the position values of the molecules are small (less than 1.0); thus. when the scale of the gradients is similar to position values, adding too much guidance results in "over-optimizing the position and distorting the molecule structures. This can be validated in Fig. [T] the scale increases, the molecules become less concentrated around O, spreading out and exhibiting higher energy above ground states and force RMS. The distribution shifts downward. indicating lower validity, and its peak moves away from O, suggesting reduced stability. The performance, worse than the baselines, implies that the molecular structures are being distorted due to excessive guidance. The results for GeoLDM with CHEMGUIDE on properties are presented in Table 14
L COMPREHENSIVE RESULTS ON NEURAL GUIDANCE
We present full numeric results of unconditional GeoLDM using noisy and clean neural guidance with various scales Table 16 is observed that larger scales yield better results while smaller scales produce results closer to unconditional GeoLDM. This is intuitive because larger scales pro- vide more guidance to the generation, which brings better properties; on the other hand, since we using unconditional GeoLDM the backbone, the guidance becomes more negligible when the scale is small and thus the performance converges to unconditional GeoLDM. X in Table I6 means that the molecules collapse in the middle of the generation, there Explanation and case study of collapse can be found in Section H.6 and Section H.4 To better in- vestigate the effect of larger scales, split 500 generated molecules into 10 groups that single
output molecules.
25

---

Published as a conference paper at ICLR 2025
Table 14: MAE of 200 generated molecules on QM9 using GeoLDM with CHEMGUIDE on all six properties, with GeoLDM as the baseline. * and bold denote the overall best and the best within different scales. Percentage changes between our results and GeoLDM are shown parentheses.
Property
J
<HOMO
LUMO
A
Units
Bohr²
meV
meV
meV
GcoLDM
4.7555
1,5490
0.62644
2.1790
2.1568
2.00911
scale 1.0 10"
4.9045
1,5335
0.7228
(-7.80%
1.9321
1.9318
scale 1.0 10-5
4.8790
1,5311
0.7182
2.0183
(-10.43%
GeoLDM w/ CHEMGUIDE
4.8781
1,5295
scale 1.0 10-2
(2.58% f)
(- (-1.26%%)
0.7449 (150)
2.0548
1,9428
scale 1.0
4,9025
1,5979
0.6961
2.0136
1.9607
0.6392
scale 1.0 10+0
5.2251
1.631
2.0298
2,0416
(2.04%%)
Table 15: MAE of 200 generated molecules on GEOM using GeoLDM with CHEMGUIdE on all six properties, with GeoLDM the baseline. and bold denote the overall best and the best within different scales. Percentage changes between our results and GeoLDM are shown parentheses.
Property
a
P
FHUMU
LLUMO
Units
Bohr²
D
meV
meV
meV
GcoLDM
24.3424
3.3676
1.1922
5,0903
1.9956
1.2939
5,0802"
scale 1.0 x 10
26.2032
3,3632
1,9381
(8.53%
(-0.20%
scale 1.0 x 10
26.1301
3,3881
1.2974
5.1477
1.9616
GeoL.DM w/ CHEMGUidE
3.3412
1.9264
scale 1.0 x 10-2
25.9853
1,3032
5.2268
(-0.78%.)
(-3.47%.1)
scale 1.0 10-1
25.8031
3,4581
1.3002
5,3704
1.9445
24.2168
scale 1.0 10+0
3.7300
1.2992
5.1828
2.1446
(-0.52%%L
collapse will not stop the generation of all molecules, we put the remaining number of molecules in the parenthesis in Table 16 and report the MAE of the molecules in groups without collapse. For example, 3. 3,2309 (450) in of scale=5 means there are 450 molecules in groups with no collapse and their MAE 3.2309.
We plot the MAE trajectories of optimizing all six properties in Fig. 12 Applying noisy neural guid- ance results in much smoother trajectories compared to the baseline. It reduces MAE significantly during the first 50 epochs and then stabilizes, making noisy guidance most effective in the early stages of generation. Clean neural guidance also lowers MAE in the first 100 steps to some extent, but it fluctuates more than noisy guidance (and sometimes more than the baseline, in the trajec- tory of optimizing (LUM).. However, clean guidance becomes more powerful in the final 50 steps, where observe sharp drop in MAE across all six properties. The difference in smoothness is because, in noisy guidance, the gradient is propagated through the denoising estimation (Eq 8) and the VAE decoder (Algorithn inherently accounting for the effect of noise in the gradient calculation, which leads to consistent guidance throughout the generation steps. On the contrary. clean guidance, the gradient is computed in the clean space (Eq(TT) and then projected back into the generation process by adding noise (Eq IOP. Since the gradient does not propagate through the denoising estimation and VAE decoder, it more sensitive to noise, resulting greater fluctuations and inconsistency during guidance. Noisy guidance stabilizes and doesn't 't exhibit sharp drop in later steps because the loss decreases as the MAE reduces the early stages, so the gradient is small, which mainly serves to maintain the current molecular structure, preventing the MAE from converging to that GeoLDM. This can be observed Table 16 where the MAE for smaller scales is closer that of unconditional GeoLDM. Although clean guidance helps reduce MAE in earlier steps of generation, it reaches its full potential in the later steps, when the molecules are more similar and close decoded molecules, leading sharp drop in MAE.
26

---

Published as a conference paper at ICLR 2025
50320
Gaddanee Clean Guidance
1900
40300
M66 of alpha
020
MM2 of mU
MOC
000
20000
200
10220
50
192
150
230
250
000
750
400
50
150
209
350
50
100
150
250
250
300
350
Guidance
1.75
13
120
IMME of bomoo
MAEE of lume
100
MAE
0.75
100
150
200
250
300
350
400
90
100
150
200
350
300
35%
002
100
350
200
250
900
350
40
Guidance Steps
Guidance Steps
Guidance Steps
Figure 12: MAE trajectories of 500 generated molecules using GeoLDM and GeoLDM with clean&noisy guidance with optimal configuration. We use the scales with no collapse as opti- mal scales; for example, the optimal scale of optimizing is 10.0 instead of 25.0.
M COMPREHENSIVE RESULTS ON BILEVEL OPTIMIZATION
In this section, we present MAE trajectories and detailed results of force metrics and MAE for ex- plicit bilevel optimization and implicit bilevel optimization with noisy guidance, and we provide full numeric results of MAE and force metrics of implicit bilevel optimization with clean guidance. Re- call that we only sample 200 molecules for experiments in this section, as mentioned in Section
Explicit guided diffusion with CHEMGUIDE The MAE and force metrics are shown in Table 18 and Table 17 respectively. Since we only add CHEMGUIDE to C-GeoLDM, our expectation that can achieve lower forces without increasing MAE. However, the results go beyond our expec- tations: our method not only achieves better force RMS, energy above ground state, and validity but also reduces MAE in the first properties. This shows the effectiveness of combining explicit property optimization with CHEMGUIDE, and verifies the feasibility of our bilevel optimization framework.
Implicit guided diffusion with noisy guidance and CHEMGUIDE The MAE and force metrics of bilevel optimization with noisy guidance are shown in Table 18 and Table [9 respectively. For ease of searching scales for noisy guidance and to better explore larger scales, employ ensem- ble method: based on the results from unconditional GeoLDM with noisy guidance in Table [6 we pick the top-4 scales that give the lowest MAE, and then sample 50 molecules using each scale and aggregate the results. For instance, for a, we pick scales of 25.0 (Top 1), 20.0 (Top 2). 10.0 (Top 3), and 5.0 (Top 4). Overall, we achieve better force metrics than the baselines. For each property, there exists at least one scale with large improvement in force metrics; for a, p. and C., we can find scales that produce improvement both force and MAE such scale=20 for a, scale=20.0 for μ. and scale=25 for C., indicating that our bilevel optimization with noisy guidance does succeed in optimizing both property and force. The results for MAE are consistent with GeoLDM with noisy guidance in the sense that top scales in general yield lower MAE and, the same as Table T6] our bilevel optimization with noisy guidance performs better than the baseline a, 14, and C, while worse than the baseline in the other three properties. Moreover, the percentage change of the best scale in both tables are similar yet bilevel optimization with noisy guidance yields slightly worse results than GeoLDM with noisy guidance, which suggests adding guidance for force will slightly contradict the guidance for MAE as the the guidance tries to find a balance between property and stability optimization.
To illustrate how bilevel optimization works, we provide MAE trajectories of each property in Fig. 13 Compared to MAE trajectories in unconditional Ge with noisy guidance in Fig. T2
27

---

Published as a conference paper at ICLR 2025
Table 16: MAE of all six properties of 500 sampled molecules from GeoLDM with noisy and clean neural guidance. We use C-EDM, C-GeoLDM,, and GeoLDM as baselines. * and bold denote the overall best and the best within different scales, respectively. X marks the situation where the guid- ance collapses. Percentage changes between our results and C-GeoLDM are shown in parenthesis
Property
C,
Caxeso
"LUMO
Units
Behr"
meV
meV
meV
Conditioal EDM
2,6308
1.1257
1.0604
0.3207
0.5940
0.6301
Conditional GeoLDM
2.5551
1.1084
0,9703
0.3327
0.5518
0.5878
GeoLiM
5.6732
1.6461
3.1046
0.6151
1.1778
scale 1.0 10"
5.6890
1.4784
3.1046
0.6136
1,1834
1.1725
scale 1.0 10"
5.6203
1.4743
3,1002
0.6155
1.1864
1.1683
scale 1.0 10-2
5.6547
1.4699
3,0923
0,5190
1,1739
1.1770
1.0 x
5.1373
1,5002
2.7785
0,6183
1.1509
1.1319
scale = 1.0 10
4.1372
1.3248
1,7806
0.5965
1.0523
1.1033
scale 2.0 x 10
3.6602
1.2153
1,5683
0.5763
0.9916
1,0504
DM w/ Noioy Guidenee
seale 1.0 10
3,2309 (450)
1.0331
1,2360
0.5644
0,9834
1 1.0362
saaee 1.0 10
2,3870
0.8201
1,0358
0.5667
0,8554
0.9102
2.0
19312 (400)
(350)
0.5504
0.8325
soale 2.5 10
(-40, (824)))
1,5284 (100)*
(360)
0.7112
0,4796
0.7436
0.8596
10
0,4733 (250)
(:57.9988.J)
(-26 (26.889693.)
0,7094 (450)
0,4725
0.8115
0.7468
scale 4.0 10
x
0.4153 (450)
(24 (24-827298-)
0.7496
0,7101 (200)
(30 (008806481)
scale 5.0 10
x
x
0.4325 (2000)
0,7015 (150)
(27. 12949771
scale 1.0 10-
5.7236
1.3925
2,9780
0,6380
1.1587
1 1,2056
soale 1.0 10
5,7373
1.3914
2,9779
0.6374
1.1578
1 1,2055
saale 1.0 10
5.6899
1.4002
2,734
0.6367
1.1558
1.2024
scale 1.0 10-1
5.4798
1.3384
2,9506
0.6343
1.1576
1,2096
scale 1.0 10
2,4960 (00)
(-2.23%%
0.7144
2,7440
0.5203
1.1292
1.1345
scale 2.0 10
7350
0.4455
2,3713
0.6040
1.0853
1 1.0699
GooLDM v/ Clean Guidance
seale 5.0 10
5809.9470 (200)
0.2091
1.6211
0.5533
0,9519
0.8676
scale 1.0 x 10-1
X
0.1631 (100))
(-85.2995
0.9668
0,4817
0.7369
0.6690
scale 2.0 x 10
(100)))
0.5311
0,3740
0.5688
0,4313
scale 2,5 10
2010.2822 (100)
0.4524 (250)
0,3359
0,3580 (300)
0.3833
0,2852 (100)*
seale 3.0 10a1
0.3151
0.3186
0.5747
(-51.48%..
scale 4.0 10-1
X
0.6815 (250)
0,3109
0.5374 (300)
(-2.6199.)
28.5487 (200)
scale 5.0 10
0.2044 (250)*
(-78.9398))
0.2386 (300))
(-26.285)
0,7159
0,4831 (100)
the trajectories of bilevel optimization with noisy guidance behave similarly in the sense that the MAE is reduced significantly during the first 50 guidance steps and stabilizes. The difference is that the trajectories of bilevel optimization with noisy guidance have slightly larger fluctuation dur- ing the early steps of guidance. We reason that in the early steps, the molecules are less stable, so CHEMGUIDE provides larger guidance such that the guidance from CHEMGUIDE and noisy guid- ance interact with each other and result in fluctuated trajectories. The trajectories are also more converged to GeoL.MM because the generation process leans less toward lower MAF, as it keeps balance between property and stability optimization, which can be verified from 19
Implicit guided diffusion with clean guidance Finally, we present the algorithm of bilevel opti- mization with clean guidance and CHEMGUIDE in Algorithm 3 The detailed results of optimizing and i are shown in Table 20 Recall that a is the most challenging to optimize and μ is the easiest according to Table In both properties, it's obvious that increasing guidance scales results in lower MAE yet larger force RMS, which is because the effect of clean guidance exceeds that of CHEMGUIDE, so it's trade-off between optimizing force and property. Comparing the two meth- ods of the two properties, we notice that optimizing a yields better force RMS yet higher MAE while optimizing μ results in worse force RMS yet lower MAE, which is another trade-off between optimizing force and property. In each property, although we can't directly compare the perfor- mance of GeoLDM with clean guidance and bilevel optimization with clean guidance because sample 500 molecules from the former and 200 from the latter, we can compare them with their respective baselines. We notice that by adding CHEMGUIDE on force in bilevel optimization,
28

---

Published as a conference paper at ICLR 2025
Algorithm 3 Bilevel guided diffusion sampling with clean neural guidance
Input: a latent diffusion model eg. VAE decoder D, a composition function F, target property score y, oracle guidance scale 8, SPSA perturbation Ç, number gradient descent steps K. Output: optimized molecule [x, h]
ær (0,
for all from T to 1 do
2g f- 40(22)
Ažn arg min l! (2 + + using K steps of gradient descend
(Eq. 10
20 - 20 + A0
U I0,
-t-1 x -V -VF(E)) (()))) 1|
1 (F([zx.0 + -U, - -[([xx, - SU, En,ol) U
(Eq. 15
26
ē - <Z( t) a
Azo
(Eg.
14-1, -1-1 VI-
(2)
31
o)
(Eq.
V-
2:-1 + N (W4-1 + s 02-19t-11 111
(Eq.
end for
[x, hi + D(z))
return [x. h]
Sileval Opc.
15002
Narey Gultancee
GeoLDM
MAl of elo1o
1030
M64 mm
25000
T0M oi
GCOLDM
20000
000
25000
420
10000
50
150
350
200
250
392
52
150
122
250
150
200
250
300
400
Steps
1.00
1.75
15.0
100
MMEE of homo
MAE of Lume
gep
90
10.0
60
50
100
350
200
250
300
200
400
50
100
150
200
250
330
350
403
50
100
350
200
250
300
350
durdance Steps
Guidance Staps
Guidence Steps
Figure 13: MAE trajectories of 200 generated molecules using GeoLDM and bilevel optimization with noisy guidance GeoLDM with optimal configuration. We use the scales with no collapse as optimal scales.
achieve lower force RMS percentage changes from the baselines. Surprisingly, the MAE of bilevel optimization with clean guidance is lower than that of GeoLDM with clean guidance, which seems conflict with our findings in bilevel optimization with noisy guidance, suspect that this is because, as the molecules are optimized to be more stable and thus more similar to the denoised How-
ones, clean guidance can obtain more power reducing the MAE, as explained in Section ever, there is a collapse in the bilevel optimization of μ4 with scale 10 while it doesn't 't happen in GeoLDM with clean guidance, we suspect that this due the exceptionally high force RMS in GeoLDM with clean guidance, so CHEMGUIDE provides much higher gradient and the guidance from CHEMGUIDE (pushing toward lower force RMS) and clean guidance (pushing towards higher force RMS) conflict, resulting in the collapse of molecular structures. Nevertheless, although the force RMS of bilevel optimization with clean guidance is still higher than the baselines, it is already reduce it adding CHEMGUIDE, which, together
high in GeoLDM with clean guidance and with lower MAE, indicates the feasibility of combining CHEMGUIDE with clean guidance and our success in obtaining both better property and force metrics.
29

---

Published as a conference paper at ICLR 2025
Table 17: Force metrics of 200 generated molecules from QM9 dataset using C-EDM, C- and explicit bilevel optimization. Each sub-table contains results for one property that the model is conditioned From up to bottom: μ,, Cu. EHOMO. MLUMO. Ae. and bold denote the overall best result and our best result, respectively. Percentage changes between our results and GeoLDM shown in parenthesis.
Metric
0,001
01051
0,01
scale
0.1
10
C-EDM
C-GeoLDM
(0)
(FuRaarr
0.0139
(9:3188)
D.0227
Validry
90.00%* (7,005
90,00/
$8,00%
87.50%
89.50%
83.50%
83.00%
Uniqueness
180,00%
100,00%
108,00% +
100,00%
99.47%
100.095
100
Molecale Subiiity
8M.5U%
(8.50%
89.00%
$7.50%
87.50%
87.00%
$2.50%
$1.00%%
(Uh)
0,0083
BU0DY' 13.93%_.
0.0884
Metric
0,0000
0.001
Gusdance scale (1)
0,01
0.1
1.0
C CED
Baseline (1)
C GeeLLD
CAURD
Validay
82.00'8
82.0095
$3.5055
85.00% 1-1 SOR
83.909
83.009:
86 5055
Unigueness
160,00%
100,005
100.00%
100,00
100,00% 5
100.0015
100,009
Alom Subiiity
98.185
98.325
(0.099
97.48%
Motecule Subility
80.00S
$0,00%
$1,5016
82.00%
2.00%)))
77.50%
79.00%
84 D01S
0.0106
(16.72911)
0,111
0L0CK11
Metric
Gustance scale (C,)
Buseline (Cr)
0,001
0.011
C-UDM
C C-GeoLLOM
[FaiReer)
0.01
(8.755LJ)
0.012
0.0117
Validry
85.00%
$6,00%
$6,5015
88.50%
(0.5056
$7.00%
75.80%
88.00%
Unigueness
100,00%
100009
B0BL0012
100.00%
100,00%
100,005
30000005
Nom Stabiliry
98.085
58.835
98.5716
98.68%
(0.11169)
98.57%
97.43%
98.57%
Melecale Stubility
N5.00%
(-1.00094))
$5.00%
$5.505
H4.30FE
$4.00%
74.50%
$6.00E
ground
Metric
Guidanoe scale (CHooarr)
Baseline Cerooaac
0,0001
C.GooDMM
Force RMS (LarBBor))
0,9106
(..999)))
0.106
OUDTOS
0,0112
0,0125
0.01I8
0.0113
Validay
$6,005
$7.00%
SILDOR
(3,00191)
85.50%
$2.0005
$7.005
Uniqueness
Alomn Stabiliyy
98.801%
98,
95.84%
(0.33581)
9S.
92.40%
98.17%
95.51%
Molecule Stubiiity
87.350%
85.0015"
(4.00951)
S8,00SE
$0.50%
Ensry above graund
(7788)
Metri
Guscutoee scale (egerro)
Bascline (Gurzzo)
CAEDM
Foree RMS (EbBiar)
0,0119
0.117
0.0115 (-1.37%))
0.0124
0,0143
0.0117
0,0117
Validity
81.50%
$1,50%
79,50%
$6,50%5
(2.0005
$5.00%
84.50%
Unigueness
100,005
100,00%
100,005
Aoom Stabiliyy
95.125%
98.15%
98L1815
97.745F
98.62%
(0.30%
20%5
95.32%
Milasale Stubilit
79.50r
75.000
84.00%
Fieery shooe groan (Fh)
BUN079 (40,60/11)
0.079
0.0100
Metric
Guidance scale (33e)
Buscine (.3e)
Force RMS (Eb]oorr)
0,0104
0.0106
0U0104
0,010E" 10.19911
0.0121
Q.D115
0,0112
Valddiy
88.500%
99.00%% (-3.50%
N7.500%
$3.50%
$6.00%
Unigueness
100,00%
100,005
100.0015
100.00%
100.00%
100,0095
10000095
Aleen Subiiiy
(0.089%
70.799
98.765
98. 46F6
92.04%
98.33%
95.79%
Melresle Stubility
88.59/%
Enegy alove grand state (Bl)
0,00661 (-16.419%1)
0.0085
0L0064
>0,0065
0.0094
0.0072
0,0073
30

---

Published as a conference paper at ICLR 2025
Table 18: MAE all properties of 200 sampled molecules from explicit bilevel optimization and implicit bilevel optimization with noisy guidance. The "Top" scales of each property can be found Table. T6 We use C-EDM and D- as baselines. and bold denote the overall best and the best within different scales, respectively. X marks the situation where the guidance collapses. Percentage changes between our results and C- are shown parenthesis
Property
CHOMO
LUMM
Ar
Units
Bohr"
D
FM
me
meV
meV
Conditional EDM
2,5089
1.0571
1.0624
0.3304
0,5046
0.6191
Conditional GM
2.5893
1.1582
0.9646
0.5927
Go
5.7481
1.4211
3.1630
0,6042
1.1602
1,1967
scale 1.0 x
2.5449
1.1151
0.9468
0.3554
0.5173
0.5327
0.9393
0.5225
scale 1.0 x 10ma
2.5304
1,1180
(-2.62%.L
0.3554
0.5306
(4.88%
Explicit Bilevel Optimization
scale x 10-
2,4684
1.1246
0.9727
0.3517
(-14.34%%)
0,5077
0.5354
1.0483
0,3404
scale 1.0 x
2.5394
(-9.49%
0.9843
(-0.09%%))
0.5317
0.5400
scale 1.0 x 10 10
(-4.54%
2.4430
1.0493
0.9694
0.3690
0.5288
0.5695
0,6364
Top
2.1493
(-45.05%%)
0.9290
0,4358
0,8359
0.7890
1.9703
0.7649
0,4217
Implicit Bilevel Optimization Noisy Guidance
Top
(-23.01%..)
0.7012
(-20.70%..
(23.17%%1
0.8037
0.6428
0.7598
Top
2.8 1.8
0.7712
0.9795
0,4375
(28.19%/1)
0.7663
Top
3.0144
0.8594
1.0742
0.4832
0.8404
(24.09%f)
0.6182
2,4860
0.7246
0.9369
0,8099
All
(-2.86%L)
(-37.43%%
(-2.87%1)
(30.47%%1)
(36.64%11
(41.33%f)
N MOLECULE VISUALIZATION
We present the visualization of generated molecules with CHEMGUIDE for stability optimization in Fig. 14 and 15| and bilevel optimization for molecular property and stability with noisy neural guidance and CHEMGUIDE in Fig. 16
31

---

Published as a conference paper at ICLR 2025
Table 19: Force Metrics of all 6 properties of 200 sampled molecules from implicit bilevel opti- mization with noisy guidance. The "Top" scales of each property can be found in Table. [6 We use C-EDM and C-GeoLDM as baselines. and bold denote the overall best and the best within different scales, respectively. Percentage changes between our results and C-GeoLDM are shown in parenthesis
Metric
Gundance scale Top (100)
Baseline
Top I (25.0)
Top (20.0)
Top (5.0)
All
C-FMM
C-Coel
Force RMS (EhlBohr)
0,0181
0,0132
0.0144
BARAN' (22.415)
00132
1,99911
0,0125
0.0335
Validiy
92.00%
58.00%
DOFR))
Uniqueness
100 80%%
100.00
100.00%
109.00%
100,00%
10000%S
100,0088
Auom Stubility
97.887%
98.54%
98.77%
99.44%5 (1.21911)
98.66% (0-4281)
98.45%
98.23%6
Molecule Stubiitt
(11.00511)
(7.5091)
Energy ahove groud state (U2)
0,0087
0,0047
0.003
0.0028 (69 26RL
0.0052
(-3.722
0,0094
0,0092
Metric
Guidence scale
Rascline (+)
Top: (20.0)
Top (1200)
Top (10.))
AlI
Force RMS (th/ltohr)
0,0122
0,0155
0.0098'
(-28.1296)) 1296)
CLOCAS
10117144.9%
0,0123
0.0123
Nalidit
92.00%%
94.00%
$4.00%
(9,50%)
91,50%
(5.0090))
36.50%
Uniquenesa
100L00
Arom Stuhility
07.4805
97.88%
98.55% (0.20008)
90.71%
90 91 6-0 189
97488%
98 79%
Molecule Stabliity
84.00%
$5.00%
86.00%
(2.0006
$4.50% (0.50901)
79.00%
$4.00%
Energy above grond vase (F3)
0.0035
0.0108
0U0GSS (-72919
0.01
0.009)
Metric
Guicance scale (C..)
Baseline (C,)
Top (20.0)
(25.00)
Top (EOD)
Force RMS (Eb/Boar)
0,0131
0,0103
0.0106
BURAY C.T. 40FE
DUDTOO K.9FRLL
0,0117
0.012]
Valldity
92.00% (4.00987)
92.00'5
85,009
92.09 S: (4.0091)
88.50% (0.50/8))
75,005
88.00%
Uniqueness
100,00S"
Almm Stability
96 95%
96.75%
98.32
98.79% (0.2291)
97.69% (-0-889L))
97435%
98.57%
Molecule Stublitt
80.00S
72.00%
$4.00%
88.00% (2.006
$1,006 (1000)))
Energy groun (ED)
0,0139
(687756))
0,0000 (1.9691))
0,0097
Metric
Gindance scale (ranomo))
Baseline Ecwoaaoo
Force RMS (Uh/Boar)
0,0102
0,010]
(11.0"(111330000 (12.3356))
GL0003
0.101 (10000
0.0LT8
0.0113
Vilidity
92,001%
(3.00/67)
$4.00%
92.00%"
(2.000((500011 (5.0001))
$4.005
$5.00%
(1,00951)
82.00%
$7.00%
Uniquenes
100,00%
Arom Stability
98.77% (0.26166))
97.9%%
98.77%
(0,20911)
98.52%
58 49%6
(0.028))
98. 17%
98.51%
Molecule Subllit
85.00%
36.00%
90.00% (6.0051)
$6,001
$7.50%
(3,50%81)
80.50%
$4.00%
Unergy groun
(68.919))
Guidatee scale (irzrano)
Buscine (621000)
Metric
Top
31400)
(500)
All
GeOLUM
(El/Boka)
0,0010
0.1000
IJIDSS
(-18.39%
0.0ORR (696))
0.0117
Villdiy
96.00 55 (1 11500)
96 06.00%%
96.00% (11.50911)
02.00%
94. SOS
(10,0091))
83.00%
$4.50%
Uniqueness
100,00%5
100,009
100,00%
100.005
Arom Stabliiy
.8.888
98.54%2
(1.22511
(0.619))
Molecale Stablitt
94.00% "(9,50167))
90000%
$4.00%
(9.50%)
90,50% (6.00R61)
$1.50%
$4.50%
0,0026
(-69.625))
Maric
Guodance scale (33r)
Baseline (3xc)
Top (40.0)
(30.0)
Top (3000)
Eap (25.0)
All
EDM
GenL.DM
0,0104
0.0109
0JHAY (-114798))
(7.1888))
Vilidily
88.005
96.00 "5"
(5.50951)
92.00%
92.00S
92.00% (1 505
86.00%
90.50%
Uniquenens
100.00%
100,005
100L005
100,095
100.00%
100,008"
100.005*
Arom
(0.22%)
98.32% 0.47%6
Malccule Subliiy
S0.00S
18L,00%
(-0.90RL
$8.00
$8.009
56.00%:2..505 2,500))
81.50%
B8.505
(E3)
0,0024
0,0026 (6446648L)
(77.325)
32

---

Published as a conference paper at ICLR 2025
Table 20: MAE and foree RMS of and μ of 500 sampled molecules from GeoLDM with clean guidance and 200 sampled molecules from implicit bilevel optimization with clean guidance. The "Top" scales of each property can found in Table. [6 We use C-EDM and C- as baselines. and bold denote the overall best and the best within different scales, respectively. marks the situation where the guidance collapses.
Guiclanoe scale
Baseline (n)
(o)
(0.1)
Top (0.01)
Top (0.0001)
C-EDM
C-GooLDM
Foroe RMS (El/Bohr)
0.1086
0.0179
0.0112
0.0122
0.0326
0,0118
0.0117
Clean Guidance
MAE
2,4980
5,4798
5.6894
5.7236
4,9992
2,6308
2.5551
Bilevel Opsimizatioon w/ Clean Guidance
Foroe RMS (El/Bohr)
0.0972
0,0205
0.0101
0.111
0.0244
0.0125
0.0135
MAE
1,6182
4.2294
4 4,9078
4,9764
4,2058
2,5089
2,5993
Guiklanos scale
Roseline (P)
Model (ja)
Top (10.0)
Top 2 (5.0)
Top 3 (2.0)
Top (1.0)
All
C-FDM
C-Giol DM
Feroe (Fl/Bohr)
0.3962
0.0122
Clean Guidance
MAF
0.1631
0.5031
0.4475
0.7144
0,4025
1.1257
1.1084
Optimization
Force RMS (El/Bohr)
0.1454
0,1524
0.1
0.1348 (150)
0,0123
0.0123
Clean Guidance
(1,55
1,0571
QM9
Figure 14: Molecules generated from QM9 using unconditional GeoLDM with CHEMGUIDE.
33

---

Published as a conference paper at ICLR 2025
GEOM
Figure 15: Molecules generated from GEOM using unconditional GeoLDM with CHEMGUIDE.
34

---

Published as a conference paper at ICLR 2025
alpha
mu
Cv
homo
lumo
gap
Figure 16: Molecules generated from QM9 using bilevel optimization with noisy neural guidance and CHEMGUIDE for unconditional GeoLDM.
35


Reviews:
[{"review_id": "rqdr3Zf6wq", "reviewer": "Reviewer_V6At", "summary": "The paper proposes CHEMGUIDE, a approach that uses non-differentiable guidance for the conditional generation of molecular structures in diffusion models. CHEMGUIDE use quantum chemistry oracles as guidance, providing gradients for sampling distributions. The method applies zeroth-order optimization to enhance molecule stability by minimizing atomic forces. Experiments demonstrate CHEMGUIDE’s effectiveness on two datasets.", "questions": "Please see Weaknesses", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. CHEMGUIDE’s use of quantum chemistry as a non-differentiable oracle in conditional diffusion models is meaningful.\n2. The paper reports improvements in stability and force metrics over baselines.\n3. Implementing zeroth-order optimization in a diffusion context is well-justified.", "weaknesses": "1. The paper could enhance its rigor by comparing CHEMGUIDE with more baseline models, such as MolGAN, and more importantly, other existing guidance methods. The current comparisons seem limited. It would also be comprehensive to experiment with more datasets.\n2. While GFN2-xTB is a reasonable compromise, comparing CHEMGUIDE results against high-accuracy methods like DFT more extensively could help validate the chemical accuracy of generated molecules.\n3. The paper lacks a thorough discussion on the limitations of using a non-differentiable oracle, such as the potential difficulty in handling certain molecular configurations or diverse chemical spaces. \n4. The use of the GFN2-xTB method and bilevel optimization adds computational complexity, which could restrict practical usage. And The guidance scale parameter lacks an adaptive mechanism. Exploring automated scale scheduling would improve usability.\n5. Code is not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nHcLH4yqll", "reviewer": "Reviewer_vmGX", "summary": "The authors propose CHEMGUIDE, a sampling algorithm to guide pre-trained (latent) diffusion models for molecule design with the goal to optimize for stable molecules indicated by smaller force norms when evaluated using the xTB oracle functions. As the xTB oracle function, which outputs the forces per atom in a molecule, is non-differentiable, the authors make use of known gradient approximation from random pertubation theory to approximate the gradients suitable for guidance during the diffusion sampling trajectory. The authors also suggest how their non-differentiable guidance can be combined with neural regressors as commonly done in the diffusion models literature. The authors show that they non-differentiable guidance when applied on GeoLDM leads to generated molecules with lower force norms compared to the samples when GeoLDM is used without the proposed guidance, indicating that their method works for the models trained on common benchmark datasets such as QM9 and GEOM-Drugs.", "questions": "Is there a typo in the numerator in Eq. 15 when approximating the gradient? It should say $\\frac{ \\mathcal{F}[z_{x,t} + c U, z_{h,t}] - \\mathcal{F}[z_{x,t} - c U, z_{h,t}] )}{2c}$.\n\nIn Algorithm 1, the approximated gradient g_{t-1} has a dependency to the state at time $t=0$. Is this is a typo, since Eq. 15 does not refer to the clean data prediction.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The usage of approximate gradients from non-differentiable oracles in diffusion guidance for molecule design is novel and interesting.\nTo evaluate their proposed method, the authors run a suite of multiple experiments showing that on the two common benchmarks the generated samples based on their sampling algorithm have improved evaluation metrics compared to the baselines.", "weaknesses": "The guidance component from the non-differentiable oracle shows small improvement on the QM9 and GEOM-Drugs dataset. While the idea is interesting, a stronger baseline to compare against is to use the samples generated from Baseline GeomLDM and perform a relaxation using xTB. As the authors mention in their appendix A - Implementation Details, the sampling time for 100 molecules is quite slow with 6 hours and 18 minutes if they perform their proposed guidance in the last 400 diffusion timesteps using xTB as oracle.\nHow does the GeoLDM baseline (right column) in Table 1 and 2 compare if xTB is used using a pre-defined number of relaxation steps?\n\nFurthermore, I found it hard to read the paper as Section 3 Methodology contains sections 3.1 and 3.2 which are not the author's contribution but already existing methods. I would move these Section 2 within the preliminaries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qEaB87oDLv", "reviewer": "Reviewer_tzoc", "summary": "This paper studies how to achieve derivative-free molecular optimization with diffusion models. The main motivation for this work is that many real-world molecular properties are sophisticated and can only be evaluated through expensive experiments or non-differential simulations. In this paper, a zero-order optimization method is constructed by perturbing the input molecular conformation and the effect on the molecular properties. The effectiveness of the proposed methods is validated on a set of quantum mechanical properties for small molecules.", "questions": "Most of my questions are about experiments, I feel the current experimental comparisons are too weak (only comparing with unconditional generation), see weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "* This paper studies an important and timely problem, how to move beyond generative models (randomly sample from learned distributions) to efficient search and optimization over the space with guidance is a pressing question in molecular generation. \n* Derivative-free guidance is very well-motivated and I agree that it is of great importance to problems in real-world molecular discovery process.", "weaknesses": "I have several concerns about this paper, mostly coming from the claim, related work, and experiments. \n\n* First of all, the idea of derivative-free guidance is not new, in molecular optimization, evolutionary algorithms have been used [1], twisted sequential Monte Carlo for protein design [2], twisted SMC for large language model inference [3], and stochastic optimal control method for music generation [4]. I believe the claim for this work to be the first of its kind is inappropriate, neither for derivative-free guidance nor in molecular design.\n\n* Given this is not new, the related work section in the Appendix only discusses the general molecule generation literature and part of the guided diffusion model literature, but misses the critical relevant literature both in molecular generation and other domains.\n\n* The experimental results are weak, even if I agree on the general motivation of derivative-free guidance, (1) there are works such as simple evolutionary algorithms [1] and twisted SMC [2] available for comparison; even if you do not want to compare against them, you need to compare with gradient-based method --- if you think about the experiment budget, you can always construct a classifier by the samples you have evaluated, e.g. a trained neural network. Despite this may not generalize OOD or perform badly, but you may still include them as baselines. For more potential baselines to compare against, you can check this benchmark [5].\n\n[1] Schneuing, A., Du, Y., Harris, C., Jamasb, A., Igashov, I., Du, W., Blundell, T., Lió, P., Gomes, C., Welling, M. and Bronstein, M., 2022. Structure-based drug design with equivariant diffusion models. arXiv preprint arXiv:2210.13695.\n\n[2] Wu, L., Trippe, B., Naesseth, C., Blei, D. and Cunningham, J.P., 2024. Practical and asymptotically exact conditional sampling in diffusion models. Advances in Neural Information Processing Systems, 36.\n\n[3] Zhao, S., Brekelmans, R., Makhzani, A. and Grosse, R.B., Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo. In Forty-first International Conference on Machine Learning.\n\n[4] Huang, Y., Ghatare, A., Liu, Y., Hu, Z., Zhang, Q., Sastry, C.S., Gururani, S., Oore, S. and Yue, Y., Symbolic Music Generation with Non-Differentiable Rule Guided Diffusion. In Forty-first International Conference on Machine Learning, 2024.\n\n[5] https://github.com/brandontrabucco/design-bench", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OB7A867nev", "reviewer": "Reviewer_wXP8", "summary": "The authors propose ChemGuide, a method for estimating diffusion guidance (i.e. gradients) from a non-differentiable property predictor. The goal is to eliminate the need for labeled training data typically required for property prediction networks. They demonstrate their approach in the context of 3D molecular generation. ChemGuide enables an unconditional diffusion model to generate more stable 3D molecular structures by incorporating guidance from quantum chemistry calculations (GFN2-xTB) that serve as a non-differentiable oracle.", "questions": "**clarifications**\n- Can you clarify which aspects of your method implement SPSA? Equation 15 combines two ideas: 1) introducing random perturbations for atom coordinates from a standard normal distribution, and 2) using these perturbations in a finite-difference approximation of the gradient. It would be helpful to explicitly state which of these constitutes SPSA and how it differs from standard finite differences.\n\n- What are the theoretical requirements for SPSA convergence? The paper mentions continuity of z as a requirement, but are there other conditions (e.g., smoothness, bounded variance) needed for the gradient estimates to be reliable?\n\n**suggestions**\n- Given the extensive appendix (sections A-K), adding a table of contents at its beginning would improve navigation.\n\n- Consider adding a brief discussion of computational overhead introduced by SPSA compared to standard finite differences.\n\n**nitpicks**\n- Stability' is unnecessarily capitalized on page 6, section 4.2", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- **Strong and relevant contribution**: The idea proposed in the paper has clearly relevant applications in a number of domains (i.e. any field with expensive oracles) and naturally extends existing efforts in the field of diffusion models. \n- **Novelty**: The method is conceptually simple yet novel, opening the door for various applications which could benefit from the guidance of a non-differentiable oracle.\n- **Thorough empirical evaluation**: The paper presents thorough empirical analysis, effectively demonstrating both the strengths and limitations of the proposed method. The experiments span multiple datasets (QM9 and GEOM), various molecular properties, and different guidance approaches (explicit, implicit, and combined).\n-  **Extensive analysis**: the empirical observations are grounded in real-world chemistry insights, with careful analysis of failure cases and performance trade-offs.\n- **Clarity of presentation**: the paper is well-written and includes many relevant and well-designed figures.", "weaknesses": "- **Justification of the zeroth-order optimization and comparison to other non-gradient optimization methods**: The paper lacks clear justification for choosing SPSA as the zeroth-order optimization method. Section 3.2 would benefit from a discussion of alternative approaches (e.g., finite differences, evolution strategies) and justification for their specific choice in terms of computational efficiency and accuracy trade-offs, and suitability for this particular application of guiding molecular diffusion models. A short pragaraph would suffice here.\n\n- **Assessing the quality of the gradients obtained with ChemGuide vs a differentiable regressor** While the paper shows final performance metrics, it lacks direct analysis comparing the gradients estimated via zeroth-order optimization to those from a differentiable regressor. Such comparison could provide insights into how reliable are CHEMGUIDE's estimated gradients compared to differentiated gradients. For example, the authors could plot the cosine similarity between CHEMGUIDE's estimated gradients and those from a differentiable regressor across different timesteps of the diffusion process\n\n- **Explain which guidance method is suitable for which property**:  The authors observe that noisy and clean guidance methods appear complementary, with properties poorly optimized by one often well-optimized by another (e.g., α vs ∆ϵ). However, the paper would be more practically useful if it provided explanations for these differences, helping practitioners choose the appropriate method for their specific use case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)