Chemistry-Inspired Diffusion with Non-Differentiable Guidance

## Abstract

Abstract Recent advances in diffusion models have shown remarkable potential in the conditional generation of novel molecules. These models can be guided in two ways: (i) explicitly, through additional features representing the condition, or (ii) implicitly, using a property predictor. However, training property predictors or conditional diffusion models requires an abundance of labeled data and is inherently challenging in real-world applications. We propose a novel approach that attenuates the limitations of acquiring large labeled datasets by leveraging domain knowledge from quantum chemistry as a non-differentiable oracle to guide an unconditional diffusion model. Instead of relying on neural networks, the oracle provides accurate guidance in the form of estimated gradients, allowing the diffusion process to sample from a conditional distribution specified by quantum chemistry. We show that this results in more precise conditional generation of novel and stable molecular structures. Our experiments demonstrate that our method: (1) significantly reduces atomic forces, enhancing the validity of generated molecules when used for stability optimization; (2) is compatible with both explicit and implicit guidance in diffusion models, enabling joint optimization of molecular properties and stability; and (3) generalizes effectively to molecular optimization tasks beyond stability optimization.

## 1 Introduction

Diffusion models have received increasing attention in molecular design. Their ability to generate novel molecules with desired properties (i.e., guided diffusion) has fostered advances in material science (Manica et al., ; Yang et al., ), chemistry (Anstine & Isayev, ), protein design (Watson et al., ; Abramson et al., ), etc. To achieve guided diffusion, one can explicitly condition the diffusion process on specific properties in training (Hoogeboom et al., ; Xu et al., ), such that the model is naturally a conditional generator during inference. Alternatively, an unconditional model can be trained without labels, and the diffusion process is guided implicitly at inference time (Dhariwal & Nichol, ; Vignac et al., ) using a property predictor that provides guidance gradients to steer the model toward sampling from the conditional distribution.

Both explicitly and implicitly guided diffusion require a labeled dataset to train the model or the property predictor. The acquisition of labels, however, can be expensive, time-consuming, and often impracticable(^1^11For example, $\text{LD}_{50}$ (Erhirhie et al., ) measures the lethal dose of a test substance, which is difficult to calculate from the molecular structure of the substance alone; thus, requiring animal testing.). When only small labeled datasets are available (Power et al., ), the model and the property predictor can potentially struggle to generalize beyond seen structures (see Appendix F.2), degrading the performance of the diffusion process when generating novel molecules. Implicitly guided diffusion can help address such challenges. This approach (i) requires no labels to train the model and (ii) can replace the property predictor with domain knowledge from quantum chemistry, fulfilled by quantum chemistry software such as xTB (Bannwarth et al., ; ) and Gaussian (Frisch et al., ). Such software can act as an expert oracle to create labeled datasets; thereby, avoiding the extrapolation shortcomings of neural networks.

Despite following certain computation procedures, quantum chemistry is by nature a non-differentiable oracle(^2^22Here it refers to the non-differentiability across the chemical space between different molecules.) that can not be backpropagated with neural networks, as those procedures usually involve proprietary algorithms or extremely computationally expensive to perform, which prevents the integration of quantum chemistry to guide the diffusion models. In this work, we aim to implicitly guide the diffusion process with a non-differentiable oracle, and employ zeroth-order optimization methods (Nesterov & Spokoiny, ; Malladi et al., ) to estimate the necessary guidance gradients. We refer to our method as ChemGuide, a method that leverages quantum chemistry to provide more accurate diffusion guidance. Specifically, given the oracle xTB with the GFN2-xTB method (Bannwarth et al., ) that conducts quantum chemistry calculation of potential energies and atomic forces, we first use ChemGuide to improve the stability of the generated molecular geometry (see Fig. 1), by guiding the diffusion process towards sampling from a distribution where the net force on the atoms is zero. Further, we formulate diffusion guidance on both molecular property and stability as a bilevel optimization (Colson et al., ) problem, to demonstrate that ChemGuide is compatible with both explicitly and implicitly guided diffusion. We summarize our contributions as follows:

- •
We propose ChemGuide, which leverages zeroth-order optimization techniques to introduce a non-differentiable chemistry oracle as diffusion guidance (Section 3). To the best of our knowledge, this is the first work of its kind.
- •
We show in a bilevel optimization framework (Algorithm 1) that ChemGuide can be utilized in conjunction with existing guided diffusion methods.
- •
Experiments and analyses (Section 4 and 5) demonstrate that ChemGuide (i) improves the stability of generated molecules when guided with domain knowledge from the semi-empirical quantum chemical method, GFN2-xTB (Section 4.2); (ii) generalizes to molecular optimization other than stability (Section 4.5); (iii) effectively improves both molecular property and stability when used with other guided diffusion methods (Section 4.4).

Figure: Figure 1: The overview of ChemGuide. On the left, we present the space of all molecules (roughly) as a unimodal distribution, where red/blue region indicates molecules for training/novel molecules generated by the diffusion model. In the middle, ChemGuide derives non-differentiable guidance from quantum chemistry to steer the diffusion process towards a conditional distribution (e.g., minimized forces). On the right, we present the average forces of 3 sets of molecules generated by GeoLDM (Xu et al., ) trained on QM9 () (left two) and GEOM () (rightmost one) without (above) and with (below) ChemGuide.
Refer to caption: x1.png

## 2 Preliminary

In this section, we introduce the key concepts of diffusion models and explore the architecture of a specific diffusion model designed for 3D molecule generation. We will explain how to achieve equivariance in the generated molecules and how the semi-empirical quantum chemical method GFN2-xTB can be utilized for guidance. Additionally, we will also outline our motivation for integrating ChemGuide with 3D diffusion models to optimize molecular stability.

##### Diffusion Models

In general, a diffusion model (Sohl-Dickstein et al., ; Ho et al., ; Song et al., ) consists of a forward diffusion process and a reverse denoising process. The diffusion process is a Markov chain that gradually adds Gaussian noises with a predefined variance schedule $\beta_{1:T}$ from timestep 1 to $T$ to the original datapoint ${\bm{x}}_{0}$, which is chosen such that ${\bm{x}}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. The forward diffusion process $q$ is usually defined as a fixed schedule by the following:

$$ $q({\bm{x}}_{1:T}\mid{\bm{x}}_{0})=\prod_{t=1}^{T}q({\bm{x}}_{t}\mid{\bm{x}}_{t -1})\;\;\;\;\;\;\;\;q({\bm{x}}_{t}\mid{\bm{x}}_{t-1})=\mathcal{N}({\bm{x}}_{t} ;\sqrt{1-\beta_{t}}{\bm{x}}_{t-1},\beta_{t}\mathbf{I})\vspace{-0.3em}$ (1) $$

The denoising process starts with ${\bm{x}}_{T}$ and recovers the original datapoint ${\bm{x}}_{0}$ by predicting the mean of ${\bm{x}}_{t-1}$ given ${\bm{x}}_{t}$ using a neural network denoted as $\mu_{\theta}({\bm{x}}_{t},t)$, where $\theta$ is the parameter.

$$ $p_{\theta}({\bm{x}}_{0:T})=p({\bm{x}}_{T})\prod_{t=1}^{T}p_{\theta}({\bm{x}}_{ t-1}\mid{\bm{x}}_{t})\;\;\;\;\;\;\;\;\;\;p_{\theta}({\bm{x}}_{t-1}\mid{\bm{x}} _{t})=\mathcal{N}({\bm{x}}_{t-1};\mu_{\theta}({\bm{x}}_{t},t),\Sigma_{\theta}( {\bm{x}}_{t},t))\vspace{-0.3em}$ (2) $$

In practice, $\Sigma_{\theta}=\sigma_{t}^{2}\mathbf{I}$ with $\sigma_{t}=\sqrt{\frac{1-\alpha_{t-1}^{2}}{1-\alpha_{t}^{2}}\beta_{t}}$, $\alpha_{t}=\sqrt{\prod_{i=1}^{t}(1-\beta_{i})}$ for all $t$. For simplicity, Ho et al. () reparametrize Eq. 1 such that ${\bm{x}}_{t}=\alpha_{t}{\bm{x}}_{0}+\sqrt{1-\alpha_{t}^{2}}\epsilon$, where $\epsilon\sim N(\mathbf{0},\mathbf{I})$. Hence, instead of predicting $\mu_{\theta}({\bm{x}}_{t},t)$, we now predict the noise $\epsilon_{\theta}({\bm{x}}_{t},t)$, where $\mu_{\theta}({\bm{x}}_{t},t)$ is parameterized as $\frac{1}{\sqrt{1-\beta_{t}}}({\bm{x}}_{t}-\frac{\beta_{t}}{\sqrt{1-\alpha_{t}^
{2}}}\epsilon_{\theta}({\bm{x}}_{t},t))$. Consequently, we have

$$ ${\bm{x}}_{t-1}\sim\mathcal{N}(\frac{1}{\sqrt{1-\beta_{t}}}({\bm{x}}_{t}-\frac{ \beta_{t}}{\sqrt{1-\alpha_{t}^{2}}}\epsilon_{\theta}({\bm{x}}_{t},t)),\sigma^{ 2}_{t}\mathbf{I})\vspace{-0.5em}$ (3) $$

##### Latent Diffusion Architecture for 3D Molecule Generation

An $N$-atom molecule can be represented as a point cloud $\mathcal{G}=[\mathbf{x},\mathbf{h}]\in\mathbb{R}^{N\times(3+d)}$, with $\mathbf{x}\in\mathbb{R}^{N\times 3}$ being the $N$ atom’s 3D coordinates and $\mathbf{h}\in\mathbb{R}^{N\times d}$ as the atom features indicating atom types. A latent diffusion architecture (Rombach et al., ; Xu et al., ) consists of a Variational Autoencoder (VAE) and a diffusion model trained consecutively. Particularly, the geometric latent diffusion model (Xu et al., ) uses the encoder of the VAE to project molecules to a continuous latent space, on which the diffusion model is then trained. Denote the encoder as $\mathcal{E}$ and the latent variable by $\mathbf{z}\in\mathbb{R}^{N\times(3+d_{z})}$, then $[\mathbf{z}_{\mathbf{x},0},\mathbf{z}_{\mathbf{h},0}]=\mathcal{E}([\mathbf{x},
\mathbf{h}])$, with $\mathbf{z}_{\mathbf{h},0}\in\mathbb{R}^{N\times d_{z}}$ and $d_{z}<d$. Let $\mathbf{z}_{t}=[\mathbf{z}_{\mathbf{x},t},\mathbf{z}_{\mathbf{h},t}]$, the latent forward diffusion process and reverse denoising process are defined by replacing ${\bm{x}}$ with ${\bm{z}}$ in Eq. 1 and 2.
We denote the decoder of the VAE as $\mathcal{D}$, which maps $\mathbf{z}_{0}$ back to the original molecular space, such that $\mathcal{D}([\mathbf{z}_{\mathbf{x},0},\mathbf{z}_{\mathbf{h},0}])=[\mathbf{x}
,\mathbf{h}]\in\mathbb{R}^{N\times(3+d)}$. The encoder $\mathcal{E}$ and the decoder $\mathcal{D}$ are parameterized with an equivariant graph neural network (EGNN) (Satorras et al., ) to translate between discrete molecular data and latent variables, such that the atom types are invariant and the positions are equivariant to transformations as the following:

$$ $R\mathbf{z}_{\mathbf{x},t}+T,\mathbf{z}_{\mathbf{h},t}=\mathcal{E}(R\mathbf{x} _{t}+T,\mathbf{h}_{t})\;\;\;\;\;\;R\mathbf{x}_{t}+T,\mathbf{h}_{t}=\mathcal{D} (R\mathbf{z}_{\mathbf{x},t}+T,\mathbf{z}_{\mathbf{h},t})\vspace{-0.3em}$ (4) $$

for any rotation matrix $R$ and translation matrix $T$, where $\mathbf{z}_{\mathbf{x},t}\in\mathbb{R}^{N\times 3}$ are required to satisfy zero center gravity and have zero-mean over $N$ atoms for each position. In addition, the latent diffusion model is also parameterized by EGNN such that transitions between each timestep in the denoising process also respect the same characteristics.

##### GFN2-xTB Method

Matter such as electrons, atoms, and molecules, interacts with matter inherently to reach configurations with lower potential energies, i.e. stability.
To formulate this as a molecular geometry optimization problem, let $h_{1},\cdots,h_{N}$ be the $N$ atoms of a given molecule and $\mathbf{x_{1}},\cdots,\mathbf{x_{N}}\in\mathbb{R}^{3}$ be their coordinates in the 3D space. Each atom $h_{i}$ is subject to a force:

$$ $\displaystyle\mathbf{f_{i}}(\mathcal{G})$ $\displaystyle=\frac{\partial E_{p}\left(\mathbf{x_{1}},\cdots,\mathbf{x_{N}} \mid h_{1},\cdots,h_{N}\right)}{\partial\mathbf{x_{i}}},\,\forall i\in[N] \vspace{-0.15em}$ (5) $$

where $E_{p}$ represents the potential energy of the conformation. The force here manifests valid physical interpretations: at a high level, an atom is pushed accordingly by the exerted force until the force reduces to zero and an equilibrium is achieved. The two necessary (but not sufficient) conditions for a stable molecular geometry are: (i) all forces on the atoms should be (close to) zero, i.e., $\forall i\in[N],\mathbf{f_{i}}(\mathcal{G})=\mathbf{0}$; and (ii) the Hessian matrix, i.e. the second derivative of the potential energy, must be non-negative.

However, the exact mathematical potential energy evaluation, i.e., the solution to the Schrödinger equation, is still a “black box” (). Over the years, different levels of theories and methods have been developed to evaluate potential energies, such as force field (FF), semi-empirical methods (e.g., xTB), and density functional theory (DFT) methods (e.g., B3LYP/6-31G(2df,p)), listed in an order of increased accuracy and cost. After trading-off between accuracy and efficiency within a feasible computation cost, we selected GFN2-xTB, a more recent and advanced semi-empirical method (Bannwarth et al., ), to calculate the forces of the generated molecular geometry in the diffusion process. More details about GFN2-xTB can be found in Appendix C.

##### Motivation for Stability Optimization with 3D Diffusion

Stable molecular geometry is favored for molecule generation. A novel but unstable molecular geometry may not be physically achieved at all and thus carries no substantial meaning. To optimize stability, one can minimize the potential energy of the molecule by iteratively minimizing the forces on the atoms (Eq. 5), motivating our choice of a 3D diffusion model. More details and discussions are provided in Appendix D.

## 3 Methodology

### 3.1 Guidance From Neural Regressor

##### Noisy Guidance

The goal of neural guidance is to direct the denoising process towards a target property value $y$. Dhariwal & Nichol () propose to modify the denoising process to achieve conditional generation with an unconditional model and using a scalar $s$ that controls the guidance strength:

$$ ${\bm{x}}_{t-1}\sim\mathcal{N}(\frac{1}{\sqrt{1-\beta_{t}}}({\bm{x}}_{t}-\frac{ \beta_{t}}{\sqrt{1-\alpha_{t}^{2}}}\epsilon_{\theta}({\bm{x}}_{t},t))+s\sigma^ {2}_{t}\nabla_{{\bm{x}}_{t}}\log p_{\phi}(y|{\bm{x}}_{t}),\sigma^{2}_{t} \mathbf{I})$ (6) $$

Here $p_{\phi}(y|{\bm{x}}_{t})$ is parameterized by a classifier, and $y$ is a categorical label, such that the modification $s\sigma^{2}_{t}\nabla_{{\bm{x}}_{t}}\log p_{\phi}(y|{\bm{x}}_{t})$ shifts the mean of the sampling distribution to provide guidance. Let $y\in\mathbb{R}$ and $f_{\eta}:\mathcal{G}\to\mathbb{R}$ be the neural regressor for the molecular property of interest (Vignac et al., ). Now, assuming $y|{\bm{x}}_{t}\sim\mathcal{N}(f_{\eta}({\bm{x}}_{t}),\sigma^{2}_{\eta}\mathbf{
I})$ and $\sigma^{2}_{\eta}=1$, we have:

$$ $\displaystyle\nabla_{{\bm{x}}_{t}}\log p_{\phi}(y\mid{\bm{x}}_{t})\propto- \nabla_{{\bm{x}}_{t}}\|y-f_{\eta}({\bm{x}}_{t})\|^{2}_{2}=-\nabla_{{\bm{x}}_{t }}\mathcal{L}(y,f_{\eta}({\bm{x}}_{t}))$ (7) $$

where $\mathcal{L}(y,f_{\eta}({\bm{x}}_{t}))$ is the Mean Square Error (MSE) between the target and the prediction of $f_{\eta}(\cdot)$.

However, in the early stage of the denoising process, ${\bm{x}}_{t}=\alpha_{t}{\bm{x}}_{0}+\sqrt{1-\alpha_{t}^{2}}\epsilon$ might not be informative enough to predict $y$ as it consists mostly of Gaussian noise. For effective prediction during the denoising process, we estimate the denoised version of ${\bm{x}}_{t}$ as Kawar et al. (); Song et al. ():

$$ $t_{0}({\bm{x}}_{t})\coloneq\hat{{\bm{x}}}_{0}=\frac{{\bm{x}}_{t}-\sqrt{1- \alpha_{t}^{2}}\epsilon_{\theta}({\bm{x}}_{t},t)}{\alpha_{t}}$ (8) $$

where $f_{\eta}(\hat{{\bm{x}}}_{0})$ is used in place of $f_{\eta}({\bm{x}}_{t})$ as the predicted molecular property.

##### Clean Guidance

Besides applying the gradient as guidance on noisy ${\bm{x}}_{t}$, we build insight from Bansal et al. () and derive guidance in the clean (=noise free) space $\hat{{\bm{x}}}_{0}$ as:

$$ $\Delta{\bm{x}}_{0}=\operatorname*{arg\,min}_{\Delta}\mathcal{L}(y,f_{\eta}( \hat{{\bm{x}}}_{0}+\Delta))\vspace{-0.3em}$ (9) $$

where $\Delta{\bm{x}}_{0}$ is approximated using $K$ steps of gradient descent starting from $\Delta=0$. Note that $\Delta{\bm{x}}_{0}$ is in clean data space, so we need to translate it back to the noisy space while recovering ${\bm{x}}_{t}$:

$$ ${\bm{x}}_{t}=\alpha_{t}(\hat{{\bm{x}}}_{0}+\Delta{\bm{x}}_{0})+\sqrt{1-\alpha_ {t}^{2}}\tilde{\epsilon}$ (10) $$

where $\tilde{\epsilon}$ is the augmented noise used to to sample ${\bm{x}}_{t-1}$ and is thus given by:

$$ $\tilde{\epsilon}=\epsilon_{\theta}({\bm{x}}_{t},t)-\frac{\alpha_{t}}{\sqrt{1- \alpha_{t}^{2}}}\Delta{\bm{x}}_{0}\vspace{-0.5em}$ (11) $$

### 3.2 Guidance From a Non-Differentiable Oracle

We aim to tackle a challenging problem where the guidance is specified by a non-differentiable oracle (e.g., the GFN2-xTB method). We change our notation from ${\bm{x}}_{t}$ to $\mathbf{z}_{t}$ at each time step to indicate diffusion in the latent space. With the guidance target $y$ being 0, our non-differentiable function $\mathbf{f}$ (Eq. 5) for force guidance is defined as follows.

$$ $\mathbf{f}(\mathcal{G})=\frac{1}{N}\sum^{N}_{i}\mathbf{f_{i}}(\mathcal{G}) \vspace{-0.5em}$ (12) $$

where the molecular graph is decoded as $\mathcal{G}=\mathcal{D}(\hat{{\bm{z}}}_{0})$ with $\hat{{\bm{z}}}_{0}=t_{0}({\bm{z}}_{t})$ estimated by Eq. 8, and we aim to achieve $\mathbf{f}(\mathcal{G}^{\prime}_{0})<\mathbf{f}(\mathcal{G}_{0})$. Here $\mathcal{G}^{\prime}_{0},\mathcal{G}_{0}$ are provided by ChemGuide and an unguided diffusion model.
As $\mathbf{f}$ is non-differentiable across different molecules, we can not directly add guidance using Eq. 7. Instead, we estimate the gradient analytically.
Recall that $\mathcal{L}(y,\mathbf{f}(\mathcal{G}))$ is the MSE loss, $\mathbf{f}:\mathcal{G}\rightarrow\mathbb{R}$ is the non-differentiable oracle, and $\mathcal{D}$ is the decoder. Let $\mathcal{F}$ be the composition $\mathbf{f}\circ\mathcal{D}\circ t_{0}$, and we estimate the gradient as:

$$ $\displaystyle\hat{\nabla}_{{\bm{z}}_{t}}\log p_{\phi}(y|{\bm{z}}_{t})$ $\displaystyle\propto-\nabla_{\mathcal{F}({\bm{z}}_{t})}\mathcal{L}(y,\mathcal{ F}({\bm{z}}_{t}))\nabla_{{\bm{z}}_{t}}\mathcal{F}({\bm{z}}_{t})$ (13) $\displaystyle\approx-\nabla_{\mathcal{F}({\bm{z}}_{t})}\mathcal{L}(y,\mathcal{ F}({\bm{z}}_{t}))\lim_{\zeta\rightarrow 0}\frac{\mathcal{F}([{\bm{z}}_{\mathbf {x},t}+\zeta\mathbf{1}_{N\times 3},{\bm{z}}_{\mathbf{h},t}])-\mathcal{F}([{\bm {z}}_{\mathbf{x},t}-\zeta\mathbf{1}_{N\times 3},{\bm{z}}_{\mathbf{h},t}])}{2\zeta}$ (14) $$

This approximation is possible because ${\bm{z}}_{t}$ is continuous and it eliminates the need to train neural regressors for properties such as forces, which comes with approximation error (Gasteiger et al., ). In addition, these regressors are often trained on ground state geometries (optimized stable geometries) instead of non-optimal geometries, which are the most common geometries during the denoising process, making them dubious options for property guidance.
However, directly adding (or subtracting) $\zeta\mathbf{1}_{N\times 3}$ to ${\bm{z}}_{\mathbf{x},t}$ would break the equivariance requirement in Eq. 4 on the latent variables, as it shifts the mean of the coordinates by $\zeta$. To maintain zero central gravity (Xu et al., ) of input to $\mathcal{F}(\cdot)$, we construct a perturbation matrix $\mathbf{U}\in\mathbb{R}^{N\times 3}\sim\mathcal{N}(0,1)$, and apply Simultaneous Perturbation Stochastic Approximation (SPSA) (Spall, ; Nesterov & Spokoiny, ) to estimate the gradient, then Eq. 14 becomes:

$$ $\displaystyle\hat{\nabla}_{{\bm{z}}_{t}}\log p_{\phi}(y|{\bm{z}}_{t})$ $\displaystyle\propto-\nabla_{\mathcal{F}({\bm{z}}_{t})}\mathcal{L}(y,\mathcal{ F}({\bm{z}}_{t}))\frac{\mathcal{F}([{\bm{z}}_{\mathbf{x},t}+\zeta\mathbf{U},{ \bm{z}}_{\mathbf{h},t}])-\mathcal{F}([\mathbf{z}_{\mathbf{x},t}+\zeta\mathbf{U },{\bm{z}}_{\mathbf{h},t}])}{2\zeta}\mathbf{U}$ (15) $$

where $\zeta$ is a small perturbation scale (e.g., $10^{-6}$). The perturbed representations $[{\bm{z}}_{\mathbf{x},t}\pm\zeta\mathbf{U},{\bm{z}}_{\mathbf{h},t}]$ obey zero central gravity required by $t_{0}$, as $\mathbb{E}[\zeta\mathbf{U}]=0$. Note that, unlike neural regressor guidance, we only add guidance to the positions (i.e. ${\bm{z}}_{\mathbf{x},t}$) and apply no gradient to the atom types (i.e. ${\bm{z}}_{\mathbf{h},t}$). We do this because the force definition (Eq. 5) is only physically grounded when the set of atoms stays constant (i.e. no matter/mass is created from or reduced to void). In other words, since energy is a relative concept, any gradient that arises from atom-type change has no physical interpretation.

### 3.3 Combine Guidance from Neural Network and Non-Differentiable Oracle

Figure: Algorithm 1 Bilevel guided diffusion sampling with noisy neural guidance

Apart from guided generation conditioned on one single property (e.g., the norm of static polarizability $\alpha$) with a regressor or a non-differentiable oracle (e.g., GFN2-xTB), we can perform joint optimization with the guidance specified by both the differentiable regressor and the non-differentiable oracle, which is encapsulated under the bilevel optimization framework. Formally, recall that $f_{\eta}$ represents the regressor and $\mathcal{F}=\mathbf{f}\circ\mathcal{D}\circ t_{0}$ is the composition of the non-differentiable oracle $\mathbf{f}$, the VAE decoder $\mathcal{D}$, the latent diffusion model, respectively, the joint optimization is formulated as:

$$ $\displaystyle\min_{{\bm{z}}_{x,t}}\quad$ $\displaystyle\mathcal{F}([{\bm{z}}_{x,t},{\bm{z}}_{h,t}^{*}])$ (16) s.t. $\displaystyle[{\bm{z}}_{x,t}^{*},\mathbf{z}_{h,t}^{*}]=\operatorname*{arg\,min }_{[{\bm{z}}_{x,t},{\bm{z}}_{h,t}]}\|f_{\eta}([{\bm{z}}_{x,t},{\bm{z}}_{h,t}]) -y\|^{2}$ (17) $$

where $y$ is the target property value (e.g., we hope the static polarizability $\alpha\rightarrow y$ after optimization). As detailed in Algorithm 1, we first optimize molecular property with $f_{\eta}$, and obtain the optimal atom positions ${\bm{z}}_{x,t}^{*}$ and features ${\bm{z}}_{h,t}^{*}$ (Eq. 17); we then optimize stability with $\mathcal{F}$ by fixing $\mathbf{z}_{h,t}^{*}$ and initializing ${\bm{z}}_{x,t}$ (Eq. 16) as ${\bm{z}}_{x,t}^{*}$. The chemistry intuition of our design is detailed in Appendix E.

## 4 Experiments

### 4.1 Experiment Setting

##### Dataset

The models in our experiment are trained on the QM9 dataset () and the GEOM dataset (). The QM9 dataset is a catalog with 134K small drug-like molecules consisting of up to nine heavy (non-hydrogen) atoms. The Geometric Ensemble Of Molecules (GEOM) dataset includes 450K molecules with up to 91 heavy atoms (on average, 24.9), where 37 million molecular conformations are generated and reported with their geometries, energies, and statistical weight.

##### Guidance Property

We study guided generation for force and thus energy optimization on QM9 and GEOM. For neural and combined guidance, we evaluate on QM9 as there are no labels in GEOM to train the regressors. We consider the following 6 properties reported in QM9: the norm of static polarizability ($\alpha$, Bohr^3), the norm of dipole moment ($\mu$, Debye), heat capacity at room temperature ($C_{v}$, cal/(mol$\cdot$K)), the energy of the electron in the highest occupied molecular orbital ($\epsilon_{\text{HOMO}}$, eV), the energy of the electron in the lowest unoccupied molecular orbital ($\epsilon_{\text{LUMO}}$, eV), and the HOMO-LUMO energy gap ($\Delta\epsilon$, eV). We choose $s$ (Eq. 6) from $\{1,10^{-1},10^{-2},10^{-3},10^{-4}\}$ for all experiments, and additionally $\{2,5,10,20,25,30,40,50\}$ for the 6 properties.

##### Model

We integrate ChemGuide with GeoLDM (Xu et al., ), an improvement from EDM (Hoogeboom et al., ). For non-differentiable guidance on force, we compare our method to unconditional EDM and GeoLDM(^3^33EDM and GeoLDM are in short of the unconditional version of the models when the context is clear.), since there is no available ground-truth force to explicitly train a conditional model. For noisy neural guidance and combined guidance, we choose conditionally trained EDM (C-EDM) and GeoLDM (C-GeoLDM) as the baselines.

##### Evaluation Metric

For non-differentiable guidance on the force, we use the Root Mean Square (RMS) of the forces calculated at GFN2-xTB as the evaluation metric. We also report validity, uniqueness, atom stability, molecule stability, and energy above the ground state.
For differentiable neural guidance on the 6 properties ($\alpha,\mu,C_{v},\epsilon_{\text{HOMO}},\epsilon_{\text{LUMO}},\Delta\epsilon$), we follow Xu et al. () and use the Mean Absolute Error (MAE) between the target property $y$ and the predicted value $\hat{y}$ from the regressor of the generated molecule as the evaluation metric.

We add guidance to the last 400 of the 1000 diffusion steps (Han et al., ), and calculate the change percentage between our method and the GeoLDM/C-GeoLDM baseline. The implementation details are provided in Appendix A.

**Table 1: Metrics of 500 generated molecules from QM9 using GeoLDM with non-differentiable oracle (i.e. xTB) guidance. * and bold denote the overall best result and our best result, respectively. Percentage changes between our results and GeoLDM are shown in parentheses.**
| Metric | Guidance Scale | Baseline |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0.0001 | 0.001 | 0.01 | 0.1 | 1.0 | EDM | GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0104^∗ (-6.76%$\downarrow$) | 0.0104 | 0.0107 | 0.0108 | 0.0125 | 0.0114 | 0.0111 |
| Validity | 91.40%^∗ (1.60%$\uparrow$) | 91.20% | 91.20% | 90.00% | 89.40% | 86.60% | 89.80% |
| Uniqueness | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 99.02% | 98.97% | 98.98% | 99.03%^∗ (0.10%$\uparrow$) | 98.67% | 98.53% | 98.93% |
| Molecule Stability | 90.60% | 90.40% | 90.20% | 91.00%^∗ (2.20%$\uparrow$) | 87.80% | 83.60% | 88.80% |
| Energy above Ground State (Eh) | 0.0042^∗ (-15.78%$\downarrow$) | 0.0042 | 0.0045 | 0.0050 | 0.0061 | 0.0072 | 0.0050 |

### 4.2 Non-Differentiable Oracle Guidance on Force

We adopt various guidance scales on QM9 and show numeric results in Table 1. It is observed that over different scales, our non-differentiable oracle guidance generates molecules with up to 6.31% lower force, 1.60% and 2.20% higher validity and molecule stability, respectively. Despite optimizing molecular stability via force instead of directly with energy, ChemGuide exhibits a smaller energy change than GeoLDM with a 15.78% decrease. These results show that our method generates molecules closer to the ground state, i.e. the most stable geometry, suggesting that our methods can generate molecules with higher validity and Stability.
Further, we explore ChemGuide on the GEOM dataset with larger molecules and report the results in Table 2, where a similar trend is observed in our favor: our method generates valid molecules closer to the ground state. Surprisingly, ChemGuide performs better on larger molecules than smaller ones (e.g., a higher decrease in force RMS), which brings potential applications in large-molecule production, e.g., protein. This is partly because larger molecules have more intricate structures and conformational space; thus, are likely to suffer more from distorted geometries than smaller ones, yielding more space for improvement.

Figure: Figure 2: Histograms and distributions of force RMS and energy change of 500 generated molecules from QM9 and GEOM using GeoLDM with ChemGuide under scale=0.0001. Energy change refers to the energy of our generated molecule above its ground state energy.
Refer to caption: x2.png

**Table 2: Metrics of 500 generated molecules from GEOM using unconditional GeoLDM with non-differentiable oracle (i.e. GFN2-xTB) guidance. * and bold denote the overall best result and our best result. Percentage changes between our results and GeoLDM are shown in parentheses.**
| Metric | Guidance Scale | Baseline |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0.0001 | 0.001 | 0.01 | 0.1 | 1.0 | EDM | GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0445 | 0.0447 | 0.0434 | 0.0411^∗ (-14.16%$\downarrow$) | 0.0513 | 0.0742 | 0.0478 |
| Validity | 47.00% | 45.20% | 51.60%^∗ (2.60%$\uparrow$) | 50.40% | 49.40% | 46.40% | 49.00% |
| Uniqueness | 100.00%^∗ | 100.00% | 100.00% | 100.00% | 100.00% | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 84.47% | 84.49% | 84.65%^∗ (0.12%$\uparrow$) | 84.36% | 83.45% | 81.22% | 84.53% |
| Energy above Ground State (Eh) | 0.2148 | 0.2085 | 0.2104 | 0.1935^∗ (-13.92%$\downarrow$) | 0.2650 | 0.3742 | 0.2248 |

Fig. 2 shows the plot of the force RMS and energy change histogram and distribution of the 500 generated molecules from EDM, GeoLDM, and ChemGuide on QM9 and GEOM. Our method is more concentrated around 0 and less spread to higher regions. Also, note that both EDM and GeoLDM generate outlier molecules with significantly large force and energy change, while no outlier is generated by ChemGuide. For example, in Fig. 2(a), an outlier with force RMS 0.155 is generated by GeoLDM, which is $\sim$15 times higher than the force RMS where the molecules are concentrated. It demonstrates that our method performs better both on average and molecule-wise, and we provide detailed plots of various scales in Fig. 10 in Appendix H.

To further validate the effectiveness of ChemGuide, we compute stability metrics at the DFT/B3LYP/6-31G(2df,p) level of theory with Gaussian16 (Frisch et al., ) for more precise and strict benchmarking. As DFT is computationally demanding, we only present the results of ChemGuide with the best scale, GeoLDM, and EDM on QM9 in Table 3.
Despite applying a less accurate but computationally feasible oracle (i.e., GFN2-xTB), ChemGuide still improves validity and stability (i.e., valid molecular geometry close to optimized ground state geometry), when measured against higher standards (i.e., DFT). On the other hand, applying DFT as the non-differentiable oracle to guide the generation process is computationally expensive and impracticable.

Figure: Table 3: Metrics of 500 generated molecules from QM9 using GeoLDM with ChemGuide using scale=0.0001, calculated at DFT/B3LYP/6-31G(2df,p) level of theory, more precise but computationally expensive than GFN2-xTB. Strict validity is defined as generated geometries within 50 kcal/mol (0.07968 Eh) of the optimized geometries. * and bold denote the overall best result and our best result. Percentage changes between our results and GeoLDM are shown in parentheses.
Refer to caption: x3.png

### 4.3 Neural Network Guidance on Properties

#### 4.3.1 Unconditional Diffusion with noisy neural guidance

We present the results of noisy guidance (Section 3.1) in Table 4 and provide further details in Table 11 in Appendix I. For all the properties $\alpha$, $\mu$, $C_{v}$, $\epsilon_{HOMO}$, $\epsilon_{LUMO}$, and $\Delta\epsilon$, noisy neural guidance outperforms unconditional GeoLDM by 57.92%, 37.24%, 77.09%, 23.18%, 36.87%, and 37.88% respectively, which verifies its effectiveness. Furthermore, GeoLDM with noisy neural guidance yields comparable performance with C-GeoLDM and C-EDM, provided that conditional models are explicitly trained for property optimization; surprisingly, we use unconditional GeoLDM as the backbone and outperforms C-GeoLDM in $\alpha$, $\mu$, and $C_{v}$. To understand the details of guided generation, we plot the MAE of $C_{v}$ on sampled molecules at each guidance step in Fig. 3. Compared to GeoLDM which oscillates in the first 200 steps and converges slowly to a high MAE, the trajectory of noisy neural guidance drops fast in the first 50 steps and the amplitude of the oscillation is less intense, which suggests the guidance smooths the MAE change and quickly steers the model toward better properties. The property MAE trajectories are detailed in Fig. 11 in Appendix I.

#### 4.3.2 Unconditional Diffusion with clean neural guidance

As presented in Table 4, clean neural guidance outperforms unconditional GeoLDM on all properties and beats C-GeoLDM in $\mu$, $C_{v}$, $\epsilon_{\text{HOMO}}$, and $\Delta\epsilon$ by $81.68\%$, $67.53\%$, $3.06\%$, and $34.84\%$. Interestingly, it seems that noisy and clean guidance complement each other on different properties (e.g., $\alpha,\Delta\epsilon$). In Fig. 3, clean neural guidance shows a similar oscillation trend in the first 200 guidance steps compared to GeoLDM, then exhibits a sharp drop in the last 50 steps and achieves the lowest MAE, which is also observed on other properties shown in Fig. 11 in Appendix I. We hypothesize this is due to that molecules are less noisy as denoising steps accumulate; thus, guidance optimization in the clean space manifests its strongest power when ${\bm{z}}_{t}$ is less dependent on the one-step estimation function $t_{0}(\cdot)$ in later steps, where it is closer and more similar to ${\bm{z}}_{0}$ (Eq. 8).

**Table 4: MAE of 500 generated molecules from QM9 using GeoLDM with noisy/clean neural guidance, where the result from the best configuration is reported. We use C-EDM, C-GeoLDM, and GeoLDM as baselines. $*$ and bold denote the overall best and the best within different scales, respectively. Percentage changes between our results and C-GeoLDM are shown in parentheses.**
| Property | $\alpha$ | $\mu$ | $C_{v}$ | $\epsilon_{\text{HOMO}}$ | $\epsilon_{\text{LUMO}}$ | $\Delta\epsilon$ |
| --- | --- | --- | --- | --- | --- | --- |
| Units | $\text{Bohr}^{3}$ | D | $\frac{\text{cal}}{\text{mol}}\text{K}$ | meV | meV | meV |
| Conditional EDM | 2.6308 | 1.1257 | 1.0804 | 0.3207 | 0.5940 | 0.6301 |
| Conditional GeoLDM | 2.5551 | 1.1084 | 0.9703 | 0.3327 | 0.5518^∗ | 0.5878 |
| GeoLDM | 5.6732 | 1.6461 | 3.1046 | 0.6151 | 1.1778 | 1.2022 |
| GeoLDM w/ Noisy Guidance | 2.3870^∗<br>(-6.58%$\downarrow$) | 1.0331 | 0.7112 | 0.4725 | 0.7436 | 0.7468 |
| GeoLDM w/ Clean Guidance | 5.4798 | 0.2031^∗<br>(-81.68%$\downarrow$) | 0.3151^∗<br>(-67.53%$\downarrow$) | 0.3109^∗<br>(-3.06%$\downarrow$) | 0.5688 | 0.3830^∗<br>(-34.84%$\downarrow$) |

### 4.4 Combined Guidance on Force and Properties

We present the results for the bilevel optimization on molecular property and stability, with MAE shown in Table 5, and the force RMS and energy change in Fig. 5. We sample 200 molecules for combined guidance, as there are 6 properties to optimize, and force optimization with xTB only supports CPU computation. We choose conditional GeoLDM (C-GeoLDM) as the baseline.

**Table 5: MAE of 200 generated molecules using C-GeoLDM with ChemGuide and GeoLDM with noisy neural guidance, where the result from the best configuration is reported. We use C-EDM and C-GeoLDM as baselines. $*$ and bold denote the overall best and the best within different scales, respectively. Percentage changes between our results and C-GeoLDM are shown in parentheses.**
| Property | $\alpha$ | $\mu$ | $C_{v}$ | $\epsilon_{\text{HOMO}}$ | $\epsilon_{\text{LUMO}}$ | $\Delta\epsilon$ |
| --- | --- | --- | --- | --- | --- | --- |
| Units | $\text{Bohr}^{3}$ | D | $\frac{\text{cal}}{\text{mol}}\text{K}$ | meV | meV | meV |
| C-EDM | 2.5089 | 1.0571 | 1.0624 | 0.3304 | 0.5046 | 0.6191 |
| C-GeoLDM | 2.5593 | 1.1582 | 0.9646 | 0.3407 | 0.5927 | 0.4982^∗ |
| Bilevel Optimization<br>w/ Explicit Guidance | 2.4430^∗<br>(-4.54%$\downarrow$) | 1.0483 | 0.9393 | 0.3404^∗<br>(-0.09%$\downarrow$) | 0.5077^∗<br>(-14.34%$\downarrow$) | 0.5225<br>(4.88%$\uparrow$) |
| Bilevel Optimization<br>w/ Noisy Guidance | 2.4860 | 0.7246^∗<br>(-37.43%$\downarrow$) | 0.9369^∗<br>(-2.87%$\downarrow$) | 0.4445 | 0.8099 | 0.7041 |

#### 4.4.1 Explicitly Guided Diffusion with ChemGuide

We employ C-GeoLDM with ChemGuide, which resembles the bilevel optimization framework as C-GeoLDM automatically optimizes molecular property during the denoising process. Compared to C-GeoLDM alone, we expect our method to reduce the force while not hurting MAE. We present the results in Fig. 5 and Table 5 (details are provided in Table 12 and Table 13 in Appendix J). For the 6 properties ($\alpha$, $\mu$, $C_{v}$, $\epsilon_{HOMO}$, $\epsilon_{LUMO}$, $\Delta\epsilon$), our method reduces the force RMS by 9.21%, 3.33%, 8.75%, 6.69%, 1.27%, and 10.19%, and for the first 5, MAEs decrease 4.54%, 9.49%, 2.62%, 0.09%, and 14.34%. The results suggest it is feasible and effective to combine explicitly guided diffusion (e.g., C-GeoLDM) with ChemGuide, which goes beyond our expectation as both force RMS and property MAE are reduced.

#### 4.4.2 Implicitly Guided Diffusion with ChemGuide

We apply two aforementioned neural network guidance approaches (noisy/clean guidance, Section 3.1) for the property optimization step in our bilevel optimization framework (Section 3.3). We expect that both the property MAE and the force metrics are better than conditional GeoLDM.

Figure: Figure 4: Force RMS and energy change of 200 generated molecules using explicit bilevel optimization, implicit bilevel optimization with noisy neural guidance, C-EDM, and C-GeoLDM. Energy change refers to energy above ground state.
Refer to caption: x4.png

##### Bilevel optimization with noisy guidance

As shown in Table 5, our method outperforms C-GeoLDM in $\alpha$, $\mu$, and $C_{v}$ by 2.86%, 37.43%, and 2.87% in MAE. It is also observed in Fig. 5 that, for the six properties, our method achieves lower force RMS than C-GeoLDM by 1.94%, 4.57%, 8.97%, 10.66%, 16.26%, and 7.18%, with smaller energy change compared to baselines. Details of MAE and force metrics are provided in Table. 13 and Table. 14 in Appendix J. Combining property optimization with force optimization, noisy guidance with ChemGuide reduces MAE and improves the stability of generated molecules, which suggests that the combined guidance prompts the diffusion model towards generating stabler molecules with better properties.

To further understand how our bilevel framework optimizes property, we provide the MAE trajectory of $C_{v}$ in Fig. 5. The trajectories of our noisy neural guidance method (Fig.3) and bilevel optimization behave similarly, such that the MAE drops fast in the first 50 steps and oscillates less compared to GeoLDM. However, bilevel optimization is slightly worse than the noisy neural guidance method (i.e. it converges closer to C-GeoLDM than noisy guidance). We suspect it is because guided by gradients from the neural regressor and then ChemGuide, the generation process leans less toward a lower MAE, as it keeps a balance between property and stability optimization. Details and explanations of the MAE trajectories of each property can be found in Fig. 12 in Appendix J.

##### Bilevel optimization with clean guidance

We present the details of this section in Appendix G.

### 4.5 ChemGuide beyond Stability

In this section, we explore the potential of using ChemGuide to optimize molecular properties other than force and energy. We sample 200 molecules using unconditional GeoLDM with ChemGuide on $\alpha$, $\mu$, $\epsilon_{HOMO}$, $\epsilon_{LUMO}$, and $\Delta\epsilon$, which can also be calculated at GFN2-xTB(^4^44$C_{v}$ is skipped as its calculation requires the Hessian matrix, which is expensive to compute.), and the results are shown in Table 6. Specifically, since the property label of a molecule from QM9 (calculated at DFT) may be different from the xTB computation, we obtain property scores of all QM9 molecules with xTB as a distribution where we sample the guidance target, which is the reason that GeoLDM has a different MAE than Table 5, and GEOM is processed similarly. It is observed that ChemGuide on properties improves the baseline in $\mu$, $\epsilon_{LUMO}$, and $\Delta\epsilon$. Our method performs less satisfyingly on $\alpha$ and $\epsilon_{HOMO}$ because they are inherently more challenging to optimize, as observed in Table 4. We can see that ChemGuide is general and applicable to a broad range of molecular optimization tasks other than stability across different datasets. Full results can be found in Table 9 and 10 in Appendix H, and our discussion on the difference between ChemGuide and neural guidance for properties is presented in Appendix F.2.

**Table 6: ChemGuide for properties ($\alpha$, $\mu$, $\epsilon_{HOMO}$, $\epsilon_{LUMO}$, $\Delta\epsilon$) with 200 generated molecules on QM9 and GEOM, where the result from the best configuration is reported. We use GeoLDM as the baseline. $*$ and bold denote the overall best and the best within different scales, respectively. Percentage changes between our results and GeoLDM are shown in parentheses.**
| Property | $\alpha$ | $\mu$ | $\epsilon_{\text{HOMO}}$ | $\epsilon_{\text{LUMO}}$ | $\Delta\epsilon$ |  |
| --- | --- | --- | --- | --- | --- | --- |
| Units | $\text{Bohr}^{3}$ | D | meV | meV | meV |  |
| QM9 | GeoLDM | 4.7555^∗ | 1.5490 | 0.6264^∗ | 2.1790 | 2.1568 |
| GeoLDM w/ ChemGuide | 4.8781<br>(2.58%$\uparrow$) | 1.5295^∗<br>(-1.26%$\downarrow$) | 0.6392<br>(2.04%$\uparrow$) | 2.0091^∗<br>(-7.80%$\downarrow$) | 1.9318^∗<br>(-10.43%$\downarrow$) |  |
| GEOM | GeoLDM | 24.3424 | 3.3676 | 1.1922^∗ | 5.0903 | 1.9956 |
| GeoLDM w/ ChemGuide | 24.2168^∗<br>(-0.52%$\downarrow$) | 3.3412^∗<br>(-0.78%$\downarrow$) | 1.2939<br>(8.53%$\uparrow$) | 5.0802^∗<br>(-0.20%$\downarrow$) | 1.9264^∗<br>(-3.47%$\downarrow$) |  |

## 5 Analyses

In Appendix F, we discuss acceleration methods for ChemGuide (Appendix F.1); generalization analyses of property regressor (Appendix F.2) and conditional diffusion model (Appendix F.3); the effect of guidance scales for noisy/clean guidance (Appendix F.4); the effect of optimization steps for clean guidance (Appendix  F.5).

## 6 Related Work

Due to space limits, we present the discussion of some related works in Appendix B.

## 7 Conclusion & Future Work

In this work, we study conditional generation for 3D diffusion models on molecules. We propose ChemGuide, which guides the generation process of an unconditionally trained diffusion model using an external, typically non-differentiable, chemistry oracle. By analytically estimating the guidance gradients using zeroth-order optimization methods, while ensuring the equivariance and invariance requirements for 3D diffusion models, we aim to enhance the stability of generated molecules without relying on labeled data to train an additional property predictor. We observe reduced atomic forces and improved molecular validity across both small and large molecule datasets when the model is equipped with ChemGuide, highlighting the effectiveness of our approach. Further experiments incorporating both ChemGuide and neural guidance within a bilevel optimization framework demonstrate improvements in both molecular properties and stability. This suggests that our non-differentiable guidance is compatible with existing methods. Moreover, when ChemGuide is used for property optimization alone, it shows strong generalization beyond force minimization.

However, since xTB runs on CPU and neural guidance requires gradients that heavily utilize GPU resources, our method may be constrained by limited or insufficient computational resources. Future work could explore ways to accelerate the guidance process while reducing computational costs without compromising performance. Additionally, we observe that the guidance scale plays a critical role in the generative process of guided diffusion for molecules, but it is generally difficult to determine in advance. This opens the possibility for research into more effective methods for selecting guidance scales, including automated scale scheduling that optimizes guidance strength.

In summary, we propose ChemGuide, which introduces guidance signals for the diffusion process from a non-differentiable quantum chemical oracle. We hope this work sparks further exploration into how domain knowledge from physics and chemistry can be more effectively integrated into neural networks to enhance molecular design.

## Appendix A Implementation Details

##### Hardware & Time

We use a 48 GiB A6000 GPU with AMD EPYC 7513 32-Core Processors for our experiments. For molecular optimization conducted with xTB (e.g., force, $\alpha$), it takes around 6 and 18 hours to sample 100 molecules on the QM9 and GEOM datasets, respectively.

##### Model

For unconditional EDM (Hoogeboom et al., ) and GeoLDM (Xu et al., ), we use the checkpoints available in the official implementations. For the conditional models (i.e., C-EDM, C-GeoLDM) and the property regressors, we follow the instructions and train them with the hyperparameters specified by the authors.

##### Evaluation Metric

We present the metrics to evaluate molecular stability and property in our experiment as follows.

For stability optimization, we report the following metrics:

- •
force RMS (Eh/Bohr): we compute the root mean square (RMS) of the forces on the atoms of a molecule; it is the lower the better.
- •
validity: it measures how many percent of the generated molecules are valid, and is calculated by either xTB or DFT and specified in the context; it is the higher the better.
- •
uniqueness: it is the percentage of unique molecules among the generated molecules (Xu et al., ); it is the higher the better.
- •
atom stability: it is the percentage of atoms that have the correct valency (Xu et al., ); it is the higher the better.
- •
molecule stability: it is the percentage of generated molecules whose atoms are all stable (Xu et al., ); it is the higher the better.
- •
energy above ground state (Eh): it is the difference between the energy before and after using xTB or DFT (specified in the context) to optimize the geometry of the molecules, and a smaller energy above ground state indicates the geometry is closer to the stable ground state of the molecule; it is the lower the better.
- •
strict validity: it is defined as generated geometries within 50 kcal/mol (0.07968 Eh) of the optimized geometries; it is the higher the better.

For property optimization on $\alpha$ (Bohr^3), $\mu$ (Debye), $C_{v}$ cal/(mol$\cdot$K), $\epsilon_{\text{HOMO}}$ (eV), $\epsilon_{\text{LUMO}}$ (eV), $\Delta\epsilon$ (eV) , we report the following metrics:

- •
MAE: we compute the Mean Absolute Error (MAE) between the target property $y$ and the predicted value $\hat{y}$ by either the regressor or xTB (specified in the context) of the generated molecules; it is the lower the better.

We use one step of guidance/optimization for our experiments (e.g., clean guidance in Section 3.1) by default. Our implementation is available at [https://anonymous.4open.science/r/ChemGuide](https://anonymous.4open.science/r/ChemGuide).

## Appendix B Related Work

##### Molecule Generation

Previous works focus on generating molecules as SMILES strings (Kusner et al., ; Gómez-Bombarelli et al., ; Segler et al., ), and 2D graphs (Li et al., ; Jo et al., ), where various models, such as VAE (Simonovsky & Komodakis, ; Jin et al., ; ), GAN (De Cao & Kipf, ; Prykhodko et al., ), diffusion models (Vignac et al., ; Kong et al., ; Zhao et al., ), have been proposed to serve as the generative backbone. To model the spatial structure of molecules, G-Schnet (Gebauer et al., ) and G-SphereNet (Luo & Ji, ) use autoregressive models to generate the 3D coordinates of the molecules, however, they are less powerful compared with diffusion models. Hoogeboom et al. () propose to use the equivariant diffusion model (EDM) for 3D molecule generation, which utilizes an equivariant graph neural network (Satorras et al., ) to model the molecules as a 3D graph, with coordinates and atom types as its node features. GeoLDM (Xu et al., ) further extends EDM to latent diffusion, and has shown improvements in stability and validity of the generated molecules. However, to achieve conditional generation, both EDM and GeoLDM need to be re-trained with labels, where the target property value is appended to the feature space as auxiliary information, which guides the generation toward fulfilling certain molecular property requirements.

##### Guided Diffusion

Guided diffusion has shown promising results in vision (Bansal et al., ), where the guidance can be provided by texts (Rombach et al., ), poses and edges (Zhang et al., ), classifiers (Dhariwal & Nichol, ), etc. Similar approaches are adapted for diffusion-based molecule generation, where explicitly guided diffusion (Hoogeboom et al., ; Xu et al., ) trains a conditional model with labels, and implicitly guided diffusion (Vignac et al., ; Bao et al., ) combines an unconditional model with a conditional property network, which provides guidance as gradients to the diffusion process at inference. Gruver et al. () propose guided discrete diffusion for
multi-objective protein sequence optimization, and
Han et al. () achieve multi-property guidance by modeling property relations as probabilistic graphs. Weiss et al. () investigate guided diffusion and demonstrate the capability to design molecules beyond the seen property distribution, while MOOD (Lee et al., ) and GCD (Klarner et al., ) incorporate out-of-distribution control to improve generalization of the property predictor. However, unlike ChemGuide, all of the above works require a labeled dataset to train a property network; thus, inevitably constrained by the generalization ability of the neural networks.

## Appendix C GFN2-xTB

GFN stands for, respectively, geometry optimization, vibrational frequencies, and non-covalent interactions. xTB refers to extended tight binding, and 2 refers to the version. In the GFN2-xTB method, the total energy expression is given by Bannwarth et al. (; )

$$ $\displaystyle E_{\text{GFN2-xTB}}$ $\displaystyle=E_{\text{rep}}+E_{\text{disp}}+E_{\text{EHT}}+E_{\text{IES}+ \text{IXC}}+E_{\text{AES}}+E_{\text{AXC}}+G_{\text{Fermi}}$ (18) $$

where $E_{\text{rep}}$ is the repulsive energy contribution from short-range interactions, $E_{\text{disp}}$ is the dispersion energy contribution from long-range interactions, $E_{\text{EHT}}$ is the energy contribution from the extended Hückel theory (EHT), $E_{\text{IES}+\text{IXC}}$ is the isotropic electrostatic (IES) energy contribution and the isotropic exchange-correlation (IXC) energy contribution, $E_{\text{AES}}$ is the anisotropic electrostatic (AES) energy contribution, $E_{\text{AXC}}$ is the anisotropic exchange-correlation (AXC) energy contribution, and $G_{\text{Fermi}}$ is the entropic contribution of an electronic free energy at finite electronic temperature $T_{\text{el}}$ due to Fermi smearing.

Its accuracy and efficiency come strictly from the element-speciﬁc and global parameters for all elements up to radon (Z = 86) Bannwarth et al. (), hence the semi-empiricism. The pre-computed tight-binding parameters and empirical corrections are utilized to approximate the electronic structure and calculate energy contributions efficiently.

## Appendix D Motivation for Stability Optimization

The geometries of the generated molecules have domain-specific implications—a molecule’s stability and properties depend significantly on its preferred quantum geometric states, i.e., atomic and molecular geometries. For example, polarity is closely related to molecular geometry. A water molecule H2O has a stable V-shaped H-O-H geometry of 104.5 degrees and is thus polar. A generated H2O molecule with a linear H-O-H geometry would instead be nonpolar but unstable. Therefore, when we discuss molecules and their properties, it is essential to start from their preferred stable geometries (ground states), which motivates our work to improve the stability of the generated molecular geometries in the diffusion process.

## Appendix E Chemistry reasoning behind the bilevel property optimization

The task of generating molecules with target properties is closely related to the field of rational chemical design such as drug design. Domain knowledge is widely used to guide the design process. For example, benzene rings that contain oxygen (e.g., dioxins) and sulfur (e.g., thiophenes) are typically known to be toxic and carcinogenic. Therefore, when designing drugs, it is typical to stay away from such species prior to performing further property analysis such as bind-affinity to a specific protein or DNA. In such design processes, the overall structures are considered by experts before further target property analysis.

The conflict behind generating stable molecules with target properties is that properties are closely related to both atom types $\mathbf{h}$ and atom coordinates $\mathbf{x}$. We are mostly concerned with the properties of a stable molecule with an optimized geometry but it is not known a priori given the atom types. In the diffusion process to generate molecules, the atom types are generated and stabilized during earlier steps, and their coordinates are further optimized in later steps. Since our target is to generate stable molecules with desired properties, there is a conflict between stability-first and property-first in the generation process.

The bilevel property and stability optimization proposed here takes inspiration from the rational design process: higher-level overall structure before lower-level property detail analysis. An overall molecular structure is generated with desired properties in earlier steps, where the diffusion model is considered as chemistry/biology/etc. experts performing the initial structure-property analysis. In the later steps, the geometries are further optimized.

## Appendix F Analyses

### F.1 Acceleration Methods for ChemGuide

GPUs can significantly accelerate computation for neural network inference and training, but xTB operates on the CPU, which lowers the running speed of the inference. Hence, we propose a skip-step acceleration method for xTB-guided optimization. Specifically, suppose we set the skip-step to be $k$, then we only calculate gradients from xTB for every $k$ step. We use the best scale (=0.0001) found in Table 1 for the skip-step acceleration experiment, and the results are shown in Table 7. As we skip more steps, the performance becomes worse and approaches unconditional GeoLDM without any guidance. Hence, there is a trade-off between time and performance: the more guidance from xTB we add to the generation, the better performance we can end with. This also demonstrates the effectiveness and necessity of the xTB guidance: more guidance almost monotonically enhances performance.

**Table 7: Metrics of 500 generated molecules from QM9 using GeoLDM with non-differentiable oracle (i.e. xTB) guidance. * and bold denote the overall best result and our best result, respectively. Percentage changes between our results and GeoLDM are shown in parentheses.**
| Metric | Every N step (s) | Baseline |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 1 | 3 | 5 | EDM | GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0104^∗ (-6.76%$\downarrow$) | 0.0106 | 0.0011 | 0.0114 | 0.0111 |
| Validity | 91.40%^∗ (1.60%$\uparrow$) | 91.40% | 89.40% | 86.60% | 89.80% |
| Uniqueness | 100.00%^∗ | 99.79% | 100.00% | 100.00% | 100.00% |
| Atom Stability | 99.02% | 99.01% | 98.63% | 98.53% | 98.93% |
| Molecule Stability | 90.60% | 90.60% | 87.20% | 83.60% | 88.80% |
| Energy above ground state (Eh) | 0.0042^∗ (-15.78%$\downarrow$) | 0.0045 | 0.0055 | 0.0072 | 0.0050 |

### F.2 Generalization Analysis of Neural Regressors

Figure: Figure 6: Percentage change of $\alpha$ predicted by the neural regressor when adding various noise.
Refer to caption: x6.png

As observed in Table 11 in Appendix I and Table 13 in Appendix J, our results are not always stable and sometimes collapse (see the collapse analysis in Section F.4). In this section, we provide an analysis of the neural network used for gradient calculation, with the results shown in Fig. 6.
We sampled 1,000 molecules from QM9 and added Gaussian noise with varying variance (1e-4, 1e-3, 1e-2, 1.0, 2.0, 3.0, 4.0) to their positions. We then evaluated the neural regressor and xTB, calculating the percentage change in predicted values from those predicted when no noise was added. For visualization purposes, we clipped the maximum percentage change to be less than 500%. The line represents the mean of the percentage changes, while the shaded area (magnitude) represents the range from maximum to minimum. As the noise scale increases, the regressors become increasingly unstable, with percentage changes reaching as high as 100%. This instability can explain the fluctuating MAE in our results, as we use the neural regressor during the generation process, where the molecules are noisier and less optimized than those in the QM9 dataset. Consequently, the regressor may provide suboptimal or even incorrect guidance during the early stages of the generation.
By comparing the performance of the regressor and xTB, it is clear that xTB provides more stable predictions, as both the mean and the magnitude of instability remain lower than those of the regressor. This helps explain the performance differences in Table 4 and Table 6, where ChemGuide performs worse than $\alpha$: the reegressor produces more stable results in $\alpha$. However, ChemGuide performs better in $\Delta\epsilon$ because the regressor becomes more stable at larger noise scales, and xTB, on average, begins to “prune” invalid molecules when the noise scale is large, ceasing to provide guidance (i.e., gradients) to these invalid molecules.

This behavior is evident from the “peak” in xTB prediction magnitude around noise scale = 1.0, after which the magnitude quickly decays and converges to the mean, while the magnitude of the regressor’s performance continues to increase. This trend is more obvious in $\Delta\epsilon$, where a -100% change is observed, indicating that xTB either throws an error on invalid molecules, in which case the gradient is 0, or returns negligible values for distorted molecules, and its guidance diminishes and stops further changes to molecular structures.

This validates our motivation for using a non-differentiable chemistry oracle to guide generation and optimization. A chemistry oracle, based on deterministic chemistry-rule calculations, can better capture the properties of molecules, even for previously unseen ones, and will not return high scores on invalid molecules, which further distort the molecular structures.

### F.3 Generalization Analysis of Conditional GeoLDM

Figure: Figure 7: Percentage changes of MAE of 200 generated molecules using C-GeoLDM with various perturbed scales.
Refer to caption: x7.png

As discussed in the introduction (Section 1), explicit guidance requires training a conditional model with labels; thus, the model is likely to be not robust and does not generalize well to distribution shifts. It performs poorly when conditioned on property scores that were not encountered during training. To demonstrate this, we perturb the sampled context from the property distribution on which C-GeoLDM is conditioned and generate 200 molecules. Since the sampled property scores are standardized with z-score normalization, we shift the context by $\pm 1,\pm 2,\pm 3$ (which means we look at the $\pm 1\cdot\sigma$, $\pm 2\cdot\sigma$, and $\pm 3\cdot\sigma$ regions of the property score distribution). We then plot the percentage change in MAE compared to the case with no distribution shift (i.e., perturb scale=0) in Fig. 7.
For percentage changes greater than 1000%, we clip the values (for example, in the case of $\mu$, the percentage change exceeds 1000%, so it is not fully displayed). The percentage change in MAE can be as high as 50% even within the standard deviation of $\pm 1\cdot\sigma$, and it increases significantly when the distribution is shifted by $\pm 2\sigma$, which suggests C-GeoLDM is prone to be affected by distribution shifts; thus, unlikely to generalize well when conditioned on scores beyond the property distribution seen during training.

### F.4 Analysis of Gradient Collapse

Figure: Figure 8: Gradient collapse of $\alpha$ using GeoLDM with noisy neural guidance.
Refer to caption: x8.png

It’s observed in Table. 11 in Appendix I that some gradients of noisy neural guidance and clean neural guidance collapse for some scales. For example, scale 5, 20, 25, 30 when optimizing $\alpha$, so we explain why such collapses happen in this section via a case study of optimizing $\alpha$ with scale 10 (optimal scale with no collapse), 30, 40, and 50. As we increase the scale, the MAE grows quickly in the first 50 steps and the molecules collapse because the added gradient is so high that it pulls the atom apart and distorts the molecule structure. Hence, the guidance scale for each property must be carefully searched and selected so that it provides strong enough guidance while maintaining a valid molecule structure.

### F.5 More Clean Guidance Optimization Steps

Recall that the clean guidance is provided in Eq.10, where we use K steps of gradient descent (GD). In this section, we experiment with more GD steps. For each property, we select the scale that results in the lowest MAE without collapse, as shown in Table11. For example, we choose a scale of 0.1 for $\alpha$. The results are presented in Fig. 9.

To investigate the effect of additional recurrent steps and to prevent a single collapse from stopping the entire generation process, we divide the molecules into groups of 50 and calculate the MAE on a group-wise basis. In the figure, ”High” indicates that the MAE is too large to be displayed properly, and a circle ($\circ$) means that the MAE is either too low or too high; thus, it is classified as an outlier. For instance, in the case of $\mu$, most MAEs are high when GD step=5, except for one group with an MAE of 0.12.
In terms of the lowest MAE a group can achieve, using more GD steps tends to decrease the MAE, as observed for $\alpha$ and $\Delta\epsilon$, where the lowest MAE is reduced by approximately 30% for $\alpha$ and 55% for $\mu$ (even though the lowest MAE for $\mu$ is already small, it can still be significantly reduced from 0.20 with GD step=1 to 0.09 with GD steps=3). However, in terms of average MAE, since clean guidance inherently has higher instability during generation (with details and analysis of this instability available in Fig.3 and Section I), adding more GD steps amplifies this effect. This is most evident in the case of $C_{v}$, where all groups with GD steps=3 and 10 collapse, and the MAE becomes exceptionally high when GD=5. Moreover, adding more steps does not guarantee a monotonically better MAE, as seen in $\epsilon_{LUMO}$, where 3 steps yield the optimal result while steps=5 increases the variance and mean. Therefore, selecting the number of GD steps and guidance scales carefully for each property is essential, which we leave for future work.

Figure: Figure 9: Box plots of MAEs of 500 generated molecules using clean guidance with more GD steps to optimize all 6 properties. First, second (median), third quantile are plotted.
Refer to caption: x9.png

## Appendix G Bilevel Optimization with Clean Guidance

As shown in Section 4.3.2, our clean guidance method can achieve significantly lower MAE than the baseline. However, its ability to drastically lower MAE also contributes to unstable molecular geometries. To illustrate this, we pick two properties as a case study from Table 4 which are easiest ($\mu$) and hardest ($\alpha$) to optimize using clean guidance, and the results are shown in Table 8. Comparing the MAE and force RMS across two experiments in each property optimization, we observe that the bilevel optimization framework helps reduce force RMS and MAE at the same time, indicating that optimizing force/property yields better results on the other. However, we can observe that the force RMS of bilevel optimization is still higher than the baseline (right block of Table 8, with 200 molecules generated), but this is because the force RMS of clean guidance is much higher than the baseline (left block of Table 8, with 500 molecules generated); thus, ChemGuide in bilevel optimization can’t minimize the force significantly to beat the baseline, which is the natural hardness of optimizing force after clean guidance as the clean guidance is so strong that it significantly changes the molecular structures, making it challenging for xTB to optimize. We notice a trade-off between force and property optimization when comparing the performance across $\alpha$ and $\mu$ in bilevel optimization: lower force yields higher MAE while lower MAE brings higher force. Hence, given the difficulty of optimizing force and MAE at the same time to beat the baseline, it’s important and tricky to pick the best scales or schedules for force and property optimization, which we leave to future work. Similarly, we report full numerical results of the case study on $\alpha$ and $\mu$ optimization to Table 15 in Appendix J.

**Table 8: Comparisons of force RMS and MAE of generated molecules from QM9 using baselines, unconditional GeoLDM with clean neural guidance, bilevel optimization with clean neural guidance.**
| Models | Clean Neural Guidance | Bilevel w/ Clean Neural Guidance |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\alpha$ $(\text{Bohr}^{3})$ | $\mu$ (D) | $\alpha$ $(\text{Bohr}^{3})$ | $\mu$ (D) |  |  |  |  |  |  |
| Force RMS<br>(Eh/Bohr) | MAE | Force RMS<br>(Eh/Bohr) | MAE | Force RMS<br>(Eh/Bohr) | MAE | Force RMS<br>(Eh/Bohr) | MAE |  |  |
| Baselines | C-EDM | 0.0118 | 2.6308 | 0.0122 | 1.1257 | 0.0125 | 2.5089 | 0.0123 | 1.0571 |
| C-GeoLDM | 0.0117 | 2.5551 | 0.0120 | 1.1084 | 0.0135 | 2.5593 | 0.0123 | 1.1582 |  |
| Ours | 0.0326 | 4.9992 | 0.3962 | 0.4025 | 0.0274 | 4.2058 | 0.1348 | 0.3150 |  |

## Appendix H Comprehensive results on non-differentiable oracle guidance

We report the distributions of force RMS and energy above ground state of 500 generated molecules from QM9 dataset and GEOM dataset using unconditional GeoLDM with ChemGuide using various scales (0.0001, 0.001, 0.01, 0.1, and 1.0) in Fig. 10, numerical results are shown in Table 1 and Table 2 in Section 4.2. We observe that smaller scales produce better results while larger scales (i.e. 1.0) produce worse results than the baselines. This is because the position values of the molecules are small (less than 1.0); thus, when the scale of the gradients is similar to position values, adding too much guidance results in “over-optimizing” the position and distorting the molecule structures. This can be validated in Fig. 10: as the scale increases, the molecules become less concentrated around 0, spreading out and exhibiting higher energy above ground states and force RMS. The distribution shifts downward, indicating lower validity, and its peak moves away from 0, suggesting reduced stability. The performance, worse than the baselines, implies that the molecular structures are being distorted due to excessive guidance. The results for GeoLDM with ChemGuide on properties are presented in Table 9.

Figure: Figure 10: Histograms and distributions of force RMS and energy above ground state of 500 generated molecules from QM9 dataset using unconditional GeoLDM with ChemGuide using various scales. Energy change refers to energy above ground state.
Refer to caption: x10.png

**Table 9: MAE of 200 generated molecules on QM9 using GeoLDM with ChemGuide on all six properties, with GeoLDM as the baseline. $*$ and bold denote the overall best and the best within different scales. Percentage changes between our results and GeoLDM are shown in parentheses.**
| Property | $\alpha$ | $\mu$ | $\epsilon_{\text{HOMO}}$ | $\epsilon_{\text{LUMO}}$ | $\Delta\epsilon$ |  |
| --- | --- | --- | --- | --- | --- | --- |
| Units | $\text{Bohr}^{3}$ | D | meV | meV | meV |  |
| GeoLDM | 4.7555^∗ | 1.5490 | 0.6264^∗ | 2.1790 | 2.1568 |  |
| GeoLDM w/ ChemGuide | scale = $1.0\times 10^{-4}$ | 4.9045 | 1.5335 | 0.7228 | 2.0091^∗<br>(-7.80%$\downarrow$) | 1.9321 |
| scale = $1.0\times 10^{-3}$ | 4.8790 | 1.5311 | 0.7182 | 2.0183 | 1.9318^∗<br>(-10.43%$\downarrow$) |  |
| scale = $1.0\times 10^{-2}$ | 4.8781<br>(2.58%$\uparrow$) | 1.5295^∗<br>(-1.26%$\downarrow$) | 0.7449 (150) | 2.0548 | 1.9428 |  |
| scale = $1.0\times 10^{-1}$ | 4.9025 | 1.5979 | 0.6961 | 2.0136 | 1.9607 |  |
| scale = $1.0\times 10^{+0}$ | 5.2251 | 1.6631 | 0.6392<br>(2.04%$\uparrow$) | 2.0298 | 2.0416 |  |

**Table 10: MAE of 200 generated molecules on GEOM using GeoLDM with ChemGuide on all six properties, with GeoLDM as the baseline. $*$ and bold denote the overall best and the best within different scales. Percentage changes between our results and GeoLDM are shown in parentheses.**
| Property | $\alpha$ | $\mu$ | $\epsilon_{\text{HOMO}}$ | $\epsilon_{\text{LUMO}}$ | $\Delta\epsilon$ |  |
| --- | --- | --- | --- | --- | --- | --- |
| Units | $\text{Bohr}^{3}$ | D | meV | meV | meV |  |
| GeoLDM | 24.3424 | 3.3676 | 1.1922^∗ | 5.0903 | 1.9956 |  |
| GeoLDM w/ ChemGuide | scale = $1.0\times 10^{-4}$ | 26.2032 | 3.3632 | 1.2939<br>(8.53%$\uparrow$) | 5.0802^∗<br>(-0.20%$\downarrow$) | 1.9381 |
| scale = $1.0\times 10^{-3}$ | 26.1301 | 3.3881 | 1.2974 | 5.1477 | 1.9616 |  |
| scale = $1.0\times 10^{-2}$ | 25.9853 | 3.3412^∗<br>(-0.78%$\downarrow$) | 1.3032 | 5.2268 | 1.9264^∗<br>(-3.47%$\downarrow$) |  |
| scale = $1.0\times 10^{-1}$ | 25.8031 | 3.4581 | 1.3002 | 5.3704 | 1.9445 |  |
| scale = $1.0\times 10^{+0}$ | 24.2168^∗<br>(-0.52%$\downarrow$) | 3.7300 | 1.2992 | 5.1828 | 2.1446 |  |

## Appendix I Comprehensive results on neural guidance

We present full numeric results of unconditional GeoLDM using noisy and clean neural guidance with various scales in Table 11. It is observed that larger scales yield better results while smaller scales produce results closer to unconditional GeoLDM. This is intuitive because larger scales provide more guidance to the generation, which brings better properties; on the other hand, since we are using unconditional GeoLDM as the backbone, the guidance becomes more negligible when the scale is small and thus the performance converges to unconditional GeoLDM. ✗  in Table 11 means that the molecules collapse in the middle of the generation, so there are no output molecules. Explanation and case study of collapse can be found in Section F.4 and Section F.2. To better investigate the effect of larger scales, we split 500 generated molecules into 10 groups so that a single collapse will not stop the generation of all molecules, we put the remaining number of molecules in the parenthesis in Table 11 and report the MAE of the molecules in groups without collapse. For example, 3.2309 (450) in $\alpha$ of scale=5 means there are 450 molecules in groups with no collapse and their MAE is 3.2309.

We plot the MAE trajectories of optimizing all six properties in Fig. 11. Applying noisy neural guidance results in much smoother trajectories compared to the baseline. It reduces MAE significantly during the first 50 epochs and then stabilizes, making noisy guidance most effective in the early stages of generation. Clean neural guidance also lowers MAE in the first 100 steps to some extent, but it fluctuates more than noisy guidance (and sometimes more than the baseline, as in the trajectory of optimizing $\epsilon_{LUMO}$). However, clean guidance becomes more powerful in the final 50 steps, where we observe a sharp drop in MAE across all six properties. The difference in smoothness is because, in noisy guidance, the gradient is propagated through the denoising estimation (Eq.8) and the VAE decoder (Algorithm1), inherently accounting for the effect of noise in the gradient calculation, which leads to consistent guidance throughout the generation steps. On the contrary, in clean guidance, the gradient is computed in the clean space (Eq.11) and then projected back into the generation process by adding noise (Eq.10). Since the gradient does not propagate through the denoising estimation and VAE decoder, it is more sensitive to noise, resulting in greater fluctuations and inconsistency during guidance. Noisy guidance stabilizes and doesn’t exhibit a sharp drop in later steps because the loss decreases as the MAE reduces in the early stages, so the gradient is small, which mainly serves to maintain the current molecular structure, preventing the MAE from converging to that of GeoLDM. This can be observed in Table 11, where the MAE for smaller scales is closer to that of unconditional GeoLDM. Although clean guidance helps reduce MAE in earlier steps of generation, it reaches its full potential in the later steps, when the molecules are more similar and close to decoded molecules, leading to the sharp drop in MAE.

Figure: Figure 11: MAE trajectories of 500 generated molecules using GeoLDM and GeoLDM with clean&noisy guidance with optimal configuration. We use the scales with no collapse as optimal scales; for example, the optimal scale of optimizing $\alpha$ is 10.0 instead of 25.0.
Refer to caption: x11.png

**Table 11: MAE of all six properties of 500 sampled molecules from GeoLDM with noisy and clean neural guidance. We use C-EDM, C-GeoLDM, and GeoLDM as baselines. $*$ and bold denote the overall best and the best within different scales, respectively. ✗ marks the situation where the guidance collapses. Percentage changes between our results and C-GeoLDM are shown in parenthesis**
| Property | $\alpha$ | $\mu$ | $C_{v}$ | $\epsilon_{\text{HOMO}}$ | $\epsilon_{\text{LUMO}}$ | $\Delta\epsilon$ |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Units | $\text{Bohr}^{3}$ | D | $\frac{\text{cal}}{\text{mol}}\text{K}$ | meV | meV | meV |  |
| Conditional EDM | 2.6308 | 1.1257 | 1.0804 | 0.3207 | 0.5940 | 0.6301 |  |
| Conditional GeoLDM | 2.5551 | 1.1084 | 0.9703 | 0.3327 | 0.5518 | 0.5878 |  |
| GeoLDM | 5.6732 | 1.6461 | 3.1046 | 0.6151 | 1.1778 | 1.2022 |  |
| GeoLDM w/ Noisy Guidance | scale = $1.0\times 10^{-4}$ | 5.6890 | 1.4784 | 3.1046 | 0.6136 | 1.1834 | 1.1725 |
| scale = $1.0\times 10^{-3}$ | 5.6703 | 1.4743 | 3.1002 | 0.6155 | 1.1864 | 1.1683 |  |
| scale = $1.0\times 10^{-2}$ | 5.6547 | 1.4669 | 3.0923 | 0.6190 | 1.1739 | 1.1770 |  |
| scale = $1.0\times 10^{-1}$ | 5.1373 | 1.5102 | 2.7785 | 0.6183 | 1.1509 | 1.1319 |  |
| scale = $1.0\times 10^{+0}$ | 4.1372 | 1.3248 | 1.7806 | 0.5965 | 1.0523 | 1.1033 |  |
| scale = $2.0\times 10^{+0}$ | 3.6602 | 1.2153 | 1.5683 | 0.5763 | 0.9916 | 1.0504 |  |
| scale = $5.0\times 10^{+0}$ | 3.2309 (450) | 1.0331 | 1.2369 | 0.5644 | 0.9834 | 1.0362 |  |
| scale = $1.0\times 10^{+1}$ | 2.3870 | 0.8201 (450) | 1.0358 | 0.5667 | 0.8554 | 0.9102 |  |
| scale = $2.0\times 10^{+1}$ | 1.9312 (400) | 0.6251 (350) | 0.8804 | 0.4735 | 0.8061 | 0.8325 |  |
| scale = $2.5\times 10^{+1}$ | 1.5284 (100)^∗<br>(-40.1824%$\downarrow$) | 0.5770 (300) | 0.7112 | 0.4796 | 0.7436 | 0.8596 |  |
| scale = $3.0\times 10^{+1}$ | ✗ | 0.4733 (250)<br>(-57.2988%$\downarrow$) | 0.7094 (450)<br>(-26.8886%$\downarrow$) | 0.4725 | 0.8115 | 0.7468 |  |
| scale = $4.0\times 10^{+1}$ | ✗ | ✗ | ✗ | 0.4153 (450)<br>(24.8272%$\uparrow$) | 0.7496 | 0.7101 (200)<br>(20.8064%$\uparrow$) |  |
| scale = $5.0\times 10^{+1}$ | ✗ | ✗ | ✗ | 0.4325 (300) | 0.7015 (150)<br>(27.1294%$\uparrow$) | ✗ |  |
| GeoLDM w/ Clean Guidance | scale = $1.0\times 10^{-4}$ | 5.7236 | 1.3925 | 2.9780 | 0.6380 | 1.1587 | 1.2056 |
| scale = $1.0\times 10^{-3}$ | 5.7373 | 1.3914 | 2.9779 | 0.6374 | 1.1578 | 1.2055 |  |
| scale = $1.0\times 10^{-2}$ | 5.6894 | 1.4102 | 2.9734 | 0.6367 | 1.1558 | 1.2024 |  |
| scale = $1.0\times 10^{-1}$ | 5.4798 | 1.3384 | 2.9506 | 0.6343 | 1.1576 | 1.2096 |  |
| scale = $1.0\times 10^{+0}$ | 2.4980 (200)<br>(-2.23%$\downarrow$) | 0.7144 | 2.7440 | 0.6203 | 1.1292 | 1.1345 |  |
| scale = $2.0\times 10^{+0}$ | 1320.7350 | 0.4475 | 2.3713 | 0.6040 | 1.0853 | 1.0609 |  |
| scale = $5.0\times 10^{+0}$ | 5309.9470 (200) | 0.2031 | 1.6211 | 0.5533 | 0.9519 | 0.8676 |  |
| scale = $1.0\times 10^{+1}$ | ✗ | 0.1631 (100)^∗<br>(-85.29%$\downarrow$) | 0.9668 | 0.4817 | 0.7369 | 0.6690 |  |
| scale = $2.0\times 10^{+1}$ | ✗ | 2179.0210 (100) | 0.5311 | 0.3740 | 0.5688 | 0.4313 (300) |  |
| scale = $2.5\times 10^{+1}$ | ✗ | 2010.2822 (100) | 0.4524 (250) | 0.3359 | 0.5580 (300) | 0.3830 |  |
| scale = $3.0\times 10^{+1}$ | ✗ | ✗ | 0.3151 | 0.3186 | 0.5747 | 0.2852 (100)^∗<br>(-51.48%$\downarrow$) |  |
| scale = $4.0\times 10^{+1}$ | ✗ | ✗ | 0.6815 (250) | 0.3109 | 0.5374 (300)<br>(-2.61%$\downarrow$) | 28.5487 (200) |  |
| scale = $5.0\times 10^{+1}$ | ✗ | ✗ | 0.2044 (250)^∗<br>(-78.93%$\downarrow$) | 0.2386 (300)^∗<br>(-28.28%) | 0.7159 (100) | 0.4831 (100) |  |

## Appendix J Comprehensive results on bilevel optimization

In this section, we present MAE trajectories and detailed results of force metrics and MAE for explicit bilevel optimization and implicit bilevel optimization with noisy guidance, and we provide full numeric results of MAE and force metrics of implicit bilevel optimization with clean guidance. Recall that we only sample 200 molecules for experiments in this section, as mentioned in Section 4.4.

##### Explicit guided diffusion with ChemGuide

The MAE and force metrics are shown in Table 13 and Table 12 respectively. Since we only add ChemGuide to C-GeoLDM, our expectation is that we can achieve lower forces without increasing MAE. However, the results go beyond our expectations: our method not only achieves better force RMS, energy above ground state, and validity, but also reduces MAE in the first 5 properties. This shows the effectiveness of combining explicit property optimization with ChemGuide, and verifies the feasibility of our bilevel optimization framework.

##### Implicit guided diffusion with noisy guidance and ChemGuide

The MAE and force metrics of bilevel optimization with noisy guidance are shown in Table 13 and Table 14 respectively. For ease of searching scales for noisy guidance and to better explore larger scales, we employ an ensemble method: based on the results from unconditional GeoLDM with noisy guidance in Table 11, we pick the top-4 scales that give the lowest MAE, and then sample 50 molecules using each scale and aggregate the results. For instance, for $\alpha$, we pick scales of 25.0 (Top 1), 20.0 (Top 2), 10.0 (Top 3), and 5.0 (Top 4). Overall, we achieve better force metrics than the baselines. For each property, there exists at least one scale with a large improvement in force metrics; for $\alpha$, $\mu$, and $C_{v}$, we can find scales that produce improvement in both force and MAE such as scale=20 for $\alpha$, scale=20.0 for $\mu$, and scale=25 for $C_{v}$, indicating that our bilevel optimization with noisy guidance does succeed in optimizing both property and force. The results for MAE are consistent with GeoLDM with noisy guidance in the sense that top scales in general yield lower MAE and, the same as Table 11, our bilevel optimization with noisy guidance performs better than the baseline in $\alpha$, $\mu$, and $C_{v}$ while worse than the baseline in the other three properties. Moreover, the percentage change of the best scale in both tables are similar yet bilevel optimization with noisy guidance yields slightly worse results than GeoLDM with noisy guidance, which suggests adding guidance for force will slightly contradict the guidance for MAE as the the guidance tries to find a balance between property and stability optimization.

To illustrate how bilevel optimization works, we provide MAE trajectories of each property in Fig. 12. Compared to MAE trajectories in unconditional GeoLDM with noisy guidance in Fig. 11, the trajectories of bilevel optimization with noisy guidance behave similarly in the sense that the MAE is reduced significantly during the first 50 guidance steps and stabilizes. The difference is that the trajectories of bilevel optimization with noisy guidance have slightly larger fluctuation during the early steps of guidance. We reason that in the early steps, the molecules are less stable, so ChemGuide provides larger guidance such that the guidance from ChemGuide and noisy guidance interact with each other and result in fluctuated trajectories. The trajectories are also more converged to GeoLDM because the generation process leans less toward a lower MAE, as it keeps a balance between property and stability optimization, which can be verified from 14.

##### Implicit guided diffusion with clean guidance

Finally, we present the algorithm of bilevel optimization with clean guidance and ChemGuide in Algorithm 2. The detailed results of optimizing $\alpha$ and $\mu$ are shown in Table 15. Recall that $\alpha$ is the most challenging to optimize and $\mu$ is the easiest according to Table 4. In both properties, it’s obvious that increasing guidance scales results in lower MAE yet larger force RMS, which is because the effect of clean guidance exceeds that of ChemGuide, so it’s a trade-off between optimizing force and property. Comparing the two methods of the two properties, we notice that optimizing $\alpha$ yields better force RMS yet higher MAE while optimizing $\mu$ results in worse force RMS yet lower MAE, which is another trade-off between optimizing force and property. In each property, although we can’t directly compare the performance of GeoLDM with clean guidance and bilevel optimization with clean guidance because we sample 500 molecules from the former and 200 from the latter, we can compare them with their respective baselines. We notice that by adding ChemGuide on force in bilevel optimization, we achieve lower force RMS percentage changes from the baselines. Surprisingly, the MAE of bilevel optimization with clean guidance is lower than that of GeoLDM with clean guidance, which seems to conflict with our findings in bilevel optimization with noisy guidance, we suspect that this is because, as the molecules are optimized to be more stable and thus more similar to the denoised ones, clean guidance can obtain more power in reducing the MAE, as explained in Section I. However, there is a collapse in the bilevel optimization of $\mu$ with scale 10 while it doesn’t happen in GeoLDM with clean guidance, we suspect that this is due to the exceptionally high force RMS in GeoLDM with clean guidance, so ChemGuide provides much higher gradient and the guidance from ChemGuide (pushing toward lower force RMS) and clean guidance (pushing towards higher force RMS) conflict, resulting in the collapse of molecular structures. Nevertheless, although the force RMS of bilevel optimization with clean guidance is still higher than the baselines, it is already high in GeoLDM with clean guidance and we can reduce it by adding ChemGuide, which, together with lower MAE, indicates the feasibility of combining ChemGuide with clean guidance and our success in obtaining both better property and force metrics.

Figure: Algorithm 2 Bilevel guided diffusion sampling with clean neural guidance

Figure: Figure 12: MAE trajectories of 200 generated molecules using GeoLDM and bilevel optimization with noisy guidance on GeoLDM with optimal configuration. We use the scales with no collapse as optimal scales.
Refer to caption: x12.png

**Table 12: Force metrics of 200 generated molecules from QM9 dataset using C-EDM, C-GeoLDM, and explicit bilevel optimization. Each sub-table contains results for one property that the model is conditioned on. From up to bottom: $\alpha$, $\mu$, $C_{v}$, $\epsilon_{HOMO}$, $\epsilon_{LUMO}$, $\Delta\epsilon$. $*$ and bold denote the overall best result and our best result, respectively. Percentage changes between our results and GeoLDM are shown in parenthesis.**
| Metric | Guidance scale ($\alpha$) | Baseline ($\alpha$) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0.0001 | 0.001 | 0.01 | 0.1 | 1.0 | C-EDM | C-GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0123 | 0.0124 | 0.0122^∗ (-9.21%$\downarrow$) | 0.0127 | 0.0131 | 0.0125 | 0.0135 |
| Validity | 90.00%^∗ (7.00%$\uparrow$) | 90.00% | 88.00% | 87.50% | 89.50% | 83.50% | 83.00% |
| Uniqueness | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 99.47% | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 98.87%^∗ (0.63%$\uparrow$) | 98.81% | 98.76% | 98.57% | 98.87% | 98.44% | 98.23% |
| Molecule Stability | 89.50%^∗ (8.50%$\uparrow$) | 89.00% | 87.50% | 87.50% | 87.00% | 82.50% | 81.00% |
| Energy above ground state (Eh) | -0.0083 | -0.0085 | -0.0079^∗ (-13.93%$\downarrow$) | -0.0084 | -0.0104 | -0.0094 | -0.0092 |
| Metric | Guidance scale ($\mu$) | Baseline ($\mu$) |  |  |  |  |  |
| 0.0001 | 0.001 | 0.01 | 0.1 | 1.0 | C-EDM | C-GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0120 | 0.0120 | 0.0122 | 0.0119^∗ (-3.33%$\downarrow$) | 0.0136 | 0.0123 | 0.0123 |
| Validity | 82.00% | 82.00% | 83.50% | 85.00% (-1.50%$\downarrow$) | 83.50% | 83.00% | 86.50%^∗ |
| Uniqueness | 100.00%^∗ | 100.00%%^∗ | 100.00%%^∗ | 100.00%%^∗ | 100.00%%^∗ | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 98.18% | 98.18% | 98.23% | 98.32%^∗ (0.03%$\uparrow$) | 97.96% | 97.48% | 98.29% |
| Molecule Stability | 80.00% | 80.00% | 81.50% | 82.00% (-2.00%$\downarrow$) | 77.50% | 79.00% | 84.00%^∗ |
| Energy above ground state (Eh) | -0.0106 (16.72%$\uparrow$) | -0.0106 | -0.0112 | -0.0111 | -0.0116 | -0.0113 | -0.0091^∗ |
| Metric | Guidance scale ($C_{v}$) | Baseline ($C_{v}$) |  |  |  |  |  |
| 0.0001 | 0.001 | 0.01 | 0.1 | 1.0 | C-EDM | C-GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0111 | 0.0111 | 0.0111^∗ (-8.75%$\downarrow$) | 0.0112 | 0.0123 | 0.0117 | 0.0121 |
| Validity | 86.00% | 86.00% | 86.50% | 88.50%^∗ (0.50%$\uparrow$) | 87.00% | 75.80% | 88.00% |
| Uniqueness | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 98.48% | 98.48% | 98.57% | 98.68%^∗ (0.11%$\uparrow$) | 98.57% | 97.43% | 98.57% |
| Molecule Stability | 85.00% (-1.00%$\downarrow$) | 85.00% | 85.50% | 84.50% | 84.00% | 74.50% | 86.00%^∗ |
| Energy above ground state (Eh) | -0.0067 | -0.0066 | -0.0063^∗ (-21.39%$\downarrow$) | -0.0066 | -0.0092 | -0.0097 | -0.0081 |
| Metric | Guidance scale ($\epsilon_{HOMO}$) | Baseline ($\epsilon_{HOMO}$) |  |  |  |  |  |
| 0.0001 | 0.001 | 0.01 | 0.1 | 1.0 | C-EDM | C-GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0106^∗ (-6.69%$\downarrow$) | 0.0106 | 0.0108 | 0.0112 | 0.0128 | 0.0118 | 0.0113 |
| Validity | 86.00% | 86.00% | 87.00% | 90.00%^∗ (3.00%$\uparrow$) | 85.50% | 82.00% | 87.00% |
| Uniqueness | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 99.46% | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 98.81% | 98.76% | 98.84%^∗ (0.33%$\uparrow$) | 98.79% | 98.40% | 98.17% | 98.51% |
| Molecule Stability | 87.50% | 87.00% | 88.00%^∗ (4.00%$\uparrow$) | 88.00% | 85.00% | 80.50% | 84.00% |
| Energy above ground state (Eh) | -0.0057^∗ (-27.87%$\downarrow$) | -0.0057 | -0.0061 | -0.0060 | -0.0125 | -0.0090 | -0.0079 |
| Metric | Guidance scale ($\epsilon_{LUMO}$) | Baseline ($\epsilon_{LUMO}$) |  |  |  |  |  |
| 0.0001 | 0.001 | 0.01 | 0.1 | 1.0 | C-EDM | C-GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0119 | 0.0117 | 0.0115^∗ (-1.27%$\downarrow$) | 0.0124 | 0.0141 | 0.0117 | 0.0117 |
| Validity | 81.50% | 81.50% | 82.00% | 79.50% | 86.50%^∗ (2.00%$\uparrow$) | 83.00% | 84.50% |
| Uniqueness | 100.00%^∗ | 100.00%^∗ | 99.45% | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 98.12% | 98.15% | 98.18% | 97.74% | 98.62%^∗ (0.30%$\uparrow$) | 98.20% | 98.32% |
| Molecule Stability | 79.50% | 80.00% | 80.50% | 78.00% | 84.00% (-0.50%$\downarrow$) | 81.50% | 84.50%^∗ |
| Energy above ground state (Eh) | -0.0079^∗ (-0.60%$\downarrow$) | -0.0079 | -0.0079 | -0.0102 | -0.0096 | -0.0083 | -0.0079 |
| Metric | Guidance scale ($\Delta\epsilon$) | Baseline ($\Delta\epsilon$) |  |  |  |  |  |
| 0.0001 | 0.001 | 0.01 | 0.1 | 1.0 | C-EDM | C-GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0104 | 0.0106 | 0.0104 | 0.0101^∗ (-10.19%$\downarrow$) | 0.0121 | 0.0115 | 0.0112 |
| Validity | 88.50% | 89.00% (-1.50%$\downarrow$) | 88.50% | 87.50% | 83.50% | 86.00% | 90.50%^∗ |
| Uniqueness | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 98.87%^∗ (0.08%$\uparrow$) | 98.79% | 98.76% | 98.46% | 98.04% | 98.33% | 98.79% |
| Molecule Stability | 88.50%^∗ | 88.00% | 88.00% | 86.00% | 81.00% | 81.50% | 88.50% |
| Energy above ground state (Eh) | -0.0061^∗ (-16.41%$\downarrow$) | -0.0065 | -0.0064 | -0.0065 | -0.0094 | -0.0072 | -0.0073 |

**Table 13: MAE of all 6 properties of 200 sampled molecules from explicit bilevel optimization and implicit bilevel optimization with noisy guidance. The ”Top” scales of each property can be found in Table. 11. We use C-EDM and C-GeoLDM as baselines. $*$ and bold denote the overall best and the best within different scales, respectively. ✗ marks the situation where the guidance collapses. Percentage changes between our results and C-GeoLDM are shown in parenthesis**
| Property | $\alpha$ | $\mu$ | $C_{v}$ | $\epsilon_{\text{HOMO}}$ | $\epsilon_{\text{LUMO}}$ | $\Delta\epsilon$ |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Units | $\text{Bohr}^{3}$ | D | $\frac{\text{cal}}{\text{mol}}\text{K}$ | meV | meV | meV |  |
| Conditional EDM | 2.5089 | 1.0571 | 1.0624 | 0.3304 | 0.5046 | 0.6191 |  |
| Conditional GeoLDM | 2.5593 | 1.1582 | 0.9646 | 0.3407 | 0.5927 | 0.4982 |  |
| GeoLDM | 5.7481 | 1.4821 | 3.1630 | 0.6042 | 1.1602 | 1.1967 |  |
| Explicit Bilevel Optimization | scale = $1.0\times 10^{-4}$ | 2.5449 | 1.1151 | 0.9468 | 0.3554 | 0.5173 | 0.5327 |
| scale = $1.0\times 10^{-3}$ | 2.5304 | 1.1180 | 0.9393<br>(-2.62%$\downarrow$) | 0.3554 | 0.5306 | 0.5225<br>(4.88%$\uparrow$) |  |
| scale = $1.0\times 10^{-2}$ | 2.4684 | 1.1246 | 0.9727 | 0.3517 | 0.5077<br>(-14.34%$\downarrow$) | 0.5354 |  |
| scale = $1.0\times 10^{-1}$ | 2.5394 | 1.0483<br>(-9.49%$\downarrow$) | 0.9843 | 0.3404<br>(-0.09%$\downarrow$) | 0.5317 | 0.5400 |  |
| scale = $1.0\times 10^{+0}$ | 2.4430<br>(-4.54%$\downarrow$) | 1.0493 | 0.9694 | 0.3690 | 0.5288 | 0.5695 |  |
| Implicit Bilevel Optimization<br>w/ Noisy Guidance | Top 1 | 2.1493 | 0.6364^∗<br>(-45.05%$\downarrow$) | 0.9290 | 0.4358 | 0.8359 | 0.7890 |
| Top 2 | 1.9703^∗<br>(-23.01%$\downarrow$) | 0.7012 | 0.7649^∗<br>(-20.70%$\downarrow$) | 0.4217<br>(23.17%$\uparrow$) | 0.8037 | 0.6428 |  |
| Top 3 | 2.8100 | 0.7712 | 0.9795 | 0.4375 | 0.7598<br>(28.19%$\uparrow$) | 0.7663 |  |
| Top 4 | 3.0144 | 0.8594 | 1.0742 | 0.4832 | 0.8404 | 0.6182<br>(24.09%$\uparrow$) |  |
| All | 2.4860<br>(-2.86%$\downarrow$) | 0.7246<br>(-37.43%$\downarrow$) | 0.9369<br>(-2.87%$\downarrow$) | 0.4445<br>(30.47%$\uparrow$) | 0.8099<br>(36.64%$\uparrow$) | 0.7041<br>(41.33%$\uparrow$) |  |

**Table 14: Force Metrics of all 6 properties of 200 sampled molecules from implicit bilevel optimization with noisy guidance. The ”Top” scales of each property can be found in Table. 11. We use C-EDM and C-GeoLDM as baselines. $*$ and bold denote the overall best and the best within different scales, respectively. Percentage changes between our results and C-GeoLDM are shown in parenthesis**
| Metric | Guidance scale ($\alpha$) | Baseline ($\alpha$) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Top 1 (25.0) | Top 2 (20.0) | Top 3 (10.0) | Top 4 (5.0) | All | C-EDM | C-GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0181 | 0.0132 | 0.0114 | 0.0099^∗ (-26.47%$\downarrow$) | 0.0132 (-1.94%$\downarrow$) | 0.0125 | 0.0135 |
| Validity | 92.00%^∗ (9.00%$\uparrow$) | 80.00% | 88.00% | 88.00% | 87.00% (4.00%$\uparrow$) | 83.50% | 83.00% |
| Uniqueness | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 97.87% | 98.54% | 98.77% | 99.44%^∗ (1.21%$\uparrow$) | 98.66% (0.42%$\uparrow$) | 98.44% | 98.23% |
| Molecule Stability | 88.00% | 86.00% | 88.00% | 92.00%^∗ (11.00%$\uparrow$) | 88.50% (7.50%$\uparrow$) | 82.50% | 81.00% |
| Energy above ground state (Eh) | -0.0087 | -0.0047 | -0.0043 | -0.0028^∗ (-69.26%$\downarrow$) | -0.0052 (-43.72%$\downarrow$) | -0.0094 | -0.0092 |
| Metric | Guidance scale ($\mu$) | Baseline ($\mu$) |  |  |  |  |  |
| Top 1 (30.0) | Top 2 (25.0) | Top 3 (20.0) | Top 4 (10.0) | All | C-EDM | C-GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0122 | 0.0153 | 0.0095^∗ (-23.12%$\downarrow$) | 0.0098 | 0.0117 (-4.57%$\downarrow$) | 0.0123 | 0.0123 |
| Validity | 92.00% | 94.00% | 84.00% | 96.00%^∗ (9.50%$\uparrow$) | 91.50% (5.00%$\uparrow$) | 83.00% | 86.50% |
| Uniqueness | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 97.48% | 97.88% | 98.55%^∗ (0.26%$\uparrow$) | 97.71% | 97.91% (-0.38%$\downarrow$) | 97.48% | 98.29% |
| Molecule Stability | 84.00% | 84.00% | 86.00%^∗ (2.00%$\uparrow$) | 84.00% | 84.50% (0.50%$\uparrow$) | 79.00% | 84.00% |
| Energy above ground state (Eh) | -0.0035 | -0.0108 | -0.0027 | -0.0025^∗ (-72.31%$\downarrow$) | -0.0050 (-45.45%$\downarrow$) | -0.0113 | -0.0091 |
| Metric | Guidance scale ($C_{v}$) | Baseline ($C_{v}$) |  |  |  |  |  |
| Top 1 (30.0) | Top 2 (25.0) | Top 3 (20.0) | Top 4 (10.0) | All | EDM | GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0131 | 0.0105 | 0.0106 | 0.0099^∗ (-18.40%$\downarrow$) | 0.0110 (-8.97%$\downarrow$) | 0.0117 | 0.0121 |
| Validity | 92.00%^∗ (4.00%$\uparrow$) | 82.00% | 88.00% | 92.00%^∗ (4.00%$\uparrow$) | 88.50% (0.50%$\uparrow$) | 75.80% | 88.00% |
| Uniqueness | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 96.95% | 96.73% | 98.32% | 98.79%^∗ (0.22%$\uparrow$) | 97.69% (-0.88%$\downarrow$) | 97.43% | 98.57% |
| Molecule Stability | 80.00% | 72.00% | 84.00% | 88.00%^∗ (2.00%$\uparrow$) | 81.00% (-5.00%$\downarrow$) | 74.50% | 86.00% |
| Energy above ground state (Eh) | -0.0139 | -0.0036 | -0.0037 | -0.0025^∗ (-68.78%$\downarrow$) | -0.0060 (-24.96%$\downarrow$) | -0.0097 | -0.0081 |
| Metric | Guidance scale ($\epsilon_{HOMO}$) | Baseline ($\epsilon_{HOMO}$) |  |  |  |  |  |
| Top 1 (40.0) | Top 2 (50.0) | Top 3 (30.0) | Top 4 (20.0) | All | EDM | GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0102 | 0.0101 | 0.0099^∗ (-12.33%$\downarrow$) | 0.0103 | 0.0101 (-10.66%$\downarrow$) | 0.0118 | 0.0113 |
| Validity | 92.00%^∗ (5.00%$\uparrow$) | 84.00% | 92.00%^∗ (5.00%$\uparrow$) | 84.00% | 88.00% (1.00%$\uparrow$) | 82.00% | 87.00% |
| Uniqueness | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 98.77%^∗ (0.26%$\uparrow$) | 97.87% | 98.77%^∗ (0.26%$\uparrow$) | 98.54% | 98.49% (-0.02%$\downarrow$) | 98.17% | 98.51% |
| Molecule Stability | 88.00% | 86.00% | 90.00%^∗ (6.00%$\uparrow$) | 86.00% | 87.50% (3.50%$\uparrow$) | 80.50% | 84.00% |
| Energy above ground state (Eh) | -0.0032 | -0.0028 | -0.0030 | -0.0028^∗ (-64.91%$\downarrow$) | -0.0029 (-62.59%$\downarrow$) | -0.0090 | -0.0079 |
| Metric | Guidance scale ($\epsilon_{LUMO}$) | Baseline ($\epsilon_{LUMO}$) |  |  |  |  |  |
| Top 1 (50.0) | Top 2 (25.0) | Top 3 (40.0) | Top 4 (30.0) | All | EDM | GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0096 | 0.0100 | 0.0100 | 0.0095^∗ (-18.39%$\downarrow$) | 0.0098 (-16.26%$\downarrow$) | 0.0117 | 0.0117 |
| Validity | 96.00%^∗ (11.50%$\uparrow$) | 94.00% | 96.00%^∗ (11.50%$\uparrow$) | 92.00% | 94.50% (10.00%$\uparrow$) | 83.00% | 84.50% |
| Uniqueness | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 99.08% | 99.22% | 97.88% | 99.54%^∗ (1.22%$\uparrow$) | 98.92% (0.61%$\uparrow$) | 98.20% | 98.32% |
| Molecule Stability | 94.00% ^∗(9.50%$\uparrow$) | 90.00% | 84.00% | 94.00%^∗ (9.50%$\uparrow$) | 90.50% (6.00%$\uparrow$) | 81.50% | 84.50% |
| Energy above ground state (Eh) | -0.0021^∗ (-73.79%$\downarrow$) | -0.0024 | -0.0026 | -0.0025 | -0.0024 (-69.62%$\downarrow$) | -0.0083 | -0.0079 |
| Metric | Guidance scale ($\Delta\epsilon$) | Baseline ($\Delta\epsilon$) |  |  |  |  |  |
| Top 1 (40.0) | Top 2 (30.0) | Top 3 (20.0) | Top 4 (25.0) | All | EDM | GeoLDM |  |
| Force RMS (Eh/Bohr) | 0.0104 | 0.0104 | 0.0109 | 0.0099^∗ (-11.47%$\downarrow$) | 0.0104 (-7.18%$\downarrow$) | 0.0115 | 0.0112 |
| Validity | 88.00% | 96.00%^∗ (5.50%$\uparrow$) | 92.00% | 92.00% | 92.00% (1.50%$\uparrow$) | 86.00% | 90.50% |
| Uniqueness | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ | 100.00%^∗ |
| Atom Stability | 97.47% | 98.04% | 98.73% | 99.02%^∗ (0.23%$\uparrow$) | 98.32%(-0.47%$\downarrow$) | 98.33% | 98.79% |
| Molecule Stability | 80.00% | 88.00%^∗ (-0.50%$\downarrow$) | 88.00 | 88.00%^∗ | 86.00%(-2.50%$\downarrow$) | 81.50% | 88.50%^∗ |
| Energy above ground state (Eh) | -0.0034 | -0.0036 | -0.0026^∗ (-64.64%$\downarrow$) | -0.0028 | -0.0031 (-57.32%$\downarrow$) | -0.0072 | -0.0073 |

**Table 15: MAE and force RMS of $\alpha$ and $\mu$ of 500 sampled molecules from GeoLDM with clean guidance and 200 sampled molecules from implicit bilevel optimization with clean guidance. The ”Top” scales of each property can be found in Table. 11. We use C-EDM and C-GeoLDM as baselines. $*$ and bold denote the overall best and the best within different scales, respectively. ✗ marks the situation where the guidance collapses.**
| Model ($\alpha$) | Guidance scale | Baseline ($\alpha$) |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Top 1 (1.0) | Top 2 (0.1) | Top 3 (0.01) | Top 4 (0.0001) | All | C-EDM | C-GeoLDM |  |  |  |
|  | Clean Guidance | Force RMS (Eh/Bohr) | 0.1086 | 0.0179 | 0.0112 | 0.0112 | 0.0326 | 0.0118 | 0.0117 |
|  | MAE | 2.4980 | 5.4798 | 5.6894 | 5.7236 | 4.9992 | 2.6308 | 2.5551 |  |
|  | Bilevel Optimization<br>w/ Clean Guidance | Force RMS (Eh/Bohr) | 0.0972 | 0.0205 | 0.0101 | 0.0111 | 0.0274 | 0.0125 | 0.0135 |
|  | MAE | 1.6182 | 4.2294 | 4.9078 | 4.9764 | 4.2058 | 2.5089 | 2.5593 |  |
| Model ($\mu$) | Guidance scale | Baseline ($\mu$) |  |  |  |  |  |  |  |
| Top 1 (10.0) | Top 2 (5.0) | Top 3 (2.0) | Top 4 (1.0) | All | C-EDM | C-GeoLDM |  |  |  |
|  | Clean Guidance | Force RMS (Eh/Bohr) | 1.1042 | 0.2684 | 0.2760 | 0.0836 | 0.3962 | 0.0122 | 0.0120 |
|  | MAE | 0.1631 | 0.2031 | 0.4475 | 0.7144 | 0.4025 | 1.1257 | 1.1084 |  |
|  | Bilevel Optimization<br>w/ Clean Guidance | Force RMS (Eh/Bohr) | ✗ | 0.1454 | 0.1524 | 0.1109 | 0.1348 (150) | 0.0123 | 0.0123 |
|  | MAE | ✗ | 0.1335 | 0.2452 | 0.5420 | 0.3150 (150) | 1.0571 | 1.1582 |  |

## Appendix K Molecule Visualization

We present the visualization of generated molecules with ChemGuide for stability optimization in Fig. 13 and 14, and bilevel optimization for molecular property and stability with noisy neural guidance and ChemGuide in Fig. 15.

Figure: Figure 13: Molecules generated from QM9 using unconditional GeoLDM with ChemGuide.
Refer to caption: x13.png

Figure: Figure 14: Molecules generated from GEOM using unconditional GeoLDM with ChemGuide.
Refer to caption: x14.png

Figure: Figure 15: Molecules generated from QM9 using bilevel optimization with noisy neural guidance and ChemGuide for unconditional GeoLDM.
Refer to caption: x15.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see Weaknesses
soundness: 3
strengths: 1. CHEMGUIDE’s use of quantum chemistry as a non-differentiable oracle in conditional diffusion models is meaningful.
2. The paper reports improvements in stability and force metrics over baselines.
3. Implementing zeroth-order optimization in a diffusion context is well-justified.
summary: The paper proposes CHEMGUIDE, a approach that uses non-differentiable guidance for the conditional generation of molecular structures in diffusion models. CHEMGUIDE use quantum chemistry oracles as guidance, providing gradients for sampling distributions. The method applies zeroth-order optimization to enhance molecule stability by minimizing atomic forces. Experiments demonstrate CHEMGUIDE’s effectiveness on two datasets.
weaknesses: 1. The paper could enhance its rigor by comparing CHEMGUIDE with more baseline models, such as MolGAN, and more importantly, other existing guidance methods. The current comparisons seem limited. It would also be comprehensive to experiment with more datasets.
2. While GFN2-xTB is a reasonable compromise, comparing CHEMGUIDE results against high-accuracy methods like DFT more extensively could help validate the chemical accuracy of generated molecules.
3. The paper lacks a thorough discussion on the limitations of using a non-differentiable oracle, such as the potential difficulty in handling certain molecular configurations or diverse chemical spaces. 
4. The use of the GFN2-xTB method and bilevel optimization adds computational complexity, which could restrict practical usage. And The guidance scale parameter lacks an adaptive mechanism. Exploring automated scale scheduling would improve usability.
5. Code is not provided.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Is there a typo in the numerator in Eq. 15 when approximating the gradient? It should say $\frac{ \mathcal{F}[z_{x,t} + c U, z_{h,t}] - \mathcal{F}[z_{x,t} - c U, z_{h,t}] )}{2c}$.

In Algorithm 1, the approximated gradient g_{t-1} has a dependency to the state at time $t=0$. Is this is a typo, since Eq. 15 does not refer to the clean data prediction.
soundness: 2
strengths: The usage of approximate gradients from non-differentiable oracles in diffusion guidance for molecule design is novel and interesting.
To evaluate their proposed method, the authors run a suite of multiple experiments showing that on the two common benchmarks the generated samples based on their sampling algorithm have improved evaluation metrics compared to the baselines.
summary: The authors propose CHEMGUIDE, a sampling algorithm to guide pre-trained (latent) diffusion models for molecule design with the goal to optimize for stable molecules indicated by smaller force norms when evaluated using the xTB oracle functions. As the xTB oracle function, which outputs the forces per atom in a molecule, is non-differentiable, the authors make use of known gradient approximation from random pertubation theory to approximate the gradients suitable for guidance during the diffusion sampling trajectory. The authors also suggest how their non-differentiable guidance can be combined with neural regressors as commonly done in the diffusion models literature. The authors show that they non-differentiable guidance when applied on GeoLDM leads to generated molecules with lower force norms compared to the samples when GeoLDM is used without the proposed guidance, indicating that their method works for the models trained on common benchmark datasets such as QM9 and GEOM-Drugs.
weaknesses: The guidance component from the non-differentiable oracle shows small improvement on the QM9 and GEOM-Drugs dataset. While the idea is interesting, a stronger baseline to compare against is to use the samples generated from Baseline GeomLDM and perform a relaxation using xTB. As the authors mention in their appendix A - Implementation Details, the sampling time for 100 molecules is quite slow with 6 hours and 18 minutes if they perform their proposed guidance in the last 400 diffusion timesteps using xTB as oracle.
How does the GeoLDM baseline (right column) in Table 1 and 2 compare if xTB is used using a pre-defined number of relaxation steps?

Furthermore, I found it hard to read the paper as Section 3 Methodology contains sections 3.1 and 3.2 which are not the author's contribution but already existing methods. I would move these Section 2 within the preliminaries.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: Most of my questions are about experiments, I feel the current experimental comparisons are too weak (only comparing with unconditional generation), see weaknesses.
soundness: 2
strengths: * This paper studies an important and timely problem, how to move beyond generative models (randomly sample from learned distributions) to efficient search and optimization over the space with guidance is a pressing question in molecular generation. 
* Derivative-free guidance is very well-motivated and I agree that it is of great importance to problems in real-world molecular discovery process.
summary: This paper studies how to achieve derivative-free molecular optimization with diffusion models. The main motivation for this work is that many real-world molecular properties are sophisticated and can only be evaluated through expensive experiments or non-differential simulations. In this paper, a zero-order optimization method is constructed by perturbing the input molecular conformation and the effect on the molecular properties. The effectiveness of the proposed methods is validated on a set of quantum mechanical properties for small molecules.
weaknesses: I have several concerns about this paper, mostly coming from the claim, related work, and experiments. 

* First of all, the idea of derivative-free guidance is not new, in molecular optimization, evolutionary algorithms have been used [1], twisted sequential Monte Carlo for protein design [2], twisted SMC for large language model inference [3], and stochastic optimal control method for music generation [4]. I believe the claim for this work to be the first of its kind is inappropriate, neither for derivative-free guidance nor in molecular design.

* Given this is not new, the related work section in the Appendix only discusses the general molecule generation literature and part of the guided diffusion model literature, but misses the critical relevant literature both in molecular generation and other domains.

* The experimental results are weak, even if I agree on the general motivation of derivative-free guidance, (1) there are works such as simple evolutionary algorithms [1] and twisted SMC [2] available for comparison; even if you do not want to compare against them, you need to compare with gradient-based method --- if you think about the experiment budget, you can always construct a classifier by the samples you have evaluated, e.g. a trained neural network. Despite this may not generalize OOD or perform badly, but you may still include them as baselines. For more potential baselines to compare against, you can check this benchmark [5].

[1] Schneuing, A., Du, Y., Harris, C., Jamasb, A., Igashov, I., Du, W., Blundell, T., Lió, P., Gomes, C., Welling, M. and Bronstein, M., 2022. Structure-based drug design with equivariant diffusion models. arXiv preprint arXiv:2210.13695.

[2] Wu, L., Trippe, B., Naesseth, C., Blei, D. and Cunningham, J.P., 2024. Practical and asymptotically exact conditional sampling in diffusion models. Advances in Neural Information Processing Systems, 36.

[3] Zhao, S., Brekelmans, R., Makhzani, A. and Grosse, R.B., Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo. In Forty-first International Conference on Machine Learning.

[4] Huang, Y., Ghatare, A., Liu, Y., Hu, Z., Zhang, Q., Sastry, C.S., Gururani, S., Oore, S. and Yue, Y., Symbolic Music Generation with Non-Differentiable Rule Guided Diffusion. In Forty-first International Conference on Machine Learning, 2024.

[5] https://github.com/brandontrabucco/design-bench

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: **clarifications**
- Can you clarify which aspects of your method implement SPSA? Equation 15 combines two ideas: 1) introducing random perturbations for atom coordinates from a standard normal distribution, and 2) using these perturbations in a finite-difference approximation of the gradient. It would be helpful to explicitly state which of these constitutes SPSA and how it differs from standard finite differences.

- What are the theoretical requirements for SPSA convergence? The paper mentions continuity of z as a requirement, but are there other conditions (e.g., smoothness, bounded variance) needed for the gradient estimates to be reliable?

**suggestions**
- Given the extensive appendix (sections A-K), adding a table of contents at its beginning would improve navigation.

- Consider adding a brief discussion of computational overhead introduced by SPSA compared to standard finite differences.

**nitpicks**
- Stability' is unnecessarily capitalized on page 6, section 4.2
soundness: 4
strengths: - **Strong and relevant contribution**: The idea proposed in the paper has clearly relevant applications in a number of domains (i.e. any field with expensive oracles) and naturally extends existing efforts in the field of diffusion models. 
- **Novelty**: The method is conceptually simple yet novel, opening the door for various applications which could benefit from the guidance of a non-differentiable oracle.
- **Thorough empirical evaluation**: The paper presents thorough empirical analysis, effectively demonstrating both the strengths and limitations of the proposed method. The experiments span multiple datasets (QM9 and GEOM), various molecular properties, and different guidance approaches (explicit, implicit, and combined).
-  **Extensive analysis**: the empirical observations are grounded in real-world chemistry insights, with careful analysis of failure cases and performance trade-offs.
- **Clarity of presentation**: the paper is well-written and includes many relevant and well-designed figures.
summary: The authors propose ChemGuide, a method for estimating diffusion guidance (i.e. gradients) from a non-differentiable property predictor. The goal is to eliminate the need for labeled training data typically required for property prediction networks. They demonstrate their approach in the context of 3D molecular generation. ChemGuide enables an unconditional diffusion model to generate more stable 3D molecular structures by incorporating guidance from quantum chemistry calculations (GFN2-xTB) that serve as a non-differentiable oracle.
weaknesses: - **Justification of the zeroth-order optimization and comparison to other non-gradient optimization methods**: The paper lacks clear justification for choosing SPSA as the zeroth-order optimization method. Section 3.2 would benefit from a discussion of alternative approaches (e.g., finite differences, evolution strategies) and justification for their specific choice in terms of computational efficiency and accuracy trade-offs, and suitability for this particular application of guiding molecular diffusion models. A short pragaraph would suffice here.

- **Assessing the quality of the gradients obtained with ChemGuide vs a differentiable regressor** While the paper shows final performance metrics, it lacks direct analysis comparing the gradients estimated via zeroth-order optimization to those from a differentiable regressor. Such comparison could provide insights into how reliable are CHEMGUIDE's estimated gradients compared to differentiated gradients. For example, the authors could plot the cosine similarity between CHEMGUIDE's estimated gradients and those from a differentiable regressor across different timesteps of the diffusion process

- **Explain which guidance method is suitable for which property**:  The authors observe that noisy and clean guidance methods appear complementary, with properties poorly optimized by one often well-optimized by another (e.g., α vs ∆ϵ). However, the paper would be more practically useful if it provided explanations for these differences, helping practitioners choose the appropriate method for their specific use case.

Meta-review: The paper proposes a novel guidance technique for molecule generation. Namely, it targets generating molecules closer to their equilibrium state (with lower force magnitude) by adding the proposed guidance term to the diffusion model. The main challenge of this problem is that the objective function (evaluating the force) is not differentiable. The authors propose to address this challenge by evaluating the finite difference of the objective along a random direction. Reviewers tzoc and V6At found the empirical study insufficient but still up to the ICLR standards. On the contrary, Reviewer wXP8 extensively argued for the acceptance of the paper.

Decision: Accept (Poster)