Equivariant Energy-Guided SDE for Inverse Molecular Design
Authors: Fan Bao
Min Zhao
Zhongkai Hao
Peiyao Li
Chongxuan Li
Jun Zhu
Venue: ICLR 2023 poster (2023)
Keywords: score-based model
diffusion model
inverse molecular design
energy guidance
molecule generation
equivariance

Abstract: Inverse molecular design is critical in material science and drug discovery, where the generated molecules should satisfy certain desirable properties. In this paper, we propose equivariant energy-guided stochastic differential equations (EEGSDE), a flexible framework for controllable 3D molecule generation under the guidance of an energy function in diffusion models. Formally, we show that EEGSDE naturally exploits the geometric symmetry in 3D molecular conformation, as long as the energy function is invariant to orthogonal transformations. Empirically, under the guidance of designed energy functions, EEGSDE significantly improves the baseline on QM9, in inverse molecular design targeted to quantum properties and molecular structures. Furthermore, EEGSDE is able to generate molecules with multiple target properties by combining the corresponding energy functions linearly.

Published as a conference paper at ICLR 2023
EQUIVARIANT ENERGY-GUIDED SDE FOR INVERSE MOLECULAR DESIGN
Fan Bao' ", Min Zhao' *, Zhongkai Hao', Peiyao Li', Chongxuan Li Jun Zhu' 'Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua-Huawel Joint Center for Al BNRist Center, State Key Lab for Intell. Tech. & Sys., Tsinghua University, Beijing, China GGooling School of Artificial Intelligence, Renmin University of China, Beijing, China Beijing Key Laboratory of Big Data Management and Analysis Methods Beijing, China bf1 98mail tsi iinguuugua.. edu. cn, gracezhaol 9976 gma com, [hzj21, lipyl 9 J@mai iimmilss.. wtw.s..ingguaaa.aeee aachuaa.edu.ccmm edu. cr, chongxuanli@rue edu. en, des2
eeszj@tsinghua, inghua. www.ghuaaaeeee....m..mmmm
ABSTRACT
Inverse molecular design is critical in material science and drug discovery, where the generated molecules should satisfy certain desirable properties. In this paper, we propose equivariant energy-guided stochastic differential equations (EEGSDE), a flexible framework for controllable 3D molecule generation under the guidance of an energy function in diffusion models. Formally, we show that EEGSDE naturally exploits the geometric symmetry in 3D molecular conforma- tion, as long as the energy function is invariant to orthogonal transformations. Em- pirically, under the guidance of designed energy functions, EEGSDE significantly improves the baseline on QM9, in inverse molecular design targeted to quantum properties and molecular structures. Furthermore, EEGSDE is able generate molecules with multiple target properties by combining the corresponding energy functions linearly.
INTRODUCTION
The discovery of new molecules with desired properties is critical in many fields, such the drug and material design (Hajduk & Greer 2007 Mandal et al.] 2009) Kang et al.] 2006; Pyzer-Knapp al. 2015). However, brute-force search in the overwhelming molecular space is extremely chal- lenging. Recently, inverse molecular design (Zunger 2018) provides an efficient way to explore the molecular space, which directly predicts promising molecules that exhibit desired properties.
A natural way of inverse molecular design is to train a conditional generative model (Sanchez- Lengeling & Aspuru-Guzik 2018). Formally, learns distribution of molecules conditioned on certain properties from data, and new molecules are predicted by sampling from the distribution with the condition set to desired properties. Among them, equivariant diffusion models (EDM) (Hooge- boom et al.] (002)) leverage the current state-of-art diffusion models (Ho et al. 2020), which involves a forward process to perturb data and a reverse process to generate 3D molecules conditionally or unconditionally. While EDM generates stable and valid 3D molecules, we argue that single conditional generative model is insufficient for generating accurate molecules that exhibit desired properties (see Table I and Table for an empirical verification).
In this work, we propose equivariant energy-guided stochastic differential equations (EEGSDE), a flexible framework for controllable 3D molecule generation under the guidance of energy func- tion in diffusion models. EEGSDE formalizes the generation process as an equivariant stochastic differential equation, and plugs in energy functions to improve the controllability of generation. Formally, we show that EEGSDE naturally exploits the geometric symmetry in 3D molecular con- formation, as long as the energy function is invariant to orthogonal transformations.
We apply EEGSDE to various applications by carefully designing task-specific energy functions. When targeted to quantum properties, EEGSDE able to generate more accurate molecules than
"qqual contribution. 'Correspondence to: C. Li and J. Zhu.

---

Published as a conference paper at ICLR 2023
Polarizhiliit a Dipole Moment y
Condition
Heat Capacity Ge
E(z.0.) e(R Invariant Energy
Timge Pingerprint
Quamumn Property
2T
ds - Uf00s + sttft Vilpp
(a(a,() () (VElle,e.) V,E(s.o,)) V.E(s, (f it glt)d(t,, in,)
ZO
Equivariant Energy Guidance
Figure I: Overview of our EEGSDE. EEGSDE iteratively generates molecules with desired prop- erties (represented by the condition c) by adopting the guidance of energy functions in each step. As the energy function is invariant to rotational transformation R. its gradient (i.e., the energy guid- ance) is equivariant
and therefore the distribution of generated samples is invariant R.
EDM, e.g., reducing the mean absolute error by more than 30% on the dipole moment property. When targeted to specific molecular structures, EEGSDE better capture the structure information molecules than EDM, e.g, improving the similarity target structures by more than 10%. Fur- thermore, EEGSDE is able to generate molecules targeted to multiple properties by combining the corresponding energy functions linearly. These demonstrate that our EEGSDE enables a flexible and controllable generation of molecules, providing smart way explore the chemical space.
RELATED WORK
Diffusion models are initially proposed by Sohl-Dickstein et al. (2015). Recently, they are bet- ter understood in theory by connecting it score matching and stochastic differential equations (SDE) (Ho et al., 2020 Song et al. 2020). After that, diffusion models have shown strong em- pirical performance in many applications Dhariwal Nichol (202T): Ramesh al. (2022): Chen et al.. 2220); Kong et al. (202)) There are also variants proposed to improve or accelerate diffusion
models NNichll Dhariwal 2021 Vahdat et al. 2021 Dockhorn et al. 202 Bao et al. 2022b
Salimans & Ho 2022 Lu al. 2022).
Guidance a technique to control the generation process diffusion models. Initially, Song et al. (020); Dhariwal & Nichol (221I) use classifier guidance to generate samples belonging to class. Then, the guidance is extended to CLIP (Radford et al., 2021) for text to image generation, and semantic-aware energy (Zhao al. 2022) for image-to-image translation. Prior guidance methods focus on image data, and are nontrivial to apply to molecules, since they do not consider the geo- metric symmetry. contrast, our work proposes general guidance framework for 3D molecules, where an invariant energy function is employed to leverage the geometric symmetry of molecules.
Molecule generation. Several works attempt model molecules 3D objects via deep genera- tive models Nesterov et al. (2020); Gebauer et al. (2019); Satorras et al. (2021a): Hoffmann & Noë (2019; Hoogeboom et (2022). Among them, the most relevant one the equivariant diffusion model (EDM) (Hoogeboom et al. 2022), which generates molecules in an iterative denoising man- ner. Benefiting from recent advances of diffusion models, EDM stable to train and is able to generate high quality molecules. We provide formal description of EDM in Section 3] Some other methods generate simplified representations molecules, such as ID SMILES strings (Weininger) 1988) and 2D graphs of molecules. These include variational autoencoders (Kusner et al.) 2017
Dai et al. 2018 Jin et al. 2018 Simonovsky Komodakis 2018 Liu et al. 2018), normalizing
flows (adhhaw al. 2019 Zang & Wang 2020 Luo et al. (021). I) generative adversarial net-
works (Bian et al. 2019 Assouel et al., 2018). and autoregressive models (Popova et al,, 2019)
Flam-Shepherd et al. 2021). There also methods generating torsion angles in molecules. For
2

---

Published as a conference paper at ICLR 2023
instance, Torsional Diffusion (Jing et al. 2022) employs the SDE formulation of diffusion models to model torsion angles in a given 2D molecular graph for the conformation generation task.
Inverse molecular design. Generative models have been applied to inverse molecular design. For example, conditional autoregressive models (Gebauer et al. 2022) and EDM (Hoogeboom et al. 2022) directly generate 3D molecules with desired quantum properties. Gebauer et al. (2019) also finetune pretrained generative models on a biased subset to generate 3D molecules with small HOMO-LUMO gaps. In contrast to these conditional generative models, our work further proposes a guidance method, a flexible way to control the generation process of molecules, Some other meth- ods apply optimization methods to search molecules with desired properties, such as reinforcement learning (Zhou et al. [2119] You et al.) (2018) and genetic algorithms (Jensen) [2019] Nigam et al.) 2019). These optimization methods generally consider the 1D SMILES strings or 2D graphs of molecules, and the 3D information is not provided.
3 BACKGROUND
3D representation of molecules. Suppose a molecule has M atoms and let x² E R (n = 3 in general) be the coordinate of the ith atom. The collection of coordinates x (x ..... xاا RMn determines the conformation of the molecule. In addition to the coordinate, each atom is also associated with an atom feature, e.g., the atom type. We use h' R represent the atom feature of the ith atom, and use h = (h' ..... hM)) R Md to represent the collection of atom features in a molecule. We use a tuple information and the atom feature information.
(z, h) to represent molecule, which contains both the 3D geometry
Equivariance and invariance. Suppose R a transformation. distribution p(ar, h)) is said to be invariant to R. if p(x, h) = p( (Ræ, h) holds for all æ and h. Here Ræ = (Rx' ..... Rx is applied to each coordinate. function (a², a²) = f (x., h) that have two components a", a in its output is said to be equivariant to R. if f [ax h) (Ra a²) holds for all æ and h. A function f(x h) is said to be invariant to R, if (Rx, h) = f(x, h) holds for all æ and h.
Zero CoM subspace. It has been shown that the invariance to translational and rotational trans- formations is an important factor for the success of 3D molecule modeling (Köhler et al., 2020 Xu et al. 2022). However, the translational invariance is impossible for distribution in the full space M (Satorras et al. 2021a). Nevertheless, we can view two collections of coordinates æ and y as equivalent if x can be translated from y, since the translation doesn't 't change the identity of a molecule. Such an equivalence relation partitions the whole space R-dnn into disjoint equiva- Ience classes. Indeed, all elements in the same equivalence classes represent the same conformation, and we can use the element with zero center of mass (CoM), i.e., M ΣM x² O, as the spe- cific representation. These elements collectively form the zero CoM linear subspace X (Xu et al.) 2022 Hoogeboom et al. 2022), and the rest of the paper always uses elements in X to represent conformations.
Equivariant graph neural network. Satorras et al. (021b) propose equivariant graph neural nenworks (EGNNs), which incorporate the equivariance inductive bias into neural networks. Specif- ically, (a", a²) = EGNN(ar, h) is a composition of I, equivariant convolutional layers. The I-th layer takes the tuple (ar,, h4) the input and outputs an updated version (I1+1, (11+1,h+1). as follows:
m -D. ///, ||o|
at 12,, e3; Om)) wh = D. On)), hit1 t (h/, Σ
u m On):
J
x of
a/41
Σ j ||||
Du (hj, (hi, h/ ||a - c' ;z)))
- x 2l2 +1
where Pm, P... Dn T are parameterized by fully connected neural networks with parameters 0m, Ow, 0n Or respectively, and et are optional feature attributes. We can verify that these layers are cquivariant to orthogonal transformations, which include rotational transformations as special cases. As their composition, the EGNN is also equivariant to orthogonal transformations. Fur- thermore, let EGNN" (z, h) a i.e., the second component in the output of the EGNN. Then EGNN" h) is invariant to orthogonal transformations.
3

---

Published as a conference paper at ICLR 2023
Equivariant diffusion models (EDM) (Hoogeboom et al. 2022) variant of diffusion models for molecule data. EDMs gradually inject noise to the molecule x = (z, h) via a forward process
(())N/1Z0== - II (=nn|Zn-1) a( kn-1) =Nx=))>xxxxx (x
(I)
where On and 3n represent the noise schedule and satisfy an + 3, = 1. and Wxx represent the Gaussian distribution in the zero CoM subspace X (see its formal definition in Appendix (A.2). Let on and Bn BnBn-1/P To generate samples, the forward process
on 01 ... On. B,
is reversed using Markov chain:
==
P(=--111e p(mn))ppZnnlnnn z2z) -Nx (2,, Ma (zn), B.). (2) n=1
Here p(=x) (m 1) (h (0, 1). The mean pm = (1 (z,, parameter-
ized by noise prediction network eØ (Zm: n2), and is trained using MSE loss, follows:
Bn
(n((n) = Van
(zn
co n)), min E,, E (os,) 1n
c..l??
where En
50 is the standard Gaussian noise injected to 2g and w(t) is the weight term. Hoogeboom et al. tional transformations if the noise prediction network is equivariant to orthogonal transformations. In Section 4.2 we extend this proposition to the SDE formulation of molecular diffusion modelling.
(022) show that the distribution of generated samples p(zo) is invariant to rota-
Hoogeboom al. (2022) also present a conditional version of EDM for inverse molecular design by adding extra input of the condition to the noise prediction network as eg (Zn, Cz,,C,n)) n.n.
EQUIVARIANT ENERGY-GUIDER SDE
In this part, we introduce our equivariant energy guided SDE (EEGSDE), as illustrated in Figure | EEGSDE is based on the SDE formulation of molecular diffusion modeling, which described in Section 4. and Section 4.2 Then, we formally present our EEGSDE that incorporates an energy function to guide the molecular generation in Section
We provide derivations Appendix A
4.1 SDE IN THE PRODUCT SPACE
Recall that a molecule is represented as a tuple z = (xc, h), where x = -w=,, xM E X E Rddd represents atom features. Here X {a E R- M E x = }} is the zero CoM subspace mentioned in Section 3 and is the feature dimension. We first introduce space X x IRMe Md which gradually adds noise to and h. This can be described by the forward SDE:
represents the conformation and (h' ........e h hM)
aattnuuous-time diffusion process {z1 (oc)sss in the product
dz = f(t)zdt + g(t)d(w.. wx),
((z)).
where (t) and g(t) are two scalar functions, wr and wA are independent standard Wiener processes in X and add respectively, and the SDE starts from the data distribution q( (2)). Note that w,, can be constructed by subtracting the CoM of a standard Wiener process w in IR Mx
1.dn,i.e i.e., W2, w w. where M ΣM w² is the CoM of u (w² (w.,.... u. M). It can be shown that the SDE has linear Gaussian transition kernel q(z,/z,) q(x,, (ar,(ar,)) (r,))(h,,h)))) from æ, to 2t, where S T. Specifically. there exists two scalars 0gjs and Bi.. . s.t., g(xx (x,) = NNxxxel (aal B11s) and q( h2) (Nhh, aajj, 3..)) Here NNx denotes the Gaussian distribution in the subspace X, and see Appendix AA. for its formal definition. Indeed, the forward process EDM in Eq. (I is discretization of the forward SDE in Eq. (3).
S
4

---

Published as a conference paper at ICLR 2023
To generate molecules, reverse Eq. (3) from T to O. Such time reversal forms another SDE, which can be represented by both the score function form and the noise prediction form:
d= [[(())= ==(()² (vn log ((() V (t(). Va log (.))))) ( i.n),
score function form
=[f(t)z+ + g())²
VPz0 Et
Jtt + +()dd((( w.)) 2T (r(T))
(4)
noise prediction form
Here (=) is the marginal distribution of 2t, Va l: is the gradient of log (z()) win.. a Va log (z) = M Ei= ae lo q: (z) the CoM of Va log (=(2), dt is the infinitesimal nega- tive timestep, a and wh are independent reverse-time standard Wiener processes in X and R-dd respectively, and E₁ 2, voro= is the standard Gaussian noise injected to Zu- Compared to the /3jo
original SDE introduced by Song al. (2020), our reverse SDE Eq. additionally subtracts the CoM of V log (z). This ensures always stays in the zero CoM subspace as time flows back.
To sample from the reverse SDE in Eq. 4.. we use noise prediction network cØ(z,, f) estimate E. Ct, through minimizing the MSE loss ming E, s: we(t) (e(o))leo(z
eall where t is uniformly sampled from [), and wi(() controls the weight of the loss term at time t. Note that the noise e, is in the product space
R R-Md, dd so we subtract the CoM of the predicted noise of x; to
ensure Eo cø(2,, (2,, t) is also in the product space.
Substituting EØ (z 1) into Eq. (4), we get an approximate reverse-time SDE parameterized by 0: dz = [f())= + g(t)²
VBelo (z, t)]dt + ((t)d(w,, wp), 2T ~ pT(=))
(5)
Gaussian prior in the product space that approxi- mates @r (z())) We define pe (zo) as the marginal distribution of Eq. (5) at time = O, which is the distribution of our generated samples. Similarly to the forward process, the reverse process of EDM
where pr pr(zT) (x ()r
in Eq. 2 is discretization of the reverse SDE in Eq. (5).
4.2 EQUIVARIANT SDE
To leverage the geometric symmetry in 3D molecular conformation, (o(2)) should be invariant to translational and rotational transformations. As mentioned in Section 3] the translational invariance of (=())) ) is already satisfied by considering the zero CoM subspace. The rotational invariance can be satisfied if the noise prediction network is equivariant to orthogonal transformations, as summa- rized in the following theorem:
Theorem 1. Let (E)(z,, t), EZ(ZE, t)) eø(z;,t) t), where (z(z,, and ( are the predicted noise of Kt and h; respectively. for any orthogonal transformation R E Raxx, cø(zi,) t) is equivariant R, i.e., co (Rx,, h,,t) (R h1,t), cg ( ht,t)), and pr(zr) is invariant to R, i.e., pr (Ræy, hr) )= = pr(xr, h.), then (z() is invariant to any rotational transformation.
As mentioned in Section 3 the EGNN satisfies the equivariance constraint, and we parameterize (z,, t) using an EGNN following Hoogeboom et al. (2022). See details in Appendix D
4.3 EQUIVARIANT ENERGY-GUIDE SDE
Now we describe equivariant energy-guided SDE (EEGSDE), which guides the generated molecules of Eq. (5) towards desired properties c by leveraging a time-dependent energy function E(z, c, t):
dz [f(t)z gg(t)(
tz
B30
(V.E(z,c,t) VaE(z, tnE( g(()d((ū. wh), ZT ~ pT(#T), (6)
energy gradient taken in the product space
While (z) is defined in X x Rdd its domain can be extended to MR x Rddd and the gradient is valid.
See Remark [ Appendix A. for details.
5

---

Published as a conference paper at ICLR 2023
which defines distribution po (z(2l)) conditioned on the property Here the CoM V Ez, c,t) of the gradient subtracted to keep the SDE in the product space, which ensures the translational invariance of (o(zo.)). Besides, the rotational invariance is satisfied by using energy invariant to orthogonal transformations, as summarized in the following theorem:
Theorem 2. Suppose the assumptions in Theorem 7 hold and E(z,c, c, is invariant to any or- thogonal transformation R. i.e., F. (R., h c,t) h, Then po is invariant any rotational transformation.
Note that we can also use a conditional model E0 e) in Eq. (6 See Appendix C for details.
To sample from pa 2g l). various solvers can be used for Eq. (6) such the Maler-Maruyama method (Song et al. 2020) and the Analytic-DPM sampler (Bao et al. 2022b We present the
Euler-Mane Malry-aa method as an example in Algorithm T Appendix
4.4 How TO DESIGN THE ENERGY FUNCTION
Our EEGSDE is a general framework, which can be applied to various applications by specifying different energy functions c,), For example, we can design the energy function according to consistency between the molecule z and the property c, where a low energy represents a well consistency. As the generation process in Eq. (6) proceeds, the gradient of the energy function encourages generated molecules to have a low energy, and consequently a well consistency. Thus, can expect the generated molecule aligns well with the property In the rest of the paper, we specify the choice of energy functions, and show these energies improve controllable molecule generation targeted to quantum properties, molecular structures, and even combination of them.
Remark. The term "energy" in this paper refers to a general notion in statistical machine learning, which is a scalar function that captures dependencies between input variables (LeCun et al., 2006). Thus, the "energy" in this paper can be set to a MSE loss when we want to capture how the molecule align with the property (as done in Section 5). Also, the "energy" in this paper does not exclude potential energy or free energy in chemistry, and they might be applicable when we want generate molecules with small potential energy or free energy.
5 GENERATING MOLECULES WITH DESIRED QUANTUM PROPERTIES
Let E be certain quantum property. To generate molecules with the desired property, we set the energy function the squared error between the predicted property and the desired property E(zz, c, = s (2, t) c|2,, where g(z,, t) is time-dependent property prediction model, and s is the scaling factor controlling the strength of the guidance. Specifically, g can be parameterized by equivariant models such as EGNN (Satorras et al. 2021b). Tr3-Transformer (Fuchs et al. 2020) and DimeNet (Klicpera et al. 2020) to ensure the invariance of E(z,, t) long they perform well in the task of property prediction. In this paper we consider EGNN. We provide details on parameterization and the training objective in Appendix E We can also generate molecules targeted to multiple quantum properties by combining energy functions linearly (see details in Appendix
5.1 SETUP
We evaluate QM9 (Ramakrishnan et al. 2014), which contains quantum properties and coordi- nates of 130k molecules with up to nine heavy atoms from (C. N, O, F). Following HDM, we split QM9 into training. validation and test sets, which include 100K, 18K and 13K samples respectively. The training set is further divided into two non-overlapping halves D.. Dd equally. The noise pre- diction network and the time-dependent property prediction model of the energy function are trained on D2 separately. By default, EEGSDE uses a conditional noise prediction network cø(z, c, t) in Eq. (6). since we find generate more accurate molecules than using an unconditional one (see Appendix G.T for an ablation study). See more details in Appendix F.3
Evaluation metric: Following EDM, we use the mean absolute error (MAE) to evaluate how gen- erated molecules align with the desired property. Specifically, we train another property prediction model Φp (Satorras et al. 2021b) on Da. and the MAE is calculated as Di (0p(z;) cal. where z; is a generated molecule, c; 18 its desired property. We generate K=10,000 samples for evaluation with dp- For fairness, the property prediction model d, is different from g( (z,, 1) used in the energy

---

Published as a conference paper at ICLR 2023
Table 1: How generated molecules align with the target quantum property. The L-bound (Hooge-
boom et al. 2022) represents the loss of Φp D, and can be viewed as lower bound of the MAE metric. The conditional EDM results are reproduced, and are consistent with Hoogeboom et al.
2022 (see Appendix G.4). "#Atoms" uses public results from Hoogeboom et al. (0022).
Method
MAE
Method
MAE]
Method
MAE]
C.
col mol
K)
# (D)
a (Bohr")
U-bound
6.879110.015
U-bound
1.61310.003
U-bound
8.9810.02
#Atoms
1.971
#Atoms
1.053
#Atoms
3,86
Conditional EDM
1.065=0.010
Conditional EDM
1.123=0.013
Conditional EDM
2.78=00044
EEGSDE (sm1)
1.037110.000
EEGSDE (s=0.5)
0,930_10.005
EEGSDE (s=0.5)
2.67100.04
EEGSDE (s=5)
0.981=0.002
EEGSDE (s= 1)
0.858=0.0006
EEGSDE (s=
2.62+10.33
EEGSDE (x=10)
0.901
EEGSDE (s=2)
0.777=00.07
EEGSDE (x=3)
2.50
L-bound
0.040
L- bound
0.043
L-bound
0.09
As (meV)
EHOMO (mcV)
ÉLUMO (mcV)
U-bound
1464+-
U-bound
645+-41
U-hound
1457:+5
#Atoms
#Atoms
426
#Atoms
813
Conditional EDM
671=5
Conditional EDM
371:2
Conditional EDM
601+7
EEGSDE (s=0.5)
574+44
EEGSDE (s=0.1)
35744
EEGSDE (==0.5)
525+:4
FRGSDD (s=l)
542+2
FRGSDR (x=0.5)
320+1
REGSDR (==l)
496+22
EEGSDE (a=3)
487-3
EEGSDE (a=1)
302:22
EEGSDE (s=3)
447:16
L--ound
65
L-bound
39
L-bound
36
Table 2: The mean absolute error (MAE) computed by the Gaussian software instead of dp-
Method
MAEJ
Method
MAEL
Method
MAE
Method
MAE,
Method
MAE,
# (D)
o (Bohr")
As (meV)
FHOMO (me
SLUMO (meV)
Conditional EDM
1.20
Conditional FDM
2.41
Conditional FDM
775
Conditional EDM
354
Conditional EDM
573
HEGSDE (s=0.5)
0.96
LEGSDE (s=0.5)
2.27
LEGSDE (x=0.5)
638
EEGSDE (s=0.1)
349
LEGSDE
495
EEGSDE (s=1)
0.78
EEGSDE (s=1)
2,03
EEGSDE (a=1)
555
BEGSDE (==0.5)
341
EEGSDE (==1)
445
EEGSDE (s=2) 0.73
EEGSDE (s=3)
1.85
EEGSDE (a=3)
532
EEGSDE (s=1) 284
EEGSDE (s=3)
Table 3: How generated molecules align with multi- ple target quantum properties.
Table 4: How generated molecules align with target structures.
Method
MAEL
MAR2J
Method
Similarity?
mal K). (D)
QM9
Conditional EDM
1.079.1.0077
1.15610.011
eG-SchNet
0.499++0.002
REGSDE (0,=10, 1-=l)
0.981++0.008
0.912-00006
Conditional EDM
0.671:10.004
EEGSDE (s=0.1) I)
0.696::0.002
Ae (me μ (D)
ERGSDE (s=0.5)
0.736:+00.002
Conditional EDM
683+11
1.130=0.007
EEGSDE (s=1.0)
0.750:10.003
BEGSDE (s)=x=11)
5631.3
0.0063
GEOM-Drug
(Bohr"), ), μ (D)
Conditional EDM
0.165:10.001
Conditional EDM
2.76:10.01 2.61+0001
1.158=0.002
EEGSDE (s=0.5)
0.185=+0.001
EEGSDE (8)=8)=1.5)
0.855=0.007
ERGSDE (s=1.0)
0.193+-0.001
function, and they are trained on the two non-overlapping training subsets Da, D3 respectively. This ensures information leak occurs when evaluating BEGSDE. To further verify the effective- ness of EEGSDE, we also calculate the MAE without relying on a neural network. Specifically, we use the Gaussian software (which calculates properties according to theories of quantum chemistry without using neural networks) to calculate the properties of 100 generated molecules, For complete- ness, we also report the novelty, the atom stability and the molecule stability following Hoogeeoomm et al. (2022) in Appendix G.2 although they are not our main focus.
Baseline: The most direct baseline conditional EDM, which only adopts a conditional noise pre- diction network. We also compare two additional baselines "U-bound" and "#Atoms" from Hooge- boom et al. (2022) (see Appendix F2 for details).
5.2 RESULTS
Following Hoogeboom et al. (022), we consider six quantum properties in QM9: polarizability a, highest occupied molecular orbital energy EHоMO, lowest unoccupied molecular orbital energy ÉLUMO, HOMÓ-LUMO gap As, dipole moment f and heat capacity C., Firstly, generate
7

---

Published as a conference paper at ICLR 2023
Taggt structure
Cand EDM
RECSDE
(ours)
Figure 2: Generated molecules on QM9 targeted to specific structures (unseen during training). The molecular structures of EEGSDE align better with target structures then conditional EDM.
molecules targeted to one of these six properties. As shown in Table with the energy guidance, our EEGSDE has significantly better MAE than the conditional EDM on all properties. Remarkably, with a proper scaling factor 8, the MAE of FEGSDE is reduced by more than 25% compared to conditional EDM properties As, ÉLUMO, and more than 30% μ.. What's more, shown in Table 2 our EEGSDE still has better MAE under the evaluation by the Gaussian software, which further verifies the effectiveness of our EEGSDE. We further generate molecules targeted to multiple quantum properties by combining energy functions linearly. As shown Table 3 our EEGSDE still has significantly better MAE than the conditional EDM.
Conclusion:
These results suggest that molecules generated by our EEGSDE align better with the desired properties than molecules generated by the conditional EDM baseline (for both the single- property and space in a guided way generate promising molecules for downstream applications such as the virtual screening, which may benefit drug and material discovery.
cases). As consequence, EEGSDE is able to explore the chemical
6 GENERATING MOLECULES WITH TARGET STRUCTURES
Following Gebauer et al. (2022), use the molecular fingerprint to encode the structure information of molecule. The molecular fingerprint c = (C1.... CL) is a series bits that capture the presence absence of substructures in the molecule. Specifically, substructure is mapped position the bitmap, and the corresponding bit q will be if the substructure exists in the molecule and will be otherwise. To generate molecules with a specific structure (encoded by the fingerprint c), we set the energy function as the squared error E(z,, c, c, 8 s||m(2, cll between time-dependent multi-label classifier m m= 1)) and c. Here s is the scaling factor, and (= () is trained with binary cross entropy loss to predict the fingerprint as detailed in Appendix E.2, Note that the choice of the energy function is flexible and can be different the training loss of m t). In initial experiments, we also try binary cross entropy loss for the energy function, but we find it causes the generation process unstable. The multi-label classifier m2 1) parameterized in a similar way to the property prediction model g( t) in Section S and present details in Appendix E.1
specific
6.1 SETUP
We evaluate on QM9 and GEOM-Drug. We train our method (including the noise prediction network and the multi-label classifier) on the whole training set. By default, we use a conditional noise prediction network (0(2, t) in Eq. (6) for better performance. See more details in Appendix F.4
To measure how the structure a generated molecule aligns with the target one, the Tanimoto similarity (Gebauer et al. 2022). which captures similarity between structures by comparing their fingerprints.
Evaluation metric:
Baseline: The most direct baseline is conditional EDM (Hoogeboom et al. 2022 which only adopts a conditional noise prediction network without the guidance an energy model. We also consider cG-SchNet (Gebauer al. 2022). which generates molecules in autoregressive manner.
6.2 RESULTS
As shown in Table 4 EEGSDE significantly improves the similarity between target structures and generated structures compared to conditional EDM and eG-SchNet on QM9. Also note in a proper range, larger scaling factor results in a better similarity, and EEGSDE with s=1 improves the sim-
8

---

Published as a conference paper at ICLR 2023
S =0
100 Bohr
S, =2
S =0
Target Structre
S =0.5
Figure 3: Generate molecules QM9 targeted to both the quantum property a and the molecular structure. As the scaling factor 82 grows, the substructure of generated molecule gradually change from the symmetric ring less isometrically shaped structure. Meanwhile the generated molecule aligns better with the target structure as the scaling factor $1 grows.
ilarity by more than 10% compared to conditional EDM. In Figure 2 we plot generated molecules of conditional EDM and EEGSDE (s=1) targeted specific structures, where EEGSDE aligns better with them. We further visualize the effect of the scaling factor in Appendix G.3] where the generated structures align better the scaling factor grows. These results demonstrate that our EEGSDE captures the structure information in molecules well.
We also perform experiments on the more challenging GEOM-Drug (Axelrod & Gomez--Bmbarellii 2022) dataset, and we train the conditional EDM baseline following the default setting of Hooge- boom et al. (2022). As shown in Table - 0.165, which is much lower than the value on QM9. We hypothesize this is because molecules in GEOM-Drug has much more atoms than QM9 with more complex structure, and the default set- ting in in/Hoogeboom et al. (2022) is suboptimal. For example, the conditional EDM on GEOM-Drig has smaller number of parameters than the conditional EDM on QM9 (15M v.s. 26M), which insufficient to capture the structure information. Nevertheless, our EEGSDE still improves the similarity by ~11 We provide generated molecules on GEOM-Drug in Appendix G.5
we find the conditional EDM baseline has a similarity of
Finally, we demonstrate that our EEGSDE is flexible framework to generate molecules targeted to multiple properties, which is often the practical case. We additionally target to the quantum property (polarizability) QM9 by combining the energy funetion for structures in this section and the energy function for quantum properties in Section 5 Here choose 100 Bohr which is relatively large value, and we expect it to encourage less isometrically shaped structures. As shown in Figure the generated molecule aligns better with the target structure the scaling factor s1 grows, and meanwhile a ring substructure in the generated molecule vanishes as the scaling factor for polarizability $2 grows, leading to less isometrically shaped structure, which is expected.
These results suggest that molecules generated by our EEGSDE align better with the target structure than molecules generated by the conditional EDM baseline. Besides, EEGSDE can generate molecules targeted to both specific structures and desired quantum properties by combining energy functions linearly. As a result, EEGSDE may benefit practical cases molecular design when multiple properties should be considered at the same time.
Conclusion:
7 CONCLUSION
This work presents equivariant energy-guided SDE (EEGSDE), flexible framework for con- trollable 3D molecule generation under the guidance of an energy function in diffusion models. EEGSDE naturally exploits the geometric symmetry in 3D molecular conformation, as long as the energy function is invariant to orthogonal transformations. EEGSDE significantly improves the conditional EDM baseline in inverse molecular design targeted to quantum properties and molecu- lar structures. Furthermore, EEGSDE able to generate molecules with multiple target properties by combining the corresponding energy functions linearly.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank Han Guo and Zhen Jia for their help with their expertise in chemistry. This work was supported by NSF of China Projects (Nos. 02061136001, 61620106010, 62076145, U19B2034, U1811461 U19A2081, 6197222); Beijing Outstanding Young Scientist Program NO. BJJWZYJH012019 10000000000000000099 a grant from Tsinghua Institute for Guo Qiang; the High Perfor- mance Computing Center, Tsinghua University; the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (22XNKJ13). J.Z was also supported by the XPlorer Prize. C. Li was also sponsored by Beijing Nova Program.
ETHICS STATEMENT
Inverse molecular design is critical in fields like material science and drug discovery. Our EEGSDE flexible framework for inverse molecular design and thus might benefit these fields. Currently the negative consequences are not obvious.
REPRODUCIBILITY STATEMENT
Our code is included in the supplementary material. The implementation of our experiment is de- scribed in Section 5 and Section 6 Further details such as the hyperparameters of training and model backbones are provided in Appendix F. We provide complete proofs and derivations of all theoretical results in Appendix A
REFERENCES
Rim Assouel, Mohamed Ahmed, Marwin H Segler, Amir Saffari, and Yoshua Bengio, De- factor: Differentiable edge factorization-based probabilistic graph generation. arXiv preprint arXiv: 1.09766, 2018.
Simon Axelrod and Rafael Gomez-Bombarelli Geom, energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data, 9(1):1-14, 2022.
Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206.07300 07309, 2022a.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang, Analytic-dpm: analytic estimate of the optimal reverse variance diffusion probabilistic models. arXiv preprint 0at::2201.0650:3 2022b.
Yuemin Bian, Junmei Wang, Jaden Jungho Jun, and Xiang- Qun Xie. Deep convolutional genera- tive adversarial network (dcgan) models for screening and design of small molecules targeting cannabinoid receptors. Molecular pharmacentics, 16(1 1):4451-4460, 2019.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients waveform generation. arXiv preprint arXiv:2009.0000 00000, 2020.
Hanjun Dai, Yingtao Tian, Bo Dai, Steven Skiena, and Song. Syntax-directed variational autoen- 2018.
coder for structured data. arXiv preprint
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.05233, 2021.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. arXiv preprint 12X00021 2021.
Daniel Wamm-Shepeerd, Kevin Zhu, and Alán Aspuru-Guzik. Keeping it simple: Language models 2021.
can learn complex molecular distributions. arXiv preprint
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se tranfffoof::::::: 3d roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:19 33:1970-1991, 2020.
10

---

Published as a conference paper at ICLR 2023
Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. Advances in neural information processing systems, 32, 2019.
Niklas WA Gebauer, Michael Gastegger, Stefaan SP Hessmann, Klaus-Robert Müller, and Kristof T Schütt. Inverse design of 3d molecular structures with conditional generative neural networks. Nature commumications, 13(1):1-11, 2022.
Philip J Hajduk and Jonathan Greer. A decade of fragment-based drug design: strategic advances and lessons learned. Nature reviews Drug discovery, (53):21-2119 2007.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 12399, 2020.
Moritz Hoffmann and Frank Noé. Generating valid euclidean distance matrices. arXiv preprint arXiv:1910.03131, 2019.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant dif- fusion for molecule generation in 3d. In International Conference Machine Learning, pp- 8867-8887. PMLR, 2022.
Jan H Jensen. graph-based genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space. Chemical science, 10(12):357733572 2019.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junetion tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2323-2332. PMLR, 2018.
Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional diffu- sion for molecular conformer generation. arXiv preprint arXiv:2206.01729, 2022.
Kisuk Kang, Ying Shirley Meng, Julien Breger, Clare P Grey, and Gerbrand Ceder. Electrodes with high power and high capacity for rechargeable lithium batteries. Science, 311(5763)9777-98, 2006.
Johannes Klicpera, Shankari Giri, Johannes T Margraf, and Stephan Günnemann. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. arXiv preprint arXiv:2011.14115. 2020.
Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: exact likelihood generative learning for symmetric densities. In International conference on machine learning, pp. 5361-5370. PMLR, 2020.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint (1Xi6:2009.09761, 2020.
Matt J Kusner, Brooks Paige, and José Miguel Hernández-Lobato. Grammar variational autoen- coder. In International conference on machine learning, pp. 1945-1954. PMLR, 2017.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. tutorial on energy-based learning. Predicting structured data, 100), 2006.
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander Gaunt. Constrained graph varia- tional autoencoders for molecule design. Advances in neural information processing systems, 31, 2018.
Cheng Lu. Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.0092. 2022.
Youzhi Luo, Keqiang Yan, and Shuiwang Ji. Graphdf: A discrete flow model for molecular graph generation. In International Conference on Machine Learning, pp. 7192-7203. PMLR, 2021.
Kaushalya Madhawa, Katushiko Ishiguro, Kosuke Nakago, and Motoki Abe. Graphnvp: An invert- ible flow model for generating molecular graphs. arXiv preprint arxiv: 1600, 2019.
11

---

Published as a conference paper at ICLR 2023
Soma Mandal, Sanat K Mandal, et al. Rational drug design. European journal of pharmacology, 625(1-3):00-100, 2009.
Vitali Nesterov, Mario Wieser, and Volker Roth. 3dmolnet: generative network for molecular structures. arXiv preprint arXiv:0010006477, 2020.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09677 2021.
AkshatKumar Nigam, Pascal Friederich, Mario Krenn, and Alán Aspuru-Guzik. Augmenting ge- netic algorithms with deep neural networks for exploring the chemical space. arXiv preprint arXiv:1909.1165:. 2019.
Mariya Popova, Mykhailo Shvets, Junier Oliva, and Olexandr Isayev. Molecularrnn: Generating realistic molecular graphs with optimized properties. arXiv preprint prXiv::905.133377 2019.
Edward O Knapp. Changwon Suh, Rafael KómezzBombarelli, Jorge and Alán Aspuru-Guzik. What is high-throughput virtual screening? a perspective from organic materials discovery. Annu. Rev. Mater. Res., 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763. PMLR, 2021.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):177,
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint (tx.i::2022.05/22.. 2022.
Benjamin Sanchez-Lengelee and Alán Aspuru-Guzik. Inverse molecular design using machine learning: Generative models for matter engineering. Science, 361(6400):360-365 2018.
Victor Garcia Satorras, Emiel Hoogeboom, Fabian Fuchs, Ingmar Posner, and Max Welling. (n) equivariant normalizing flows. arXiv preprint arXiv:2105.0900 2021a.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural net- works. In International conference on machine learning, pp. 9323-9332. PMLR, 2021b.
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International conference artificial neural networks, pp. 412-422. Springer, 2018
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-22662222 2265. PMLR, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. arXiv preprint arXiv:2 2021.
David Weininger. Smiles, chemical language and information system. 1. introduction method- ology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-2)) 1988.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geo- metric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923; 2022.
12

---

Published as a conference paper at ICLR 2023
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems. 31, 2018.
Chengxi Zang and Fei Wang. Moflow: invertible flow model for generating molecular graphs. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 617-626, 2020.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. arXiv preprint arXiv:2207.0666 2022.
Zhenpeng Zhou, Steven Kearnes, Li Li, Richard N Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning. Scientific reports, 9(1):1-10, 2019.
Alex Zunger. Inverse design in search of materials with target functionalities. Nature Reviews Chemistry, 2(4):1-16, 2018.
A DERIVATIONS
A. ZERO COM SUBSPACE
The zero CoM subspace {i E RM EE x² 0} is (M 1)n dimensional subspace of R Therefore, there exists an isometric isomorphism ø from RMM-1) X, i.e., @ linear bijection from 1)M to X, and [| ||ا|| for all â (MM-))n We use Ag MMM represent the matrix corresponding to Φ. so we have ((t) Agă. An important property of the isometric isomorphism is that AgAJx x for all E Mr. We show the proof following.
Proposition 1. Suppose is isometric isomorphism from RMM-1ja X, and let Ag. MM be the matrix corresponding to d. Then we have Ag
x: for all E RR,,
where M E
Proof. We consider a new subspace of Mn X- (x RM x 1 X}. i.c. the orthogonal component of X. We can verify that X = fx RRn, I1 = 22 = ..... aM }. and X is n dimensional. Thus, there exists an isometric isomorphism & from TK" X- Let Ag R Muxa rep-
resent the matrix corresponding to th. Then we define A(ă, 9) = d(.) + (Ø)) where & (M-1) RM Therefore, X is linear bi-
and ý E R² The image of À is {ær + y x X.y E X-) jection from R MM to RM and the matrix corresponding to is A [As Av). Furthermore, is an isometric isomorphism, since [\(@; 9)ll [ 1|2 + [ (d)))), (D)) || + 1
112 This means A is orthogonal transformation. Therefore, and AgAJa x + A = æ. Since A x X and
AxA
AgA x + A
A E XT, we can conclude that AaAJa x is the orthogonal projection of x to X, which is exactly x æ.
Since T (M-1)x and X are two intrinsically equivalent spaces, an equivalence of distributions in these two spaces can also be established, as shown the following propositions.
Proposition 2. Suppose x is a random vector distributed in X, and à = o (x:) is its equivalent representation in (M)n,, If x ~ q(x), then i ~ ((Ø), where @()) q()(i)).
Proposition 3. Suppose y is are random vectors distributed in X, and a --)()) (y) their equivalent representations in (M-1)n If x:l (( then a ÿ g()lg).
where (( = g(o(#)lo(ū)
A.2 SDE IN THE PRODUCT SPACE
Definition (Gaussian distributions in the zero CoM subspace). Suppose μ X. Let Nx 0²) (2mo²)-(M-11)) (ippi//////ppppieepppl 202 || Hll3), which is the isotropic Gaussian distribu- tion with mean ya and variance o² in the zero CoM subspace X.
13

---

Published as a conference paper at ICLR 2023
Proposition 4 (Transition kernels of a SDE in the zero CoM subspace). Suppose dæ f(t)adt + g(t)dw,, Zo ~ ((xa) is a SDE in the zero CoM subspace. Then the transition kernel from w, 22 (0 I < t S T) can be expressed q(z,|xx))
yanpa Wa Stys), where ddls exp(2 S [(r)dr) and Bule = orj.s Sx u(T)²///))/)))/ //rrj/rr are two scalars determined by [(-) and g())-
Proof. Firstly, map the process {ar in the zero CoM subspace to the equivalent space R(M-1)m through the isometrie isomorphism Φ introduced in Appendix A. This produces new process { where 2t Φ x (æx). By applying o- to the SDE in the zero CoM subspace, we know et,- = o (x=:) satisfies the following SDE in (R)M-11n;;
dē f(t)&ddt + gt)ddū Do ~ ((80),
where v is the standard Wiener process in (M and ((x)) = q( According to Song et al. (2020). the transition of Eq. 7 from E. to E. (s t) can be expressed as q( (
(a yarpa Dt where odls = exp(2 (r)ddr) and Sdl Otte (())
/ar1.dr / are two we know the transition kernel
scalars determined by f()) and g(.). According to Proposition 3
((x))2z)) from x to H, satisfies
q(2:/2x)) - -00 (x:)) (xx)) = (2T ) (M-1))/2 exp(
11 (ax) - vanlo (xx))
23 s
=(2 (MM-1) eexp
1 (x) - [
// lincarity of o o-
2/11/
-12 ( 1M1/2 exp(
1
||
varl
|/ norm preserving of Φ-1
2311/
-Nx (arloa Stjo)
U
Proposition 5 (Transition kernels of the SDE in the product space). Suppose dz = f())zdt + g(O)d(w,, Way Wp), Za ((20) is the SDE in the product space X x Raid as introduced in Eq. ( Then the transition kernel from 2s to zt (0 I < < T) can be expressed Nx (arl yo Bels) and q(
as (Z((2))) = q(zrela.)) (()hhh))) where (x,)
(hellaanhhh Sajo). Here Otls and 3elp are defined as in Proposition 4
Proof. Since wT and wA are independent each other, the transition kernel from 2, to 2, can be factorized as g(Z, [Z.) = g( (ts (h, h2), where q( xx.) is the transition kernel of dæ = f (t)ærdt + dd., and qhe (h.) the transition kernel of dh = f(t)hdt g(t)d According to Proposition Fand (Sng et al. (2020), we have g(xx(xx,) = NNx (I a. Br a) and qh, hh)) N/hel hoighhh Seje)
Remark 1. The marginal distribution of ze is [
(h)) q((z))= = - Jgma" (hto
where X is the Lebesgue measure in the zero CoM subspace X. While gi(2e) is a distribu- tion in X x R Md it has a natural differentiable extension to Rinn x R Md since q (xo) =
Nx a0. has a differentiable extension R RM according Definition | can take gradient of qe (z.) w.rt. Be in the whole space R
Thus, we
Proposition 6 (Time reversal of the SDE in the product space). Suppose dz ft)zztt +
g(t)d(w: w,, wp). Za q(20) is the SDE in the product space X x R&dd as introduced in Eq. (3).
Then its time reversal satisfies the following reverse-time SDE, which can be represented by both the score function form and the noise prediction form:
dz [t)( - =(t)² (Va log ((()) - Va log (2), Vn log ())((())))t + g(t)d(( (w, wx),
score function form
g(t)2 VBejo
=[S(1)z
E Be(
Jdl + g())d(w,,, wn), 2T ~ >g~gr(Er)
noise prediction form
14

---

Published as a conference paper at ICLR 2023
where wx and Wi, reverse-time standard Wiener processes in X and RMd respectively, and Gt 21 y is the standard Gaussian noise X RMdd injected to Zu-
Furthermore, we have V log qe(x) - V log (+(a)) = - - yl m.) w.) where /arpoa
V8010
VBuo
is the standard Gaussian noise in the CoM subspace injected to æo.
Proof. Let 2, = ( he), where d. = (xx), introduced the proof of Proposition { IT is process in IR (M - 1)n
Then
x IR Md determined by the following SDE
d f(t g(t)drë; E0 ((ău).
(8)
where v is the standard Wiener process in (M x addd and ((20) = ((((
According to SSng al. (2020), Eq. ( has time reversal:
d = [f()) g(t)² log d (2) g(t)du; 2T (r (2r):
(9)
where (x) ) is the marginal distribution of 2, which satisfies ((t) = 9: (0(#)) h2) according to Proposition 2 and i is the reverse-time standard Wiener process in RIM 1)m Md
Then apply the linear transformation T (o(i), h) to Eq. ( which maps &, back to z.. This yields
dz = (V())((((()))g)))) (z) + g())d( i.));
2T (T()))
(10)
Here wa and wr are reverse-time standard Wiener processes and Rad respectively, and T(V log (x)) can be expressed as (ø(V log (x))), Vn log (2)).
Since v log (=)
Ao Va log (x h) = Al v log we have Φ(Va log q (5))
A V log (z), where Ag represents the matrix corresponding to ø. According to Propo-
sition 1 we have σf V a log (&)) = Va log (z) Va lo). Besides, Vn log (2) = V log qe (z). Thus, Eq. (10) be written
dz [f(t) g (V log (z) V log) vn log (z()))d g(t) wn))
which is the score function form of the reverse-time SDE.
We can also write the score function in Eq. as Ve log (2) = (g(aola)) V log (() 2.
Reo (, where é
vaquc is the standard Gaussian noise injected to 2g. With VBelo
this expression, we have T(V log (2))
E E.( C4, where
Er papoo is the standard Gaussian noise in X x Md injected to zo. Thus, Eq. 10 can also Vaga
be written
g(t)² VBroo E (x e/jt + g(t) wx)).
dz [f(t)=
which is the noise prediction form of the reverse-time SDE.
A.3 EQUIVARIANCE
Theorem 1. Let (E)(2,, (), e,, 0)) = 60(21,)) 1), where (Z, and e/y( Zz, are the predicted noise of It and h, respectively. If for any orthogonal transformation R E gaxn co(xi,t) t) is equivariant to R, i.e., E RR,, he t) (Re) hr, t), (0(,,, ht,t)), and (x() is invariant to R, i.e. pr pr(Ray Rxr,hr) = pr px(T hr), then po(2o) is invariant to any rotational transformation.
Proof. Suppose R E R is orthogonal transformation. Let 2. (x.. hτ) (0 < T) be the process determined by Eq. (5). Let = Ræę and u.. = (vf h²) ). We use (u, and p (z)) to denote the distributions of u and 2 respectively, and they satisfy pt (31, h.) pp (R-ly. h4)-
15

---

Taking gradient to both side of Published as a conference paper at ICLR 2023
By applying the transformation Tz (Rx, h)) to Eq. (5) we know the new process {ut Tu=)T= satisfies the following SDE:
9(1) (Re t), (2)(())))) + ((t) w.)
du = Td=== [f(t)u+
VBuo
=If(t)u= g(t)²
Rx, h, dt + g(t)d wi) by equivariance (Eo (o, t), cØ (Rx,
1
=Is(1)++ + g(t)2 60 ( 1)]dl g(L)d( in))
// Winner process is isotropic
VB00
and its initial distribution is (uT) = (yT hT) P² yyr hr) = pr (Rt(R- hT)
pr (yr, hr)) = (r(rr). Thus, the SDE of {ut "u)) is exactly the same to that of [=( [201T 11-
This indicates that the distribution of ug is the same the distribution of = i.c. p6 (wo)
P5( uo) = pe (un). Also note that (uo) = P yo. yo. ho) = pe R RR. yo. ho). Thus,
Po uo) po R yo, ho), and consequently Po Rog, ha) (o ha). This means pe (z))
is invariant to any orthogonal transformation, which includes rotational transformations as special cases.
Theorem 2. Suppose the assumptions in Theorem 7 hold and c,,,c,l) is invariant any or- thogonal transformation R, i.e. E Rax h e, El h, C, Then po(2olc) is invariant to any rotational transformation.
Proof. Suppose R E Rax" is an orthogonal transformation. B(Ræ, h, e) = E(v, h c, t) w.r.t. æ, we get
R R e, D Ea
Multiplying R both sides, we get
V -Y RV
Let ((2, (V=E(z,c,t)
Then we have
(iRa,h,c,, = - F. R= n
=( RV RV.F(a.. e, V,E(RR, c,), B( h, c, E ()), VaE(o, c,tt).
Thus, (0z, t) is equivariant to R. Let éo c, cs(z,t) + v Bqo c, t),, which is linear combination of two equivariant functions and is also equivariant to R.
Then, Eq. (6) can be written
dz [5(1z+ + g(t)²
BBo0 60(2, e, ()]dt + g())d(),, wx), 2r pr(2T).
According to Theorem 1 we know marginal distribution at time 0, i.e., pe (zo (c), is invariant to any rotational transformation.
B SAMPLING
In Algorithm [ present the Falyr-Maruyama method to sample from EEGSDE in Eq.
16

---

Published as a conference paper at ICLR 2023
Algorithm 1 Sample from EEGSDE using the Euler-Maruyama method
Require: Number of steps N
Al
N
- (x 2, h), where -
(Sample from the prior pr (zT))
for ; = N to do
iAt
9r t- V F2, f, 9h + CnF(2,c,t)
{Calculate the gradient of the energy function)
g 9z. 92. 9h)
(Subtract the CoM of the gradient)
F - z(t) + gt)
E0 1) +
E0
V810
Ex E En), where Ex (0, 1), en (0,,1)
2 F + g(t)
{Update z according to Eq. 6
end for return 2
C CONDITIONAL NOISE PREDICTION NETWORKS
In Eq. 6 we can alternatively use a conditional noise prediction network cø(2, e, for a stronger guidance, as follows
d= = [1(== (000))((____) cz,)
VByo
+ +Vv2E(2,c,t) V Cz t)) + g(t)d(w,, wx), 27 ~ (T(2))
The conditional noise prediction network is trained similarly to the unconditional one, using the following MSE loss
min (x - c.122.
D PARAMETERIZATION OF NOISE PREDICTION NETWORKS
We parameterize the noise prediction network following Hoogeboom et al. (022), and we pro- vide the specific parameterization for completeness. For the unconditional model cø t), first concatenate each atom feature and f, which gives h" (h',t) t). Then we input æ and the EGNN follows
h' ( M²)
(a ad) EGNN(æ, h
(x, 0).
Finally, we subtract the CoM of a", and gets the parameterization of c0(2,
e0(2,t) (a"
where a comes from discarding the last component " that corresponds to the time. For the conditional model eø c)t), additionally concatenate to the atom feature h i.e., h = (h', t, c), and other parts the parameterization remain the same.
E DETAILS OF ENERGY FUNCTIONS
E. PARAMETERIIAATION OF TIME-DEPENDENT MODELS
The time denendepnddnnt property prediction model g(2,, t) is parameterized using the second compo- nent in the output of EGNN (see Section 3 followed by a decoder (Dec):
g( t) Dec Dec(EGNN' (xc, h()), h; concatenate(hy,.f t),
(11)
where the concatenation is performed on each atom feature, and the decoder is a small neural network based on Satorras al. (2021b). This parameterization ensures that the energy function
17

---

Published as a conference paper at ICLR 2023
(ze, c, is invariant to orthogonal transformations, and thus the distribution of generated samples also invariant according to Theorem
Similarly, the time-dependent multi-label classifier is parameterized by EGNN
m( t) a (Dec(EGNN" (z,, k()),, h concatenate h, t).
The multi-label classifier has the same backbone to the property prediction model in Eq. 111, except that the decoder outputs vector of dimension L, and the sigmoid function is adopted for multi- label classification. Similarly to Eq. (TT). the HGNN in the multi-label classifier guarantees the invariance of the distribution of generated samples according to Theorem 2
E.2 TRAINING OBJECTIVES OF ENERGY FUNCTIONS
Time-dependent property prediction model. Since the quantum property is scalar, we train the time-dependent property prediction model g t) using the & loss
Ee (l. cl.
where / is uniformly sampled from [0.
Time-dependent multi-label classifier. Since the fingerprint is bit map, predicting be viewed as multi-label classification task. Thus, we use time dependent multi-label classifier m(z,, t), and train it using the binary cross entropy loss
EgE
ci log (z., t) + (1 at) log([1 mz(z,,t))
where is uniformly sampled from [0, T). and mi (zt,t) is the I-th component of m t).
F EXPERIMENTAL DETAILS
F.1 HOW TO GENERATE MOLECULES TARGETED TO MULTIPLE QUANTUM PROPERTIES
When we want to generate molecules with K quantum properties c = (c1, (2,,,2,...,cx Ck), we com- bine energy functions for single properties linearly as E'z,, c, = Σ Ex Ck, t), where Eg (Z,, Cs2 t) factor and gu (2,,4 t) is the time-dependent property prediction model for the k-th property. Then use the gradient of E () to guide the reverse SDE as described in Eq. (6)
(a t) C& 2 is the energy function for the k-th property, St is the scaling
F.2 THE "U-BOUND" AND "#ATOMS" BASELINES
The "U-bound" and "#Atoms" baselines are from Hoogeboom shuffles the labels in Dj and then calculate the loss of de on it, which can be regarded as an upper bound of the MAE. The "#Atoms" baseline predicts a quantum property e using only the number of atoms in a molecule.
2022). The "U-bound" baseline
F.3 GENERATING MOLECULES WITH DESIRED QUANTUM PROPERTIES
For the noise prediction network, we use the same setting with EDM (Hoogeboom al. 2022 for a fair comparison, where the models is trained 2000 epochs with batch size of 64, a learning rate 0.0001 with the Adam optimizer and an exponential moving average (EMA) with a rate of 0.9999.
The EGNN used in the energy function has 192 hidden features and layers. We train 2000 epochs with a batch size of 128, learning rate of 0.0001 with the Adam optimizer and an exponential moving average (EMA) with a rate of 0.9999.
During evaluation, we need to generate set of molecules. Following the EDM (Hoogeboom al. 2022). we firstly sample the number of atoms in molecule M ~ p(M) and the property value M(c (or C1 p(c, MM p(cy M)...., .. CK p(cx (M) for multiple properties). Here p(M) is the distribution molecule sizes on training data, and p(c[M) ) the distribution of the property training data. Then generate molecule given M,c. c.
18

---

Published as a conference paper at ICLR 2023
F.4 GENERATING MOLECULES WITH TARGET STRUCTURES
Computation of Tanimoto similarity. Let S be the set of bits that are set to 1 in the fingerprint of generated molecule, and S be the set of bits that set in the target structure. The Tanimoto similarity is defined as |Sa n | U 5"), where denotes the number of elements in set.
Experimental details on QM9, For the backbone of the noise prediction network, we use three- layer MLP with 768, 512 and 192 hidden nodes as embedding to encode the fingerprint and add the output of with the embedding of atom features h. which is then fed into the following EGNN. The EGNN has 256 hidden features and 9 layers. We train it 1500 epoch with batch size of 64, learning rate of 0.0001 with the Adam optimizer and an exponential moving average (EMA) with rate of 0.9999. The energy function is trained with 1750 epoch with batch size of 128, learning rate of 0.0001 with the Adam optimizer and an exponential moving average (EMA) with a rate of 0.9999. Its EGNN has 192 hidden features and layers. For the baseline eG-SchNet, it using the public code. Since the default data split in eG-SchNet different with ours, we train the cG-SchNet under the same data split with ours for fair comparison. We report results 200 epochs (there no gain on similarity metric after 150 epochs). We evaluate the Tanimoto similarity on the whole test set.
reproduce
Experimental details on GEOM-Drug, We use the same data split of GEOM-Drug with Hooge- boom et al. (2022), where training, validation and test set include 554K, 70K and 70K samples respectively. We train the noise prediction network and the energy function on the training set. For the noise prediction network, we use the recommended hyperparameters of EDM (Hoogeboom itt al. 2022). where the EGNN has 256 hidden features and 4 layers, and the other part is the same as that in QM9. We train 10 epoch with a batch size of 64, learning rate of 0.0001 with the Adam optimizer and an exponential moving average (EMA) with a rate of 0.9999. The backbone of the energy function is the same as that in QM9 and we train the energy function for 14 epoch. We evaluate the Tanimoto similarity randomly selected 10K molecules from the test set.
G ADDITIONAL RESULTS
G.1 ABLATION STUDY ON NOISE PREDICTION NETWORKS AND ENERGY GUIDANCE
We perform an ablation study on the conditioning, i.e., use conditional or unconditional noise pre- diction networks, and the energy guidance. When neither the conditioning nor the energy guidance is adopted, single unconditional model can't perform conditional generation, and therefore we only report its atom stability and the molecule stability. As shown in Table [5] Table 6 and Table 7 the conditional noise prediction network improves the MAE compared to the unconditional one, and the energy guidance improves the MAE compared to sole conditional model. Both the conditioning and the energy guidance do not affect the atom stability and the molecule stability much.
Table 5: Effects of conditioning and energy guidance single quantum property y (D).
Conditional
Guidance
Scale
MAEJ
Noveltyt
ASt
MS
82.92
98.37
81.74
X
v
0.1
1.415
83.60
98.33
81.16
X
V
0.5
1.241
83.84
98.27
80.75
1.138
84,04
98.14
80.04
V
0.1
1.071
84.36
98.18
80.08
V
V
0.5
0.935
84.06
98.20
80.05
19

---

Published as a conference paper at ICLR 2023
Table 6: Effects of conditioning and energy guidance on a single quantum property C. (
cal mol
K).
Conditional
Guidance
Scale
MAEI
Novelty f
AS1
MS1
X
X
82.92
98.37
81.74
2.360
84.15
98.36
81.44
10
1.459
84.25
98.07
79.06
V
X
1.066
83.64
98.25
80.50
1.038
83.71
98.21
80.61
0.939
83.92
98.06
79.21
Table 7: Effects of conditioning and energy guidance on multiple quantum properties Cv. Jr.
Conditional
Guidance
Scale
MAEL
MAE2
Noveltyt
ASt
MS
82.92
98.37
81.74
1,0.1 10,1
2.381
1.421
83.81
98.31
80.52
1.501
1.157
84.11
98.02
78.20
1.075
1.163
85.45
97.99
77.02
V
1,0.1
1.053
1.097
85.20
97.96
77.08
/
V
10,1
0.982
0.916
85.51
97.58
73.94
G.2 RESULTS ON NOVELTY, ATOM STABILITY AND MOLECULE STABILITY
For completeness, we also report novelty, atom stability and molecule stability on 10K generated molecules. Below we briefly introduce these metrics.
Novelty (Simonovsky Komodakis 2018 is the proportion of generated molecules that do not appear in the training set. Specifically, let G be the set of generated molecules, the novelty is calculated reproduced conditional EDM and our method are trained Dg. This leads an inflated value compared to the one evaluated on the whole dataset (Hoogeboom et al. 2022).
|G D.!
[G] Note that the novelty is evaluated D'. since the
Atom stability (AS) (Hoogeboom et al. 2022) is the proportion of atoms that have the right valency. Molecule stability (MS) (Hoogeboom et al. 2022) is the proportion of gener- ated molecules where all atoms stable. We use the official implementation for the two metrics from the EDM paper (Hoogeboom et al. 2022).
As shown in Table 8 and Table 9 conditional EDM and EEGSDE with small scaling factor have a slightly better stability, and EEGSDE with large scaling factor has slightly better novelty in general. The additional energy changes the distribution of generated molecules, which improves the novelty of generated molecules in general. Since there is tradeoff between novelty and stability (see the caption of Table the EDM paper (Hoogeboom et al. 2022)), a slight decrease on the stability is possible when the scaling factor is large.
20

---

Published as a conference paper at ICLR 2023
Table 8: Additional results on the novelty, the atom stability (AS) and the molecule stability (MS) of generated molecules targeted to single quantum property.
Method
Noveltyf
AS
MS1
Method
Novelty?
AST
MSt
Conditional EDM
83.64=0.30
98.25:10.02
80.82:+:0.2
Conditional EDM
83,93=0.111
98.17:00.04
80.25:1:0.40
EEGSDE (s=1)
83,53=0.18
98.25-10.06
80.83:10.33
EEGSDE (s=0.5)
83,85=0.20
98,18=0.02
80.25::0.18
EEGSDE (s=5)
83.57=0.17
98. 98.16:10.04
80,22:10.34
EEGSDE (s=1)
84.43=0.21 18.17.0.04
80.06::0.35
ERGSDE (s=10)
98.03
79.07++0.24
HEGSDE (s=2)
84.62+0.31
98.06+0.002
78.92++0.30
Ar
EHOMO
Conditional EDM
83,93=0.45
98.30:10.00
81.95:10.227
Conditional EDM
84.35=0.31
98.17=0.07
79.61=10.32
EEGSDE (s=0.5)
84.09=0.27
98. 18.10:0006
80.99:10.29
EEGSDE (s=0.1)
84.44:0.33 98.19=0.03
79.81=0.200
EEGSDE (s=1)
83.91=0.38
98.08:10.04
79.85:10.29
EEGSDE (s=0.5)
84.57=0.28
08.1320.002
79.40:10.32
ERGSDE (s=3)
85.17-0.05
97.76+H0.03
77.43++0.37
FEGSDE (s=1)
84.77++0.25
08.08+00.02
78.90+
ELUMO
Conditional EDM
84.56-
98.13++0.04
79.33++0.30
Conditional EDM
84.62:+0.28
98.26=0004
01.34
EEGSDE (s=0.5)
84.35=0.31
98.19:10.3
80.08:1.334
EEGSDE (s=0.5)
84.37=0.31
98.25:10.44
81.18:10.31
EEGSDE (s=1)
84.45=0.33
98. 18.10.40004
80.04:1.336
EEGSDE (s=1)
84.70=10.34
98.27=0.05
81.23:10.75
EEGSDE (s=3)
84.19-0.32
98.26
80.95++0.35
FEGSDE (s=3)
84.83+00.0
98.14+0.01
80.00+0.21
Table 9: Additional results on the novelty, the atom stability (AS) and the molecule stability (MS) of generated molecules targeted to multiple quantum properties.
Method
Noveltyt
ASt
MSt
C.. f
Conditional EDM
85.31 -043
98.00+0.07
7..42+0.80 74.56.10.54
EEGSDE 1s,=10, s2=1)
85.62 0.86
97.67 10.08
As.
Conditional EDM EEGSDE (+==8==1)
85.06
97.96 10.00
10.30
85.56=0.56
97.61=0.04
72.72+00227
Conditional EDM EEGSDE (11m8=1155)
85.18=0.35 35
98.00=0.06
77.96=10.3
85.36-0.03
97.99100.0
77.77-1..26
G.3 VISUALIZATION OF THE EFFECT OF THE SCALING FACTOR
We visualize the effect of the scaling factor in Figure 4 where the generated structures align better as the scaling factor grows.
Target
S=0
0=0.5
structure
Figure 4: Visualization of the effect of the scaling factor on QM9. As the scaling factor grows, the generated structures align better with the target structure. corresponds to the conditional EDM.
21

---

Published as a conference paper at ICLR 2023
G.4 REPRODUCE
We compare our reproduced results and the original results of conditional EDM (Hoogeboom et al., 2022 in Table TO The results consistent.
Table 10: The reproduced results of conditional EDM and I--ound are consistent with the original ones.
Quantum property
Aa
ЕHOMO
ÉLUMO
C
Unit
Bohr
meV
mcV
meV
D
cal K mo!
Conditional EDM (Hoogeboom et al. 2022
2.76
655
356
584
1.111
1.101
Conditional EDM (reproduce)
2.79
674
371
593
1.118
1.054
L-bound Hoogeboom et al. 2022
0.10
64
36
0.043
0.040
L-bound (reproduce)
0.09
65
39
36
0.043
0.040
G.5 GENERATING MOLECULES WITH TARGET STRUCTURES GEOM-DRUG
We plot generated molecules on GEOM-Drug in Figure 5 and can be observed that the atom types of generated molecules with EEGSDE often match the target better than conditional EDM.
Taggt structure
EDM
BEGSDE
(ours)
Figure Generated molecules on GEOM-Drug.
G.6 THE DISTRIBUTIONS OF PROPERTIES
We plot the distribution of the properties of the training set, as well as the distribution of the prop- erties of generated molecules (calculated by the Gaussian software). As shown in Figure 6 these distributions match well.
22

---

Published as a conference paper at ICLR 2023
(A)
Takik Dar
IEGS00 (ed)
[R]
0.2
Tokkii Dute
ED0500 (-:3)
(C)
0:3
Tuskk Dul
1E0508 (nit)
613-
605
608
0.80
608
02
(D)
(E)
Track Date
Tea wiy Data
825-
Danity
R13
811-
80
Figure 6: The distribution of the properties of the training the distribution of the properties of molecules generated by EEGSDE.
23


Reviews:
[{"review_id": "quw3kN7Y3mh", "reviewer": "Reviewer_1ipW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present an energy-guided score model formulation for the task of equivariantly generating molecules. This work builds upon lots of recent references on SDE score matching models, equivariant networks (EGNNs) and specifically the work of Hoogeboom et al (2022), EDM. The main contribution of the paper is the addition of an energy function in the score matching algorithm, which can be used to generate molecules with desired properties.", "summary_of_the_review": "Overall, this is a good contribution to ICLR. Although the approach is marginally incremental, I believe the work is novel and shows how to tackle a fundamental problem in atomistic ML, how perform inverse molecular design using denoising diffusion models on a regression setting.\n\nI would recommend acceptance upon revision of several minor issues.", "strength_and_weaknesses": "Overall, the paper is somewhat incremetal but in general technically sound and a pleasure to read.\n\nStrengths:\n* Clear motivation to solve a real problem (inverse molecular design)\n* A lot of effort went into explaining previous methods. This is a fantastic introductory paper that requires no reference reading for context.\n* Simple and elegant formulation\n* Code provided in the supplementary material\n\nWeaknesses:\n* Main results on Tables 1, 2, 3 could be provided with standard deviations to assess how stable results are depending on the chosen seed\n* Supplied code could be cleaned / improved with clearer instructions on how to run the experiments in the paper.\n\nOther comments:\n\n* It would be fantastic if the authors physically motivated the choice for using energy functions (Eq. 6). The reverse SDE is presented without a lot of justification\n* In Section 6 the authors claim that they tried a binary cross-entropy loss for generating molecules with desired structures, but ended up using a mean squared error loss. In Appendix D, however, the binary cross entropy is the one described for this task.\n* It is somewhat unclear to me whether the time-dependent property prediction model is jointly trained with the score model or whether this is trained separately. It would be great if the authors could clarify along these lines.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cvOlhFu-tGU", "reviewer": "Reviewer_ChXk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method for 3D molecular generation called energy-guided stochastic differential equations (EEGSDE), which introduces an equivariant diffusion model based on SDE formulations. The authors first introduce the problem setting of 3D molecular design, as well as past work in molecular design and related work including diffusion models and guidance of diffusion models. Subsequently, the paper introduces relevant background and formulations (3D molecule representation, equivariance and Center of Mass modeling) and outlines the mathematical details of EEG-SDE, including the SDE formulation and an extension for an equivariant SDE formulation based on an equivariant neural network backbone (EGNN). After outlining the entire algorithmic flow with all relevant and priorly introduced components, the authors introduce a set of molecular design experiments based primarily on the QM9 dataset.\n\nIn their experiments, the authors first train EEG-SDE on the QM9 dataset and compare it's performance to equivariant diffusion models (EDM) along a couple of metrics (MAE, Novelty, Atomic Stability, Molecular Stability). In their formulation, the authors also include a scaling factor, $s$, which serves to modulate the strength of a given loss signal and is varied in many of the authors' experiments. The first set of presented results show general competitiveness with EDM with outperformance along some metrics and lower performance on others. The authors then show an experiment generating molecules based on a target structure where the similarity of the generated molecule to the target structure is based on a molecular fingerprint. Based on the similarity metric EEG-SDE appears to outperform SDE.", "summary_of_the_review": "While I appreciated the thoroughness of the description of the method and all its different parts, in my view the reasons to reject outweigh the reasons to accept the paper in its current form. The novelty appears a bit incremental compared to EDM and the presentation of the methods and results is a bit unclear to be able to see clear differentiation. \n\n------\nScore revised from 5 -> 6 based on discussion and adjustments made during author response.", "strength_and_weaknesses": "**Strengths**\n\n* The paper introduces a new equivariant molecular diffusion method and describes the different pieces, including various definitions and formulations, in great detail. \n* The results presented in the experiments seem to indicate some advantages of using EEG-SDE compared to EDM.\n\n**Weaknesses**\n\n* Some of the figures, especially Figure 1 and Figure 4, could benefit from greater detail to the overall message they are trying to convey. In Figure 1, the different images could have additional labels to guide the reader as to what the different sections represent. In Figure 4, it is unclear what the authors are trying to show with the caption not being clear about the purpose and it being hard accompanying text being hard to find.\n* The results on the GEOM-Drug dataset are only briefly mentioned in the text. It would have been nice to see them included in more cohesive manner, potentially in one of the prior tables. \n* The overall message from the results is somewhat unclear. It would be nice if the paper could summarize some key takeways for EEGSDE based on the results of the experiments?\n\n**Additional Questions**\n\n* Fitting to properties appears to be applicable to one property at a time given the scalar signal. How does the multi-property case shown in Figure 4 work in that instance? Is it mainly a loss over a vector? If so, how is it implemented?\n* Did you consider trying other equivariant models (SE3-Transformer, DimeNet, etc)? Why, why not?\n* It would good to have the definitions for the metrics in Table 1 and Table 2 (Similarity, Novelty, Atomic Stability (AS), Molecular Stability (MS)). Putting them in the appendix is fine.\n* What is the distinction between EDM and this method? A clear distinction would provide more clarity on the novelty - from what I can tell it is the inclusion of SDE formulation. As such, it would be good to clarify the difference between SDE and diffusion models in general terms first. Section 4.1 has SDE definitions, but not a contrast to standard diffusion.\n* How would you compare your work to Torsion Diffusion (Jing, Bowen, et al. \"Torsional Diffusion for Molecular Conformer Generation.\" arXiv preprint arXiv:2206.01729 (2022).)?\n* Could you provide more details on the molecular fingerprint - how does it capture 3D similarity related to positions?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bfwngiOlXN8", "reviewer": "Reviewer_ZV34", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new diffusion model for generating 3D molecules using the guidance energy function. In order to incorporate various energy functions into the proposed framework, the authors theoretically demonstrate that those functions are invariant to transformations, which is an essential feature for 3D molecule representations. The proposed model outperforms its predecessor by a wide margin. Additionally, the representation makes it easier to comprehend how the model performs as intended.", "summary_of_the_review": "Despite the fact that the novel part is minor, it is theoretically supported, which is a plus. However, generating molecules with target structures appears to be unfair and requires improvement.", "strength_and_weaknesses": "[Strength]\n* Incorporating energy functions is well supported by the theories.\n* In the task of generating molecules with desired quantum properties, the proposed framework significantly improves its baseline, EDM. \n* The paper is easy to read and well organized.\n* The visualizations help to understand how the model behaves in different settings.\n\n\n[Weaknesses]\n* Although their approach is novel, the core model relies on the existing model EDM with just adding energy functions.\n* EDM seems to be generating more stable molecules. Insights behind this would be helpful.\n* Why not compare the model by Gebauer et al. 2022 in the task of generating molecules with target structures? This task seems to be motivated by that paper. \n* The authors, however, include EDM as a baseline. How did you create a molecule with target structures using EDM? This comparison appears to be unfair because there are no explicit terms for target structures in EDM.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "W7aK_4PM2l", "reviewer": "Reviewer_BGNw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors present an extension of equivariant Diffusion Models (as presented by Hoogeboom et al.) for conditional molecular generation. The method, named “Equivariant Energy-Guided SDE” (EEGSDE), introduces an additional property prediction network that acts as a guiding force in the generation process. The property prediction network is trained in parallel with the reverse diffusion process, and the network's gradient is taken to be the additional force. \n\nThe authors evaluate their conditional generation on multiple metrics—generation novelty, atom stability, molecule stability and the mean absolute error given by the trained property classifier. Under these metrics, the main improvement of the work is in terms of MAE and novelty, with the note that it is unclear how representative the novelty metrics are due to the training procedure of the prediction network. ", "summary_of_the_review": "At this moment in time, I do not believe the paper to be ready for publication at the ICLR conference. Most importantly, the current questions regarding the metrics used for evaluating the method make it hard to judge the significance of the work. With these questions addressed in the rebuttal, I would be willing to increase my score. \n\n--- \n\nBased on the newly presented results using Gaussian software to compute the properties of generate molecules I have raise my score (5->6). ", "strength_and_weaknesses": "With their proposed method the authors tackle the currently relevant and exciting problem of conditional molecular generation. The explicit focus on conditional generation is an important contribution to the molecular design space. The paper is very well structured and sufficiently places their contribution in the context of prior work on diffusion models. Although, this connection could be strengthened by making this connection explicit in the name of the method (eg. Equivariant Energy-Guided Diffusion Models). \n\nHowever, there are a number of issues with the paper that I would like to see addressed:\n\n1. Conceptually, how does the conditional generation using the prediction method differ from the conditional generation as done by Hoogeboom et al. Most importantly, what are the differences between conditional samples generated by the presented method and the EDM method? \n2. I believe it would be interesting/needed to include a note as to why the loaded term “Energy Model” is used to refer to the property prediction model. Aside from the gradient of the property prediction network being used as a guiding force, I do not see any direct relation between the MSE loss and the molecular energy. Using the term “Energy” in this context might give the reader the impression some form of potential energy or free energy guidance is used. \n3. It is unclear to me how representative the MAE as a metric for evaluation is. Am I correct in assuming that the trained property prediction model of the presented method is used to evaluate the others? \n4. Related, it would be good if the authors could clarify how representative the current novelty scores are and which subsets of the dataset are used for calculating the novelty score for each model. As I understand it, only a subset of the data is used and as such the novelty score should be inflated compared to when using the entire dataset. Contrary to this expectation, the novelty score for the SDM method seems lower than presented in the original paper (Table 2 of Hoogeboom et al.)\n5. Also related, I’m unsure about how representative atom stability and molecular stability are as a metric. Especially in the context of this work where the bonds have to be estimated heuristically. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster