Latent 3D Graph Diffusion
Authors: Yuning You
Ruida Zhou
Jiwoong Park
Haotian Xu
Chao Tian
Zhangyang Wang
Yang Shen
Venue: ICLR 2024 poster (2024)
Keywords: 3D graphs
latent diffusion models
in/equivariant representations
Primary area: generative models

Abstract: Generating 3D graphs of symmetry-group equivariance is of intriguing potential in broad applications from machine vision to molecular discovery. Emerging approaches adopt diffusion generative models (DGMs) with proper re-engineering to capture 3D graph distributions. In this paper, we raise an orthogonal and fundamental question of in what (latent) space we should diffuse 3D graphs. ❶ We motivate the study with theoretical analysis showing that the performance bound of 3D graph diffusion can be improved in a latent space versus the original space, provided that the latent space is of (i) low dimensionality yet (ii) high quality (i.e., low reconstruction error) and DGMs have (iii) symmetry preservation as an inductive bias. ❷ Guided by the theoretical guidelines, we propose to perform 3D graph diffusion in a low-dimensional latent space, which is learned through cascaded 2D–3D graph autoencoders for low-error reconstruction and symmetry-group invariance. The overall pipeline is dubbed latent 3D graph diffusion. ❸ Motivated by applications in molecular discovery, we further extend latent 3D graph diffusion to conditional generation given SE(3)-invariant attributes or equivariant 3D objects. ❹ We also demonstrate empirically that out-of-distribution conditional generation can be further improved by regularizing the latent space via graph self-supervised learning. We validate through comprehensive experiments that our method generates 3D molecules of higher validity / drug-likeliness and comparable or better conformations / energetics, while being an order of magnitude faster in training. Codes are released at https://github.com/Shen-Lab/LDM-3DG.

Published as a conference paper at ICLR 2024
LATENT 3D GRAPH DIFFUSION
Yuning You', Ruida Zhou², Jiwoong Park', Haotian Xu', Chao Tian', Zhangyang Wang", Yang Shen'
Texas A&M University, UUniversity of California, Los Angeles, University of Texas at Austin [yuning. you, siwoong.park. hx105, chao.tian, yshen]@tamu. ruida www.uclllaacc. edu, atlaswang@utexas..
ABSTRACT
Generating 3D graphs of symmetry-group equivariance is of intriguing potential in broad applications from machine vision to molecular discovery. Emerging ap- proaches adopt diffusion generative models (DGMs) with proper re-engineering to capture 3D graph distributions. In this paper, we raise an orthogonal and fun- damental question of in what (latent) space we should diffuse 3D graphs. We motivate the study with theoretical analysis showing that the performance bound of 3D graph diffusion can be improved in a latent space versus the orig- inal space, provided that the latent space is of (i) low dimensionality yet (ii) high quality (i.e., low reconstruction error) and DGMs have (iii) symmetry preserva- tion as an inductive bias. @ Guided by the theoretical guidelines, we propose to perform 3D graph diffusion in low-dimensional latent space, which is learned through cascaded 2D-3D graph autoencoders for low-error reconstruction and symmetry-grout invariance. The overall pipeline is dubbed latent 3D graph diffusion. @ Motivated by applications in molecular discovery, further tend latent 3D graph diffusion to conditional generation given SE(3)-invariaan attributes equivariant 3D objects. + We also demonstrate empirically that out-of-distribution conditional generation can be further improved by regular- izing the latent space via graph self-supervised learning. We validate through comprehensive experiments that our method generates 3D molecules of higher validity drug-likeliness and comparable while being an order of magnitude faster in training. Codes are released at aEtps://qithut. com Shen- Lab/ Lab/LLa-b/D- 3DG
better conformations / energetics,
INTRODUCTION
Generative AI is shifting the paradigms in broad appli- cations, with its intriguing potential to simulate various real-world data (Ho et al.] 2020 Song al. 2020). In this paper, the research focus is 3D graph generation (see Fig. 1 and Sec. for definition), which is of significant needs in scientific discovery ranging from biomolecules (Morris) & Lim- Lilly 2008) spatial transcriptomics (Burgess 2019). The emerging approaches (Hoogeboom et al.
3D Graph M
1 M = (9.C)
2022 Wu al. 2022 Vignac et al., 2023; Morehead &
Topo-Feature g
Cheng 2023) adopt diffusion generative models (DGMs)
(Cao al., 2022 Croitoru et al. [2233] Yang et al.) 2022).
class of generative Al that witnessed startling empirical
Geom-Feature C
successes in generating Fuclidean data, with additional architectural redesigns preserve permutation and SE(3)
Figure 1: 3D graph is composed of the topo- logical (connectivities) and geometric fea- tures (spatial coordinates).
infequivariance for non-Euclidean 3D graphs.
Inspired by the seminal works (Hoogeboom et al. 2022
Wu et al. 2022 Vignac et al. 2023 Morehead & Cheng 2023) on how to build 3D graph DGMs, this paper drives the exploration by asking orthogonal, under- investigated yet fundamental question: In what space should we diffuuee 3D graphs? Typically DGMs are built directly in the product space of 3D graph topology (i.e. nodes and edges) and geometry (i.e. coordinates). Such space is

---

Published as a conference paper at ICLR 2024
intuitive choice where symmetry structures exist, but can be arduous for DGMs to capture resulting distributions (De Bortoli 2022 Liu et al. (023). The reason is that real-world data usually distribute on constrained, low-dimensional manifolss (De Bortoli 2022b. For instance, arbitrarily perturbing atom types or coordinates could lead to sterenchemically invalid molecules (Mislow) (2022) and randomly distorting expression profiles would result in biologically aberrant transeriptomics (Marsh iet al., 2022). We hypothesize that a more compact, yet informatiiv and symmetry-preserved space w.r.t. such manifold can benefit 3D graph diffusion generation in both quality and speed.
Contributions.
What (latent) space for 3D graph diffusiin? (Sec. 3.2) We provide a motivational analysis show- ing that 3D graph diffusion performance bound is related to (i) dimensionality of latent embeddings, (it) quality of latent spaces w.r.t. distributions, and (iii) symmetry preservation of DGMs. Our anal- ysis indicates that, the performance for 3D graphs can be improved in latent space, compared to that in the original space if the latent space is constructed with (i) low dimensionality and (ii) low reconstruction error, which present a trade-off, while (iii) preserving group symmetry.
How to construct compact and informative space for 3D graph diffusion with symmetry pre- served? (Sec. 3.1 Compared to Fuclidean-structure
data (Rombach et al. 2022 Jing et al. 2022), 3D graph data are much more challenging to "parametrize" due to the symmetry groups of permuta- tion and SE(3) transformations (Hoogeboom et al. 2022; Wu et al. 2022) and the abstraction of data of diverse natures Youu et al., 2020al 2021). Our solution is to learn low-dimension latent space parametrization in a data-driven manner, by pretraining a 3D graph autoencoder (AE). We innovate in building the AE architecture strategically the decomposed topological and geometric features such that the symmetry constraints of permutation and SE(3) transformations are disentangled and properly tackled in separate but cascaded AE models. DGM is then trained in the resulting latent space to model distributions, and the overall pipeline is dubbed latent 3D graph diffusion.
@ How to introduce latent diffusion for conditional 3D graph generation with additional symmetry structures w.r.f. conditions? (Sec. 3.3 Through the motivating example of molecular discovery, we further extend our pipeline to the more challenging conditional generation, where the generated 3D graphs need to be invariant scalar attributes or equivariant to 3D objects. We achieve this by redesigning 3D graph AEs and diffusion with conditional in/equivariance preserved.
How to regularize latent space for better "generalizabilit" of latent 3D graph diffasion? We empirically observe that the out-of- -f-tt-i-itiiitiion robustness can be improved with appropriate regu- larization of the latent space via graph self seff-supervieed learning; and the extent of semantic alignment Kwon al. 2022 in the latent space is indicative of generation quality.
Experimental results (Sec. 4 demonstrate that, our method is capable of generating 3D molecular graphs of higher validity/ / druu-likeelnees and comparable conformations / energetics compared to olate-of-the-arr competitors, while being at least order of magnitude faster in diffusion training. The advantage in speed can increase as the molecule size / complexity increases.
2 RELATED WORKS
3D graph generation. A 3D graph M = (G E M is composed of topology and geometry C (Barthélemy) (011) as shown in Fig. [ One inspiring example is molecules (Hoogeboom et al.)
2022 Wang et al. (022)) whose topology is G (A, B), where a e AA (nodes) contains information of atom identities and features and b E B (edges) of bond types, and geometry is e E C denoting the 3D coordinates of node, Therefore, the aim of 3D graph generation to learn the underlying distri- bution Po parametrized by 8, of the set of observed samples D = M..M.M.......
~ Pdata (M) M2.......] is the set of generated samples that M, ~ Po( (MM), and E(-) is objective function. The emerging approaches adopt varied generative models to the 3D graph space M. including autoregressive (Gebauer et al.. (201)) normalizing flows
with the optimization ming E(D, (DD,D). where D
(Satorras et al. 2021), ) and recently diffusion models (Hoogeboom et al. 2022 Wu et al., 2022
Vignac et al. 2023 Morehead & Cheng 2023), with additional architectural redesigns to certify permutation and SE(3) in/equivariance.
Diffusion models. Diffusion generative models (DGMs) (Sohl-Dickstein et al.
2015 Ho et al,
2020 (Song et al. 2020 Chen et al. (022a) have received an explosion of interest recently, for their
startling empirical successes in modeling distributions of Euclidean data, denoted as Pdata(X) in the
RRD space with dimension D. The forward diffusion process perturbs data with noises in time T
through a stochastic differential equation (x(!) ( + V2dw, ()) ~ Pdaaaa (x), x(1) ~ 7000)x)
2

---

Published as a conference paper at ICLR 2024
where w is the Wiener process (Ross 1995, and the reverse reconstructs from noisy data via dx(()) (x(t) + 2Vx In (x()))))(() dt + /2d x(0) (x). DGMs thus are targeted estimator fo RR R20 - RR parametrized by 0 with the optimization ming pxpc), (x
training
V
Inp (11 In practice, the continuous processes are
ally discretized into S steps with step size
(Sohl-Dickstein et al. 2015 Ho et al. 2020).
Latent diffusion and autoencoders for structured data. Recent work (Rombach et al. 2022 Yu et al. 2020) has shown improved generative modeling by exploring expressive generative models the latent space of Euclidean data, which is parametrized with autoencoders (AEs). As well- studied data-driven dimension reduction technique, AEs (Kramer 1991 Kingma & Welling 2013
have been recently explored for 2D topological graphs (Simonovsky & Komodakis 2018 Jin al.
2020 Tang et al. 2022
et al. 2018; De Cao & Kipi, 2018 [im et al. 2018
Woodward Huang et al. [022]] Chen et al. (022b) and point clouds (Anvekar et al.J [2022] Zeng
et al. 2022; Zhu et al. 2022a). The investigation 3D graph AEs and their impact on 3D graph
diffusion however remains open.
3
METHODS
We begin with the conclusion of theoretical analysis to elucidate the motivation behind the proposed latent 3D graph diffusion (Proposition 2 in Sec. 3.2), described informally follows:
3D Graph Diffusion Performance < Latent Space Reconstruction Quality
+ Symmetry Preservation x Data Dimensionality.
One end of the spectrum is equivariant diffusion (e.g. (Hoogeboom al. 2022 Wu et al. 2022) in the original 3D graph space which results in the highest dimensionality but lossles reconstruction. We aim at better DGMs in the constructed latent space that balances reconstruction and dimension- ality, while maintaining symmetry for better generation.
With the above guidelines, the proposed method comprises the following three components, with an overview depicted in Fig. [[... The pivotal one is the cascaded 2D-3D graph autoencoder that constructs a high-quality and lower-dimensional latent space data-driven manner, and invariant to permutation and SE(3) transformations, where the diffusion model is built upon.
Pretrained 3D graph autoencoder (AE) to map between non-Euclidean structures M and latent embeddings z: 2 h ø1 M = To oz (z). We build qualified 3D graph AE via (i) de- composing features into topological and geometric views and (ii) auto-encoding, to bypass the optimization barrier the complex data structures regarding symmetry (to achieve sufficiently
low reconstruction errors versus non-cascaded AEs, Sec. 3.1 & Fig. b)). while preserving permutation- and equivvarrre respectively.
Diffusiio generative model (DGM) to capture data distributions po in the latent space. The implemented DGM in the pipeline is standardized described in Sec.
Conditional infequi-variance fed to DGM. In SE(3) invariant conditional generation, we featur- ize -Evariainarraant conditions Xcond and overwrite the score estimator as (z( Xcond, t). We also extend the pipeline -quiivaquv generation conditioned on 3D objects (Sec. [3.3 & Fig. 3.. Besides, we investigate how to improve "generalizability" of conditional generation via regularizing latent spaces with graph self-supervised learning (Fig. ()).
The sampling procedure is cascaded with latent sampling ž ~ po and then decoding M = -hhh/ho (z). Please refer Algs. & 2 for details.
3.1 CASCADED AUTO-ENCODING ON DECOMPOSED VIEWS OF 3D GRAPHS
Our preliminary efforts attempt to build 3D graph AE directly in "one shot", that embeds and reconstructs topology and geometry simultaneously, via min e az ( on (M)), (M) The permutation- and SE(3)-invariance is guaranteed in the objective functions E(:) via graph match- ing Yu et al.) 2020) and superimposition algorithms (Kabsch) 1976) (see Append. B.1. However, substantial difficulty is encountered optimizing the 3D graph AE with the intertwined symmetry constraints (of permutation and SE(3)) on such complex data structures (see Tab. [ one-shot AE reconstruction performance on 3D molecules and Tab. 16 for more variants).
3

---

Published as a conference paper at ICLR 2024
00
(b)
(a)
Topo- Dacoder
Pretrained Encoder
Pretrained
Decoder
Latent Space
ac
Geom-
3D Graph
Diffusion Model
Decoder
ELBO
Condition
Neural Network
egg. Polarizability
Reconstruction
Optimization
Embedding
(J)
Grapph Self Supervisied Learnnng
(c)
Figure 2: (a) Pipeline overview of self-supervised latent 3D graph diffusion where the latent space is (b)
learned by an autoencoder architecture on the decomposed 3D graph topologies and geometries and can be (c)
regularized graph self-supervision through auxiliary objective during autoencoder training.
Cascaded AEs on decomposed 3D graph features with disentangled symmetry constraints. Instead of auto-encoding the 3D graph features M straightfor- wardly in one shot, propose build AEs the de- composed features topology G and geometry C where disentangled: topological
Table 1:
Reconstruction performance of AE workflows on molecule data, evaluated with topological accuracy (Recon.), Tani- moto score (Tani.), and geometric root-mean- square error (RMSE).
the symmetry constraints AE are optimized with the permutation constraint and geometric AE with SE(3) (Fig. b)). We are thus able bypass the optimization barrier in a "divide and conquer" manner, mainly inspired by the sub-band coding theory
Methods
Recon.1
Tani.
RMSE +
Random Init.
0%
9.50%
One-Shot
0.80%
1.80
Cascaded AF
79.57%
95.68%
0.69
[Vetterli] 1984): signals are divided into different frequency bands before/after encoding/decoding to be transmitted in the bandlimited system. The auto-encoding processes are formulated as:
Encoding: ZG h or (9), Zc h (,,
[cg:
Decoding: G ho C= -0, (9,2),
(1)
where the topological and geometric latent encoders ho Norar To Koi.c are constructed to be permutation- and SE(3)-invariant, respectively, together with the invariant training objec-
tives in optimization
minø1,a.02. Ep(g)sa de ( ())).) (Jin et al. 2020) and
min. Entg.cycc $2.c (9, h (1.c CC)).) Zhu et al. 2022a
Note that the 3D graph cascaded AE is pretrained for once and utilized repetitively. We adopt architectures from (Jin al.) (020) for the topological AE and (Zhu et al.) (2022a) for the geometric (see Append. B.2 for details). Tab. I demonstraatss its qualified reconstruction.
3.2 ANALYSIS OF LATENT 3D GRAPH DIFFUSION
We would like to understand how the choice of diffusion space impacts the generation quality of 3D graphs. We provide an analysis extended from prior works (Chen et al.) [022a Lee et al. 2023 Liu et al. [222cc] Block al. 2020) Lee al. (022) to the non-Euclidean space of 3D graphs, by further considering the symmetry structures via introducing symmetry- group equivariance.
Our reaching conclusions are ()) 3D graph diffusion performance could be related data dimen- sionality (Propos. 1), revealing the motivation for why needing latent 3D graph diffusion; and (ii) performance could be improved if mappings to the lower-dimensional space 3D graphs are ap- propriately constructed (Propos. 2), identifying the guidelines to building 3D graph AEs.
Setup. The featurization of a 3D graph M ((A, of size N is instantiated as XM (XA, xc) where XA E N xD the node feature matrix of dimension DA and xc E R x3 the coordinate matrix. The connection information be later determined with certain domain rules as B rule(A, (A, C). Such featurization widely adopted (Hoogeboom et al. 2022 Wu et al. 2022 Morehead & Cheng 2023) and therefore serves analysis.
Different from Euclidean data assessing the DGM performance with the "flat" distribution discrep- ancy (ps Pdata). e.g., the Ê-infinity, total variation distance (Chen et al.) 2022a or Wasserstein
4

---

Published as a conference paper at ICLR 2024
2022), there exist symmetry structures in the 3D graph space. We character- ize it by introducing the equivariance class [xalu, == {-r w (xxx) Vι E II E [), where II, 92 are the N --pemuutation and SE(3) groups. respectively. i... T(XM) = (TXA, Txc), (xM)) = (XA, xcw), VT E l E Q. Thus, the assessment for 3D graphs is formulated with the distribu- tion discrepancy on equivariance classes as (jo (atsaa)) where (o (xx]))),,))) = Pr(x's s XM [XM],.0; XM pad and Paata (xMM) = Pr(xM XM E [xM]]_.,, XM Pdata
metric (De Bortoli
We first illustrate the potential factors relevant to diffusion performance in the 3D graph space.
Proposition 1. (Performance bound of 3D graph diffusion is related to feature dimensional- ity of data and model inductive bias. See proof in Append. [A.I] Assume DGM is trained to model Pdata (XM) with po(XMM) Vt the score Vx(n((t) is I-Lipsshitz, for some n) the moment (pansa(Xas) |xa||2 2+4 is finite, and Vt 2 0 the score estimation error is bounded
that I (x (0, Illfa (XM .t) - Vx (x11)) I Eécore. Denote the second moment m = (Xsee) XMIl and suppose the DGM step size is 1. Then, holds for the 3D graph DGM assessment (total variation distance or TV below);
Tpo Pdata) S (po, II, S2) (Kaa + (LVD' + Lm + Escore) VT)
(2)
where ax-)) depends both the score estimator architecture and the symmetric groups, and N is the normal distribution of dimension D' that D' N (DA + 3).
However it is not necessary to model the distribution spanning the full RD' space, if (i) the dis- low-dimensional manifold, a.k.a. the manifold hypothe- sis (De Bortoli, [022 Fefferman et al. 2016) (which applies to molecules), and (ii) the manifold (Do Carmo & Flaherty Francis| 1992) can be parametrized. We next illustrate the factors relevant to 3D graph diffusion performance in the latent space.
tribution is supported constrainedly
Proposition 2. (3D graph diffusion could benefit from the lower-dimensional latent space if ap- propriately constructed. See proof in Append. A.2) Assume there existing mappings RD' RD". h RD" RDD that D" < D' and h is injective. Assume DGM now is trained in RD" to model p data(z) = Pr(xM: : (Xxa) = x, ~ Pdata] with pe(z), and is evaluated in R on E n ((xx]1,,)) = Pr(z: T (z) [xM]u,., z pod (as in Propos. 1), and the assumptions in Propos. I retain for the score estimator fo and mapping distribution. Then, holds:
TV e. Pdnta) I TV(P) data: Pdata)
a(po. R, Th II, (V aKLaFaaallNorr
+ +LVD" D" + Lm + Escore) VT).
(3)
where T data (()M),,))) Pr(x's T ((h((sa)) E [xM]n. X'y pdatal, and a(-) depends on both the latent diffusion architecture that a(po, h TT IT,) = a T IT) if T data = Pdata-
Both latent space and DGMs influence the generation quality. By comparing the bounds in Ineq. (original space) and Ineq. (3) (latent space), assuming that IT and ft equivariance is satisfied in both DGMs, the performance bound in the latent space could outperform that in the original space only D" D' (dimensionality) and TV data Pdata) > 0 is low (reconstruction error).
Guided by the theoretical analysis, we are motivated to seek such "qualified" manifold parametriza- tions for 3D graph DGMs, that is, Th and h with low latent dimension 1)" and low reconstruction error, which presents non-trivial trade-off. The intertwined symmetries of 3D graphs present ad- ditional challenge to the latent space. Our solutions are cascaded 3D graph AEs on decomposed features and separate symmetries, as described in Sec. 3.1
3.3 3D GRAPH GENERATION WITH EQUIVARIANT CONDITIONS
We are further interested in conditional generation of 3D graphs given another 3D object (e.g. a 3D graph M.ond), for instance, generating 3D molecules binding to given target-protein structures (Liu et al. 2022a Guan et al. 2023a). Here the -quiivaana constraint needs to be satisfied, i.e., generated 3D graphs would lrtnstate/rotate accordingly w.r.t. conditional objects.
Since the latent space is constructed invariant to permutation and SE(3) transformations, latent 3D graph diffusion be directly extended to conditional generation equivariant attributes, featur-
5

---

Published as a conference paper at ICLR 2024
(a)
Pretrained
(0)
Pretrained Decoder
Topo-
Encoder
Decoder
Z
Diffusion Model
Invariant Information
Equivariant
Information
Bc
Geom-
Decoder
ELBO
(b)
Condition
Figure 3: (a) Pipeline overview of latent 3D graph diffusion conditional equivariant 3D objects. (h) Au-
toencoder architecture the 3D graph topology and complex geometry.
ized as Xcond, by simply overwriting the score estimator fo (z(z() Xeood; f). We extend our pipeline
accordingly into equivariant conditional generation (see Fig. 3[a) for overview).
Complex encoding. We keep the topological AE as in Sec. 3.1 and feed the geometrical AE with the 3D graphs the complex (of generated molecules and conditional proteins) (Stärk et al. 2022).
Equivariant decoding. The latent space is still constructed invariant to (i.e. insensitive to) the rigid- body pose conditional objects, i.c., the absolute geometry of conditions. Since conditional objects always accessible during decoding/geneen
we additionally feed them into the geometric decoder to provide such information as shown in Fig. B). where the decoder implemented to be equivariant based upon (Stärk et al. 2022). The auto-encoding processes are then formulated as:
Encoding: ZG h eac. (9)),
he, Mcoud),
z [zg; zcl;
Decoding: Ç - de
Th dec, [ManG zc).
(4) Thereafter, during latent diffusion, we learn to restore the latent distribution of topology and relative geometry given conditional objects po(z)Meona). which is achieved by overwriting the score estimator (z (t) Mecond, t). See Append. C.1 for more details of architectures.
Graph self-supervised learning as auxiliary pretraining objectives. We also explore how 3D graph AEs constructed differently would impact latent diffusion performance. We leverage graph self-supervised learning You et al.] 2020a) to regularize the latent space, by acting as the auxiliary objective (i.e. multi-task learning) during topological and geometric AE pretraining. See Append. B.3 for more details. Inspired by (Kwon et al., (022). we conjecture the better "semantics-awwre" latent space is more amenable to DGMs capturing distributions. We thus quantify the extent of and verify it is indeed indicative of 3D graph generation quality (Sec.
EXPERIMENTS
We evaluate the proposed pipeline, (self-supervised) latent 3D graph diffusion, three scenarios of real-world applications 3D molecules: unconditional generation (Hoogeboom et al. 2022) (Sec.
4.1
invariant generation conditioned on quantum properties (Sec. 4.2). and equivariant generation
conditioned on protein targets (Guan et 2023a (Sec. 4.3) Before detailing the results, briefly
summarize our main findings as follows.
Latent 3D graph diffusion generates topologically more valid and geometrically stable molecules against competitors, for which AE's reconstruction quality critical (results (i,vi).
Latent diffusion can better capture data distribution, even the diiit-fffdsttiutiion"" scenario. Such capability is related to "semantic awareness" of the latent space (results (ii,iv,v,vii))
An additional bonus of latent diffusion is training efficiency: It trained faster in the lower- dimensional latent space, by an order of magnitude than competitors (results vi)).
4.1 UNCONDITIONAL GENERATION OF 3D MOLECULES
Configurations. We pretrain our topological and geometric AEs on the sarge-scale public databases ChEMBL (Gaulton et al. 2012) and PubChemQC (Naatt & Shimazaki, 2017), respectively, which can be repetitively utilized in almost all later experiments. We benchmark our method on the datasets of QM9 (with 100K smaller molecules up 29 atoms) (Ramakrishnan et al. 2014) and

---

Published as a conference paper at ICLR 2024
Drugs (with 450K larger molecules up to 181 atoms) (Axelrod & Coomez-oomaarel) 2022 follow-
ing (Hoogeboom et al. 2022; Wu et al. 2022 Vignac et al. 2023 Morehead & 2023) The
performance is evaluated both validness of generated molecules (whether obeying stereochem- ical constraints) and distribution discrepancy with training data (whether capturing the distribution
of observed molecules). See Append. [.1 for more details.
We compare with state-of-the-art (SOTA) competitors, including non-diffusion based models ENF (Satorras et al. (2221, G-Schnet (Gebauer et al., 2219) and diffusion models GDM, EDM (Hooge- 2022). MiDi [Vignac el al. 2023), GCDM (Morehead & Cheng 2023).
boom et al. 2022] Wu et al.
GraphLDM, and GeoLDM (u et 2023).
Results. The results of unconditional generation are shown in Tabs. 2 We obtain the following observations based on the validness and distribution assessments.
Table 2: Unconditional generation evaluation on validness of 3D molecules. Valid: proportion of (POF)
chemically valid molecules; Valid&Uni: POF chemically valid and unique molecules; AtomSta: POF atoms with correct valency: MolSta: POF molecules without unstable atoms. Numbers(std) in red are the best results.
QM9
Drugs
Methods
Mean
Valid
Valid&Uni
AtomSta
MolSta
Valid
AtomSta
ENF
40.2
39.4
85.0
4.9
42.37
G-Schnet
85.5
80.3
95.7
68.1
82.40
GDM
97.0
63.2
90.8
75.0
81.50
GDM-Aug
90.4
89.5
97.6
71.6
91.8
77.7
86.43
FDM
91.9(0.5)
90.7(0.6)
98.7(0.1)
82.0(0.4)
92.6
81.3
89.53
EDM-Bridge
92.0
90.7
98.8(0.1)
84.6(0.3)
92.8
82.4(0.8)
90.21
GCDM
94,8(0.2)
93.3(0.0)
98.7(0.0)
85.7(0.4)
89.0(0.8)
92.30
MiDi
97.9
97.0
97.9
84.0
78.0
82.2
89.50
GraphI.DM
83.6
82.7
97.2
70.5
97.2
76.2
84.56
GruphLDM-Aug
90.5
89.5
97.9
78.7
98.0
79.6
89.03
GeoLDM
93.8(0.4)
92.7(0.5)
98.9(0.1)
89,4(0.5)
99.3
84.4
93.08
Ours
100.00(0.00)
95.27(0.25)
97.57(0.02)
86.87(0.23)
100.00(0.00)
80.51(0.08)
93.37
Table 3: Unconditional generation evaluation distribution discrepancy with test data. NLL denotes negative log-likelihood values estimated by diffusion models, and other metrics represent total variation distances (x le- 2) of certain molecular properties, between generated and observe molecules, lower the better.
Methods
NLL
MW
ALogP
PSA
QED
FCD
Energy
EDM
-1.22
(..38.38)
0.85(0.12)
2.37(0.18)
58.04(0.39) 0.87(0.05)
2.81(
Ours
-3.48
2.52(0.39)
0.91(0.10)
1.22(0.12)
1.04(0.05)
47.66(3.42) 1.87(0.18)
(i) Quality of AEs is critical for latent diffusion validness performance. Our latent diffusion generates more valid and diverse molecules topology and stable molecules in geometry versus SOTAs, as shown in Tab. 2 We realize these two perspectives of validness measurements are actually linked to the reconstruction quality of topological and geometric features of 3D graphs, that means, the quality of AEs is the bottleneck of validness assessment. As numerically reflected in Tab. I. the topological AE is better trained for reconstruction than geometric in the sense of molecular data, which interprets the above validness results. This phenomenon also echoes motivational analysis in Sec. 3.2 on AEs. Overall, our method leads to the best validness performance on average compared to SOTAS, credit the well-constructed and trained 3D graph AE.
(ii) Diffusion in latent space better captures data distri- butions. Compared to EDM in the original 3D graph space, latent diffusion generates molecules of more similar property distributions training data in 5 out of metrics, shown in Tab. 3 Results in other metrics such as Hellinger distance and Wasserstein metric are presented in Append. E. We also pro- vide the probabilistic measure of negative log-likelihood (nor- malized on variables to remove the dimensionality bias) esti- mated by diffusion models, demonsirate that our approach appropriately captures data distributions. The results again echo our motivational analysis in Sec. o.2 latent diffusion. Another more intuitive interpretation is that the lower-dimensional latent em- beddings preserve the more semantically important bits data (Rombach et al.) 2022), facilitating diffusion to learn distributions of molecules with similar properties.
Table 4: Training time per epoch (im- provement) in unconditional genera- tion. seconds; minutes.
Methods
QM9
Drugs
EDM GeoLDM
146s
349m
356s
1307m
Ours
12s (112x)
10m (134x)
7

---

Published as a conference paper at ICLR 2024
Furthermore, since latent diffusion is trained in the (Euclidean) space of lower dimensionality and does not rely on the complicated message passing mechanism in score estimators, it provides an additional bonus of (iii) 12 / 34 times faster training on the smaller / larger dataset in Tab, A
4.2 CONDITIONAL GENERATION ON (INVARIANT) QUANTUM PROPERTIES
Configurations. We benchmark our approach on QM9 (Ramakrishnan et al. 2014) following (Hoogeboom et al., 2022) where molecules are annotated with six quantum properties which are invariant to SE(3) transformations. Models take additional featurized properties as inputs, and are evaluated with the mean absolute error (MAE) between the conditional and oracle-predicted prop- erties. We also examine whether models can correctly generate molecules conditional on properties different from training, which we refer as the "out-of-distribution" setting (OOD, versus ID).
We compare with the representative EDM (Hoogeboom et al. 2022), which however poses a strong bias between molecular sizes and property ranges. We argue restricts the applicability of diffusion models, and thus remove the bias for a more realistic evaluation. See Append. D.2 for more details.
The results of conditional generation on invariant attributes are shown in Tab. 5 We achieve the following observations through comparing between w/o and w/ latent diffusion in ID and OOD, and w/o and w/ applying graph self-supervieed learning (GSSL) during AE training.
Results.
Table 5: Conditional generation on six quantum properties(unit) evaluation. Numbers represent the mean absolute error between conditional and oracle-pedicted properties (Satorras et al. 202T). the lower the better. ID: in di-iistributon;; OOD: out-of- distrrbbbtt GSSL: graph supervisdd learning.
Methods
o(Bohr")
Ac(meV)
(me
er. (me
p(D)
C, mol K)
Random
41.00
193.36
103.30
121.83
8.40
13,56
EDM
20.15
287.00
158.70
166.20
7.01
13.63
ID
Ours
15.56
107.14
54.62
63.08
6,33
13.66
Ours-GSt Ours-GSSL
16.43
113.15
55.03
66.53
9.22
13.65
Random
73.03
344.43
183.00
217.01
14.96
24.15
EDM 55.70
1561.9
1196.80
228.20
19.13
38.42
OOD
Ours 32.06
363.13
109.30
178.69
22.18
31.12
Ours-GSSL 30.07
388.31
103.86
179.41
26.89
40.82
(iv) Latent diffusion leads to better generation quality conditional on ID or OOD properties. Our latent diffusion achieves lower MAES than EDM for 5 out of ID and OOD properties. The greater challenges appear in the OOD setting (where the random baseline is pretty strong) and also the benefits of latent diffusion. This again demonstrates the compact comantics-focuseed latent space is more amenable for diffusion model learning, than the 3D graph space.
For a deeper understanding of what is learned by latent diffusion, we visualize the generated molecules conditional on different polarizability values (a)) in Append. F.2 which are expected less isometrically shaped for larger a (Wu et al. 20012). We find our model learns to generate molecules of larger a by increasing the chain length (Khan al. 2002, Rodrigues al. 2023). This behavior is different from that in EDM, which needs to prescribe the molecular size before generation. It points a direction to explicitly bridging between latent and semanties, as in (Liu al. 20226).
(v) Graph self-supervised learning (GSSL) improves OOD generation quality if further incor- porating "semantics-awareenss"" Applying GSSI. during AE training improves out metrics in the OOD setting. We additionlly explore on when GSSL would benefit latent diffusion. Based on the previous results and inspired by (Kwon et al. 2022). we conjecture the more "semantics- aware" latent space is easier for DGMs to capture distributions. We thus quantify such concept with the homogeneity ratio (Kwon et al. 2022) in the latent space, i.e. how consistently certain direction contributes increasing property values for all data points. We find only for properties a and E.. GSSL improves the homogeneity ratio for certain direction as shown in Figs. I0 & [I] corresponding to the observed improvements of OOD results. See Append. E.2 for more details. This indicates a future potential to design semantics-specific GSSL. tasks to boost latent diffusion, prior knowledge presented for the designated generation scenario.
4.3 CONDITIONAL GENERATION BINDING TO (EQUIVARIANT) PROTEIN TARGETS
Configurations. We retain the topological AE, pretrain complex geometric AE and benchmark method on the dataset of CrossDocked (100K complexes) (Francoeur et al. 2020) following
8

---

Published as a conference paper at ICLR 2024
(Guan al. 2023a The evaluation is the potentness of the generated 3D molecules justified by (in topology) drug- trug-likelinnes (QED), synthesizability (SA), and (in geometry) binding affinity with protein targets computed with AutoDock Vina (Huey et al. 2012) (HiAff: the proportion of generated molecules with higher affinity than reference). See Append. D.3 for more details.
We compare method with SOTA competitors including non-diffusion based models LiGAN (Ragoza et al. 2022), GraphBP (Liu et al. 2022a). AR (Luo et al. 2021), Pocket2Mol (Pngg et al, 2022) and diffusion models TargetDiff (Guan et al. (023a), DecompDiff' (Guan al. 2023b). DifTS- BDD (Schneuing et al. 2022). Since competitors are all pocket-dependent that pocket information utilized during generation, for fair comparison, we also use via filtering the closest molecules to pocket residues after generation. The result of the pocket-free version without filtering is also reported for reference.
Table 6: Conditional generation on protein binding targets evaluation. Assessment metrics QED/SA & Vina scores are caleulated with RDKit (Landrum) 2013) & AutoDock (Huey et al. 2012 respectively.
Methods
Time!
QEDt
SAT
HiAff
Vinal
VDock|
Vina (Top-10%)
Diversityt
LiGAN
0.39
0.59
21.1%
-6.33
0.66
GraphBP
0.43
0.49
14.2%
-4.00
-7.16
0.79
AR
21
0.51
0.63
37.9%
-5.75
-6.75
0.70
Pocket2Mol
390m
0.56
0.74
48.4%
-5.14
-7.15
-8.71
0.69
TargetDiff"
340m
0.48
0.58
58.1%
-5.47
-7.80
-9.66
0.72
Dif
0,46
0.55
-7.33
-9.92
0.75
DecompDiff
0.45
0.61
64.4%
-5.67
-8.39
0.68
Ours
6m (135x)
0.60
0.71
48.08%
-5.23
-6.85
-12.34
0.80
(vi) Latent diffusion generates more potent molecules for target proteins. For conditional gen- eration of 3D molecules equivariantly conditional protein structures (Tab. (). latent diffusion gen- erates molecules of the highest drug- -ile-likeinness and diversity and the second-highes synthesizability. This superior performance in topology generation is consistent with our previous experiments. and not sensitive to using protein pockets or not. As to geometry generation, although our method un- derperforms in the population evaluation of binding affinity, without AutoDock Vina docking (Vina) or with docking to change geometries (VDock), partly due to increasing diversity, it significantly outperforms others in the top 10% most potent binders, following the DiffSBDD evaluation. Fur- thermore, pre-training 3D graph AE with GSSL (Sec. 3.3 which regularizes the learning of the latent space, improves the Vina Dock score (Tab. 17
Qualitatively, also find (vii) latent diffusion has the potential to model the pocket distribu- tion. We visualize the generated 3D molecules before pocket filtering in Fig. [2] where the circle area is potential pocket that is not reflected in the reference molecule, and method generates molecules appropriately covering both the reference and potential pockets. This of significance in real-world drug discovery when exploring druggable protein pockets is pressing (Pérot al. 2010)
5 CONCLUSIONS
This paper proposes an effective and efficient pipeline for 3D graph generation, dubbed latent 3D graph diffusion. It captures the 3D graph distribution by first encoding them into the latent space, training diffusion model accordingly, and then decoding back to 3D graphs. The key questions explored in this paper are: why latent diffusion is useful for 3D graphs (motivational analysis, Sec. 3.2 how to parametrize the 3D graph latent space (auto-encoding decomposed views, Sec. 3.1).
and what latent spaces better benefit latent 3D graph diffusion (semantics-awareneee Sec. 4.2
In applications, we also extend latent diffusion to the more significant conditional generation on invariant attributes (Sec. 4.2 equivariant 3D objects (Sec. 4.3) Experiments in drug discovery verify the superiority of our pipeline in both generation quality and diffusion training efficiency.
Although study illuminates the potential of generative modeling in the latent space for complex non- non-Euclidean data. it remains intriguing how to explicitly regularize the latent space with specific priors beyond our GSSL strategies.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This project was in part supported by US Army Research Office Young Investigator Award W91 INF2010240 and the NSF Al Institute for Foundations of Machine Learning (Z. Wang) as well as the National Science Foundation (CCF-1943008 to Y. Shen). Portions of this research were conducted with the advanced computing resources provided by Texas A&M High Performance Re- search Computing.
REFERENCES
Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems, 34: 480-493, 2021.
Tejas Anvekar, Ramesh Ashok Tabib, Dikshit Hegde, and Uma Mudengudi. Vg-vae: A venatus geometry point-cloud variational auto-encoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2978-2985, 2022.
Simon Axelrod and Rafael Gomez-Bombarelli. Geom, mnnoyyannottaed molecular conformations for property prediction and molecular generation. Scientific Data, 9(1):185, 2022.
Marc Barthélemy. Spatial networks. Physics reports, 499(1-3):1-101, 2011. 2011.
Normand J Beaudry and Renato Renner. An intuitive proof of the data processing inequality. arXiv preprint 1rX 100.0040, 2011.
Adam Block, Youssef Mroueh, and Alexander Rakhlin. Generative modeling with denoising auto- encoders and langevin sampling. arXiv preprint arXiv:2002.00107, 2020.
Darren Burgess. Spatial transcriptomics coming of age. Nature Reviews Genetics, 20(6):317-317, 2019.
Hanqun Cao, Cheng Tan, Zhangyang Gao, Guangyong Chen, Pheng-Ann Heng. and Stan Z Li. A survey generative diffusion model. arXiv preprint arXiv:2209.02646, 2022.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.1 2022a.
Xiaohui Chen, Yukun Li, Aonan Zhang, and Li-ping Liu. Nvdiff: Graph generation through the diffusion of node vectors. arXiv preprint arXiv:22/ 10794, 2022b.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: survey. IEEE Transactions Pallern Analysis and Machine Intelligence, 2023.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv: 2019.
Valentin De Bortoli. Convergence denoising diffusion models under the manifold hypothesis. arXiv preprint arXiv:2208.053: 2022.
Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. arXiv preprint 1rXiv:1805. 11973, 2018.
Manfrreo Perdigao Do Carmo and J Flaherty Francis. Riemannian geometry, volume 6. Springer, 1992.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 2944):983-1049 2016.
Paul G Francoeur, Tomohide Masuda, Jocelyn Sunseri, Andrew Jia, Richard B lovanisci, Ian Snyder, and David R Koes. Vinee-dimeesional convolutional neural networks and cross-docked data set for structure- based drug design. Journal of chemical information and modeling, 60(9):4200- 4215, 2020.
10

---

Published as a conference paper at ICLR 2024
Anna Gaulton, Louisa Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, (ODDI::DI10-DDII 2012.
Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapted generation 3d point sets for the targeted díscovery of molecules. Advances in neural information processing systems, 32, 2019.
Rafael Kömezz-Bombaree Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Ryan P Adams, and Alán Aspuru-Guzik Automatic chemical design using data-driven contin- uous representation of molecules. ACS central science, 4(2):268 276, 2018.
Timothy D Hirzel,
Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma. 3d equiv- ariant diffusion for target-aware molecule generation and affinity prediction. arXiv preprint arXiv:2303.03543 2023a.
Jiaqi Guan, Xiangxin Zhou, Yuwei Yang, Yu Bao, Jian Peng, Jianzhu Ma, Qiang Liu, Liang Wang, and Quanquan Gu. Decompdiff: Diffusion models with decomposed priors for structure-base drug design. 2023b.
Thomas A Halgren. Mmff mmff94s option for energy minimization studies. Journal of compu- lational chemistry, 20(7):720-729, 1999.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
Neural Information Processing Systems, 33:6840-6833 2020.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant dif- fusion for molecule generation in 3d. In International Conference on Machine Learning, pp- 8867-8887. PMLR, 2022.
Yinan Huang, Xingang Peng, Jianzhu Ma, and Muhan Zhang. 3dlinker: an (3) equivariant varia- tional autoencoder for molecular linker design. arXiv preprint
2022.
Ruth Huey, Garrett Morris, and Stefano Forli. Using autodock 4 and autodock vina with autodock- tools: tutorial. The Scripps Research Institute Molecular Graphics Laboratory, 10550(92037): 1000, 2012.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International conference on machine learning, pp. 4839-4848. PMLR, 2020.
Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion gen- erative models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIII, pp. 274-289. Springer, 2022.
Wolfgang Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallo- graphica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, 32 (5):922-922 1976.
Malek O Khan, Torbjörn Åkesson, and Bo Jönsson. Simple approach to polyampholytes based on chain polarizabilities. The Journal of chemical physics, 116(9))391-3322 2002.
Diederík Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
1rXiv:1312.6114, 2013.
Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal, 37(2):233224 1991.
Mingi Kwon, Jaescok Jeong, and Youngjung Uh. Diffusson models already have semantic latent space. arXiv preprint 1rXiv:2210.1044 2022.
Greg Landrum. Rdkit documentation. Release, 1(1-79):4, 2013.

---

Published as a conference paper at ICLR 2024
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for soore-based generative modeling with polynomial complexity. arXiv preprint arXiv:2206.06227,2222 2022.
Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory, pp- 946-985, PMLR. 2023.
Jaechang Lim, Seongok Ryu, Jin Woo Kim, and Woo Youn Kim. Molecular generative model based on conditional variational autoencoder for de novo molecular design. Journal of cheminformatics, 10(1):1-9, 2018.
Meng Liu, Youzhi Luo, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji. Generating 3d molecules for target protein binding. arXiv preprint arXiv:2204.09410 09410, 2022a.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. arXiv preprint 1rXiv:2110.07700 2021.
Shengehao Liu, Chengpeng Wang, Weili Nie, Hanchen Wang, Jiarui Lu, Bolei Zhou, and Jian Tang. Grapheg: Unsupervised discovery of steerable factors in graphs. In NeurIPS 2022 Workshop: New Frontiers in Graph Learning, 2022b.
Xingehao Liu, Lemeng Wu, Mao Ye, and Qiang Liu. Let us build bridges: Understanding and extending diffusion generative models. arXiv preprint arXiv:2208. 14699, 2022c
Xingchao Liu, Lemeng Wu, Mao Ye, et al. Learning diffusion bridges on constrained domains. In The Eleventh International Conference Learning Representations, 2023.
Shitong Luo, Jiaqi Guan, Jianzhu Ma, and Jian Peng. A 3d generative model for basture-bbase drug design. Advances in Neural Information Processing Systems, 34:6229-6239, 2021.
Xuezhe Ma. Chunting Zhou, Xian Li. Graham Neubig, and Eduard Hovy. Flowseq:
Non-autoregressive conditional sequence generation with generative flow. arXiv preprint arXiv:1909.02488) 2019.
Samuel E Marsh, Alec J Walker, Tushar Kamath, Lasse Dissing-Olesen, Timothy R Hammond, Yvanka de Soysa, Adam MH Young, Sarah Murphy, Abdulraouf Abdulraouf, Nacem Nadaf, et al. Dissection of artifactual and confounding glial signatures by single-cell sequencing of mouse and human brain. Nature Neuroscence, 25(3):306-316, 2022.
Kurt Martin Mislow. Introduction to stereochemistry. Courier Corporation, 2002.
Alex Morehead and Jianlin Cheng. Geometry-complete diffusion for 3d molecule generation. arXiv preprint arXiv:2302.044:1 2023.
Garrett M Morris and Marguerita Lim-Wilby. Molecular docking. Molecular modeling of proteins, pp. 365-382, 2008.
Maho Nakata and Tomomi Shimazaki. Pubchemqe project: large-scale first-principles electronic structure database for data-driven chemistry. Journal of chemical information and modeling, 57 (6):1300-1308, 2017.
Xingang Peng, Shitong Luo, Jiaqi Guan, Qi Xie, Jian Peng, and Jianzhu Ma. Pocket2mol: Effi cient molecular sampling based on 3d protein pockets. In International Conference on Machine Learning. pp. 17644-17655. PMLR, 2022.
Stéphanie Pérot, Olivier Sperandio, Maria A Miteva, outreix. Druggable pockets and binding site centric chemical space: a paradigm shift in drug discovery. Drug discovery today, 15(15-16):656-6f) 2010.
Camproux, and Bruno O Vill-
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, et al. Molecular sets (moses): a benchmarking platform for molecular generation models.
Frontiers in pharmacology, 11 11:566644, 2020.
12

---

Published as a conference paper at ICLR 2024
Matthew Ragoza, Tomohide Masuda, and David Ryan Koes. Generating 3d molecules conditional on receptor binding sites with deep generative models. Chemical science, 13(9):2701-2713, 2022.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):-77, 2014.
José L Rodrigues, Raphael F Ligorio, Anna Krawczuk, Renata Diniz, and Leonardo HR Dos Santos. Distributed functional-group polarizabilities in polypeptides and peptide clusters toward accurate prediction of electro-optical properties of biomacromolccules. Journal of Molecular Modeling, 29(2):49, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Sheldon M Ross, Stochastic processes. John Wiley & Sons, 1995.
Victor Garcia Satorras, Emiel Hoogeboom, Fabian Fuchs, Ingmar Posner, and Max Welling. (n) equivariant normalizing flows. arXiv preprint arXiv:2105.0900 2021.
Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blun- dell, Pietro Lió, Carla Gomes, Max Welling, et al. Structure-based drug design with equivariant diffusion models. arXiv preprint arXiv:22 13695, 2022.
Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference Machine Learning, pp. 9377-93388 PMLR, 2021.
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In Artificial Neural Networks and Machine Learning-ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part 27, pp. 412-422. Springer, 2018
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-22662 2265. PMLR, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.1344 134566 2020.
Hannes Stärk, Octavian Ganea, Lagnajit Pattanaik, Regina Barzilay, and Tommi Jaakkola. Equibind: Geometric deep learning for drug binding structure prediction. In International Conference Machine Learning, pp. 30503-20522 PMLR, 2022.
Mingyue Tang, Carl Yang, and Pan Li. Graph auto-encoder via neighborhood wasserstein recon- struction. arXiv preprint arXiv:2202.0902. 2022.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:1 14:11287-1138 1302, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710. 10903, 2017.
Martin Vetterli. dimen-iimeniional sub-band coding: Some theory and algorithms. Signal process- ing, 612):97-112, 1984.
Clement Vignac, Nagham Osman, Laura Toni, and Pascal Frossard. Midi: Mixed graph and 3d denoising diffusion for molecule generation. arXiv preprint arXiv:2302.09048, 2023.
13

---

Published as a conference paper at ICLR 2024
Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. Comenet: Towards complete and efficient message passing for 3d molecular graphs. arXiv preprint arXiv:2206.085/3 2022.
Tianxin Wei, Yuning You, Tianlong Chen, Yang Shen, Jingrui He, and Zhangyang Wang. Aug- mentations in hypergraph contrastive learning: Fabricated and generative. Advances in neural information processing systems, 35:1999-1922, 2022.
Amelia Woodward. Evaluating autoencoder methods for building a molecule graph autoencoder. Chemistry, (12)::044-10))
Johnny C. Wu, Gaurav Chattree, and Pengyu Ren. Automation of amoeha polarizable force field parameterization for small molecules. Theoretical chemistry accounts, 131:1-11, 2012.
Lemeng Wu, Chengyue Gong, Xingchao Liu, Mao Ye, and Qiang Liu. Diffusion-based molecule generation with informative prior bridges. arXiv preprint arXiv:2209. (00::2209000865,, 2022.
Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. IEEE transactions on pattern analysis and machine intelligence, 45(2):2412-244 22299 2022.
Haotian Xu, Yuning You, and Yang Shen. Multi-modal contrastive learning for proteins by com- bining domain-informe views. In ICLR 2024 Workshop on Machine Learning for Genomics Explorations.
Minkai Xu, Alexander Powers, Ron Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. arXiv preprint arXiv:2305.01 140, 2023.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.0077 00799, 2022.
Yuning You and Yang Shen. Cross-modality and self--uper protein embedding for compound- protein affinity and contact prediction. Bioinformatics, 38 38(Sppplement 9):i68--i74, 2022.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33: 5812-5823, 2020a.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? In international conference on machine learning, pp. 10871- 10880. PMLR, 2020b.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning auto- mated. In International Conference on Machine Learning, pp. 12121-12132. PMLR, 2021.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. Bringing your own view: Graph contrastive learning without prefabricated data augmentations. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 1300-1309, 2022.
Tianshu Yu, Runzhong Wang, Junchi Yan, and Baoxin Li. Learning deep graph matching with channel-independent embedding and hungarian attention. In International conference on learning representations, 2020.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint arXiv:2110.06978, 2022.
Jinhua Zhu, Yingee Xia, Chang Liu, Lijun Wu, Shufang Xie, Tong Wang, Yusong Wang, Wengang Zhou, Tao Qin, Houqiang Li, al. Direct molecular conformation generation. arXiv preprint 22020:2200.01355. 2022a.
Zhaocheng Zhu, Chence Shi, Zuobai Zhang, Shengchao Liu, Minghao Xu, Xinyu Yuan, Yangtian Zhang, Junkun Chen, Huiyu Cai, Jiarui Lu, et al. Torchdrug: A powerful and flexible machine learning platform for drug discovery. arXiv preprimt arXiv:2202.08320, 2022b.
14

---

Published as a conference paper at ICLR 2024
A PROOFS FOR ANALYSIS
A.1 PROOF FOR PROPOSITION
Proposition 1. Assume DGM is trained to model Paata(Xx) with Po Vt 2 O the score Vx In p() is L-Lipschitz, for some > 0 the moment E (xaaa((xx) 12+ is finite, and Vt 2
the score estimation error is bounded that E ppe)((()) o M .t) Vx In (() (xM Il I Scoree De-
note the second moment m = Ema (Xxr) (oaa))|Ix |xx and suppose the DGM step size is Then, it holds for the 3D graph DGM assessment:
TV (fo, Pdata) Σ a<pe, (2) (V (KL(pasaallND)))
+ (LVD' + Lm Escore VT)
V
where a-)) depends on both the score estimator architecture and the symmetric groups, and ND is the normal distribution of dimension D' that D' N x (Da + 3).
Proof. We bound the distribution discrepancy of the total variation distance between generated and training 3D graph features with the following inequalities:
Tpe Pdataa (
a(pe, 11, 92) (pe, Pdata)
2 aalo..II, 12) (T..))((KKKI)) (VKI IN + (LVD' + Lm Escore) VT).
where (a) results from the definition of a(po, II TV Pares) and (b) is the result from TV(ps-Paatal
(Chen et al. 2022a) Theorem 2. Note by data processing inequality (Beaudry & Renner 2011),
a (po, 11.2) 1|. We thus reach the conclusion.
Symmetry-related term a(p,, I1, 92. How does the DGM architecture regard symmetry structures affects the generation performance of 3D graphs, reflected in the term a(po, I). Specifically, the DGM Po constructed in/equivariant to permutation II and SE(3) transformations Q e.g. in Hoogeboom al. 2022), apps II, (2) reaches the minimal value. That is, Po satisfies:
(xM) Po (xx])1,)). if I(xM) =1 otherwise.
O, where T(xM) E 1] is the indicator function which specifies only one xM in each [xM]n. with value and 0 for others. This means the DGM treats all members in one equivalent class the same and thus only learns to capture one data point in each class. The ultimate distribution can be recovered later via p/g (Xx) = po (x PuPa where X [xM]u,n, the probability of the transformation operators which are uniformly sampled. Given arbitrary learned distribution go that retains d = Po, we have:
and pn: po denote
TV (go, Pdata)
ta)
[Pr[XM XM E S, XM ~ {0 Pr(xM: : XM E S, XM ~ Pdlta) SCRD'
sup
(b)
Σ
go (XM1)
Paata (IxM)n,))P..Pg
Xx
Σ
Σ
(xe(x>)) Panta (aaeaa((xxx ((xslun)))
Kylun Xx XM Exxlllln
Σ
To (o([xM],,)) Paata (1MM),, p.P))
Ixsdn o. (aaln +f0
Σ
Σ
(po(x>s)= - Paaca (xx pa)
Ix o. Ial xu E waggininnnnnn i ((xa))=
(po.Paatal
where (a) results from the definition of the total variation distance, (b) results from the equiv-
alent definition of the total variation distance in the & distance, and denoting S+ = {XM : go (XM) (xxxa((Xx]]u.o) Pn, pa). (c) results from grouping the summation in the individual equivalent classes, (d) results from the inequality (xM) < ((((x)),,))) (c) results from the in/equivariant construction of po, and (f) results from the definition of the total variation distance.
15

---

Published as a conference paper at ICLR 2024
Thus, given an arbitrary learned distribution go without altering ge. always can reframe it to be in/equivariant as po that a(pe, 12) S a(go, Q), leading to tighter bound in Prop.
A.2 PROOF FOR PROPOSITION 2
Proposition Assume there existing mappings h RDD IR"", Th RDD + DDD that D" D' and h is injective. Assume DGM now is trained in RD" to model 7 data = Pr(XM : (xM) n, XM Pdata] with po(z), and it is evaluated in RDD' T (x((xM)),)) Pr(z (z) E [xllu, z pe) (as in Propos. 1), and the assumptions in Propos. retain for the score estimator fo and mapping distribution. Then, it holds:
TV(P o. Pdata) X TV(P
data: Pdata)+
a(po. R, h. 11, (v anallWWe + (LVD" + Lm + Escore) (scoree(TT
where T data (XM))))) = Pr(x To (x(x)) E [xM]L,, Xy1 ~ Pdstal. and ā(-) depends on both the latent diffusion architecture that appo, h, h II, a a II, if data = Pdata-
Proof. We bound the distribution discrepancy of total variation distance between generated (from submanifold then reconstructed) and training 3D graph features with the following inequalities:
TV o, Pdata)
(a)
TV(P data Pdata) TV o. Faatal
TV data Pdata) + TV o. Di datal
(Ve d' data)
TV dataa)
TV( data Pdata) + a(pe. h. Th (1, (po, Pdata)
TV( data Pasta) appo, h. h II, .2) KL andlllWD (LVD" + Lm Escare) /T)
where (a) is achieved by constructing an intermediate distribution and applying the triangle equality,
TV a. F. that we have < (b) comes from the rewriting by denoting a(po. h 11, TV [pe. passs)
ā (po. h. Th 11) < 1 due to the data processing inequality, and (c) is the result from Chen et al, 2022a) Theorem 2. We thus reach the conclusion.
Besides. when the latent mappings are capable of recovering the complete data distribution that & data = Pdata. we have a(pe. h h 11, = o.
a.ess) = a 1 e, II, ?2)
TV(pp. data) TV (V(ppoPaass))
where (a) results from the ideal reconstruction of h, h injectivity of h and data processing in- equality, which matches the symmetry-related term in Prop. I.
B ADDITIONAL. DETAILS FOR 3D GRAPH AES
B. PRELIMINARY EFFORTS IN BUILDING 3D GRAPH ONE-SHOT AES
Our preliminary efforts attempt to build 3D graph AE simply following the topological graph AE workflow (Simonovsky & Komodakis 2018), as shown in Fig. 4 It compromises the following components.
Encoders & decoders. Topology and geometry information is encoded in the latent embedding, through separate encoding processes and then concatenation. The topological encoder is built with graph attention networks (Veličković al. 2017) and geometric with polarizable atom interaction neural networks (Schütt et al. (022)). The decoder takes the concatenated latent embedding as in- put reconstruct topological and geometric features, built with transformer architectures (Vaswani et al. 2017) followed by three multi-layer perceptrons for nodes, edges, and coordinates, respec- tively.
Permutation- and SE(3)-invariant loss. We enforce permutation and SE(3) invariance in the op- timization objective. Specifically, before calculating the mismatch measurement between the input
16

---

Published as a conference paper at ICLR 2024
Encoder
Nodes
Latent Embedding
Topo- Encoder
Decoder
MLP-C
Coordinates
Topo&Geont
Decoder
Cicom- Encoder
MLP-B
Edges
Graph Matching: Permutationt-Invaal Invariiin Loss
Superimposition: SEi Invvarana Loss
Figure Pipeline overview of preliminary one-shot 3D graph AE.
and reconstructed 3D graph, execute the neural graph matching algorithm Yu et al. 2020 to sort out the correct node-to-node mapping, which leads to the permutation-invanne ward, the superimposition algorithm (Kabsch 1976 is executed align the input and reconstructed coordinates with the global rotation and translation operations, which leads to the SE(3)- -E(3)-invariant loss. The final optimization objective then the weighted of these losses.
loss. After-
B.2 ADDITIONAL DETAILS FOR 3D GRAPH CASCADED AES
Related works: Latent diffusion models. Considerable research (Dai & Wipf 2019; Yu et al,
2020) has focused on enhancing generative modeling capacity by exploring expressive generative models over the latent space. VQ-VAES (Ragoza et al. 2022) proposed discretizing latent variables and leveraging autoregressive models to establish other hand, employed flow-based models as the latent prior, particularly for non-autoregressive text generation. Another line of research was inspired by the limitations of simple Gaussian priors in variational autoencoders (VAEs), which could not accurately match encoding posteriors and resulted in poor sample generation. To address this issue, (Dai & Wipf 2019 Aneja al. (021) proposed using VAEs and energy-based models, respectively. to learn the latent distribution. More recently, various works have successfully developed latent diffusion models with promising results across different applications, such as image generation (Vahdat et al., 2021 and point clouds (Zeng et al.) 2022]. Among these methods, stable diffusion Models (Rombach et al. 2022) stand out for their impressive success in text-guided image generation, demonstrating remarkably realistic results.
expressive prior. (Ma et al. 2019). the
Encoders & decoders. The topological AE is built with (Jin et al. 2020p. Specifically, the topo- logical encoder is composed of hierarchical layers of node message passing, attachment message passing, and motif message passing, and the decoder composed motif decoding, attachment de- coding, and graph decoding. The geometric AE is built with (Satorras al. 202 Zhu et al. 2022a Halgren] 1999). Specifically, the geometric encoder is composed of E(n) equivariant graph neural networks, and the decoder is composed of local node/edge message passing, global representation message passing, coordinate decoding and refinement Landrum 2013). We train both topological and geometric AES for days.
The topological encoder (Jin et al. 2020) consists of three Message Passing Networks (MPNs) encode the hierarchical graph's 's three layers. Specifically, Atom Layer MPN encodes the atom layer of the hierarchical graph (HG). It takes embedding vectors of atoms and bonds as inputs. The network propagates message vectors between atoms over several iterations, culminating in the output of atom representation for each atom. In the topological decoder: (i) Decoder Function: The graph decoder incrementally expands the hierarchical graph generate a molecule. uses the same hierarchical MPN architecture to encode motifs and atoms in the partial hierarchical graph. (ii) Motif and Atom Vectors: At each generation step, the decoder produces motif vectors and atom vectors for the existing motifs and atoms in the graph. (iii) Motif Prediction: The model predicts the next motif to attached, formulated as a classification task over the motif vocabulary. (iv) Attachment Prediction: The model predicts the attachment configuration of the next motif. focusing on the intersection of the motif with its neighbor motifs. This too is a classification task over an attachment vocabulary.
17

---

Published as a conference paper at ICLR 2024
The geometric AE (Zhu et al. 2022a) consists of the following components. (i) Bond Representation Layer: Incorporates coordinate information into bond representations. The updated bond represen- tation is derived from the previous bond representation, MLP operations coordinate differences, and global representation. (ii) Atom Representation layer: Atom representations are updated using Graph Attention Network version (GATv2) to aggregate bond representations and then further refined with MLP operations and global representation. (iii) Global Molecule Representation: The global representation is updated by aggregating updated atom and bond representations with MLP operations. (iv) Conformation Construction: Each atom's conformation is predicted and normalized by centering the coordinates at the origin, ensuring numeric stability. The final prediction of the conformation is outputted by the last block with refinement (einnementLandrunn) (2013).
Training via teacher forcing. As depicted in Fig- [[a), the decoding process is sequentially con- ducted, since the geometry decoding relies on the topology input. We remove such reliance during training to improve efficiency (that AEs can be trained in parallel and sampling not needed during training), via applying teacher forcing: The topology input for the geometry decoding is the ground truth graph rather than generated, formulated as:
Encoding: ZG h enc, (9),
zc = h C(C), C= de,
[za;
g h G
Decoding:
Reconstruction quality 3D graph AEs. The results in Tab. D demonstrate the qualified recon- struction capability of 3D graph AEs. We further plot the topology and geometry reconstruction sults for 3D graphs (3D molecules here) varied properties, as shown in Figs. and respectively. Properties include number of heavy atoms (AtomNum), number of atoms (AtomNum molecular weight (MW), octanol-water partition coefficient (ALOGP), number of hydrogen bond donors (HBD), number of hydrogen bond acceptors (HBA), polar surface area (PSA), number of rotatable bonds (ROTB), number of aromatic rings (AROM), and structural alerts (ALERTS). We do not see a strong correlation between certain properties and reconstruction quality.
We include in Append. E.I more details on conceptual and numerical comparisons between oneshot AEs and our cascaded AES.
B.3 GRAPH SELF-SUPERVISED LEARNING IN AE TRAINING
Related works: Graph self-supervised learning. Graph self- self-supervised learning, surging recently, learns empirically more generalizable representations through exploiting vast unlabelled graph data You al. 2020a by 202T 2022 You & Shen 2022, Wei al., 2022 Xu et al.) (please refer to (Xie et al. (2022) for a comprehensive review). The success of self supervision hinges on big data and carefully designed pretext tasks, to enforce specific prior knowledge in graph models.
We apply graph self-supervised learning (GSSL) during AE training to regularize the 3D graph la- tent space. The adopted GSSL, graph contrastive learning You al. (020a)), implemented as the auxiliary objective (i.e. multi-task learning), during which the node dropping augmentation is con- ducted on topological/ geometric graphs, and the contrastive loss is optimized on the representations of augmented views.
C 3D GRAPH GENERATION WITH EQUIVARIANT CONDITIONS
C. ADDITIONAL DETAILS FOR 3D GRAPH AES
3.1 The geometric AE built with (Guan et al. 2023a Satorras et al. 202T: Stärk et al.. 2022 Huey al. 2012). Specifically, Specifically, the geometric encoder is composed of E(n) equivariant graph neural networks, and the decoder is composed of independent SE())-equivariant graph matching networks, multi-head SE(3)-equivariant attentions, docking transformation and refinement.
Encoders & decoders. We retain the topological AE same as in Sec.
The conditional geometric AE (Guan et al. 2023a Stärk et al. 2022) consists of the following components. (i) K-NN Graph Representations. Ligand Graph: Represented spatial k-nearest neighbor (k-NN) graph, using atoms as nodes. The graph includes atom pairs within a 4 A distance cutoff. Receptor Graph: Constituted by residues nodes, connected the closest 10 other nodes
18

---

Published as a conference paper at ICLR 2024
T
211
130
180
121
OH
100
11.
Figure Topology reconstruction results versus dif- ferent molecular properties.
Figure Geometry reconstruction results versus dif- ferent molecular properties.
19

---

Published as a conference paper at ICLR 2024
within a 30 Å distance. (ii) IEGMN (Independent B(3))-Euivariam Graph Matching Network): This network combines Graph Matching Networks and E(3)- E Graph Neural Networks, allow- ing for joint transformation of features and 3D coordinates. (iii) Transformation Process: Ensures that any independent rotation and translation of input structures are precisely mirrored in the output, which is crucial for data-scarce problems like structural drug binding. (iv) Layer Architecture: Each layer involves specific operations that update the features and coordinates of the nodes, maintaining SE(3) invariance and employing shallow neural networks for various transformation functions.
D EXPERIMENTAL SETTINGS
D.1 UNCONDITIONAL GENERATION
Algorithm Latent 3D Graph Diffusion Pipeline
Input: 3D graph data D (M,,..,,Mwi
(9v, Cx)}, initial 2D en- h and diffusion model Polo)- To +01 3D encoder/decoder h %y $2.
coder/decoder h 10.1
> Autoencoder training: for iteration k= 1 to do
1. Latent encoding that 2 = {( g
(9) h (()-n(()) (G,C)
2. Cascaded decoding that D' {(h $2.
(ZG); $2 (G, (c) Z}
3. Updating parameters that U.U U
[K-1]
V rrc D D'), i E [1, U CG.
end for
> Diffusion model training:
1. Latent encoding that Z = =1 11. Iin (C) (G E D}.
(9), h
2. Updating parameters that for k- to K": ( flk- 1] 1) V Voeain (2, a1).
>> Sampling:
1. Sampling latent that (zc: zc) ~ Ppck
2. Reconstructing 3D graph M (G,C) that 9 = To (g (zG), C= To o/,
(G, 2c).
Background. 3D molecule generation is field within computational chemistry and drug discovery that focuses on the automated generation of three-dimensional structures of molecules. plays a crucial role in understanding the properties and behavior of chemical compounds, as well as in the development of new drugs and materials. The process of generating 3D molecular structures computationally involves predicting the spatial arrangement of atoms and bonds in molecule while satisfying various constraints, such as bond lengths, bond angles, and dihedral angles. 3D molecule generation has become essential tool in various scientific and industrial applications. In drug discovery, is employed to explore and optimize potential drug candidates, predict their binding to target proteins, and assess their pharmacokinetic properties. In materials science, 3D molecule generation aids in designing novel materials with specific properties, such as improved strength, flexibility, or conductivity.
Evaluation Hoogeboom et al. 2022). Our model's performance is assessed by evaluating the chemical feasibility of the generated molecules to determine if the model effectively learns chemi- cal rules from the data. The "atom stability' metric measures the proportion of atoms in the generated molecules that have the correct valency, ensuring that the atom configurations are chemically valid. On the other hand, the 'molecule stability' metric represents the proportion of generated molecules in which all atoms maintain stable configurations. In essence, this metric assesses whether the en- tire molecular structure is chemically feasible. In addition to the stability metrics, we also report validity and uniqueness measurements. The 'validity' metric represents the percentage of gener- ated molecules that are deemed valid according to RDKIT, a widely used software for molecular validation. This ensures that the generated molecules comply with well-established chemical rules and constraints. The 'uniqueness' metric, on the other hand, indicates the percentage of unique molecules among all the generated compounds, allowing us to quantify the model's ability to pro- duce diverse and novel chemical structures.
20

---

Published as a conference paper at ICLR 2024
The featurization of 3D molecules follows (Zhu et al. 2022b Liu et al. 2021). The validness measurement is evaluated in the chemical feasibility of generated molecules, indicating whether the model can learn chemical rules from data (Xu et al. 2023), including topologiaa validity and uniqueness, and geometric stability in atoms and molecules. The distribution measurement quanti- fies the discrepancy between the properties of observed and generated molecules, including molec- ular weight (MW), octanol-water partition coefficient (ALOGP), polar surface area (PSA), drug- likeness (QED), Fréchet ChemNet distance (FCD) (Polykovskiy et al. (0200) and conformer energy. We also provide the probabilistic measure of negative log- likeelodddood (NLL) estimated by diffusion models, which is normalized on variables to remove the dimensionality bias (Hoogeboom et al.)
2022).
D.2 CONDITIONAL GENERATION ON (INVARIANT) QUANTUM PROPERTIES
Evaluation. In this task, our objective is to perform controllable molecule generation while adher- ing to desired properties. This capability proves valuable in practical applications such material and drug design, where the focus is on discovering molecules that exhibit specific property pref- erences. To assess the performance, we use the QM9 dataset, which encompasses six properties: polarizability, orbital energies and their gap. Dipole moment, and heat capacity. To evaluate the model's 's ability to conduct property-condee conditioned generation, we follow the approach introduced by (Hoogeboom et al. 2022). We initially split the QM9 training set into two halves, each containing 50,000 samples. Next, we train a property prediction network on the first half of the dataset and subsequently train conditional models on the second half. During evaluation, we employ a range of property values, to conditionally draw samples from the generative models. We then use the property prediction network to calculate the corresponding predicted property values. To gauge the perfor- mance of actual property values and their predicted counterparts. A lower MAE indicates that the generated molecules closely align with the specified property conditions. Furthermore, to evaluate the bias of the property prediction network, directly apply it to the second half of the QM9 dataset. A smaller MAE gap between the property-conditionn cates better property-conditionin performance, demonstrating the model's efficacy in generating molecules that closely match the desired property preferences.
generation, we report the Mean Absolute Error (MAE) between the
generation results and the QM9 dataset indi-
The conditional properties are ox(Bohr'), tendency of molecule to acquire an electric dipole mo- ment when subjected to anexternal electric field; (e the energy difference between HOMO and LUMO; (me V), highest occupied molecular orbital energy; EL (me lowest unoccupied molecular orbital energy; (D),, dipole moment; C,( mol K), heat capacity at 15K.
We compare with the representative diffusion baseline (Hoogeboom et al. 2022 Different from (Hoogeboom et al. (022) that poses overly strong inductive bias between the molecule size and properties during evaluation, i.e. p(M[(xcona) x P(MMNN Xcona (N))))(((((X))o)))) (xxcond NN. MM is the molecule with N atoms, where the range of Xcond given certain N could be as narrow as (ver- sus random baseline 9.01 in (Hoogeboom et al. 2022)). ). We argue that such bias seriously restricts the applicability of the models. We thus conduct a more realistic evaluation for (Hoogcboom et al.) p(MN 2022) on p(M Xcond) PN [N, Xcond (pNN))((N) by removing the size correlation, and our model can implicitly learn the size distribution without prescribing it as prior p(N) as in (Hooge- boom et al. (022). We also construct the random baseline by randomly selecting molecules from the training set.
D.3 CONDITIONAL GENERATION BINDING TO (EQUIVARIANT) PROTHIN TARGETS
Evaluation. The evaluation metrics are on the potentness of the generated 3D molecules justified by (in topology) drug- likg-likelinnes (QED), synthesizability (SA), and (in geometry) binding affinity with protein targets (HiAff: the proportion of generated molecules with higher affinity than the reference ligand). During the evaluation, we utilize pocket information by filtering the (top 33%) closest molecules to pocket residues after generation, where the pocket of the target protein is defined by the residues surrounding the reference ligand within 10Å (Liu et al. 2022a Guan et al. 2023a). The Vina score is computed with AutoDock Vina, before docking (Vina) and after docking (VDock). Top- means only the lowest 10% of scores are evaluated.
21

---

Published as a conference paper at ICLR 2024
Algorithm 2 Conditional Latent 3D Graph Diffusion Pipeline
Input: Paired 3D graph data D
{(MI. (Mv, MMoud,N)) 3D
C.), ... (GN,,NN) Meond,))) initial 2D encoder/decoder h o/1a" 22.
encoder/decoder h. $1.0
To i and diffusion model Pojo)
> Autoencoder training: for iteration k = 1 to do
o/.00 1. Latent encoding that z {((Th (9) To d Me
((G) C), Meoud) E D).
2. Cascaded decoding that D' ((( [ To d 1 (9,99,Mann Meond, ac)),
k 2,0
o
((zc. 2c), E 2}
3. Updating parameters that L UiU end for
[k-1]
ve se D'),
> Difflusio model training:
1. Latent encoding that Z = {((h (9), h Line (C, Meond Meona)) ((G,C), Menna)) E
11.G
D).
2. Updating parameters that for is = 1 to K": 00 = 0lk-1) - 17 Vocait (2, 21).
>> Sampling:
1. Sampling latent that (zg, Zc)l Polx'I
2. Reconstructing 3D graph M = -9,C) that G = To
(N) (ZG), o/xe 2,0
(9.Mcond 2c).
E
MORE RESULTS AND DISCUSSIONS
E.1
UNCONDITIONAL GENERATION
Unconditional generation Evaluated in alternative metries. We also provide the results for un- conditional generation evaluated in Hellinger distance and Wasserstein distance in Tabs. 7 & respectively. Our improvement is consistent under various metrics.
Table Unconditional generation evaluation distribution discrepancy with training data. Metrics represent Hellinger distances (x (e-) of certain molecular properties, between generated and observe molecules.
Methods
MW
ALogP
PSA
QED
Energy
EDM
17.04(0.30)
8.37(0.20)
11.44(0.15)
6.45(0.43)
13.54(0.35)
Ours
8.29(0.19)
7.04(0.09)
9.76(0.38)
11.49(0.22)
8.25(0.24)
Table 8: Unconditional generation evaluation distribution discrepancy with training data. Metrics represent Wasserstein distances (x (x1e-2) of certain molecular propertics, between generated and observe molecules.
Methods
MW
ALogP
PSA
QED
Energy
EDM
176.7709.4)
17.53(0.36)
0.57(0.01) 192.25(12.01)
261.21(9.75)
28.56(0.45)
Ours
60.98(3.59)
10.64(0.31)
1.81(0.05)
25.08(0.26)
Summary of differences across 3D-graph diffusion methods. We summarize the conceptual dif-
ference across different methods in Tab. 9 We observe that:
Lower LD latent space, better generation quality: Shifting from EDM and GeoLDM by reducing LDD per node, better generation quality is achieved.
Overly low LD latent space challenges its quality (in reconstruction) for generation: Shift- ing from GeoLDM to our one-shot model by further reducing the node factor in LD, is difficult to achieve sufficient low reconstruction errors for for generation.
22

---

Published as a conference paper at ICLR 2024
More qualified latent space (in reconstruction), better generation quality: Shifting from our one-shot model to cascaded model, we achieve state-of of-tte-artt generation results by building the 3D graph AE with low reconstruction errors.
More "well-structured" latent space (in semantics), better OOD generation quality: Shift- ing from our cascaded model self-supervised cascaded model, we achieve better OOD conditional generation results when the homogeneity metric is higher.
Table 9: Comparison
different methods, assuming the 3D graph has N nodes, each of which is equipped with a D-dimensional invariant feature and 3-dimensional equivariant feature. Then the feature dimension of data is N x (D 3), and we denote the latent dimension as D).
Methods
Topo/Geom Diffusion
Latent Dimension
EDM GeoLDM
Joint
N x (D +3)
Joint
N (D' +3)
Ours (One-Shot)
Joint Separate
D'
Ours (Cascaded)
D'
Visualization of latent embeddings. We provided t-SNE visualization of the latent embedding in Figs 7 & g We annotate data with four 2D and six 3D property values of molecules. Our visualization shows information of 2D properties is more preserved in topological embeddings, that data with similar properties tends to cluster. In addition, information of 3D properties is more preserved in geometric embeddings. The observation is also confirmed by quantitative evaluation, by computing the silhouette score of the latent embeddings w.r.t properties.
MW, SS: 3.8
ALogP, S5: 0.3
PSA, 55: 4.4
QED, SS: 3.9
Topological Embeddings
MW, 55: 1.3
ALogP, 55: 0.1
PSA, 55: 4.2
QED, S5: 2.2
Geometric Embeddings
Figure I-SNE visualization topological and geometric latent embeddings. The colors are annotated by four 2D molecular properties: MW, ALogP, PSA, and QED. SS: Silhouette Score 100).
More ablation studies. We here provide additional ablation studies. The experimental results of replacing the diffusion model with VAE are shown Tabs. 10
[T and [2] The experimental results
of omitting the 3D latent space shown in Tab. 13 The experimental results of training on less And the experimental results of
data (only 10% of the original data) are shown in Tabs.
14
15
one-shot AE of different variants are shown in Tab. 16
E.2 CONDITIONAL GENERATION
Visualization of conditionally generated molecules. Please refer to Figs. molecules conditional on different quantum properties and protein targets.
12 for generated
23

---

Published as a conference paper at ICLR 2024
alpha, SS: 0.3
mu, SS: 1.4
homo, SS: 4.4
lumo, SS: 1.0
SS: 2.4
Cv. SS: 7.2
Topological
alpha, SS: 0.7
mu, SS: 0.7
homo, SS: 3.7
lumo, SS: 3.4
gap. SS: 5.4
Cv. SS: 7.7
Geometrie
Embeddings
Figure 8: T-SNE visualization for topological and geometric latent embeddings. The colors are annotates by six 3D molecular properties. SS: Silhouette Score (x 100).
Table 10: Unconditional generation evaluation on distribution discrepancy with training data, Metries represent total variation distances (x 1e-2) of certain molecular properties, between generated and observe molecules, the lower the better. Reported are the mean values followed by the standard deviations in parentheses.
Methods VAE 64.30(22.78)
MW
ALogP
PSA
QED
FCD
Energy
66. 66.29(23.57)
64.25(23.37)
66.60(23.57)
2813.26(3.35)
64.83(23.02)
Diffusion
2.52(0.39)
0.91(0.10)
1.22(0.12)
1.04(0.05)
47.66(3.42)
1.87(0.18)
Unconditional generation evaluation on distribution discrepancy with training data. Metries represent Hellinger distances (x le-2) of certain molecular properties, between generated and observe molecules, the lower the better. Reported are the mean values followed by the standard deviations in parentheses.
Table 11:
Methods
MW
ALogP
PSA
QED
Energy
VAE 87.93(5.32)
92.94(2.81)
88.36(2.68)
97.52(1.38)
90.49(3.24)
Diffusion
8.29(0.19)
7.04(0.09)
9.76(0.38) 11.49(0.22)
8.25(0.24)
Table 12: Unconditional generation evaluation distribution discrepancy with training data. Metries represent Wasserstein distances (x le-2) of certain molecular properties, between generated and observe molccules, the lower the better. Reported are the mean values followed by the standard deviations in parentheses.
Methods
MW
AoogP
PSA
QED
Energy
1861.87(1944.16 60.98(3.59)
114.75(47.24)
1983,42(610.91)
13.67
585.78(327.65 58)
Diffusion
10.64(0.31)
192.25(12.01) 1.81(0.05)
25.08(0.26)
Unconditional generation evaluation on distribution discrepancy with training data. Metries represent total variation distances (x le-2) certain molecular properties, between generated and observe molecules, lower the better. Reported are the mean values followed by the standard deviations in parentheses.
Table 13:
Methods
MW
ALogP
PSA
QED
FCD
2D
6.06(0.85)
1.80(0.17)
2.74(0.59)
2.14(0.30)
133.04(2.63)
2D+3D
2.52(0.39)
0.91(0.10)
1.22(0.12)
1.04(0.05)
47.66(3.42)
Rationale enforce "semantic-awarrness". Several studies demonstrate if the latent space is aware of the specific semantic, there existing particular direction (for latent embeddings) to strength/weaken such semantic (Kwon et al. 2022 Liu et al. 2022b). We thus conjecture that such characteristics of the latent space could be linked to conditional generation performance: If certain latent direction stands out to correlate with conditional property increasing/decreasnng diffusion models should easily capture it. We then leverage graph self-supervised learning (You et al. 2020a
24

---

Published as a conference paper at ICLR 2024
Unconditional generation evaluation distribution discrepancy with test data. Metrics represent total variation distances (x le-2) of certain molecular properties, between generated and observe molecules, the lower the better. Reported are the mean values followed by the standard deviations in parentheses.
Table 14:
Methods
MW
ALogP
PSA
QED
FCD
Energy
EDM
2.89(0.38)
0,85(0.12)
2.37(0.18)
0.87(0.05)
58.04(0.39)
2..1()
100% Data
Ours
2.52(0.33)
0.91(0.10)
1.22(0.12)
1.04(0.05)
47.66(3.42)
1.87(0.18)
Ours-GSSL
3.09(0.29)
1.24(0.02)
1.9((0.33)
1.23(0.19)
114.16(3.61)
1.84(0.12)
EDM
12.33(1.70)
4,98(0.20)
5.25(1.07)
4.73(0.09)
598.83(14.18)
7.56(
10% Data
Oury
11.63(1.90)
2.72(1.23)
4.06(1.02)
3.25(0.44)
401.30(31.8
6.80(1.06)
Ours-GSSL
10.04(2.08)
3.74(0.20)
3.65(0.40)
2.24(0.24)
800.8)))
5.53(0.65)
Conditional generation on polarizability evaluation. Numbers represent the mean absolute error between conditional and oracle-pres peeeeeere properties (Satorras et al. 2021). the lower the better. Reported are the mean values followed by the standard deviations in parentheses. GSSL: graph self-supervised learning.
Table 15:
Methods
100% Data
10% Data
Random
41.00
EDM
20,15
24.07
Ours
15.56
21.02
Ours-GSSL 16.43
20.49
Table 16: Reconstruction performance of AE workflows molecule data, evaluated with topological accuracy (Recon.), and geometric root-mean-square error (RMSE).
Methods
Recon.t
RMSE +
Random Init.
0%
1.86
One-Shot AE
0.80%
1.80
One-Shor AE w/ Higher Topo Weight
0.30%
1.66
One-Shot AE w/ Higher Geom Weight
0.60%
1.60
One-Shot AE w/ More Layers
0.70%
1.74
Cascaded AE
79.57%
0.69
which is an effective way to regularize the latent space with specific priors. We adopt the standard node dropping prior. We further adopt the homogeneity ratio (Kwon al. 2022) to quantify it. We surprisingly find the homogeneity ratio correlates with conditional generalization performance (Sec. result (v)).
Homogeneity ratio to quantify semantic-awarenne We follow (Kwon et al. 2022 to calculate the homogeneity ratio to quantify semantic-an awareness samantic-awareness in the latent space. The idea is to check how the increasing/decrrasingg of the conditional properties is consistent toward certain direction in the latent space. Specifically, given a latent embedding and its annotated property, (i) we search for its 50 nearest neighbors, (ii) calculate the angles between the center and neighbors and bin them into 10 groups, (iii) calculate the ratio (%) of property increasing within each group, and (iv) average the values the dataset. If there existing outstanding ratios in specific direction, it indicates the model can easily generate data toward the direction for the higher property, which we can it semantic-aware; otherwise the latent space is not aware of the specific semantic and will reach 50% homogeneity ratios in all directions. The visualization of homogeneity ratios w.r.t. all six conditional properties are shown Figs. and
Ablation studies. We include more ablation studies on conditional 3D molecule generation for given protein structures: Tab, reported the impact of graph self-supervised learning (GSSL) and the resulting latent space regularization: topological evaluation of QED and topological plus geo- metric evaluation of Vina were actually worse, although VDock (after Vina docking thus changing geometries) improved. We expect that improving the cascaded AEs, especially the geometry AE, and testing the designs on new protein targets would better manifest the benefits of GSSL. Tab, [8] reported the impact of filtering generated molecules based protein pockets (if known): high-
25

---

Published as a conference paper at ICLR 2024
ID
0OD
01
70
90
110
130
140
150
Figure Visualization of molecules generated by latent 3D graph diffusion, conditional different polariz- ability values (a). Molecules of higher a are expected less isometrically shaped.
w/o GSSL h-ratios y-4.95%
W/ GSSL h-ratios a. y-28.90%
54.08
2231
2239
272
w/o GSSL hrrties of y=0.18%
GSSL h-ratios of y=1.17%
112
382
1221
Figure 10: Polar plot for homogeneity ratios of a and Eh, w/o and w/ GSSL, 7 is calculated as the absolute difference between maximum and median ratios.
affinity designs increased from 42% 48% whereas topological evaluations such as QED and SA
did not significant changes. However, when no binding site information is available, our default latent diffusion model could uncover binding pockets (potentially novel), as observed Fig. T2
Table Conditional generation on protein binding targets evaluation with graph self-supervised learning (GSSL). Numbers QED/SA & Vina scores are calculated with RDKit (Landrum 2013 AutoDock (Hee
al. 2012), respectively.
Methods
QEDt
SAT
Vinal.
VDockJ.
Vina (Top-10%)
Ours
0.60
0.71
-5.23
-6.85
-12.34
Ours-GSSL
0.46
0.73
-4.00
-8.26
-11.84
26

---

Published as a conference paper at ICLR 2024
w/o h-raties y=4.95% op"
W/ GSSL
y=28.90%
GSSL hiratiss μ. Y=0.17%
GSSL of y=0.00%
30"
$4.88
223
223
225
255
372
278
210'
270"
w/o GSSL hrraties of y=0.18%
w/ GSSL hrratcs of y=1.17%
wio GSSL hratios of y=0.98%
w/ GSSL h-ratios of 4t, y=0.511%
55 APA
JMP
GSSL h-ratios
w// GSSL Prretios de. ym ym0.22%
h-ratics
h-roties
30"
30
20
0122%
54.7%
8.90%
229
455
2707
310
2117
Figure 11: Polar plot for homogeneity ratios of a, Ae, EH, Co, w/o and w/ GSSL. 7 is calculated as the absolute difference between maximum and median ratios.
Table 18: Conditional generation on protein binding targets evaluation. Numbers of QEDISA & HiAff are calculated with RDKit (Landrum 2013) & AutoDock (Huey al., 2012), respectively. Pocket Unknown: latent diffusion without knowing the binding pocket and 3D molecules are generated for the conditional protein structures, which applics when no pocket information is known novel pockets desired (as discovered and illustrated in Fig. [2). Pocket Known: 3D molecules are still generated for the conditional protein structures but filtered based on the known binding pockets of protein structures.
Methods
QED
SA"
HiAlf
Pocket Known (Filtering molecules)
0.60
0.71
48.08%
Pocket Unknown (Default)
0.61
0.72
42.65%
Reference Molecule
Generated Molecules
Figure Visualization reference molecule and molecules generated by latent 3D graph diffusion, conditional on protein binding target. The circled area is potential binding pocket that is not reflected by the reference molecule.
27


Reviews:
[{"review_id": "0qUxIZlYUb", "reviewer": "Reviewer_N14e", "summary": "This paper investigates the appropriate latent space for generating 3D graphs. The authors derive several conclusions through theoretical analysis:\n\n- The lower the dimensionality of the latent space, the higher the performance limit of diffusion.\n\n- Higher quality of the latent space (i.e., lower reconstruction error) corresponds to a higher performance limit of diffusion.\n\n- Preserving symmetry (maintaining graph properties after translation, rotation, etc.) is an inductive bias of the latent space, contributing to an increased performance limit of diffusion.\n\nGuided by these theoretical insights, the authors propose a method named \"latent 3D graph diffusion.\" This approach utilizes cascaded 2D-3D graph autoencoders to learn a latent space with low-error reconstruction (learning topological graphs) and symmetry invariance (learning geometric graphs). Furthermore, the authors extend this method to conditional generation given SE(3) invariant properties (rotation-translation invariance) or equivariant 3D objects.\n\nExperimental results demonstrate that appropriate regularization of the latent space through graph self-supervised learning can further enhance the robustness of conditional generation. The comprehensive experimental findings indicate that, compared to existing competitive methods, this approach can generate 3D molecular graphs with enhanced effectiveness/drug similarity and is at least an order of magnitude faster in diffusion training. The speed advantage increases with the size/complexity of molecules.", "questions": "- How to improve generalization capability?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Demonstrates superior generation quality, rapid generation capabilities, and conditional generation proficiency in 3D graph generation, while enhancing robustness through regularization.", "weaknesses": "- Limited generalization capability in comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EAlwbmeaO8", "reviewer": "Reviewer_oqpy", "summary": "This study investigated the generation of 3D graphs from latent space using diffusion models and conditioned on different properties. The model is based on a cascaded 2D-3D graph autoencoders combined with diffusion models. The authors investigated the usefulness of generating 3D graphs from the latent space with symmetry preserved, and explored latent diffusion for conditional 3D graph generation.", "questions": "The authors trained the topological AE and the geometric AE separately with different constrains. I understand the difficulty in training them in one shot, but I'm wondering the influences from each of the AE, e.g., would it be possible information about the topology is lost while training using the geometric AE? If so, how much is lost/kept?\n\nWhile GSSL improves the results on OOD, it seems it can worse the results on ID. Can the authors provide more insights on this?\n\nConsidering one utility of the model is for generating new drugs, model interpretability could be important. In the latent space, would it be possible for the authors to provide some visualization on the learned features and if available, colored by some topological and geometric features of the graph? If there are certain patterns there then it may be useful to help better understand what features are mostly kept in the latent space and enhance the interpretability of the model.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and the rational is straightforward and convincing. The authors have conducted multiple numerical experiments and the conditional generation based on various properties show great potentials of such method. In general I find this is an interesting paper.", "weaknesses": "Please see my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mNQkvvjXJR", "reviewer": "Reviewer_kaPc", "summary": "This paper proposes an algorithm for diffusion-based 3D graph generation through latent space. This paper claims that low-dimensional projection is the key factor of the graph generation due to satisfy the equivariant property. The experiments are conducted using 3D molecules datasets.", "questions": "Please address my concerns listed in Weakness section.\n\n# Q-1. Proposition 1\n\nWhile the authors describe full of equations with comments, the concept itself is highly straightforward. For instance in the manuscript, the authors said that '__Proposition 1.__ _Performance bound of 3D graph diffusion is related to feature dimensionality_'.\nIn my opinion, simply if we increase the network capacity by adding more layers or increasing the channel-length with lots of training data, the diffusion models surely have the performance gain. Honestly, I cannot catch the authors' intension from this propositions.\n\n# Q-2. Ablation study\n\nWhat if the authors intentionally omits the latent space encoding? I mean instead of using 2D-3D autoencoders, what happens if the network only consists of the 3D autoencoders? How much gain could we obtain if we adopt the latent space encoding?\n\n# Q-3. Clear difference between the previous works.\n\nCan the authors create one table to clearly demonstrate the difference? I read the paper (Hoogeboom et al., 2022) and it seems like this paper also split the geometry and topology for the diffusion process. Not just this factor itself, I hope to know the clear footsteps that this paper newly takes.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The writing is clear. The introduction section is wonderful. I clearly understand the scope of this work and what factors that the authors mainly addresses about. The meaning of the latent space and the 2D-3D autoencoder is well aligned in that the latent space could be understood as down-dimensional projection from 3D to 2D. Also, the experiments look okay and outperforms the previous works. \n\nHowever, I am not sure of the clear contribution of this work. Let me write down my questions and worries in the following sections.", "weaknesses": "# W-1. Why latent space?\n\nAccording to this paper, I did not clearly catch how the authors project the 3D molecules into the 2D latent space. The technical details are missing. It naively mentioned that the authors borrow the architectures from previous works and such simple comment is not self-contained, in my opinion.\n\nHowever, let's assume that this issue is okay with me. Then, the following question is ... why the latent space is typically important to the graph generation? I know that the several recent studies utilize the latent space for the representation of the diffusion-based generative models. Nonetheless, I could not find any theoretical analysis behind the bridge between \n\n- _'the necessity of the latent space'_  and \n- _'the properties of the 3D graph representation, typically for the 3D molecules'_. \n\nFor me, this is naive extension of latent diffusion models into graph representation.\n\n# W-2. Overclaim.\n\nCan the authors exactly prove this equation in Sec 3 of this manuscript?\n\n_3D Graph Diffusion Performance <= Latent Space Reconstruction Quality + Symmetry Preservation * Data Dimensionality_\n\n # W-3. Lack of experiments\n\nIf the main contributions of this paper is about the analysis of the latent space into the 3D graph representation, I think that the authors should have provided the clear results or ablation study about the this contribution. \n\nFor instance, using the same baseline model from (Hoogeboom et al., 2022), the authors could slightly modify the architecture while maintaining the number of parameters for fair comparison. However, I could not find such kind of analysis or experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f00T6xzCHB", "reviewer": "Reviewer_7uXM", "summary": "The paper proposes to perform 3D Graph (molecule) generation in latent space instead of during it directly in the 3D space as usually done now. The author show that diffusion in a lower dimensional space should be more efficient for diffusion. They also propose to use a cascaded auto encoder (AE) instead of a one-shot AE to build such latent space. These changes are shown to help improve the quality of generated samples and are also applicable in the conditioned case.", "questions": "In section 3.1 it is stated that graph matching is used for the AE training loss to ensure permutation and SE3 invariance. This can be computationally expensive. How fast is the AE training? In terms of wall time, but also asymptotically (in O notation)? Without knowing this the comparison of training time of the proposed method vs EDM is also a bit complicated. As I understand Table 4. does not account for the AE training time (yes, it can be trained once, but it still needs training).\n\nAlso, since AE is trained on a much larger dataset, it might be fair to also pre-train the EDM on general molecule genration. It has been shown that pretraining helps molecule generation (https://arxiv.org/pdf/2309.17296.pdf). While this paper is quite recent it would still make more sense to compare against such properly tuned state-of-the-art molecular diffusion setup instead of vanilla EDM, which is an older baseline (first ever diffusion model for 3D molecule generation).\n\nWhy is MIDI, which is cited numerous times in the paper is not compared against in the experiments? It does in certain metrics perform a lot better than the EDM, which is used as the main baseline here.\n\nAlso, in 3.2 Setup sections authors say that connectivity is commonly later determined based on domain rules and cite MIDI among others. Its true, that such domain-based rules are used to recover connectivity in e.g. EDM, but in MIDI connectivity is modeled explicitly if I remember correctly?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors provide a nice and sensible motivation, that smaller latent spaces should be more efficient for diffusion. The proposed approach also outperforms the chosen baselines. The proposed formulation is nicely extended to include conditioning on both scalar and 3D properties, which allows for a wider range of applications.", "weaknesses": "The novelty of the work is somewhat limited. It takes existing 3D molecule genration setup with diffusion models and transforms it into latent diffusion, which is a known concept in general and some latent 3D models do exist (e.g. GEOLDM), using mostly off the shelf architectures.\n\nOne of the main contributions of this work is the cascaded auto encoder and main motivation for why it is needed stems from the fact that one-shot AE effectively failed to train. I find it a bit strange that the one-shot AE fails to train completely as there are 3D molecule generative models that generate 2D and 3D graph features jointly (e.g. MIDI or the example I reference in the paragraph below). They do usually re-weight the losses for 2D and 3D terms, but they do work. So I would like to see a more detailed analysis on why it doesn't work and potentially an ablation on the one-shot AE architecture and losses. I can understand one-shot AE working worse than the proposed cascaded AE with teacher forcing, but it essentially performing like a random initialization makes me wonder if it was tested sufficiently carefully.\n\nAs I also point out below, it would make sense to use for example https://arxiv.org/pdf/2309.17296.pdf as the baseline instead of an older EDM model, even though that paper is somewhat recent, so I understand its exclusion for the initial version of the paper. Still it would be nice to have for the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j0FdiDgZbH", "reviewer": "Reviewer_N36h", "summary": "In this paper, the authors introduce the generation of 3D graphs characterized by symmetry group equivariances, a feature with useful implications for machine vision and molecular discovery.The paper focuses on the optimal latent space for 3D graph diffusion, emphasizing its benefits over traditional methods. By strategically using cascaded 2D-3D graph autoencoders, the authors reveal a model called \"latent 3D graph diffusion\". Notably, this innovative approach demonstrates remarkable efficacy when adapted to the molecular context and enhanced by graph self-supervised learning. The experimental results highlight its ability to rapidly generate remarkable 3D molecular conformations.", "questions": "1. The design of latent diffusion appears to be complex. Have you considered simplifying it by merely applying KL divergence to $z^{{0}}$ and deriving a VAE-like model using the encoder and decoder as proposed in this paper? An ablation study should be added here.\n\n2. Could there be a more comprehensive validation of GCL's efficacy in Section 4?\n\n3. Given your assertion that your model \"produces superior 3D molecules faster\" owing to latent diffusion, could you perhaps contrast it with GeoLDM, a previously introduced 3D latent diffusion model?\n\n4. Is it possible to conduct a more in-depth evaluation and comparative study of experiments related to the generation of conditions related to equivariant protein targets?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This study provides a detailed analysis of the underlying methodology. It illustrates the overall relationship between diffusion performance, latent space reconstruction quality, symmetry conservation, and data dimensionality. The depth of this analysis demonstrates the thoroughness and precision of the research.\n\n2. In this work, researchers adopt graph contrastive learning as a strategy to refine and enhance latent space representations in 3D graph autoencoders (AEs). The approach stands out for its innovative nature.\n\n3. In terms of application evaluation, the authors conduct an exhaustive evaluation of the model for various scenarios. This includes unconditional generation of 3D molecules, conditional generation based on (invariant) quantum properties, and conditional generation in relation to (equivariant) protein targets. These diverse evaluations reinforce the model's laudable versatility and adaptability.", "weaknesses": "1. For the objective of unconditional 3D molecular generation, I noticed the absence of results from \"Ours-GSSL\". Including these would be crucial to validate the effectiveness of GCL.\n\n2. Regarding the unconditional 3D molecule generation, the 'molsta' metric, which gauges molecular integrity, seems paramount. From the data presented, your findings exhibit significant variances when juxtaposed against the previously introduced 3D latent diffusion model, GeoLDM. Additionally, the AtomSta and MolSta metrics for the Drug dataset appear to be omitted.\n\n3. In the context of conditional generation based on (invariant) quantum properties, it's essential to evaluate both ID and OOD. Yet, the \"Ours-GSSL\" performance doesn’t exhibit a noteworthy distinction compared to \"ours\", \"Random\", and similar methods. In some instances, it even underperforms. This might cast a shadow on the robustness of conclusions articulated in Section 4.2, especially point (v).\n\n4. Concerning the experiments related to conditional generation associated with (equivariant) protein targets, I'd recommend broadening the evaluation scope by incorporating metrics such as the Vina Score and Vina Dock in your findings. Furthermore, beyond Targetdiff, newer techniques like DiffSBDD[1] and DecompDiff[2] have emerged. Comparing your approach with these could provide a more holistic view. As per the metrics currently displayed, the method delineated in your manuscript doesn't seem to lead the pack.\n\nMinor:\n\n1. In Figure 4, there appear to be some omission errors; the circled references seem to be missing.\n\n2. The structure of \"Proofs for Analysis\" in Appendix A could be refined for better readability.\n\n[1] Schneuing, Arne, et al. \"Structure-based drug design with equivariant diffusion models.\" arXiv preprint arXiv:2210.13695 (2022).\n\n[2] Guan, Jiaqi, et al. \"DecompDiff: Diffusion Models with Decomposed Priors for Structure-Based Drug Design.\" (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lttcpgNuDi", "reviewer": "Reviewer_HVJQ", "summary": "The paper \"Latent 3D Graph Diffusion\" explores using generative AI for 3D graph generation with symmetry-group equivariance. It emphasizes the importance of choosing the right latent space for diffusion, proposing a compact and symmetry-preserving space called \"latent 3D graph diffusion.\" They extend this to conditional generation, showing its potential in molecular discovery with improved speed and quality compared to existing methods. The paper's contributions include motivational analysis, latent space construction, and extensions to conditional generation, all supported by experimental results. The conclusion suggests future research areas, including semantics-specific regularization of the latent space.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n1.\tNovel Approach: The paper addresses an important and under-investigated question regarding the latent space for 3D graph diffusion, introducing a novel concept of latent 3D graph diffusion. This approach is innovative and could lead to significant advancements in the field.\n\n2.\tTheoretical Foundation: The paper provides a theoretical analysis to motivate the use of latent spaces for 3D graph diffusion. The performance bound of 3D graph diffusion in a latent space is discussed, which adds a valuable theoretical perspective to the work.\n\n3.\tComprehensive Contributions: The paper offers a multi-faceted approach to 3D graph generation, addressing the choice of latent space, the construction of compact and informative spaces, conditional generation, and regularization of the latent space. This comprehensive approach demonstrates the authors' commitment to advancing the field.\n\n4.\tEmpirical Validation: The paper supports its claims with empirical results, showing that the proposed method outperforms existing techniques. Various settings are adopted, including unconditional generation, invariant generation conditioned on quantum properties, and equivariant generation conditioned on protein targets.", "weaknesses": "I have several questions:\n\n1. What's the difference between a one-shot AE and a Cascaded AE? The paper claims that \"a one-shot AE embeds and reconstructs molecule data, evaluating both structure topology and geometry simultaneously.\" How does a Cascaded AE differ? Are topology and geometry independent from each other in a Cascaded AE?  \n\n2. I would recommend adding an algorithm section to improve clarity. Based on my current understanding, a graph is initially encoded by an encoder in the latent space, then it undergoes diffusion in the latent space, and finally, it is transformed back to the graph space. Please correct me if I'm mistaken. Could you also clarify which diffusion model is used for the diffusion process in the latent space?  \n\n3. The paper asserts that \"a latent space should possess (i) low dimensionality, (ii) low reconstruction error, and (iii) preserve group symmetry.\" How do you ensure that the autoencoder can meet these objectives? Have you conducted any analysis in this regard?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)