Symphony: Symmetry-Equivariant Point-Centered Spherical Harmonics for 3D Molecule Generation
Authors: Ameya Daigavane
Song Eun Kim
Mario Geiger
Tess Smidt
Venue: ICLR 2024 poster (2024)
Keywords: molecule
spherical harmonics
equivariant
symmetry
generation
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose a new method for generating molecules in a symmetry-preserving manner using spherical harmonic projections.

Abstract: We present Symphony, an $E(3)$ equivariant autoregressive generative model for 3D molecular geometries
that iteratively builds a molecule from molecular fragments.
Existing autoregressive models such as G-SchNet and G-SphereNet  for molecules utilize rotationally invariant features to respect the 3D symmetries of molecules.
In contrast, Symphony uses message-passing with higher-degree $E(3)$-equivariant features.
This allows a novel representation of probability distributions via spherical harmonic signals to efficiently model the 3D geometry of
molecules. We show that Symphony is able to accurately generate small molecules from the QM9 dataset, outperforming existing
autoregressive models and approaching the performance of diffusion models. Our code is available at https://github.com/atomicarchitects/symphony.

Published as a conference paper at ICLR 2024
SYMPHONY: SYMMETRY-EQUIVARIANT POINT-
CENTERED SPHERICAL HARMONICS FOR 3D
MOLECULE GENERATION
Ameya Daigavane', Song Kim', Mario Geiger"; Tess Smidt
ammyyd,, songk) somit.edi geiger www.iarrmma...... gmail com, t dt @mit edu Massachusetts Institute of Technology 2NVIDIA
ABSTRACT
We present Symphony, an equiivaarar autoregressive generative model for 3D molecular geometries that iteratively builds molecule from molecular frag- ments. Existing autoregressive models such as G-SchNet (Gebauer et al. 2019) and G-SphereNet (Luo & Ji, 2022) for molecules utilize rotationally invariant features to respect the 3D symmetries of molecules. In contrast, Symphony uses message-passing with higher-degree E(3)--quivariant features. This allows novel representation of probability distributions via spherical harmonic signals to effi- ciently model the 3D geometry of molecules. We show that Symphony is able accurately generate small molecules from the QM9 dataset, outperforming existing autoregressive models and approaching performance of diffusion models.
INTRODUCTION
In silico generation of atomic systems with diverse geometries and desirable properties is important to many areas including fundamental science. materials design, and drug discovery (Anstine & Isayev 2023). The direct enumeration and validation of all possible 3D structures is computationally infeasi- ble and does not in itself lead to useful representations of atomic systems for guiding understanding or design. Machine learning methods offer promising avenue to explore this landscape by learning to generate 3D molecular structures.
Effective generative models of atomic systems must learn to represent and produce highly-correlated geometries that represent chemically valid and energetically favorable configurations. To do this, they must overcome several challenges:
The validity of an atomic system ultimately determined by quantum mechanics. Generative models of atomic systems are trained on 3D structures relaxed through computationally-ineens. quantum mechanical calculations. These models must learn to adhere to chemical rules, generating stable molecular structures based solely on examples.
The stability of atomic systems hinges on the precise placement of individual atoms. The omission or misplacement of a single atom result in significant property changes and instability.
Atomic systems have inherent symmetries. Atoms of the same element are indistinguishable, so there is no consistent way to order atoms within an atomic system. Additionally, atomic systems lack unique coordinate systems (global symmetry) and recurring geometric patterns occur in variety of locations and orientations (local symmetry).
Taking these challenges into consideration, the majority generative models for atomic systems operate point geometries and use permutation and Euclidean symmetry-invariant or equivariant
methods. Thus far, two approaches have been emerged as effective for directly generating general 3D
geometries of molecular systems: autoregressive models (Gebauer et al. 2019 2022 Luo & Ji 2022
Simm et al. 2020 2021) and diffusion models (Hoogeboom et al. 2022)
In this work, introduce Symphony, autoregressive generative model that uses higher-degree equivariant features and spherical harmonic projections to build molecules while respecting the
Work performed when Massachusetts Institute of Technology.

---

Published as a conference paper at ICLR 2024
E(3) symmetries of molecular fragments. Similar to other autoregressive models, Symphony builds molecules sequentially by predicting and sampling atom types and locations of new atoms based on conditional probability distributions informed by previously placed atoms. However, Symphony stands out by using spherical harmonic projections to parameterize the distribution of new atom locations. This approach enables predictions be made using features from single 'focus' atom, which serves as the chosen origin for that step of the generation process, It allows for the simultaneous prediction of the radial and angular distribution of possible atomic positions in direct manner without needing to use additional atoms.
To test our proposed architecture, we apply Symphony to the QM9 dataset and show that outperforms previous autoregressive models and is competitive with existing diffusion models on a variety of metrics. We additionally introduce metric based on the bispectrum for assessing the angular accuracy of matching generated local environments to similar environments in training sets. Finally, we demonstrate that Symphony can generate valid molecules high success rate, even when conditioned on unseen molecular fragments.
1. Compute node embeddings
Predict
Predict target number
4a. Prediot position distribution
position
Ramdorrty sample radlus and ocoition from 3D density:
My beus
Zn-1
MLP
MLP
Embed
unfinished molecule S"
posiioo
softmax
posiioms, species
2 Embeddeers
eSnn-Linear
porus fueli S");;
yooee Zuc Iuaii 8
0.96
softmax
HB
0.99
-0
0.01
posttoon
in Zooti 8
pocttio
position:
A
focue
Repeat from slop unti STOP predicted in
Randomly sample species:
Randomly sample e loo
Zue1
One iteration of the Symphony molecular generation process, in which one atom is sampled given the positions and atom types of an unfinished molecular fragment S". The complete molecule after all iterations is shown in the bottom right of the figure.
Figure 1:
2
BACKGROUND
((13)-Eqiivarian Features: We say a E(3)-equivariant feature 2R++1 transforms as the irreducible representation under rotation R and translation T:
R.T D²(R)
where D' is the irreducible representation of SO(3) of degree 21 11 DR)R) E 12 (21++1) is
referred to as the Wigner D-matrix of the rotation R. As (R) = 1 and (R) = R, invariant 'scalar" features correspond to degree = 0 features, while 'vector' features correspond to / 1 features. Note that these features are invariant under translation T.
Spherical Harmonies:
The real spherical harmonics (0, Φ) are set of real-valued orthogonal functions defined on the sphere 5", indexed by two integers 1 and m such that > 0, |m < I. Here and @ come from the notation for spherical coordinates, where is the distance from an origin, 0 E [0, T is the polar angle and ó E [0, 2nT) is the azimuthal angle. The relation between Cartesian and spherical coordinates is given by: x = 7 sin cos o, y = r sin 0 sin d, 2 = r cos
I corresponds to angular frequency: the higher the I, the more rapidly Yt. changes over S², This can intuitively be seen by looking at the functional form of the spherical harmonics. In their Cartesian form, the spherical harmonics are proportional simple polynomials. In common choice of basis, = 0 is proportional to 1, = 1 is proportional to (Ir, y, 2) and = 2 is proportional to (xyy, yz, 222 x² y²= z., x y2²), seen in Figure33D.r
2

---

Published as a conference paper at ICLR 2024
One important property of the spherical harmonics is that they be used to create - equivariant
features. Let Yi( Φ) = (9, ó).................. Yi,(0, 6) E R 21+1 represent the collection of all spherical harmonics with the same l. Then, (0 o) transforms an E(3)-qquivariant feature of degree I under rotation: (0, d)) = D' ( Y2 (0,4 <), where R. an arbitrary rotation, and (0, の) is interpreted the coordinates of point on S².
The second important property of the spherical harmonics that we employ is the fact that they form an orthonormal basis for functions on the sphere S Thus, for any function S R, we can express f as a linear combination of the Yi.m. Formally, there exists unique coefficients E IR 2+++ for each / E N, such that f(0,, Φ) = Ei=o Cr YY (((0,,)) Φ). We term these coefficients as the spherical harmonic coefficients of they are obtained by projecting onto the spherical harmonics.
3 METHODS
We first describe Symphony, our autoregressive model for 3D molecular structures, with a comparison to prior work in Section 3.6
3.1 BUILDING MOLECULES VIA SEQUENCES OF FRAGMENTS
First, we create sequences of fragments using molecules from the training set CREATERRAGGENTSEEUENNC
Given molecule M and random seed constructs a sequence of increasingly larger fragments (S!,....!! M such that |S*| = n for all n E [...).!IM] and SIMi = M exactly. Of course, there are many ways to create such sequences of fragments; simply builds a molecule via a minimum spanning tree.
via
Symphony attempts recreate this sequence step-by-step, learning the (probabilistic) mapping S" + g In particular, we ask Symphony to predict the focus node fn+1, the target atomic number Zn41 and the target position Fa41, providing feedback every step.
Algorithm I A AGMENTSFQUENCE
NeXTFRACMENTT
NEXTFRACMENT
Input: Molecule M. PRNG Seed
Sample an atom (F1, Z1) from M using the PRNG seed
S + (F,, 2
function NEXTFRAGMENT(S"" s)
(J,a) < Closest atom pair s.t. f E S" and E S-
NextFaacaaent
(Ties are broken randomly using seed 8)
fo+1 + The index of the atom f in S
fralion (r. 0,4)
Zn+1 < The atomic number atom a
F Fn+1 <- The relative position of atom w.r.t. atom
Sn+1 <- S" U (fr++1, 1n+1)}
s' +- Update PRNG Seed
return (S"+1, s')
S"
S
for t 1 to M do (8a+1, 8) + NEXTFRAGMENT(S", 8) return SIM....MM
applied to methane (CH4). From S", there are thus two valid positions to place the next atom around the focus fu+ (Bottom-Left) The true probability distribution posstion (1) is smoothly projected onto (Bottom- Right) radial shells of spherical signals according to EEqutiin 7 All the radial shells show the same angular distribution, but the shell corresponding ri is the most probable.
Figure 2: (Top) Fragments from FREEATEPRAMMENTSSOOEE
3.2 HANDLING THE SYMMETRIES OF FRAGMENTS
Here, highlight several challenges that arise because S" must be treated unordered set of atoms that live in 3D space. In particular, let RS + T {(Ri, + T, Zi..... (RF, T, 7n)) be the description of the same of atoms in S but in coordinate frame rotated
3

---

Published as a conference paper at ICLR 2024
by R- and translated by T- Similarly, let be any permutation of {1....,)) n and TS" ((Fa()) Zz(1))...., "F...,,(())) (Fx(n), Z. Fundamentally. RS" + T, S" and TS" all represent the same set of atoms. Thus, would like Symphony to naturally accommodate the symmetries of fragment S", under the group E33 of Fuclidean transformations consisting of all rotations R and translations T, and the group of all permutations of constituent atoms. Formally, we wish to have:
. Property (1): The focus distribution placus and the target species distribution piuccies should be E(3) -in)ainnn::nnt:
p p (fa+1; R + T) = p 1o (n+1:8"))
(1)
p specics (Zu+1 fat1; R + T) p species (Zu+1 Sn)
Property (2): The target position distribution prosition should be -e)iiaaai::i;:nt:
ppunition RF T fx+1, Zn+1: RS + = = (Fa+1 fnt1, Zn+1;5")
* Property (3): With respect to the ordering of atoms in S", the map p p should be permutation eeuiiiiriiiivviiianttnn while pxpecies and prosition should be permutation-invarian invariant..
We represent glocus p and prosition as probability distributions because there may be multiple valid choices of focus fn+1, species Zn+1 and position Fa+11
3.3 THE DESIGN OF SYMPHONY
The overall working of Symphony is shown graphically in Figure 1 Symphony first computes atom embeddings via an EMBEDDER. Here, we assume that EMBEDDER(S")
(hr.r | E S", 0 < < m returns a set of eqqiiaraiv features hy / of degree upto a predefined degree 4max. for each atom in S In Theorem B.I we show that Symphony guarantee Properties (1), (2) and
(3) as long as EMBEDDER is permuuration-qquivariann and ((3))-qqiivarian
From Property (1) focus and pppcciss should be invariant under rotation and translations of S Since the atom types and the atom indices are discrete sets, we can represent both of these distributions as vector of probabilities. Thus, compute fnems and yrPecies by applying multi aulti-layer perceptron MLP on only the rotation and translation invariant features of EMBEDDER(S"))
P focus (fttiiS)) MLP (ELEEEBEEDDER(S") )
(4)
p species (Za+1 ST) = MLLPEEBBEDAATOTTTPPEEE.... EMBEDDER (S") ("))m"id))))))
(5)
Alongside the node-wise probabilities for Jocus, we also predict a global STOP probability, indicating that no atom should be added.
On the other hand, Property (2) shows that prosition transforms non-identically under rotations and translations. We describe a novel parametrization of 3D probability densities such as poosition with spherical harmonic projections.
The position F'r is represented by spherical coordinates (r, A, Φ) where r is the distance from the focus f. 0 is the polar angle and ø is the azimuthal angle. Any probability distribution p positionn over positions must satisfy the normalization and non-new negaaiivvt constraints: Jq p prositionn (r: +) dV
I and prosition (r, 0. Φ) 2 O where dV = rdr sin 0d0do is the volume element and = {r e [0, oo). 0 [0, T), E [x, represents all space in spherical coordinates. Since these con- straints are hard to incorporate directly into a neural network, we predict the unnormalized logits pposiiio(r (r, 0 Φ) instead, and take the softmax over all space: prosition (r, 0, Φ) & exp prostiion (r,, 0, () To model these logits, we first discretize the radial component r into a set of discrete values. We choose 64 uniformly spaced values from 0.9A 2.0A, which covers all of the bond lengths in QM9. For each fixed value of r, we obtain a function on the sphere 82, which we represent in the basis of spherical harmonic functions Yi, (0, Φ), as described in Section and similar the construction of Cohen & Welling 2015). As we have a radial component r here, the coefficients also depend on
pposition (r. 0,0 fnt1, S")
Σ 1=0
Zn-1, S
4

---

Published as a conference paper at ICLR 2024
A
B
softmax
c
D
Co
C1
C1,0
C1,1
C2 2
C2 02.0
+ C2,1
C2.2
Figure 3: Symphony represents the atom position logits using radial shells of spherical signals. (A) illustrates an example angular distribution for given radial shell prior to applying the softmax function. The softmax enhances signal peaks and prediction precision. (B) presents two ways to plot the signals for single shell: as a colored sphere, or as surface where distance from the origin represents signal magnitude, enhancing peak visibility. (C) breaks the pre-activated signal into contributions from contributions into the 2/ + 1 spherical harmonics for = 0, 1 and 2.
= spherical harmonics. (D), (E), and (F) further break these
Symphony predicts these coefficients from the degree features of the focus node EMBEDDER (5") Jn 1. i.1s and the embedding of the target species Zn+1:
arr: fa+1, Zn+1, 8*) LINEAR(EMBEDDERS ) EMBEDATOMTYPEEZZ411) (6)
By explicitly modelling the probability distributions y focus ppocccss and prosition Symphony learns to represent all possible options of completing S into valid molecule.
3.4 BYPASSING THE ANGULAR FREQUENCY BOTTLENECK
For computational reasons, we are often limited to using finite number of spherical harmonic projections (ic, up to some lmax). Due to the way the spherical harmonics constructed, this means we can only represent signals upto some angular frequency. For example, to represent a signal on the sphere with peaks separated by d radians,
need spherical harmonic projections with mmax 2 d' This similar to issues faced when using the first few terms of the Fourier series; cannot represent high frequency components. To bypass the bottleneck of angular frequency, we propose using multiple channels of spherical harmonic projections, which are then summed over after non-linearity: prosition (r, Φ; S*)) log E Echannele ch exp E (")"' (0, See Appendix F for concrete example where adding multiple channels effectively increases the angular frequency capacity of our model. For Symphony, we find that 2 channels is sufficient, as demonstrated in
2x
3.5 TRAINING AND INFERENCE
utilize teacher forcing to train Symphony. At training time, the true focus fn+1 and atomic number Zn+i are provided as computed in NEXTFRAGMENT Thus, no sampling occurs training time. The true probability distributions focus and species are computed empirically from the set of unfinished atoms and their corresponding neighbors in M. The true probability distribution position computed by smoothly approximating Dirac delta distribution upto some cutoff frequency Imux at the target position F, around the focus atom. Further details about the training process and representing Dirac delta distributions are provided in Section C.2 and Appendixx H
||
||F,t1]
position
(F)
exp
&t,m (f - (f-Fnt1)
20
At inference time, both the focus fn+1 and atomic number Zn+1 are sampled from p (.; pppocics (-1f++1; (::+1:") S respectively. These are used sample F. F, from posiiionn (-) (-Is+t1, 7
S*) and
5

---

Published as a conference paper at ICLR 2024
Molecules generated by starting from initial fragment S' and repeatedly sampling from p placss p and prasition until a STOP is predicted or Naax = 35 iterations have occurred, based on the maximum size of molecules in the QM9 dataset as 30 atoms.
3.6 RELATION TO PRIOR WORK
Most methods for 3D molecular structure generation fall into one of two broad categories; autore- gressive and end-to-end models. G-SchNet (Gebauer et al. 2019 2022) and G-SphereNet (Luo & Ji
2022) were the first successful attempts at autoregressive generation of molecular structures.
G-SchNet uses the SchNet framework (Schütt al., (017) perform message- paasage-passing with rotation- ally invariant features and compute node embeddings. A focus node is then selected as the center of 3D grid, All of the atoms the current fragment then vote on where to place the next atom within this grid by specifying radial distance to the next atom. Because of the use of only rotationally invariant features, at least three atoms are needed to be present in the current fragment to specify the exact position of the next atom without any degeneracy due to symmetry; this procedure is called triangulation. This requires several additional tokens to break symmetry. Similarly, G-SphereNet learns normalizing flow to perform triangulation procedure once there atleast 3 atoms in S".
We wish to highlight two observations that guided the development of Symphony:
Rotationally invariant features centered at single point cannot capture the orientations of geo- metrical motifs (Pozdnyakov & Ceriotti 2022). To handle the degeneracies inherent when using rotationally invariant features to predict positions, G-SchNet uses unphysical auxiliary tokens (which are multiple spatial positions that are not atoms) to break symmetry.
G-SchNet queries all of the atoms in 8 at each iteration, which means distant atoms can have an undue influence when placing the next atom. Similarly, G- SphereNet predictions are not a smooth function of the input fragment; when the input is perturbed slightly, the choice of atoms used in the triangulation procedure can change drastically.
Recently, E(3) eqqivariant neural networks that build higher-degree ((q))eqquvariant features have demonstrated improved performance on a wide range of atomistic tasks (Batzner et al., [022] Geiger & Smidt 2022 Owen et al. 2023). Our key contribution is to show the benefit of higher-degree F(3)- equivariant features for the molecular generation task allowing for novel parametrization of 31) probability distributions using spherical harmonic projections. [Simm et al. (2021) also uses spherical harmonic projections with a single channel for molecule generation, but trained with reinforcement learning, and sampled using rejection sampling. We discuss these details in Appendix D
Among end-to-end generation methods, Hooggboom et al. (2022) developed EDM, a state-of-the- art -qqiivarrt diffusion model. EDM significantly outperformed the previously proposed E(3) eqquivarint normalizing flow (ENF) models for molecule generation (Satorras et al. 2022a).
EDM learns gradually denoise initial configuration of atoms into valid molecular structure. Both EDM and ENF are built on the [(nn)-qqiivrian) Graph Neural Networks (Satorras et al. 2022b framework which can utilize only scalar and vector features (and interactions between them).
MiDi (Vignac et al.] (202)) improves EDM by utilizing bond order information (and hence, a 2D molecular graph to compare to), which do not assume access here. While expressive, diffusion models are expensive to train, requiring x 3.5x more training on the QM9 dataset to outperform autoregressive models. Unlike autoregressive models, diffusion models do not flexibly allow for completion of molecular fragments, because they are usually trained in setups where all atoms are free to move. Current diffusion models use fully-connected graphs where all atoms interact with each other. This could potentially affect their scalability when building larger molecules. On the other hand, Symphony and other autoregressive models use distance cutoffs to restrict interactions and improve efficiency. In Appenddix I we find that EDM with distance cutoffs performs quite poorly.
Furthermore, diffusson models are significantly slower to sample from, because the underlying neural network is invoked æ 1000 times when sampling single molecule. Flow matching (Lipman et al. 2023) has also emerged as a competitor for diffusion models for 3D molecule generation (Song et al. 2023 but suffers from the same drawbacks listed above.

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTAL RESULTS
major challenge with generative modelling is evaluating the quality of generated 3D structures. Ideally, generative model should generate physically plausible structures, accurately capture training set statistics and generalize well to molecules outside of its training set. We propose a comprehensive set of tests evaluate Symphony and other generative models along these three aspects.
4.1 VALIDITY GENERATED STRUCTURES
All of the generative models considered here output set of atoms with 3D coordinates; bonding cheminformatics tools designed information is not generated by the model. Before Multiple algorithms exist for bond order for molecules, we need to assign bonds between atoms.
assignment: xyz2mol (Kim & Kim 2015), OpenBabel (Banck al. (011)) and simple lookup table based on empirical pairwise distances in organic compounds (Hoogeboom et al.) (022). Here, we perform the first comparison between these algorithms for evaluating machine-learingg learning generated 3D structures, In Table we use each of these algorithms to infer the bonds and create a molecule from generated 3D molecular structure. We declare molecule as valid if the algorithm could successfully assign bond order with no net resulting charge. We also measure the uniqueness to see how many repetitions were present the set of SMILES (Weininger) 1988) strings of valid generated molecules. Ideally, want both the validity and the uniqueness be high.
While EDM (Hoogeboom et al. 2022) is still superior on the validity and uniqueness metrics, we find that Symphony performs much better both validity and uniqueness than existing autoregressive models, G-SchNet (Gebauer et al. 2019) and G-SphereNet (Luo & Ji 2022). for the xyz 1 and OpenBabel algorithms. Note that the lookup table does not account for aromatic bonds and is quite sensitive to exact bond lengths; we believe this penalizes Symphony due to its coarser discretization compared to EDM and G-SchNet. Of note is that only xy22mol finds almost all of the ground truth QM9 structures to be valid.
Metric 1
QM9
Symphony
EDM
G-SchNet
G-SphereNet
Validity via xy xy22mol
99.99
83.50
86.74
74.97
26.92
Validity via OpenBabel
94.60
74.69
77.75
61.83
9.86
Validity via Lookup Table
97.60
68.11
90.77
80.13
16.36
Uniqueness via xy22mol
99.84
97.98
99.16
96.73
21.69
Uniqueness via OpenBabel
99.97
99.61
99.95
98.71
7.51
Uniqueness via Lookup Table
99.89
97.68
98.64
93.20
23.29
Table 1: Validity and uniqueness (among valid) percentages of molecules.
Recently, Buttenschoen et al. (2023) showed that the predicted 3D structures from machine- protein-ligand docking models tend to be highly unphysical. For Table 2 we utilize their Posters framework to perform the following sanity checks to count how many of the predicted 3D structures are reasonable. We see that the valid molecules from all models tend to be quite reasonable, with Symphony performing better than all basclines on generating structures with reasonable UFF (Rappe) cct al. 1992) energies and respecting the geometry constraints of double bonds. Further details about the PoseBusters tests provided in Section E.
Test
Symphony
EDM G-SchNet G-SphereNet
All Atoms Connected
99.92
99.88
99.87
100.00
Reasonable Bond Angles
99.56
99.98
99.88
97.59
Reasonable Bond Lengths
98.72
100.00
99.93
72.99
Aromatic Ring Flatness
100.00
100.00
99.95
99.85
Double Bond Flatness
99.07
98.58
97.96
95,99
Reasonable Internal Energy
95.65
94.88
95.04
36.07
No Internal Steric Clash
98.16
99.79
99.57
98.07
Table Percentage of valid (as obtained from xyz2mol) molecules passing each PoseBusters test.
7

---

Published as a conference paper at ICLR 2024
4.2 CAPTURING TRAINING SET STATISTICS
Next, we evaluate models on how well they capture bonding patterns and the geometry of local environments found in the training set molecules. In previous work (Luo & Ji 2022 Hoogehoom et models were compared based on how well they capture the true bond length distributions observed in QM9. However, such statistics only deal with pairwise bond lengths and cannot capture the geometry of how atoms are placed relative to each other. Here, we utilize the bispectrum (Uhrin 2021) as a rotationally invariant descriptor of the geometry of local environments. Given a local environment with central atom u, first project all of the neighbors of according to the inferred bonds onto the unit sphere 5². Then, we compute the signal f as a sum of Dirac delta distributions along the direction of each neighbor: [(F) = - drmm (f-r f.p.). The bispectrum B(J) of is then defined as: Bf) = EXTRACTSCALARS ffffffffff f). Thus, f captures the distribution of atoms around and the bispectrum B(f) captures the geometry of this distribution. The advantage of the bispectrum is that it varies smoothly when f is varied and is guaranteed to be rotationally invariant. We compute the bispectrum of local environments with atleast 2 neighboring atoms. Note that exclude the pseudoscalars in the bispectra.
For comparing discrete distributions, as employed in [Hoogeboom et al. (2022). Given the true distribution Q and the predicted distribution P, the Jensen-Shannon divergence between them is defined as: Dus(Q P) DKL (Q || M) + 2/2 DEL (PI I| M) where Dx is the
use the symmetric Jensen-Shannon divergence (JSD)
divergence and M Q+P is the mean distribution. For continuous distributions, estimating the Sensen-Shamme
2 divergence
from samples is tricky without further assumptions on the distributions. Instead, we use the Maximum Mean Discrepancy (MMD) score from Luo & Ji 2022) instead to compare samples from continuous distributions. The MMD) score the distance between means of features computed from samples from the true distribution Q and the predicted distribution P. A model with smaller MMD score captures the true distribution of samples better. We provide details about the MMD score in Section E.2
From Table 3 we see that Symphony and other autoregressive models struggle to match the bond length distribution of QM9 as well as EDM. This is the case except for the single C-H and single N-H bonds. On the bispectra, however, Symphony attains the lowest MMD for several environments. To gain some intuition for these MMD numbers, we also plotted the bond length distributions, samples of the bispectra, atom type distributions and other statistics in Appendix A for each model.
4.3 GENERALIZATION CAPABILITIES
All of the metrics discussed so far can be maximized by simply memorizing the training set molecules. Now, we propose a new metric to evaluate how well the models have actually learned to generate valid chemical structures. We compare models by asking them to complete fragments of 1000 unseen molecules from the test set, with one hydrogen atom removed. We then check how many final molecules were deemed valid. Since the valid completion rate (VCR) depends heavily the quality of the model, we compute the valid completion rate for fragments of molecules from the training set as well. If the performance is significantly different between the two sets of fragments, this indicates that the models do not generalize well. Diffusion models such as EDM are more challenging to evaluate for this task, since we would need a way to fix the initial set of atoms, so we compare only Symphony and G-SchNet. Encouragingly, both models are able to generalize well to unseen fragments, but Symphony's overall completion rate is higher for both seen and unseen fragments. We notice that the performance of Symphony on this task seems to decrease as training progresses, the reason for which remains unclear.
4.4 MOLECULE GENERATION THROUGHPUT
One of the major advantages of autoregressive models (such as Symphony) over diffusion models (such as EDM) is significantly faster inference speeds. As measured on single NVIDIA RTX A5000 GPU, Symphony's 's inference speed is 0.293 seconds/molecule, compared to EDM's 0.930 sec/mol. Symphony is much slower than existing autoregressive models (G-SchNet is at 0.011 sec/mol, and G-SphereNet 0.006) because of the additional tensor products for generating higher-degree E(3)- equivariant features, but is still approximately 3x faster than EDM. However, our sampler is currently bottlenecked by some of the limitations of JAX (Bradbury et al. 2018); we believe that Symphony's 's inference speed reported here can be significantly improved to match its training speed.
8

---

Published as a conference paper at ICLR 2024
MMD of Bond Lengths +
Symphony
EDM
G-SchNet
G-SphereNet
C-H: 1.0
0.0739
0.0653
0.3817
0.1334
C-C: 1.0
0.3254
0.0956
0.2530
1.0503
C-O: 1.0
0.2571
0.0757
0.5315
0.6082
C-N: 1.0
0.3086
0.1755
0.2999
0.4279
N-H: 1.0
0.1032
0.1137
0.5968
0.1660
C.0:
0.3033
0.0668
0.2628
2.0812
C-N: 1.5
0.3707
0.1736
0.5828
0.4949
O-H: 1,0
0.2872
0.1545
0.7899
0.1307
C-C: 1.5
0.4142
0.1749
0.2051
0.8574
C-N: 2.0
0.5938
0.3237
0.4194
2.1197
MMD of Bispectra 4
Symphony
EDM
G-SchNet
G-SphereNet
C: C2,H2
0.2165
0.1003
0.4333
0.6210
C: C1,13
0.2668
0.0025
0.0640
1.2004
C: C3.H1
0.1111
0.2254
0,2045
1.1209
C2,H1O011
0.1500
0.2059
0.1732
0.8361
C1,2,0011
0.3300
0.1082
0.0954
1.6772
O: C1,I
0.0282
0.0056
0.0487
0.0030
C2,HI,1
0.1481
0.1521
0.1967
1.3461
C2,H1
0.2525
0.0468
0.1788
0,2403
C: C1,H2NI
0.3631
0.2728
0.1610
0.9171
N: C2HHI
0.0953
0.2339
0,2105
0.6141
-heenn-Saanon Divergence |
Symphony
EDM
G-SchNet
G-SphereNet
Atom Type Counts
0.0003
0.0002
0.0011 0.0150
0.0026
Local Environment Counts
0.0039
0.0057
0.1016
Table 3: Comparing statistics of generated molecules those found in QM9. (Top): The MMD of bond lengths for the 10 most frequent bonds. The notation 'X-Y: T means that a atom was bonded to Y atom with bond of type T. (Middle): The MMD of bispectra for the 10 most occurring local environments. The notation 'X: Yn,Zm' means that an X atom was the central atom, surrounded by Y atoms and 7. atoms. (Bottom): The JSD of occurrence counts for atom types and local environments. 4 indicates that lower is better for the metrics.
Symphony Symphony 500K steps
Symphony 1000K steps G-SchNet
Valid Completion Rate 1
800K steps
Training: VCR
98.53
96.65 96.30
95.57 95.43
97.91 98.15
Testing: VCR
98.66
Table 4: Comparing the difference between fragment completion rates on (seen) training and (unseen) testing fragments with one hydrogen removed.
CONCLUSION
We have proposed Symphony, new method autoregressively generate 3D molecular geometries with spherical harmonic projections and higher-degree equivaraia features. We show promising results on molecular generation and completion, relative to existing autoregressive models. However, one drawback of our current formulation is that the discretization of our radial components is too coarse, so our bond length distributions are not as accurate as EDM or G-SchNet. This affects our validity when using lookup tables to assign bond orders as they are particularly sensitive to exact bond lengths. Further, Symphony incurs increased computational cost due to the use of tensor products to create higher degree =(3))cqquvariant features. As a highlight, Symphony is trained on only = 80 epochs, while G-SchNet and EDM are trained for 330 and 1100 epochs respectively. Further exploring the data efficiency of Symphony remains to be seen. In the future, plan to explore normalizing flows to smoothly model the radial distribution without any discretization, and placing entire local environment motif' once which would speed up generation.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
Our JAX code containing all of the data preprocessing, model training and evaluation metrics is available at httts:://gttttiiee http:////....oo..oooooootttt//aa
describes the hyperparametters used the training process for Symphony. Details about the metrics used can be found in Appendix E Section contains all of the information regarding the QM9 dataset used in this work. Further information about the baseline models and the sampled structures can be found in Section C.4
Section C.2
7 ETHICS STATEMENT
Generative models for molecules such as Symphony have the potential to be used for discovering novel drugs and useful catalysts. While harmful uses of such generative models exist, the synthesis of a molecule given only its 3D geometry is still extremely challenging. Thus, we do not anticipate any negative consequences of our research.
ACKNOWLEDGMENTS
Ameya Daigavane was supported by the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions), and the NSF Graduate Research Fellowship program. Song Kim was supported by Analog Devices as an Undergraduate Research and Innovation Scholar in the MIT Advanced Undergraduate Research Opportunities Program (SuperUROP). Mario Geiger and Tess Smidt were supported by the Integrated Computational and Data Infrastructure (ICDI) program of the U.S. Department of Energy, grant number DE- DE-SC0022215 The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper.
REFERENCES
Dylan M. Anstine and Olexandr Isayev. Generative Models as Emerging Paradigm in the Chemical Sciences. Journal of the American Chemical Sociery, 145(16):8736-8)) 8750, 2023. doi: 10.1021/ jacs.2c 13467. URL
PMID: 37052978.
Michael Banck, Craig A. Morley, Tim Vandermeceschh,ndd and Geoffrey Geoffryy R. Hutchison. Open Babel: An open chemical toolbox, 2011. URL htt.atts///oi..r///110. 1ttp://.1..111111131 0005558-246--- 2946-34-63- 3-33--3333
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P. Mailoa, Mordechai Korblluth, Nicola Molinari, Tess E. Smidt, and Boris Kozinsky. {(3))-quivariant graph neural networks for data-efficient and accurate interatomic potentials. 13, May 2022. URL https: //dois org/
10. 12011088/88 /4147414477-7 022-577022-299222
James Bradbury, Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: Composable Transformations of Python+NumPy programs, 2018. URL
http::/ /ttp::///ithhb. com/ wtt.ccommmooo/...o goomm/gogle//ax
Martin Buttenschoen, Garrett M. Morris, and Charlotte M. Deane. PoseBusters: Al-based docking methods fail generate physically valid poses or generalise novel sequences, 2023.
Taco S. Cohen and Max Welling. Harmonic exponential families manifolds, 2015.
Ameya Daigavane, Balaraman Ravindran, and Gaurav Aggarwal. Understanding Convolutions on Graphs. Distill, 2021. doi: 1/d391//iitill.000
Niklas Gebauer, Michael Gastegger, and Kristof Schütt.. Symmetry-adapted generation generaattiooninnnnnnnnnnnnoo of 3d point sets for the targeted discovery of molecules. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL out.o//ooooootttpp/o//oooeooennn
fi f11e://ppprr22111114a44/4ee00063
10

---

Published as a conference paper at ICLR 2024
Niklas W. A. Gebauer, Michael Gastegger, Stefaan S. P. Hessmann, Klaus-Robert Müller, and Kristof T. Schütt. Inverse design of 3d molecular structures with conditional generative neural net- works. 13, February 2022. URL
Mario Geiger and Tess Smidt. e3nn: Euclidean Neural Networks, 2022.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. Kernel Two-Sample Test. Journal of Machine Learning Research, 13(25);723-77,, 2012. URL
http:// /mir wwtp vi 3tt/3///retotaaaaaaa/eaa ntml
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant Diffusion for Molecule Generation in 3D, 2022.
Yeonjoon Kim and Woo Youn Kim. Universal Structure Conversion Method for Organic Molecules: From Atomic Connectivity to Three-Dimensional Geometry. Bulletin of the Korean Chemical Society, 36(7):1769-177,, 2015. doi: https://doi..rg/11.. 1002//0k0...0...ss.. 10334. URL https://
onn onlinelibrary wiwey. www.eyycccoooooooommmmo www.///.ooo..oo
10334
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, 2017.
Greg Landrum, Paolo Tosco, Brian Kelley, Ric, David Cosgrove, sriniker, gedeck, Riccardo Vianello, Nadine Schneider, Eisuke Kawashima, Dan N, Gareth Jones, Andrew Dalke, Brian Cole, Matt Swain, Samo Turk, Alexander Savelyev, Alain Vaucher, Maciej Wójeikowski, Ichiru Take, Daniel Probst, Kazuya Ujihara, Vincent F. Scalfani, guillaume godin, Juuso Lehtivarjo, Axel Pahl, Rachel Walker, Francois Berenger, jasondbiggs, and strets rdkit/rdkit: 2023 03 2 (QI 2023) Release, June 2023. URL htt.::ttp///doo org 10. 22281//zeodo. 8C53810
Yaron Lipman, Ricky T. Q. Chen, Heli Ben- Hem-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023.
Youzhi Luo and Shuiwang Ji. An Autoregressive Flow Model for 3D Molecular Geometry Generation from Scratch. International Conference on Learning Representations, 2022. URL https
//oggn/eeeiieieeee///ooooooooaoa htt/://ooommd0000meeeeeoeo NS5W
//openreview.co net.
Kieran Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia.
Implicit-pdf: Non-parametric representation of probability distributions the rotation manifold, 2022.
Cameron J. Owen, Steven B. Torrisi, Yu Xie, Simon Batzner, Jennifer Coulter, Albert Musaelian, Lixin Sun, and Boris Kozinsky, Complexity of Many-Body Interactions in Transition Metals via Machine-Learned Foree Fields from the TM23 Data Set, 2023.
Sergey N. Pozdnyakov and Michele Ceriotti. Incompleteness of graph neural networks for points clouds in three dimensions, 2022.
Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu. Deep Learning for the Life Sciences. O' O'Reilly Media. 2019, https: www.. amazon. com/ SereeSSciienn
Deep-Learning-
A. K. Rappe, C. Casewit, K. S. Colwell, W. A. III Goddard, and W. M. Skiff. UFF, full periodic table force field for molecular mechanics and molecular dynamics simulations. Journal of the American Chemical Society, 10224)): 12 1992. doi: 10.1021/j000011a440 URL
https: http://///......00011111/2//////
Sereina Riniker and Gregory A. Landrum. Better Informed Distance Geometry: Using What We Know To Improve Conformation Generation. Journal of Chemical Information and Modeling, 55 (12):2562 2574, 2015. doi: 10. jcmi////ccc http:00......tttttt////....////.
acs. jeim. 5b00654
PMID: 26575315.
M. Rupp, A. Tkatchenko, K.-R. Müller, and O. A. von Lilienfeld. Fast and accurate modeling of molecular atomization energies with machine learning. Physical Review Letters, 108:058301, 2012.
11

---

Published as a conference paper at ICLR 2024
Benjamin Sanchez-Lengeling. Emily Reif, Adam Pearce, and Alexander B. Wiltschko. A
Gentle Introduction to Graph Neural Networks. Distill, 2021. doi: 1022311/disttil.000033. https:/diisilllpbbb222//mmmmmm
Victor Garcia Satorras, Emiel Hoogeboom, Fabian B. Fuchs, Ingmar Posner, and Max Welling. E(n) Equivariant Normalizing Flows, 2022a.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) Equivariant Graph Neural Networks, 2022b.
Schrödinger, LLC. The PyMOL Molecular Graphics System, Version 1.8. November 2015.
Kristof T. Schüitt, Pieter-Jan Kindermans, Huziel E. Sauceda, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. SchNet: A continuous-filter convolutional neural network for modeling quantum interactions, 2017.
Gregor Simm, Robert Pinsler, and Jose Miguel Hernandez-Lobato. Reinforcement learning for molecular design guided by quantum mechanics. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pp. 8959-8969, PMLR, 13-18 Jul 2020. URL
https: /kttss//pproee http:///pccceeeg...c/ mlr http://.ngg...mm.pppss/////1111 1ttps/////19//im2mm22 immm20m.... html
Gregor N. C. Simm, Robert Pinsler, Gábor Csányi, and José Miguel Hernández-Lbalt Symmetry- aware actor-critic for 3d molecular design. In International Conference on Learning Representa- fions, 2021. URL httttttss///ppeeeeetetet
net id= XYN
Yuxuan Song, Jingjing Gong, Minkai Xu, Ziyao Cao. Yanyan Lan, Stefano Ermon, Hao Zhou, and Wei- Ying Ma. Equivariant flow matching with hybrid probability transport, 2023.
Martin Uhrin. Through the eyes of descriptor: Constructing complete, invertible descriptions of atomic environments. Phys. Rev. B, 104:1441111 10, Oct 2021, doi: 10.1 1033/hhssss.. 104.144110 URL http:tsts///i.k.ap...... org/
wwwp://w..ppp.o.ooo/.ooo.ooo 1103 1ttp:///o...../////eee.e....
Clement Vignac, Nagham Osman, Laura Toni, and Pascal Frossard. MiDi: Mixed Graph and 3D Denoising Diffusion for Molecule Generation, 2023.
David Weininger. SMILES, chemical language and information system. 1. Introduction to method- ology and encoding rules. Journal of Chemical Information and Computer Sciences, 28(1):31-36, 02 1988. doi: 10.102 (0.1021/ci00057aa00
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A ADDITIONAL ANALYSES
For all of the analyses performed this section, we used all the valid molecules for each model as computed by xyz2mol.
A.1 BISPECTRA LOCAL ENVIRONMENTS IN SAMPLED MOLECULES
As seen in Figure 4 we see that Symphony's 's sampled bispectra (second from left) have a slightly different distribution relative to those from QM9 in the two most frequent local environments.
Central Atom Type: C Neighbors: C2,H2
0.00
0.04
Enviroreentt
10
0.02
0.00
20
10
10
10
Symphany
GSphoroNet
Central Atom Type: C Neighbors: C1,H3
20
Local Envirreent
0.02
40
0.00
80
70
10
10
5
10
10
0
10
QM9
Symphany
EDM
GSchNet
GSpheraNet
Figure 4: Bispectra of local environments of type C: C2,H2 and type C: C1.H3 respectively. Each row corresponds a sample of the bispectrum (an array of length 15). Every entry of the bispectra colored by value according the colorbar the right.
A.2 BOND LENGTHS IN SAMPLED MOLECULES
From Figure 5 and Figure 6 that Symphony's bond length distribution tends to be wider than those QM9, hurting its MMD score relative to EDM. Improving this aspect is an ongoing effort; but we believe that the bond lengths still quite reasonable.
A.3 ATOM TYPE COUNTS
As seen in Figure all models are able to reasonably capture the distribution of atom types in QM9; Symphony performs especially well here.
A.4 RING SIZES
We also extracted all rings using RDKit Landrum et al., 2023) and counted their relative frequency, in In/Figuee 8, G-SphereNet seems to produce either very large or very small rings. The other models seem to capture the distribution of ring sizes well.
13

---

Published as a conference paper at ICLR 2024
Histogram Bond Lengths for C-H (Bond Type
Density
Symphony
80
EDM
GSccNee
GSphhreNoe
1.5
Bond Length
Bond Length (A)
Bond Length (A)
Bond Length (A)
Bond Length (A)
Histogram of Bond Lengths for C-C (Bond Type 1.0)
10
GMB
Symphony
G.ScNWi
GSphereNNe
1,0
1.5
2.0
1.0
1.5
2,0
1.0
1.5
2.0
1.0
1.5
2.0
1.0
1.5
2.0
Bond Length
Bond
Bond (A)
Bond Longth
Histogram
Lengths (Bond Type 1.0)
12.5
10.0
Symphony
EDM
GBchiee
GSphereNee
2.5
Bond Length (A)
Bond Length (A)
Bond Length (A)
Bond Length (A)
Bond Length
Histogram of Bond Lengths for C-N (Bond Type 1.0)
GMS
Symphony
GSchNer:
GSppeenNNt
1.5
2.0
1.5
2.0
1.0
2.0
1.0
1.5
2.0
1.0
1.5
Bond Length
Bond Length (A)
Band Length (A)
Band Length (A)
Bond Length
Histogram
Lengths (Bond Type 1.0)
15
Symphony
EDM
GSphercNet
Bond Length (A)
Bond Length (A)
Bond Length (A)
Bond Length (A)
Bond Length (A)
Figure Histogram bond lengths for the five most frequent bonds in QM9.
14

---

Published as a conference paper at ICLR 2024
Histogram Bond Lengths for C-O (Bond Type 2.0)
15.0
12.5
10.0
GMB
Densty
Symphony
7.3
3.0
GSchivr
GSphereNot
25
0.0
1.0
1.5
2.0
1.0
1.5
2.0
1.0
1.5
2.0
1.5
2,0
1.5
Bond Length
Bond Length
Bond Longth
Bond
Bond
Histogram of Bond Lengths (Bond Type 1.5)
Densty
Symphony
EDM
GSpheraNat
1.0
1.5
2.0
1.0
Bond Length (A)
Bond Length (A)
Bond Length (A)
Bond Length (A)
Bond Langth (A)
Histogram Bond Lengths for O-H (Bond Type 1.0)
Dentity
15
Symphory
EDM
10
GSchNe:
GSphereNee
Bond Length (A)
Bond Length (A)
Bond Length (A)
Bond Length (A)
Bond Longth (A)
Histogram of Bond Lengths for C-C (Bond Type 1.5)
GMB
Symphony
GSicheee
GophereNot
1.0
1.5
2.0
1.0
1.5
2.0
1.0
1.5
2.0
1.0
1.5
2.0
1.0
1.5
2.0
Length
Bond
Length (A)
(A)
Bond Length
Histogram Bond Lengths for C-N (Bond Type 2.0)
15
Symphony
EDM
GSphereNNt
1.5
1.0
2.0
1.0
2.0
1.5 Bond Longth (A)
Bond Length
Bond Length (A)
Bond Length (A)
Band Length (A)
Figure Histogram bond lengths for the sixth to tenth most frequent bonds in QM9.
15

---

Published as a conference paper at ICLR 2024
Atom Type Counts
model
QM9
Symphony
EDM
10
GSchNet
GSphereNet
Fraction
10%
Atom Type
Figure 7: Frequency of atom type counts in generated molecules log-scale.
Frequency of Ring Sizes
10
Model
QM9
Symphony
EDM
10
GSchNet
GSphereNet
Fraction
10
10
10
9
10
Ring Size
Figure 8: Frequency of ring sizes in generated molecules log-scale.
16

---

Published as a conference paper at ICLR 2024
B PROOF OF E(3)-EOUIVARIANCE
Theorem B.I. Suppose EMBEDDER produces -quivvvraaa and translation-invariant features
he EMBEDDER(S*) uud for every atom v. Then, p prosition -quiivaann and translation- invariant (and hence, E(3) equiivarrr):
position
RE RF + T fu+1, Zu+1: R T) position (Fu+1 fut1, Z=+1:8")
Proof: We first show that yposition is -3)-eiuuivariant. We have:
MABEDDEE(RS")) )u. (R()" "RRE"EDDDDRR(")
for every atom e and degree Note that because Zn+1 is rotationally invariant, it immediately follows from Equation and the above, that e₁ is also (33)-qquivariant with degree
a(;; R&S", SatI, 7m+1) = cr; S" Snti, ?nt1)
Now, as the Wigner D-matrices are always unitary, we have:
pposition RR; RS" , 2n+1)
a(r; fn+1,, Zu+1) ( (((R);)))
1=0
arr; fn+1, Zm+1) D Σ
cr(r; S : Zn Zn+1)"
froittion (*)
by definition. Thus, we are guaranteed that frosition is ((3)-equivarin)) Note that applying pointwise non-linearity (exp) frpsition and a rotationally invariant normalization does not change ((3) equivariance. Thus, prsition is -quuivarrii as well.
For translations, note that prosition is described relative to the focus atom fu+1. Thus, EMBEDDER is translation-invariae
EMBEDDER(S" T),. EMBEDDER(S")
pooiiiion will be translation
position [n+11 + T fn+1, S + T) = position (Te+1 fn+11 Zn+1;
In conclusion, prssition -quuivraia and mmnslattoon-equiariiant and hence E(3) equiivariat Thus, Property (2) is satisfied.
Theorem B.2. Suppose EMBEDDER produces crinutaaioo-equivarian features h
EMBEDDER( )S.) for every atom c. Then, focus is permuaation-cqqiiarraan,, while pspecies and pposition are permutation-invariant.
tocus (T(f+1):= rocus (fn+1:8")
p spocies ZZn+1 T7 (f)+1)):TT)) = p species (n++1 fn+1i5")
position (ra+1 T ((n+1), Zn+1: = prosition (Fxt1 fntt1 St)
where represents permutation of the atoms of S
Proof: Because EMBEDDER is permutaaion-equuivrraatt equivariant:
EMBEDDER ST))) )=()), EMBEDDER(S")...
for each atom v. Then, from Equation 5
Jucus (ff++)):*"")
( (F((4)))))
-MLP((BMBEDDER(S") fm410))))
focus (fn+1):
17

---

Published as a conference paper at ICLR 2024
claimed. Similarly, p speciss (Zu+1 x (ff+t));=8") MLY(EMMEEDATOOMMYYYZZZZ)) (TBE (=((f=-1).))
MLP.EMBBDAATOMTYPPIZHH EMBEDDER(S"))
ppssies (Zn+1 Sn+1;8")
For prositiin, it is sufficient to show that the coefficients (r(r) are permutationnequiivriaatt
(frt(() LIBEAR(EEBEDDER ) @ EMBEDAATOMTYPE((2H+))))
LINEAR(EMBEDDER(S") fn4r EMBEDATTOTTYPE(2 (2++))
fm-1; 2n+1,
Thus, all distributions transform as expected.
C DETAILS OF MODELS
C.I EMBEDDERS
Here, we describe E3SchNet and NequlP (Batzner et al. 2022) which we use to embed the atoms in each fragment into -equivaavva features. As shown Appendix we require these models to be E(3) -quivvaaaa
Both of these models are geometric message message-passing neural networks, type graph neural network (Sanchez-Lengeling et al., 202T;; Daigavanc et al. (021) that respects the symmetries of 3D structures. In particular, E3SchNet as the EMBEDDER for the focus and atom type prediction, and NequlP as the EMBEDDER for the position prediction. Unlike previous autoregressive models which utilized shared embedder for all tasks, we found that using different embedders for these two tasks performed much better in our experiments.
Given the fragment S", we define neighbour of each atom E S by Euclidean distance cutoff I dmaxi
N(11) = Li E S || S dmax
(8)
Initially, the features h, of each atom in S are set as the embedding of its atomic number Z;- At each iteration the features h( is updated using the atom's 's features (1(-11 and its neighbour's features hft (t-1) where (())) from the previous round. The final embedding for atom is returned
as (7) where T is the number of cressage-passingg iterations. Algorithm 2 formally shows the operations of general message passing neural network.
Algorithm 2 General Operation of Message Passing Neural Network
Input: Fragment S", Message Passing Iterations T, Cutoff dmax. Update Function UPDATE Compute neighbor lists for each atom in St according to Equation 8
for é 1, 2, do:
T.(0) SCALAREMBEDDING (Z:)
for t = 2, T do:
for 1, 2,....... do:
h (t-1)
(N(1) < (1) lj (jE(()))
6/9) UPDATE(M (1-1),
(6))
return (M. (1(P)p
Different message-passing networks differ in their choice of UPDATE function. Following Batzner et al. (002). the UPDATE for NequlP is defined as:
UPDATE(M(()) (t-1) (n = n. (1-1)
Σ
Σ
Re Ro((F
(1-1)
JEN(()
1=0
18

---

Published as a conference paper at ICLR 2024
(-(-) is a learned multi-layer perceptron (MLP). We set C = 20, dmax 5A, Imax 5, and T = here. For clarity, we assume the decomposition of the tensor product into a direct sum irreducible representations of ((3) above.
E3SchNet is our generalization of the SchNet model (Schitt et al., 2017) that was used in GGebauer al. 2019 to produce higher-degree -33)-vquuvariint features. The UPDATE function for E3SchNet 1s defmeed as:
UPDATE(R (t-1) ( (t-1) LINEAR
Wizt (
(EN( t=
where Wayt are scalars computed via:
Wife LINEAR ([(CCOFFF(()) RADIALBASSIS(FF;))))
We use the Gaussian radial basis functions, following SchNet. In fact, for (mxx = O, E3SchNet reduces exactly to the standard SchNet. We set Luax 2, find that the benefits of using even higher degree features for the focus and atom type prediction task are minimal. The cutoff is again 5A.
We that NequlP and E3SchNet guarantee crmmutation-equivriinn -quuvivaiaa,,,,,, and hence satisfy the requirements for EMBEDDER in Appendix
translation invariance and
We implement Symphony with the e3nn-jax library that utilizes the JAX (Bradbury al. 2018) framework for creating efficient ((3)_equivariant machine learning models.
C.2 TRAINING DETAILS
We Tue 10-$ and express the Dirac delta distribution in the spherical harmonic basis upto Imux = 5, as explained in Appendix H The predicted distributions p pfpecies and presition are learned by minimizing the KL divergence to their true counterparts. We found that adding small amount of zero zero-centered Gaussian noise o² = 2.5 x 10 to all input atom positions helped with robustness. All parameters in the EMBEDDER, MLP and LINEAR layers are trained with the Adam (Kingma & Ba 2017) optimizer with learning rate of x 10-1, We chose the parameters that achieved the lowest loss on the validation set over 8000000 training steps with a batch size of 16 fragments.
C.3 DATA DETAILS
Following EDM (Hoogeboom et al. 2022), obtained the QM9 (Rupp et al. 2012) dataset using the
DeepChem library (Ramsundar et al. 22019), and filtered out 3054 uunchaacttriied molecules (avail- able ht/ttss: atlttps::/springesssss /stp//innnnnaaazzceeee
figshare. com/ http///cowooaeeeeeeeeeeeeeeee 31155404)
which rearranged significantly during geometry optimization, giving us exactly 130831 molecules. Symphony was trained used the same splits as EDM: 100000 molecules train, 13083 molecules for validation and 17748 molecules for test, obtained from a random permutation of the molecules.
C.4 BASELINE MODEL DETAILS
For the baseline models, used the pretrained EDM model https: thub ccm//ehocgeboom/e3333. e fatsfffffoooffiieeooeeoooooolnee for cotrrmolcculee and the pretrained G-SphereNet model
ggraph3D G SphereNet We retrained the G-SchNet model on the EDM splits fol- lowing https:: https://githul. cwm amistic machine- learning/G SccNNett The samples (in xxyz format) of all models used for evaluation is available at this URL:
https://Eigshi: wwwp////alceaaaeeeeee222...
D LEARNING AND SAMPLING FROM POSITION DISTRIBUTIONS
In this section, we drop the superscript from position it should be clear from context.
19

---

Published as a conference paper at ICLR 2024
D.1 TRAINING
To recap Section 3.3 Symphony predicts coefficients (r;"") represent the position distribution
((r,0,0) = log Σ
q (r:80))"
exp
chamnel
1=0
p(r, (1) = z exp (r,
where Z the partition function.
As mentioned in Section C.2 the coefficients are learned by minimizing the KI. divergence to the target distribution q:
KL [p)) J
((F)f(F)dF log Z / (()) log log nn
(() log
dr
Following the notation of Section 3.3 Ω represents the set fr E 10, E [0, E (0,2 which is all space in spherical coordinates.
For training, we only need the unnormalized logits f and not the normalized distribution This is identical the log sum-exp trick when training with cross-entropy loss for classification problem. Unlike the classification case where the number of classes is finite, the integral above must be computed over all r, and & which an infinite set. To numerically approximate this integral, we use uniform grid on and a Spherical r. As discussed in Section 3.3] the uniform grid on r spans values from 0.9A to 2.0A which is more than sufficient cover all bond lengths in organic molecules. The Spherical Gauss- LegennLe quadrature is a product of two quadratures: a ID Gauss-Legendre quadrature with 180 points over cos 0 E [-1, 1], and uniform grid of 359 points over [0, 2x) for ø.
quadrature on the sphere at each value of
Symphony predicts the coefficients (r) which can be used to evaluate frr, 0, Φ) at any point. This evaluation for spherical grid of (0, Φ) values can be done quickly via Fast Fourier Transform (FFT) that is implemented in e3nn-jja jax. We perform this FFT procedure for each sphere defined by radial grid point
D.2 SAMPLING
Once the model is learnt, we need to sample from the distribution p. A key advantage of predicting the coefficients (r) of fr, 0 Φ) is that different resolution of angular grid can be chosen for sampling than that of training. We simply evaluate frr 0, Φ) on the quadrature grid as before, apply the exponential, and normalize via numerical integration get p(r, 0, Φ). We first marginalize over 0, o to obtain a distribution p(r) to sample a radius r. Then, we sample one of the angular grid points (0, () for the sphere corresponding to this radius Overall, this procedure gives sample from p(r, 0, o).
In Section G.2 we assess how the validity of molecules generated by Symphony varies as the grid resolution is varied.
Note that our sampling procedure is much simpler than that of Simm et al. (202I). which uses rejection sampling with a uniform base distribution. We perform some quantitative experiments with the parametrization Simm et al. (202T) in Section F.2.
While we are primarily interested in learning distributions over 3 which are equivariant under E(3). there has been prior work in learning distributions over manifolds [Cohen Welling (2015); Murphy al. (2022), where the issue of estimating the partition function are also solved by discretizing over an appropriate domain.
E DETAILS OF METRICS
E.1 POSEBUSTERS
Table 5 provides details of the Posebusters tests used in Table We use the default parameters from their framework.
20

---

Published as a conference paper at ICLR 2024
Test
Description
All Atoms Connected
There exists a path along bonds between any two atoms in the molecule.
Reasonable Bond Lengths
The bond lengths the input molecule are within 0.75 of the lower and 1.25 of the upper bounds determined by distance ge- ometry.
Reasonable Bond Angles
The angles in the imput molecule are within 0.75 of the lower and 1.25 of the upper bounds determined by distance geometry.
Aromatic Rings Flatness
All atoms in aromatic rings with 5 or members are within 0.25A of the closest shared plane.
Double Bonds Flatness
The two carbons of aliphatic carbon-carbon double bonds and their four neighbours are within 0.25A of the closest shared plane. The calculated energy of the input molecule is no more than 100 times the average energy of ensemble of 50 conformations generated for the input molecule. The energy is calculated using the UFF (Rappe al., 1992) in RDKit (Landrum et al.] (2233 and the conformations are generated with ETKDGv3 (Riniker & Landrum 2015) followed by force field relaxation using the UFF with up to 200 iterations.
Reasonable Molecule Energy
No Internal Steric Clash
The interatomic distance between pairs non-covalently bound atoms is above 0.8 of the lower bound determined by distance geometry.
Table 5: Description of each intramolecular PoseBusters test, taken from Table 4 of Buttenschoen et al. 2023).
E.2
MAXIMUM MEAN DISCREPANCY
The Maximum Mean Discrepancy (MMD), introduced in Gretton et al. (2012), measures how different two distributions px and py are, given kernel function A. Formally, y, the MMD is defined as:
MMD(px,Py) PY)
E V X.X'xxx
X()]
+
E Y
[KYY,
E X-px Apy
[xX,, Y)]
From the above equation, we see that the MMD can be easily estimated with samples from each distribution. We choose is the sum of Gaussian kernels at different scales:
(XX = Σ exp((10((-3) -|X -x
F THE ADVANTAGE OF USING MULTIPLE CHANNELS OF SPHERICAL HARMONICS
F.I AN EXAMPLE WITH THE OCTAHEDRON
Figure 9 shows how adding a second channel helps reduce the effective mmxx needed to represent posmio The atoms depicted by red circles have been placed already, and the atom at the center of the octahedron has been chosen as the focus. To accurately capture the positions of the three remaining atoms (depicted by two stars and square), we would need projection upto l = 4, because the angle made by the 'star", central atom and the 'square' is z z radians. However, we used one channel to represent the 'stars' and one to represent the 'square', we can get away by only using projections upto mmx = 2, because the 'stars' are diametrically opposite each other.
F.2 A STUDY ON LEARNING RANDOM SIGNALS
To quantitatively show the effect of having multiple channels, we see how well the model is able to learn random distribution the sphere. We randomly sample N target points with coordinates
21

---

Published as a conference paper at ICLR 2024
Figure 9: Usually, would require mmax 4 to represent prosition for the 'stars' and "square' atoms, centered at the red central atom. With two channels, we only need up to mmax = 2 each.
(R:) iil on the sphere, and then define the distribution:
(f)
exp(oi it))
with the same Dirac delta distribution approximation as described in Appendix H. We use mmax = throughout this section. Then, we randomly initialize coefficients to minimize the KL divergence to
min K L(q || p.)
where Pe is the probability distribution defined by coefficients c, as before:
Lax
(10,6) log Σ
T Y((
exp
chamnel ch
1=0
1
p(0, Φ) exp (0,
This corresponds a simpler setting where we have only one radius r.
We assess the KI. divergence a function of number of position channels ch and 4mxx in Figure 10) We see a consistent improvement across different |max as the number of position channels are increased.
KL Divergence Random Signal
10'
10
Position Channels
KL Divergence
10
10
2
5
Figure 10: Final KL divergence K L(q || Po) for learned coefficients e as function number of position channels ch and 4max.
22

---

Published as a conference paper at ICLR 2024
We also experimented with the parametrization from Simm al. 2021 who define:
p(0, Φ) Z exp (-x (10,0000)
where k = =i=in |al2. This extra factor of was proposed by Simm et al. (0021) to "regularize the distribution so that it does not approach delta function". In the left panel of Figure 11 show that this regularization hurts the model. Even adding multiple channels does not help, because the regularization term 'switches' off multiples channels. However, as shown in the right panel of Figure 11 removing this regularization significantly helps the model, with further improvement as the that parametrization performs similarly to Simm et al. (2021) without the regularization term. Based on this experiment, we plan to experiment with non-lincarities for the logits future versions of Symphony.
number of channels are increased. For mmx 5,
Divergence on Random Signal
Simm at al. (2021) Parametrization
Simm et al. (2021) Paramotrization without Regularization
10¹
Position Channels
Divergence
10
KL Divergence
10
2
1
Max L
Max
Figure 11: Final KL divergence K LLq || Pe) for learned coefficients c as function of number of position channels ch and lmnx, with the parametrization proposed by (Simm al. (202I). Removing the regularization term helps the model learn better.
G ABLATION STUDIES
G. 1ax AND NUMBER OF POSITION CHANNELS
To understand the practical effect of adding multiple position channels to Symphony, as well as the impact of increasing (max. trained variants of Symphony varying (max for the focus embedder E3SchNet from to 2, the number position channels from to and mmxx for the position embedder NequlP from 1 to 5.
Due to computational constraints, we trained these models for 1, 000, 000 steps each, which 8x lesser than the original model reported in Section 4 Thus, the validity numbers slightly lower overall. However, we believe we can still observe important trends from this experiment.
We report the validity measured by xyz2mo1 for each of these models in Figure 12
For the focus embedder E3SchNet, we do not significant increase in validity when going from Imax to Imax 2.
For the position embedder NequlP, we find large jump when going from Imxx 1 to Imax Further increasing /mux seemed to help slightly. For computational reasons, we kept (max 5.
Increasing the number of position channels helps for Imxx in particular.
23

---

Published as a conference paper at ICLR 2024
Ablation Study: Validity Molecules
Max L for Focus = 1, Position Channels
60
L Focus Position Channels 2
Validity %
Max L for Focus 1, Position Channels
Max L for Focus 1, Position Channels 4
40
Max L for Focus 2. Position Channels
L Focus Position Channels 2
20
Max L for Focus Position Channels
Max L for Focus Position Channels 4
3
Max Position
Validity as function of /mxx for the position and focus embedders. Models for which mmax = for the focus embedder are marked in blue. Models for which Imax = 2 for the focus embedder are marked in red. The intensity of colours increases with the number of position channels.
Figure 12:
G.2 RESOLUTION
Here, we take the trained Symphony model, freeze all weights, and measure the validity of molecules across range of grid resolutions. The original grid resolution for model training was re) (180, 359) as described above. From Figure 13 we see that the validity is within the expected variation even when using upto 10x smaller grids. Further amplification of the resolution also does not seem to affect the validity. We hypothesize that this is due to sampling with lower temperature than ideal making the target distribution more diffuse; future work will seek to understand this effect better.
Validity Molecules for Different Resolutions (ra, rg)
(360, 179)
83.8
(90,
83.7
83.6
Validity %
83.5
(180, 89)
83.4
(45, 719)
(45, 179)
(90, 719)
83.3
(90, 89)
(180, 179)
(45, 359)
83.2
(360, 359)
(360, 89)
(45, 89)
(90, 359)
(180, 719)
83.1
(180, 359)
83.0
10
10
Total Resolution (re rg)
Figure 13: Validity as a function of sampling grid resolution (ro.re).
The previous experiment measured the effect of the grid resolution for sampling. We also seeked to understand the effect of the grid resolution for training. For this, we reuse the task of Section F.2 and vary the grid resolution. All other hyperparameters were kept fixed, with 4mxx = 2 and 2 position channels. From Figure 14 we see that the learning is not affected even at low resolutions. In fact, from KL divergence perspective, it is easier learn at lower resolutions because localization is easier. However, lower resolutions come with decreased accuracy when sampling, as shown by the rightmost plot of Figure 14]
24

---

Published as a conference paper at ICLR 2024
Leamiing Random Distribution on the Sphere
RL Divergence Loss
Gracient Nomms
Mean Distance to Closest Target Point
10
Mean Disanne Orooot Torpet Purk
107
3.5
10 Deiigette Loee.
Goclet Nome
Reaskrtonn
Irurgl=110.
10"
Irurgl= 1180, 1791
2591
Pruingl 1360, 7191
79
02
0.1
10
2000
4000
8500
5000
10003
0
2000
4000
6000
5000
10000
2000
4000
6000
8000
10000
Number of Inering Shace
Number of Inering Shace
Number of Inering Shace
Figure 14: The effect of resolution when learning the random signal from Section F.2 model was trained with a resolution of To) (180, 359).
Our original
G.3 TEMPERATURE
Again, take the trained Symphony model, freeze all weights, and measure the validity of molecules across a range of temperatures T. This means scaling all the logits by a factor of 7 Higher temperatures make the model more diffuse, while lower temperatures make the model more peaked. We see that while the validity improves significantly at lower temperatures, the uniqueness tends to suffer. As seen Figure 13] this experiment suggests more careful sampling of the temperature better understand Pareto frontier between validity and uniqueness.
H REPRESENTING DIRAC DELTA DISTRIBUTIONS
Suppose we have the function ff) = 8(F to) defined on the sphere S and we wish to compute its spherical harmonic coefficients Ct.m
Lax
f(0.
Σ of ())000))
Σ
Σ
C. (0,
m=-1
By orthonormality of the spherical harmonics, and the annihilation property of the Dirac delta:
Cl.m = / [(0, Φ) Vi,m T. (0, () sin 0d0ds
/ 8(F - (() (0, o) sin 0d0do
(fo)
Thus, can easily compute the spherical harmonic coefficients for the Dirac delta distribution upto any required max. This is implemented in the e e3nn jax package. Due to the frequency cutoff, the Dirac delta distribution thus obtained is smooth approximation of true Dirac delta.
I THE EFFECT OF DISTANCE CUTOFFS ON DIFFUSION MODELS
As described in Section 3.6) EDM (Hoogeboom et al. 2022) and other diffusion models use fully- connected graphs to denoise the 3D molecular graph at each timestep. Here, investigate the performance of EDM as we apply a distance- based cutoff c to the edges:
(i,j) E E
|| Fy S
In Table 6 we vary e on the QM9 dataset to measure how many of the edges in the fully-connected graphs are kept after the radial cutoff.
In Figure 16, we measured the 'atom stability' as computed by Hoogeboom et al. (022) on the FDM-generated molecules is varied, when trained on QM9. We notice sharp drop-off in this
25

---

Published as a conference paper at ICLR 2024
Validity of Molecules for Different Temperatures Trccus
0.01 0.1
80
60
Validity %
20
10
100
30
40
50
60
70
80
90
100
Uniqueness %
Validity of Molecules for Different Temperatures T Tposttion
100
0.01
0.1
80
60
Validity %
40
20
10010
30
40
50
60
70
80
90
100
Uniqueness
Figure 15: Validity function of temperature applied to the focus (above) and position (below) distribution logits.
26

---

Published as a conference paper at ICLR 2024
Radial Distance Cutoff
Percentage of Edges Preserved
2,0 A
26%
3.0 A
61% %
5.0 A
R 93%
10.0 A
100%
Table 6: Percentage of edges preserved by the radial distance cutoff for molecular graphs from QM9.
metric as (and hence, the number of edges) reduces, despite the use of many peessag--psssingg layers in the underlying graph neural network. Even for graph still remain, the quality of the EDM-generated molecules drops significantly. On observing the trajectories of the individual atoms, we noticed that the system tends to explode for lower values of c. Note that Symphony uses radial cutoff of 5 A well.
A, where 93% of the edges in the
Effect of Radial Cutoff on EDM Molecule Quality
1.0
0.8
Atom Stability
Radial Cutoff
0.6
2.0
3.0
0.4
5.0
10.0
0.2
0.0
0
1
2
3
4
Training Step
1e5
Figure 16: Atom stability on QM9 as measured by Hoogeboom et al. 2022 cutoff c, showing that the model is quite sensitive.
function of radial
GENERATED MOLECULES FROM SYMPHONY
Figure 17 exhibits random non-cherry-picked samples from Symphony.
27

---

Published as a conference paper at ICLR 2024
Figure 17: Molecules generated by Symphony and visualized with PyMOL (Schrödinger, LLC 2015).
28


Reviews:
[{"review_id": "nzhpTvAovJ", "reviewer": "Reviewer_UhEf", "summary": "The authors propose to generate molecules autoregressively, invariant to Euclidean transformations. First, the authors transform the dataset of molecules into a dataset of sequences, adding one atom per time step. This is then turned into a autoregressive generative problem. Iteratively: a \"focus\" atom in the past is sampled, then the atomic charge of the next atom, then the position of the next atom relative to the focus atom. By having the focus selection and atomic charge distributions invariant to Euclidean transformations, and the position distribution equivariant, the resulting distribution is invariant to Euclidean transformations. The procedure is made permutation invariant by making the target sequences choose the nearest neighbour next, starting from a random atom, and breaking ties randomly.\n\nThe network uses higher-order representations of the rotation group to allow for more precision in the position distribution, which is an energy based model / harmonic exponential family on the spherical manifold. In the experiments, the method outperforms other autoregressive methods, while being outperformed by all-on-one diffusion methods.", "questions": "- It appears to me that the position distribution is an instance of a Harmonic Exponential Family [1] on a manifold, which probably should be cited.\n- Can the authors clarify why Eq (4) is smooth wrt the radius, but has a Dirac delta in the direction component? This should mean that the KL divergence is infinite almost always? Or is the smoothening handled implicitly by a band-limited Fourier transform? If so, it's not actually a Dirac delta that's used. The authors should clarify that.\n- As high frequencies may be necessary to precisely predict positions, it'd be great if the authors can show in an ablation study that $l_{max}=5$ suffices.\n- The authors write about $E(3)$ symmetry, which includes mirroring, but subsequently only talk about rotations and translations. Did the authors mean the group $SE(3)$, excluding mirroring?\n- Can the authors comment on the runtime of autoregressive generation vs diffusion methods in terms of the number of atoms? Might it be that autoregressive methods are only faster for small molecules?\n\n\nRefs:\n- [1] Cohen, Taco S., and Max Welling. 2015. “Harmonic Exponential Families on Manifolds.” arXiv [stat.ML]. arXiv. http://arxiv.org/abs/1505.04413.\n\n-------\nMy concerns have been convincingly addressed and I have increased my score.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The code is provided and is readable.\n- Autoregressive generation of molecules can be a fast alternative to e.g. diffusion methods.\n- The method outperforms other autoregressive methods.\n- The translational symmetry is elegantly handled via the focus atom. Also the rotational equivariance relative to the focus atom is sensible, compared to rotations around a center of mass, as is commonly done.", "weaknesses": "- in effect, the authors define an energy based model for the positions. It appears from the code that training and sampling of positions is done by discretizing $f^{position}$ on a grid. This seems like a shortcoming of the method, as I suspect that precise positioning is important in molecular generation. The paper should be transparent about this discretization. What type and resolution of grid is used? It'd be interesting to see how the resolution affects the sample quality. If the quality is highest with a very fine grid, it'd be great if the authors could consider and evaluate alternative learning and sampling strategies for energy based models (contrastive divergence / Hamiltonian Monte Carlo / Langevin dynamics etc).\n- While the approach predicting the sequence index for the focus point is in/equivariant to permutations of the resulting molecule, it still appears to me that an opportunity for additional permutation equivariance is missing. Why didn't the authors use a permutation equivariant network on $S^n$ to select one focus atom?\n- The method performs worse than diffusion methods and don't convince that autoregressive generation is the appropriate method for molecular generation.\n\nWhen the authors address the gridding issue convincingly, I will raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SDZi8cbuNi", "reviewer": "Reviewer_XBL9", "summary": "This submission provides an E(3)-equivariant autoregressive generative model for 3D molecule generation. The use of multiple channels of spherical harmonic projections is novel. The results show the performance is comparable to THE diffusion-based model EDM, but slightly worse.", "questions": "1. Why did you choose 64 uniformly spaced values from 0.9A to 2.1A? I think the model can learn the distribution automatically by equation (3).\n\n2. Since there are 64 spaced values for $r$, I am wondering how you solve them during the sampling process.\n\n3. Can you give some explanation as to why the EDM outperforms the proposed framework for many metrics?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The whole framework for autoregressive generation is reasonable: running message-passing on the current molecular fragments (current state), selecting the focus node, predicting atom species, and finally predicting the position of the added atom. This framework is similar to many previous autoregressive models.\n\n2. The main contribution is the definition of the distribution for the position of the added atom. The representation for $r$ by spherical coordinates and the use of spherical harmonic functions is novel and reasonable for the target task.\n\n3. Using multiple channels of spherical harmonic projections is novel and the authors provide valid reasons for using it. However, it lacks an ablation study to show how the multiple channels influence the final performance. For example, if the channel is 3 or 4.", "weaknesses": "1. I think the Property (3) in page 4 is very important. And many previous autoregressive models can not solve it well. Actually I think it is not difficult to define an E(3)-equivariant model currently as many previous works have solved it well. The diffusion model doesn't face the permutation-invariant problem. But it can be serious for the autoregressive model. The main paper claims the three properties have been proved in Theorem B.1. Unfortunately, I found the authors only proved the first two properties and ignored the property (3). I DON'T believe the current framework can guarantee the permutation-invariant even the embedder is E(3)-equivariant.\n\n2. I think many criticisms of the diffusion model such as EDM on page 6 are unfair. \n\na. \"Unlike autoregressive models, diffusion models do not flexibly allow for completion of molecular fragments\". Actually, it is not difficult for diffusion models to do completion sampling by replacement guidance without any extra training. Currently, the diffusion model works well on image completion and I also try it on EDM, it works too.\n\nb. \"To avoid recomputation of the neighbor lists during diffusion, current diffusion models use fully-connected graphs where all atoms interact with each other. This could potentially affect their scalability when building larger molecules.\" I believe it is even more challenging for the autoregressive model on the large molecules as the sequence could be too long. And the molecules in QM9 used in the experiment are very small. There is no evidence that the proposed framework performs better on large molecules compared with EDM.\n\nc. \"diffusion models are significantly slower to sample from, because the underlying neural network is invoked ≈ 1000 times when sampling a single molecule.\" It is true that for EDM it needs 1000 times denoising process. But actually, it runs very fast and supports batch sampling. I tested EDM on a single GPU A100, it takes 20 seconds to sample 50 molecules (a batch). I think the submission should provide their own sampling time to support this claim. \n\n3. Though the definition of the position distribution is very clear and reasonable, the current submission doesn't provide any details on how to sample from it after training. People can know $f^{position}$ for any given position, but it is not easy to sample from an energy function. I doubt this sampling process can be very time-consuming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mxa7hTv9y5", "reviewer": "Reviewer_cp6D", "summary": "The authors present Symphony, an E(3)-equivariant autoregressive generative model, that leverages a unique parametrization of 3D probability densities with spherical harmonic projections to make predictions based on features from a single focus atom. This approach captures both radial and angular distributions of potential atomic positions. Additionally, the model incorporates a novel metric based on the bispectrum to assess the angular accuracy of generated local environments. The authors claim that the model demonstrates superior performance on the QM9 dataset compared to previous autoregressive models and is competitive with existing diffusion models across various metrics.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Symphony uses a novel parametrization of 3D probability densities with spherical harmonic projections, allowing for predictions based on features from a single focus atom (novelty). This method overcomes the limitations of traditional approaches, such as G-SchNet and G-SphereNet, which require at least three atoms to precisely determine the position of the next atom, eliminating uncertainties due to symmetry.\n\n2. The NN for probability distribution over positions satisfies normalization and non-negativity constraints by applying softmax functions.\n\n3. A novel metric based on the bispectrum is introduced to assess the angular accuracy of matching generated local environments to similar environments in training sets.\n\n4. Symphony is able to generate valid molecules with a high success rate, even when conditioned on unseen molecular fragments.", "weaknesses": "1. Complexity: Spherical harmonic projections and inverse projections involve complex mathematical operations, potentially increasing the computational complexity of the model. There is no comparison regarding the time spent by the network among different autoregressive generation models, such as G-SchNet and G-SphereNet.\n\n2. Approximation Error: Spherical harmonic projections are an approximation method and may introduce errors, especially when a limited number of spherical harmonics are used.\n\n3. Dependence on Focus Atom: This method relies on the selection of an appropriate focus atom; if the focus atom is poorly chosen, it may affect the accuracy of predictions. (not specific for the paper).\n\n4. Challenges in Handling Symmetry: When dealing with the symmetry of molecules, ensuring that the predicted distributions are symmetrically equivariant(although proved) may increase the complexity of the model.\n\n5. The experiments are comprehensive, but the improvement is marginal because\n\n(1) In Table 1, the EDM performs best;\n\n(2) In Table 3, the MMD of Bond lengths from Symphony is worse than from EDM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CH5YCwswGn", "reviewer": "Reviewer_msmy", "summary": "This manuscript introduces Symphony, an autoregressive technique designed for the generation of 3D molecular structures via spherical harmonic projections. Unlike prevailing autoregressive models like G-SchNet and G-SphereNet, which employ rotationally invariant features of degree 1, Symphony leverages higher-degree E(3) equivariant features. The proposed model demonstrates superior performance in generating small molecules from the QM9 dataset and offers competitive results when compared to the E(3)-equivariant diffusion model.", "questions": "Please refer to the previous section", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well written and well organized.\n2. Distinct from other autoregressive equivariant generative models, the proposed approach employs higher-degree E(3) equivariant features. This enhances the flexibility in representing probability distributions.\n3. In terms of performance, the proposed method is on par with existing diffusion models. However, it boasts a significant advantage in computational efficiency, as current diffusion models rely on fully-connected graphs that may pose scalability challenges.", "weaknesses": "1. The manuscript employs a coarse discretization of the radial component for predicting the relative positions of subsequent atoms, a limitation acknowledged in the conclusion section. It would be intriguing to explore the effects of employing a finer discretization. Specifically, would such a refinement significantly compromise the computational efficiency of the method? Furthermore, could this potentially improve the distribution of bond lengths?\n2. n Equation (1), would it be more precise to describe this as a conditional distribution of $f$ given $S$? does the embedder at $f_{n+1}$ use both $h^{\\text{position}}$ and $h^{\\text{focus}}$?\n3. Typo: In the final equation on page 4, the term should correctly be denoted as $p^{\\text{position}}$.\n4. On the first line of page 6, the mechanism for predicting the STOP condition remains unspecified. Could you elaborate on this aspect?\n5. Typo: The first equation in Section 2 misses a translation $T$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)