Score-based 3D molecule generation with neural fields
Authors: Matthieu Kirchmeyer
Pedro O. Pinheiro
Saeed Saremi
Venue: NeurIPS 2024 poster (2024)
Keywords: score-based generative model
neural fields
3D molecule
Primary area: machine_learning_for_healthcare

Abstract: We introduce a new representation for 3D molecules based on their continuous atomic density fields. Using this representation, we propose a new model based on walk-jump sampling for unconditional 3D molecule generation in the continuous space using neural fields. Our model, FuncMol, encodes molecular fields into latent codes using a conditional neural field, samples noisy codes from a Gaussian-smoothed distribution with Langevin MCMC (walk), denoises these samples in a single step (jump), and finally decodes them into molecular fields. FuncMol performs all-atom generation of 3D molecules without assumptions on the molecular structure and scales well with the size of molecules, unlike most approaches. Our method achieves competitive results on drug-like molecules and easily scales to macro-cyclic peptides, with at least one order of magnitude faster sampling. The code is available at https://github.com/prescient-design/funcmol.

Score-based 3D molecule generation with neural fields
Matthieu Kirchmeyer', Pedro O. Pinheiro', Saeed Saremi Prescient Design, Genentech
Abstract
We introduce new representation for 3D molecules based on their continuous atomic density fields. Using this representation, we propose new model based on walk-jump sampling [1] for unconditional 3D molecule generation in the continu- ous space using neural fields. Our model, FuncMol,, encodes molecular fields into latent codes using conditional neural field, samples noisy codes from a Gaussian- smoothed distribution with Langevin MCMC (walk), denoises these samples in a single step (jump), and finally decodes them into molecular fields. FuncMol per- forms all-atom generation of 3D molecules without assumptions on the molecular structure and scales well with the size of molecules, unlike most approaches. Our method achieves competitive results on drug-like molecules and easily scales to macro-cyclic peptides, with at least one order of magnitude faster sampling."
Introduction
Generative modeling of 3D molecular structures, if deployed successfully, can help many problems in material and life sciences. Recently, state-of-the-ar image and text generative models were adapted to 3D molecule generation, achieving some degree of success [2, 3]. However, unlike other domains where the data modality is defined by the representation itself (e.g., a digital image a tensor of pixels), there are multiple ways to represent molecule. Therefore, an important problem to consider when modeling 3D molecules is: what constitutes a good representation, for molecules?
Recent methods for 3D molecule generation usually represent molecules as point clouds of atoms [4] or discrete grids of atomic densities [5]. which we will refer as voxel grids. Point clouds are processed by graph neural networks (GNNs), usually based on equivariant architectures [6, 7J. GNNs are known to be less expressive than other architectures due to the message passing formalism [8. 9, 10] and often scale quadratically with the number of atoms. On the other hand, voxel grids are compatible with more expressive models (e-g., convnets and transformers) but computation and memory scales cubically with the volume occupied by the molecules. These limitations in expressivity and scalability hinder the scope of application of these models.
In this work, we propose a new representation for molecules that overcomes those limitations. Inspired by the 3D computer vision community [11]. we represent molecules as fields encoding atomic occupancy, i.c., continuous functions that map 3D coordinates to atomic densities. Arguably this representation more natural for molecules than for visual data: while visual data is obtained via discrete measurements, molecular fields are continuous by nature. We handle these fields as such, by parameterizing the molecular occupancy field with neural network, shared among all molecules. and modulation codes, specific to each molecule. Fields that are parametrized by neural networks are referred as neural fields, implicit neural representations (INR) or coordinate-based neural networks. The former models common molecular structures (e.g., bonds, angles, valencies, symmetries) while later encodes variations that make each molecule unique. Given modulation code, we decode the
Equal contribution. Correspondence to kirchmeyer..o kirchmeyorr.matthieeeee www.weeeuttttttttttomomoo com, oliveira sivveirr_pinhiio... pedroggene. com, saremi. saremi.saceddgene saeeddgene. com 'The code is available at https:
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

OoOoO0OO
f
④ P().
(ii) 4 Rd1g).
(ii) decode(
(a)
(b)
Figure I: (a) a conditional neural field encodes molecular field into low dimensional latent code 2. (b) using learned score function go. FuncMol performs sampling in latent space via Langevin MCMC. These codes are decoded back into molecules.
molecular field by predicting the occupancy of each atom at given 3D coordinates (see Figure (a)). This decodes the molecules into explicit representations (such as discrete grids at arbitrary resolution or .dd format file), useful for downstream tasks.
We perform generative modeling in the continuous function space simply by sampling new modulation codes. Our proposed approach, FuncMol, leverages modulation code denoiser sample molecules following the (score-based) walk-jump sampling (WJS) approach [1]. WJS enjoys many properties such as fast-mixing, simplicity for training and fast sampling speed. Sampling is composed of three steps: (i) (walk) sample noisy modulation codes with Langevin Markov chain Monte Carlo (MCMC), (ii) (jump) estimate the "clean" modulation codes, and (iii) (decode) convert the estimated codes into molecule. Figure 1(b) illustrates WJS chain, with samples generated by our model trained on a macrocyclic peptides dataset [12].
The neural molecular field representation has many advantages over prior representations: (i) it represents complex high nigh-dimensional data in relatively low-dimensional compact space, (ii) is scalable (w.rt. the number of points, size of molecules and resolution) and has low memory footprint, (iii) does not make any assumptions on molecular structure or geometry, (iv) can represent molecular structures at arbitrary resolutions and for free-form discretization, (v) it is compatible with expressive machine learning architectures, and (vi) it is domain-agnostic and can used for variety of molecular design problems that can be expressed over fields, e.g., atomic densities, surfaces, pharmacophores, molecular orbitals, electron densities etc.
In summary, our contributions are as follows. We introduce new way to represent molecular structures with neural fields. These representations are dinedmensional, compact, scalable and do not make any assumptions on the molecular structure. We then propose FuncMol, a score-based model for 3D molecule generation that leverages these representations. We show that FuncMol performs competitively against representative baselines on the drug-like molecules dataset GEOM- -rr0g [13]. based on wide set of standard and new metries that we introduce to better measure the generation quality. These results were achieved with one order magnitude faster sampling time.? 2 Finally, we illustrate FuncMol's 's ability to scale larger 3D molecules by training on CREMP [12], a recent macro-cyclic peptide dataset, to which our basclines currently unable to scale.
Related work
Neural fields. Neural fields, also referred to as implicit neural representations (INRs), are coordinate-based neural networks that map coordinates (c.g., pixels on an image or coordinates in 3D Euclidean space) to features (e.g., RGB values or atomic occupancies). The idea of representing data points implicitly as neural networks dates back to the work of [14]. Recently, these representa- tions have been successfully applied to model continuous signals, e.g., 2D images [15, 16, 17], 3D shapes [18, 19, 20, 21], 3D scenes [22, 23]. videos [24, 25], physics [26, 27]. due to their appealing properties. Recently, two concurrent seminal work lead to a fast progress of neural fields by overcom- ing the spectral bias of coordinate-based neural networks [28]. Sitzmann el al. [29] propose SIREN, a neural network that uses periodic activation functions, while Tancik el al. [30] considers posi-
Sampling time includes the "decoding" step convert the generated code into molecule.

---

tional encoding based on Fourier features. Built on top of those architectures, multiplicative filter networks (MFNs) [31] represent fields as a simple linear combination over an exponential number of basis functions (e.g. Fourier or Gabor basis). Due to their simplicity and strong performance, we use MFNs to model the atomic occupancy fields.
Generative models of fields. Generative models for neural fields were first applied in 3D computer vision problems. Mescheder et al. [19] learn the distribution of shape occupancy fields with VAEs [32], while [18, 33] achieves similar objectives using GANs [34]. Diffusion models [35] have also been applied to learn the distribution of neural fields [36, 37, 38. 39]. Some work [37. 40] parameterize the neural field with the vector of all the corresponding weights. However, when the signal is complex and the neural fields have large number of parameters (e.g.. in the order of millions), it preferable to parameterize the field with a latent code with much lower dimension [36, 41, 42, 43]. Dupont er al. [36] fit the whole dataset with a shared coordinate- based network and learn a latent modulation code for each field with gradient-based meta learning [44]. Similarly to them, we parameterize neural fields with latent modulation codes. However, instead of applying meta learning, we learn the latent codes through stochastic optimization, either following the "auto-encoding" [19] or the "auto-decoding" [20] framework.
3D molecule generation. Most 3D molecule generation approaches represent atoms as points (with coordinates and atom types) and molecules as a set of points. For example, [45, 46, 47] propose autoregressive approaches to sample atoms, while [48, 49] use normalizing flows [50]. Hoogeboom er al. [4] propose EDM, diffusion model [35] applied point cloud of atoms with E(3) equivariance [6]. Many follow-up works extend EDM [51, 52, 53]. For example, [54, 55, 56] improve its performance by leveraging extra information during training (such as molecular graph and formal charges). This contribution is orthogonal to ours and can potentially be incorporated into our generative model. Other approaches [57, 58] map atomic densities on discrete 3D regular grids and leverage computer vision techniques for generation. Recently, VoxMol [5] (and its latent version [59]), coore-based generative model based on walk-jump sampling [1], shows that voxel-based representations can achieve mate-of-the-an results on 3D drug-like molecule generation. However, these methods scale cubically with the volume occupied by molecules, which limits its scope of application. Neural fields the continuous generalization of discrete 3D grid representations: they achieve good performance on 3D molecule generation and are more efficient in terms of memory and computation.
Conditional molecule generation. Voxels and point-clouds have also been used for conditional 3D molecule generation, usually by building upon an unconditional model. The authors in [57] condition generation 3D pharmacophores features, [60, 61, 62, 63] generate ligands conditioned on protein pockets, [64] generate molecules conditioned on fragments and [65, 66] generate 3D conformations conditioned molecular graphs. We are aware of only one other work that uses field-based representation for molecules [67]. There are several differences between our works: they use different data representation, neural network architecture and noise model. While they consider the problem of generating molecule conformations given a molecular graph, we handle the more general problem of unconditional 3D molecule generation (without access to molecular graph). Our model can easily be adapted to conformer generation by conditioning the generative model to the molecular graph. Moreover, approach can also be conditioned to tasks where we do not have access to molecular graphs, such as structure-base based drug design or electron density generation.
3 Neural atomic occupancy fields
We now describe how we represent molecules as continuous occupancy fields, how we approximate them with neural fields and how decode the neural fields to retrieve molecular conformations. We finish the section by providing some useful properties of our neural field representations.
3.1 Molecules continuous occupancy fields
We represent atoms as continuous Gaussian-like shapes in 3D space, centered around their atomic coordinates. Molecules are defined as fields mapping every point in the 3D space to the atomic densities of each atom type, v 3 - R², where the number of atom types the dataset D. We follow previous work [68, 69, 70], and compute the occupancy field v, for each atom type by
3

---

000000000
HDg
H
FC
A
f(x, 2)
hd-1))
FC
FILM
h
(a)
(b)
Conditional neural field f using the multiplicative filter network architecture. (a) A latent code 2 and some coordinates are given as input the model that outputs the occupancy field that location for the corresponding molecule, (a, 2). (b) The code and coordinates are processed via FiLM layers and Hadamard products. We denote the overall operation at layer as H(!)
Figure 2:
integrating the occupancy generated by all atoms of this type as follows:
Ve R Ua (z) = II exp (-(11= Taz ? )²)
(I)
.93r
where a; is the 7th atom of type a, for total of nh atoms. We set the atoms' radius to be r = .5A for all atom types. Molecular fields are smooth functions taking values between 0 (far away from all atoms) and (at the center of atoms).
3.2 Molecular neural fields
Each molecule the dataset is mapped to a modulation code 2 E IRd and we parameterize the molecular occupancy with conditional neural field fo 3 x RR RR Our objective is to learn the parameters & and the modulation code z such that for any molecular field v and coordinate R (2r 2) v(z). We approximate the molecular fields with linear combination of an exponential large number parameterized basis functions, such that amplitudes are modulated by the individual codes This parametrization is achieved by modeling the neural field with multiplicative filter networks (MFN) [31], a type of coordinate-bar assumptions the basis functions. We introduce the parameters associated with these functions in Equation (4).
network that provides an elegant way to
perform this linear combination under
Our conditional MFN is network with L multiplicative blocks, as illustrated on Figure 2(a). We implement conditioning of its parameters with FiLM [71]. Each multiplicative block is composed of fully-connected layer, FiLM modulation layer and elementwise product with basis, as illustrated on Figure 2(b). The neural field can be expressed by the following recursive expression:
A 10 (x) swi (x),
(x)= = (7a-1) @ (0 + (bu-1) ( +(- 2 (w)) LE (1, 1),
fo(a,2) 2) 2 h (x)) = w (1- (x) 6(1--1)
where swm is spatial basis function parameterized by ( 3 denotes the Hadamard product and 8(-11) = w 1),, and 7 (1-1) w: (((-1)
are the bias and scale modulation terms. We propose two approaches to lear the neural field's parameters (w/l), ((0) (1) will w(l)) and the codes (one per each molecule in D). Auto-decoding. In this setting, introduced by [20], we initialize each code randomly and directly learn them (together with the parameters of the neural field) with backpropagation. This is achieved by solving the following optimization problem:
arg min Σ (15o(1, v(x) da,
c JoED RED
where the integral is approximated by sampling finite sets of points X' C R3, While auto-decoding was usually applied in settings with relatively few samples, we were able scale the training to large datasets of one million samples (see Appendix B). See Algorithm I on Appendix B for more details.

---

This approach, introduced by [19] and illustrated in Appendix B Figure 4, generates the modulation code via an encoder Sy. parameterized by & and decodes the neural field back. In this work, Sy is (trainable) 3D convolutional network encoder that takes (low-resolution) voxel grids G as inputs. This approach flexible and compatible with other encoder architectures and molecule representations (e.g., GNN/point clouds). The parameters of the encoder and the neural field are learned with the following objective:
Auto-encoding.
arg min 0.10
Σ VED
/
[1 v(>) dr.
(3)
Once training done, we generate the code with the trained encoder. See Algorithm on Appendix B for more details. Instead of learning the codes individually, this approach learns an encoder, which allows to leverage data augmentation more efficiently. latent space. These benefits are reflected empirically in our experiments.
result helps learn a more structured
3.3 From codes to atomic coordinates
By leveraging the modulation codes and the neural field fe. we have access to the (learned) continuous occupancy field, fe(:, However, in many useful applications in chemistry and biology, are more interested in the 3D conformation of molecules. Next, describe how the molecular conformation from a learned (or generated, as we will see next) modulation code. This the decoding step outlined in the introduction.
extract
We start by identifying all atoms in the field, their approximate locations and their type. To this end. render discretized voxel grid from the molecular field using uniform discretization of space and the neural field fo (). We then apply a peak finding algorithm to infer the number atoms in the molecule each channel of the grid (each representing separate atom type) and their (discretized) coordinates. Finally, we introduce a new continuous refinement to find the local maximum of the neural field. For each identified atoms a, refine its coordinates around the neighborhood of the coordinates found with the peak detector x.
Ca arg Folx z)]a: 21
where (a, 2) denotes the field restricted to the channel corresponding to the atom type. This continuous refinement finds atomic coordinates that lie beyond the initial coarse uniform discretization. In practice, we batch the refinement process across molecules and use L-BFGS. We demonstrate in Appendix E.2 its efficiency compared to prior non-continuous refinement approaches from [72, 5].
3.4 Molecular neural fields properties
The proposed conditional neural field enjoys many properties that make it natural choice for handling large 3D molecules represented as continuous fields.
Flexibility w.r.f. basis. Conditioning MFNs gives the flexibility to choose any type of spatial basis that satisfies a multiplicative-sum property (see the definition in [31]). In our preliminary experiments, we observed that setting the spatial basis to Gabor filters performed better than Fourier filters as they account for the sparse nature of occupancy fields. For each layer I, we consider the following Gabor parameterization, also used in [27]:
Suto (z) = exp (- 6(
2 ||I- 1|=-~1011 11 (cos((2())) x). sin(
(4)
where (1) is the mean of the Gabor filter, ( is the scale, (2 is the frequency and (:--) refers the concatenation operator. Equation (4) combines both real and imaginary parts of the complex Gabor filter. This allows to remove phase parameters and reduce the overall parameter count of MFNs [31]. Other choices of basis are also possible and are left for future work.
Parameter efficiency. Our overall conditional MFN formulation is parameter efficient and shares parameters across molecules and channels (i.e. atom types). As [27], we excluded the basis functions parameters from FiLM to further decrease the parameter count.
5

---

Memory efficiency. Our conditional neural field can be trained on any free-form discretization of the input field. Occupancy values are computed on the fly. This allowed to train FuncMol with large batch size even on large 3D molecules. We found that training the neural field by up-sampling points close to the atoms' center improved training time as further detailed Appendix B. Alternative approaches like VoxMol [5] cannot be trained efficiently in this setting: for reference, on the macro- cyclic peptide generation task of Section 5.4, on 4 A100 GPUs VoxMol's training cost per epoch was 10 hours while our neural field's training cost was less than 12 minutes.
Reconstruction quality and robustness noise. Finally our neural field reconstructs accurately the input data as demonstrated in Appendix E. Moreover, operating on these latent codes makes our model extremely robust noise in code space. We demonstrate this property in Appendix E.3 by reporting the sampling metrics when perturbing the codes by a Gaussian noise.
Sampling efficiency. We use the latent codes for generative modeling explained in Section 4. Most sampling operations are done on a small dimensional latent space, while decoding into a full molecular field is done only after sampling. As show in Section our approach (which involves sampling latent code followed decoding them into molecules) achieves at least one order magnitude faster molecule sampling time than previous methods.
Score-based generative modeling
We use our latent modulation representations for a downstream generative modeling task. Section 4. describes the neural empirical Bayes (NEB) formalism used in our method and Section 4.2 explains how we perform sampling.
4.1 Neural empirical Bayes
Let p(z) be the distribution of codes and p(v) ()) be the (unknown) distribution of molecular fields, defined more formally as the pushforward of p(z) via the mapping ~>> (-) NEB estimates the score function of smoothed density of the codes p(y), (y R V p(y) Indeed sampling from a smoothed density p(y) benefits from faster mixing than the original density p(=) [1, 73, 74). This smoothed distribution is defined by transforming the random variable Z with an additive isotropic Gaussian noise with known noise level o, Y = Z + N, where N (O, l2). The noise level plays a key role, trading-off simplicity of the denoising objective and the sampling quality.
NEB is based on empirical Bayes view of (denoising) score- -aaee models that relates the estimator of clean data (denoiser) and the score function of the smoothed density at a fixed noise level [75, 76, 1]. The denoiser is taken to be the ie. B(y) = E| Y = y). Under Gaussian noise, denoiser and smoothed score function are related by the Tweedie-Miyyne
estimator of Z given Y y which is the Bayes estimator,
formula:
E(y) y 2 pyy)
The denoiser parameterized by neural network and learned by minimizing the following objective:
[())= Egs-p(s).e || 2o(= +)
(6)
The score function is recovered from learned denoiser via Equation (5) and is used for sampling smoothed codes (see Section 4.2). In practice, we optimize the empirical loss based on the latent codes inferred from set of molecular fields D. See pseudo-code in Appendix B, Algorithm 3.
4.2 Walk-jump sampling
We use the score function go to sample codes using the walk-jump sampling (WJS) scheme [1, 77, 73, 78]. This approach samples molecules from p(z) using the learned score function of noisy data instead of clean data. It consists of two main steps: walking and jumping as detailed in Appendix B, Algorithm 4. Figure 1(b) illustrates these two main steps in a WIS chain: walking consists in generating noisy codes while jumping consists in generating clean codes 2.
(initializaaiin) To improve mixing, as [77], we initialize the chains by adding uniform noise to Gaussian noise (with the same used when training the denoiser). In practice sample the uniform

---

2, DEEE 21), where D; is the training dataset of codes, arriving at yo + e, (0,
noise over the range of code values, jt ~ - www.m//mmmmmmmpp.mmmmm
(walk) Noisy codes are sampled from p(y) with Langevin MCMC algorithms that discretize the underdamped Langevin diffusion [79] starting from yo and uo 0:
du, = -y + go (ye)dt + 2 dB.,
dy. = u,d
where B; is the standard Brownian motion in IR and y is the frriction (the "mass" is set to 1). We discretize this SDE using the ABOBA scheme from Sachs et al. [80]. given a discretization step and fixed number of walk steps K. We analyze the impact of K in Appendix E.4.
(jump) At given time step K, clean samples are estimated by denoising the smooth code, i.e., ZK = (y())). These codes are then used to obtain the atomic coordinates as detailed in Section 3.3.
5
Experiments
We now evaluate model for unconditional generation. We start with description of our exper- imental setup (Section 5.1), then present our results on two popular small molecule datasets (Sec- tions 5.2 and 5.3) and recent macro-cyclic peptide dataset (Section 5.4).
5.1
Experimental setup
Datasets. We evaluate FuncMol on three datasets: QM9 [81], GEOM-drugs [82] and CREMP [12]. QM9 contains an enumeration of all possible molecules up to 9 heavy atoms (29 including hydrogens) satisfying some constraints [83]. GEOM-drugs contains multiple conformations for 430K drug-sized molecules (computed with semi-empirical density functional theory), with an average of 44 heavy atoms per molecule. CREMP is recent dataset that contains multiple conformations of macrocyclic peptides 4-6 residue long, with an average of 74 heavy atoms per molecule. We model hydrogen explicitly and consider 5 chemical elements for QM9 (C. H, O, N, F), 6 for CREMP (C. H, O, N, F, S) and 8 for GEOM-drugs (C, H. O, N. F, S. CI and Br), ignoring the P. and B elements that occur extremely rarely. We GEOM-drus and 409K/10K/ 10K/9K on CREMP for train, validation and test, respectively. We use the same pre-processing and splits in [54] for QM9 and GEOM-drugs and in [84] for CREMP.
split of 100K/20K/13K molecules for QM9, 1.IM/1 (46K/ on
Implementation details. Our main model, FuncMol, follows the auto-encoding approach described in Section 3.2. The codes are computed with an encoder that takes as input low-resolution voxelized representation of the molecular field with grid dimension of 16x 16x 16. The encoder is a 3D CNN containing 4 residual blocks, where each block contains 3 convolutional layers followed by BatchNorm, ReLU and pooling (max pooling on the first three blocks and average pooling on the last one) layers. We consider modulation codes with dimension 1024 on QM9 and 2048 on GEOM-drugh and CREMP: We use the same neural field network for all datasets: a conditional MFN with Gabor filters and 6 --odullttdeedd layers, where each tully-connected layer has 2048 hidden units. We augment the training set by applying random rotations on the three Euler angles. The weights of the latent code encoder and neural field decoder are trained jointly.
We also show results for the auto-decoding based model, FuncMolaec- In this setting, we initialize the codes randomly and optimize them together with the neural field weights. This approach is less fit for performing large amounts of augmentation as solves a costly per-sample optimization problem; thus we did not apply data augmentation. As a consequence, we observed that this model is more prone to memorization than the auto-encoding approach (c.g., on GEOM-drugs, around 33% of the generated molecules are copies from the training set).
We normalize the codes to have zero mean and unit variance. We choose a noise level in normalized space of a 1.2 for GEOM -rugs and CREMP, version of the denoiser used in [36]: a fully-connecto 2.0 for QM9. Our code denoiser is modified network with 18 residual blocks (each with two linear layers with 6144 hidden units) and skip connections. We remove the bias of all layers and use ReLU activations as in [85]. To limit memorization in FuncMolaces we add dropout (ratio 0.3) between the fully-connected layers in each residual block, For QM9, we consider smaller network (6 residual blocks and 4096 hidden units). We initialize the MCMC chains with noise and use the following sampling hyperparameters y 1.0 and 8 a/2 in [5, 78]. For evaluation purposes,
7

---

generate one sample per chain. We consider 1000 steps for QM9 and GEOM-druge -drugs and 10000 steps for CREMP. See Appendix B for more details on the implementation.
Baselines. We compare FuncMol and FuncMoldre to three state-of-the-a arrt approaches. EDM [4] and GeoLDM [53] are diffusion models operating on point clouds (the latter is latent-space extension of the former). VoxMo! [5] is a voxel-based generative model that uses neural empirical Bayes, similar to our generative approach. All of the methods generate molecules coordinates. EDM and GcoLDM apply diffusion directly to point clouds, while Vox and FuncMol rely on an additional (cheap) post-processing step to extract atomic coordinates from voxel grids or modulation codes, respectively. We follow previous work [58, 54, 5, 62, 86], and use standard cheminformaties software (OpenBabel [87]) determine the molecule's atomic bonds given their atomic coordinates. The same post-procsssing is applied to all models for faimess of comparison.
set of atom types and their
Metries. We consider several metrics used in previous work [5] to benchmark unconditional molecule generation for the standard QM9 and GEOM-drugs datasets (for the CREMP metrics, see Section 5.4): stable mol and stable atom, the percentage of stable molecules and atoms (as defined in [4]); validity, the percentage of generated molecules that passes RDKit [88]'s sanitization filter; uniqueness, the proportion of valid molecules that have different canonical SMILES; valency Wi, the Wasserstein distance between the distribution of valencies in the generated and test set; atoms TV and bonds TV, the total variation between the distribution of atom types and bond types; bond length W1 and bond angle W1. the Wasserstein distance between the distribution of bond and lengths. We also report the average sampling time per molecule. In the case of our method, this time includes the MCMC "walk" steps, the denoising "jump", , the rendering, peak detection and bond inference.
To further investigate the quality of molecular conformations and other molecular properties on GEOM-drugs, we consider some additional metrics. These include: single fragment, the percentage of molecules that contains only a single fragment; median strain energy [89], the difference between the internal energy of the generated molecule's pose and a relaxed pose of the molecule using RDKit's Universal Force Field [90], computed over all molecules; ring size TV, the total variation between the empirical distribution of ring sizes (i.e. number of heavy atoms in rings) in generated and test sets: number of atoms/mol TV, the total variation between the empirical distribution of number of atoms per molecule in generated and test sets (in the case of molecules with multiple fragments, we consider only the largest fragment); QED, SA and logp, measure the frug-likeness score [91], the synthesizability score [92] and the lipophilic efficiency, respectively (computed with RDKit).
Ablations. In Appendix E we report series of ablation studies for the neural field and the generative model. Appendix E. measures the reconstruction quality of the training molecules. Appendix E.2 illustrates the improvements due continuous atomic coordinate refinement. Appendix E.3 shows that our field-based decoder is robust to noise, making it an ideal choice for generative modeling. Appendix E.4 ablates the impact of the number of walk steps in the WIS scheme of Section 4.2. Finally, Appendix E.5 ablates the impact of the chosen resolution when sampling codes and decoding them back to molecules. In practice, we observe that 0.25A provides a good trade-off between the sampling time and the quality of the generated molecules.
5.2 Results on QM9
As pointed by previous authors [93, 4], this dataset is not fully suited for unconditional generative models: model that captures the training distribution will have to generate samples from training set, due to the enumeration. However, many previous work report results on this dataset. Therefore, we also show results for completeness.
Table I report the metrics described in Section 5. We see that FuncMol slightly improves VoxMol and both models perform worse compared the equivariant point-cloud based baselines. We note that sampling time of FuncMol is an order of magnitude better than baselines.
5.3 Results on GEOM-drugs
Table 2 reports the same set of metrics as the previous dataset. FuncMol performs favorably over point cloud diffusion models and is close to VoxMol's performance. In particular, FuncMol and VoxMol generate molecules that are significantly more stable and better capture the distribution of bond angles. Table 3 shows results on additional metrics (described in Section 5.1). We also include
8

---

Table 1: QM9 results w.r.t test for 10000 samples per model. t/4 indicate that higher/lower numbers are better. The row data are randomly sampled molecules from the validation set. We report I-sigma error bars over sampling runs.
stable
stable
valid
unique
valency
atom
bond
bond
bond
time s/mol.J
mol %,
atom%-
%6
%6
W1
TV_
TV4
len Wi
Wi
data
98.7
99.8
98.9
99.9
.001
.003
.000
.000
.120
EDM
97.9
99.8
99.0
98.5
.011
.011
.002
.001
.440
0.54
GcoLDM
97.5
99.9
100.
98.0
.005
.017
.003
.007
.435
0.65
VoxMol
89.3
99.2
98.7
92.1
.033
.029
.009
.003
1.96
0.83
FuncMolaee
88.6
99.2
100.
81.1
.022
.066
.032
.006
1.21
0.05
FuncMol
89.2
99.0
100.
92.8
.021
.012
.006
.005
1,56
0.05
(9:4)
(4:37)
(80)
(=0.3)
(#9000)
(40000)
(4-0009)
(=00009)
(=0,06)
Table 2: GEOM-drugs results, standard metrics w.r.t test set for 10000 samples per model. 1/4 indicate that higher/lower numbers are better. The row data are randomly sampled molecules from the validation set. We report I-sigma error bars over sampling runs.
stable mol %6
stable atom%
valid
unique
valency
atom
bond TV;
bond len W4
bond
%1
%1
W11
TV₁
Wi
s/mol.J
data
99.9
99.9
99.8
100.0
.001
.001
.025
.000
0.05
EDM
40.3
97.8
87.8
99.9
.285
.212
.048
.002
6.42
9.35
GeoLDM
57.9
98.7
100.
100.
.197
.099
.024
.009
2.96
8.96
MoxMol
75.0
93.4
99.6
.254
.033
.024
0.64
7.55
FuncMolaee
69.7
95.3
100.
77.5
.268
.035
.028
.003
2.13
FuncMol
69.7
03:50
98.8
[5:11
(3:00
100.
(1:61
04:000
(4:0013
.1099
05,00
.052
(=:000
.003
(4:01
0.29
(2:7))
(1.9)
(d.1))
(1.000)
(3 000)
(200000)
122 0000
(1. (2.00)
the following plots of Appendix F: Figure 9 shows the cumulative distribution function of strain energies for generated molecules and Figures 10 and show the histograms of the other metrics.
The results are clear: FuncMol samples better drug-like molecules than point-cloud diffusion models. In fact, about half the molecules of point cloud methods have multiple fragments, they have an order of magnitude higher median strain energy. the distribution of ring sizes is off and the QED, SA and logp scores are lower. The results of FuncMol are close VoxMol in most but all metrics. However, our approach is much more scalable and efficient: FuncMol generates molecules an order of magnitude faster than previous methods (see the last column Table 2). Appendix H shows some molecules generated by FuncMol on GEOM-drugs.
Table 3: GEOM-drugs results, additional metrics w.r.t. test set for 10000 samples per model. 1/4 indicate that higher/lower numbers are better. The row data are randomly sampled molecules from the validation set. We report 1-sigma error bars over 3 sampling runs.
single frag %4
median
ring TV1
atms/mol
QED
SA
logp f
energy
TV1
data
100.
54.5
.011
.000
.658
.832
2.95
EDM
42.2
951.3
.976
.604
.472
.514
1.11
Geol
51.6
461.5
.644
.469
.497
.593
1.05
VoxMol
82.6
69.2
.264
.636
.659
.762
2.73
FuncMolace
80.2
96.4
.324
970
.67
.88
2.87
$1.02
1
(6.000
1 0039
(4:000
FuncMol
(#-02)

---

Data
10
Data
Data
Data
8
FuncMol -.094)
6
FuncMol (KL-0.1219)
12
FuncMol (KL-)
5
8
6
2
4
1
-2
2
1.6
1.8
2.0
2.2
2.4
1.6
1.8
2.0
2.2
2.4
1.6
1.8
2.0
2.2
2.4
8
8g
1.6
1.4
Data
1.2
1.0
Data
1.75
Data
FuncMol
110
1.2
FuncMal (KL-0.1997)
0.8
FuncMol (KL=0.1218)
1.25
FuncMol (KL=0.1244)
0.6
0.8
0.4
0.4
0.6
0.50
0.75
0
0.2
0.2
0.25
-3
-2
-1
0
1
2
0.0
0.00
3
-2
-1
1
2
Figure 3: Qualitative evaluation on CREMP following [84]. Left: Comparison of the bond angles (01,
02. 0g) in each residue and dihedral distributions (ó, e), w) for each residue from the reference test set (gray) and the generated samples (blue). KL divergence is calculated as KL || sampled). Right: Ramachandran plots [94] (colored by density where darker tones represent high density regions).
5.4 Results on CREMP
To showcase the scalability of FuncMol, we train dataset of larger molecules. We choose the macrocylic peptides of CREMP, that contains on average 74 atoms, making it challenging to train models using point-clouds. These molecules also pose serious limitations to voxel-based approaches as they require modeling volume of 24 cubic Angstroms. We tried to train VoxMol on this dataset using the official implementation, but did not succeed: takes around 10 hours per epoch on 4 A100 GPUs, while FuncMol takes less than 12 minutes per neural field epoch and 15s per denoiser epoch. We use the same code dimension and neural field architecture in GEOM-drugs, therefore the computational training cost of FuncMol remains unchanged.
Figure 3 shows that FuncMol captures well the underlying distribution of macrocyclic conformations. We show the distribution of bond angles (0,, 02. 03) and dihedrals (s), 1h, w) of both molecules from test and generated molecules. We also show the KL--divergence between test and generated samples. Approximately 65% of the generated molecules were valid peptides (that is, we could extract a sequence of amino acids from the SMILES strings). The Ramachandran plots [94] show that FuncMol recovers the main modes of the distribution. We note that the bond angles and dihedrals distributions are learned without having any explicit priors on the structure of these peptides. Appendix H shows some generated macrocylic peptides. Finally, our model takes around 1.5s to generate molecule. For reference, should VoxMol be trained successfully, it would take over a minute to sample single molecule (assuming similar sampling parameters as in other datasets). This a substantial speedup that showcases the potential of FuncMol scale to even larger molecules.
Discussion
We introduce new continuous representation of 3D molecules based on their atomic occupancy field and score-based generative model operating on this representation. Each molecule is assigned code that modulates a shared neural field. We demonstrate that we can build an all-atom generative model of 3D molecules, FuncMol, with state-of-the-art sampling time and competitive performance on challenging drug-like datasets. We believe that this model introduces a new paradigm for all- atom 3D modeling of molecules that has many useful properties, namely scalability, expressivity, and ficxibility, as can model various molecular design problems (involving structure, electron densities, etc.) with minor architecture changes. Future research directions include exploring different neural field architectures, adapt the model for conditional generation (e.g., structure conditioning) or model the molecular bonds alongside the atomic coordinates Moreover, the scalability of FuncMol can be a potential alternative for all-atom representations of large biomolecules.
'Recent work [54, 95] show that this improves generation quality. See Appendix G to see how our method compares with representative baseline that uses the bond information.
10

---

We would like to thank the Prescient Design team for helpful discussions and Genentech's HPC team for providing a reliable environment to train and analyze models.
Acknowledgements
References
Saeed Saremi and Aapo Hyvärinen. Neural empirical bayes. JMLR, 2019. (cit. on pp. 1, and 6)
[2] Camille Bilodeau, Wengong Jin, Tommi Jaakkola, Regina Barzilay, and Klavs F Jensen. Generative models for molecular discovery: Recent advances and challenges. Computational Molecular Science, 2022. (cit.
on p. 1)
[3] Benoit Baillif, Jason Cole, Patrick McCabe, and Andreas Bender. Deep generative models for 3d molecular structure. Current Opinion in Structural Biology, 2023. (cit. on p.
[4] Emiel Hoogehoom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In ICML, 2022. (cit. pp. 3, and 8)
[5] Pedro O. Pinheiro, Joshua Rackers, Joseph Kleinhenz, Michael Maser, Omar Mahmood, Andrew Martin Watkins, Stephen Ra, Vishnu Sresht, and Saeed Saremi. 3d molecule generation by denoising voxel grids. In NeurIPS, 2023. (cit. on pp. 5, 6, 7, 8, 18, and 21)
[6] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In ICML, 2021. (cit. on pp. and 3)
[7] Mario Geiger and Tess Smidt. c3nn: Euclidean neural networks. arXiv:2207.09455. 2022. (cit. p. 1)
[8] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? ICLR, 2019. (cit. p. 1)
[9] Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman neural: Higher-order graph neural networks. In AAAI, 2019. (cit. p. 1)
[10] Sergey N Pozdnyakov and Michele Ceriotti. Incompleteness of graph convolutional neural networks for points clouds in three dimensions. arXiv:2201.07136, 2022. (cit. 1)
[11] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum, 2022. (cit. on p. 1)
[12] Colin A Grambow, Hayley Weir, Christian N Cunningham, Tommaso Biancalani, and Kangway V Chuang. CREMP: rontoreer-rotamer ensembles of macrocyclic peptides for machine learning. arXiv:2305.0805: 2023, (cit. on pp. 2, and 18)
[13] Simon Axelrod and Rafacl property prediction and molecular generation. Scientific Data, 2022. (cit. on pp. 2 and 18)
Geom, energy-annotated molecular conformations for
[14] Kenneth O Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines, 2007. (cit. p.
[15] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, 2021. (cit. on p.
[16] Kai Wang, Zhaopan Xu. Yukun Zhou, Zelin Zang. Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion. arXiv:2402. 13144, 2024. (cit. on p. 2)
[17] Yilun Du Katie Collins, Josh Tenenbaum, and Vincent Sitzmann. Learning signal- agnostic manifolds of neural fields. NeurlPS, 2021. (cit. on p.
[18] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In CVPR, 2019. (cit. pp. and 3)
[19] Lars Mescheder, Michael Oechsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019. (cit. on pp. 2, 3, and 5)
[20] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In CVPR, 2019. (cit. on pp. 2, 3, and 4)

---

[21] Mateusz Michalkiewicz, Jhony K Pontes. Dominic Jack, Mahsa Baktashmotlagh,, and Anders Eriksson. Implicit surface representations layers in neural networks. In ICCV, 2019. (cit. on p. 2)
[22] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous -d-structure--wwre neural scene representations. NewrIPS, 2019. (cit. p. 2)
[23] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020. (cit. on p.
[24] Zhengqi Li. Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. In CVPR, 2021. (cit. on p.
[25] Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang. Videoinr: Learning video implicit neural representation for continuous space-time super-resolution. In ICCV, 2022. (cit. on p. 2)
[26] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informe neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential cquations. Journal of Computational Physics, 2019. (cit. on p. 2)
[27] Yuan Yin, Matthieu Kirchmeyer. Jean-Y Yves Franceschi, Alain Rakotomamonjy, and patrick gallinari. Continuous PDE dynamics forecasting with implicit neural representations. In ICLR, 2023. (cit. on pp. 2, 5, and 18)
[28] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In ICML, 2019. (cit. on p.
[29] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NewrIPS, 2020. (cit. p. 2)
[30] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurlPS, 2020. (cit. on p. 2)
[31] Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and Zico Kolter. Multiplicative filter networks. In ICLR, 2021. (cit. on pp. 3, 4, 5, and 18)
[32] Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICML, 2014. (cit. on p. 3)
[33] Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models distributions of functions. AISTATS, 2022. (cit. on pp. 3 and 17)
[34] Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014. (cit. on p. 3)
[35] Jascha using nonequilibrium thermodynamics. In ICML,, 2015. (cit. on p. 3)
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning
[36] Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is function and you can treat it like onc. In ICML, 2022. (cit. on pp. 3, 7, and 18)
[37] Ziya Erkoç. Fangchang Ma, Qi Shan, Matthias NieBner, and Angela Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. In ICCV, 2023. (cit. on 3)
[38] Gene Chou, Yuval Bahat, and Felix Heide. Diffusion-sdf: Conditional generative modeling signed distance functions. In ICCV, 2023. (cit. on p.
[39] Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Transactions on Graphics (TOG), 2023. (cit. on 3)
[40] Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In ICMI, 2023. (cit. on p. 3)
[41] Matthias Bauer, Emilien Dupont, Andy Brock, Dan Rosenbaum, Jonathan Richard Schwarz, and Hyunjik Kim. Spatial functa: Scaling functa to imagenct classification and generation. arXiv:2302.03) 2023. (cit. p. 3)
[42] Luca De Luigi, Adriano Cardace, Riccardo Spezialetti, Pierluigi Zama Ramirez, Samuele Salti, and Luigi di Stefano. Deep learning on implicit neural representations of shapes. In ICLR, 2023. (cit. on p. 3)
12

---

[43] Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. In NeurIPS, 2023. (cit. on p. 3)
[44] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learnnng for fast adaptation of deep networks. In ICML, 2017. (cit. on 3)
[45] Niklas WA Gebauer, Michacl Gastegger, and Kristof T Schütt. Generating equilibrium molecules with deep neural networks. arXiv: 18100.880..11 11347, 2018. (cit. on 3)
[46] Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapteed generation of 3d point sets for the targeted discovery of molecules. NewrIPS, 2019. (cit. on 3)
[47] Youzhi Luo and Shuiwang Ji. An autoregressive flow model for 3d molecular geometry generation from scratch. In ICLR, 2022. (cit. on p. 3)
[48] Jonas Köhler, Leon Klein, and Frank Noć. Equivariant flows: exact likelihood generative leamning for symmetric densities. In ICML, 2020. (cit. on p. 3)
[49] Victor Garcia Satorras, Emiel Hoogeboom, Fabian Fuchs, Ingmar Posner, and Max Welling. E(n) equivari- ant normalizing flows. NeurlPS, 2021. (cit. on 3)
[50] Danilo Rezende and Shakir Mohamed, Variational inference with normalizing flows. In ICMI,, 2015. 2015. (cit. on p. 3)
[51] Lei Huang, Hengtong Zhang, Tingyang Xu, and Ka- Ka-Chun Wong. Mdm: Molecular diffusion model for 3d molecule generation. In AAAI, 2023. (cit. on p. 3)
[52] Lemeng Wu, Chengyue Gong, Xingchao Liu, Mao Ye, and Qiang Liu. Diffusion- based molecule generation with informative prior bridges. NeurIPS, 2022. (cit. on p. 3)
[53] Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. ICMI., 2023. (cit. on pp. 3 and 8)
[54] Clement Vignac, Nagham Osman, Laura Toni, and Pascal Frossard. Midi: Mixed graph and 3d denoising diffusion for molecule generation. ECML, 2023, (cit. on pp. 3. 7, 8, 10, 18, and 24)
[55] Chenqing Hua, Sitao Luan, Minkai Xu, Zhitao Ying, Jie Fu, Stefano Ermon, and Doina Precup. Mudiff:: Unified diffusion for complete molecule generation. In Learning on Graphs Conference, 2024. (cit. on p.
[56] Xingang Peng, Jiaqi Guan, Qiang Liu, and Jianzhu Ma. Moldiff: Addressing the atom-hond inconsistency prohlem in 3d molecule diffusion generation. In ICMI,, 2023. (cit. on pp. and 24)
[57] Miha Skalic, José Jiménez, Davide Sabbadin, and Gianni De Fabritiis. Shape-based generative modeling for de novo drug design. Journal of chemical information and modeling, 2019. (cit. on p.
[58] Matthew Ragoza, Tomohide Masuda, and David Ryan Koes. Learning continuous representation of 3d molecular structures with deep generative models. NeurlPS, MLSB Workshop. 2020. (cit. pp. 3 and 8)
[59] Ewa Nowara, Pedro Pinheiro, Sai Mahajan, Omar Abul' atta, Andrew Watkins, Saeed Saremi, and Michael Maser. Nebula: Neural empirical bayes under latent representations for efficient and controllable design of molecular libraries. ICMI.. Worshop on A14Sciences, 2024. (cit. p.
[60] Matthew Ragoza, Tomohide Masuda, and David Ryan Koes. Generating 3D molecules conditional on receptor binding sites with deep generative models. Chemical science, 2022. (cit. p.
[61] Xingang Peng, Shitong Luo, Jiaqi Guan, Qi Xie, Jian Peng, and Jianzhu Ma. Pocket2mol: Efficient molecular sampling based 3D protein pockets. In ICML, 2022. (cit. on 3)
[62] Jiaqi Guan, Wesley Wei Qian. Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma. 3D equivariant diffusion for target-aware molecule generation and affinity prediction. ICI.R, 2023. (cit. on pp. 3 and 8)
[63] Pedro O Pinheiro, Arian Jamasb, Omar Mahmood, Vishnu Sresht, and Saced Saremi. Structure- based drug design by denoising voxel grids. JCML, 2024. (cit. on p.
[64] Ilia Igashov. Hannes Stärk, Clément Vignac, Arne Schneuing, Victor Garcia Satorras, Pascal Frossard, Max Welling, Michael Bronstein. and Bruno Correia. Equivariant 3d-conditional diffiusion model for molecular linker design. Nature Machine Intelligence, 2024. (cit. 3)
[65] Minkai Xu, Lantao Yu, Yang Song, Chence Shi. Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. In ICLR, 2022. (cit. on p.
13

---

[66] Bowen Jing. Gabriele Corso, Jeffrey Chang, Regina Barzilay. and Tommi Jaakkola. Torsional diffusion for molecular conformer generation. NeurlPS, 2022. (cit. on p.
[67] Yuyang Wang, Ahmed A. Elhag, Navdeep Jaitl, Joshua M. Susskind, and Miguel Angel Bautista. Gener- ating molecular conformer fields, 2023. (cit. p.
[68] Lin Li, Chuan Li, and Emil Alexov. On the modeling of polar component of solvation energy using smooth gaussian-based dielectric function. Journal of Theoretical and Computational Chemistry, 2014. (cit. p. 3)
[69] Michael J Willatt, Félix Musil, and Michele Ceriotti, Atom-density representations for machine learning. The Journal of chemical physics, 2019. (cit. on 3)
[70] Gabriele Orlando, Daniele Raimondi, Ramon Duran-Romañia Yves Moreau, Joost Schymkowizz, and Frederic Rousseau. Pyuul provides interface hetween biological structures and deep learning algorithms. Nature communications, 2022. (cit, on p. 3)
[71] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FILM: Visual reasoning with general conditioning layer. AAAI, 2018. (cit. on p. 4)
[72] Matthew Ragoza, Tomohide Masuda, and David Ryan Koes. Learning continuous representation of 3d molecular structures with deep generative models, In Neurips, Structural Biology workshop, 2020. (cit. on 5)
[73] Saeed Saremi, Rupesh Kumar Srivastava, and Francis Bach. Universal smoothed score functions for generative modeling. arXiv:2303. /6669, 2023. (cit. on p. 6)
[74] Saced Saremi, Ji Won Park, and Francis Bach, Chain of log-coneave Markov chains. ICLR, 2024, (cit. on p. 6)
[75] Herbert Robbins, An empirical bayes approach to statisties. In Breakthroughs in Statistics: Foundations and basic theory. 1992. (cit. on p. 6)
[76] Koichi Miyasawa. An empirical Bayes estimator of the mean of a normal population. Bulletin of the International Statistical Institute, 1961. (cit. on 6)
[77] Saeed Saremi and Rupesh Kumar Srivastava. Multimeasurement generative models. ICLR, 2022. (cit. on p. 6)
[78] Nathan C Frey, Dan Berenberg. Joseph Kleinhenz, Isidro Hotzel, Julien Lafrance- Vanasse, Ryan Lewis Kelly, Yan Wu, Arvind Rajpal, Stephen Ra, Richard Bonneau, Kyunghyun Cho, Andreas Loukas, Vladimir Gligorijevic, and Saeed Saremi. Protein discovery with discrete walk-jump sampling. In ICLR, 2024. (cit. on pp- and 7)
[79] Xiang Cheng, Niladri S. Chatterji, Peter L. Bartlett, and Michael I. Jordan. Underdamped Langevin MCMC: non-asymptotic analysis. In COLT, 2018. (cit. on p.
[80] Matthias Sachs, Benedict Leimkuhler, and Vincent Danos. Langevin dynamics with variable coefficients and nonconservative forces: from stationary states to numerical methods. Entropy, 2017. (cit. p. 7)
[81] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg. Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 2018. (cit. on pp. and 18)
[82] Simon Axelrod and Rafael Bombarell Gcom, onergy-annotated molecular conformations for property prediction and molecular generation. Scientific Data, 91):185, 2022. (cit. on p. 7)
[83] Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 2014. (cit. on p. 7)
[84] Colin Grambow, Hayley Weir, Nathaniel L. Diamant, Alex M. Tseng, Tommaso Biancalani, Gabriele Scalia, and Kangway V. Chuang. Ringer: Rapid conformer generation for macrocycles with sequence- conditioned internal coordinate diffusion. arXiv, 2023. (cit. on pp. 7 and 10)
[85] Sreyas Mohan, Zahra Kadkhodaie, Eero P. Simoncelli, and Carlos Fernaneez-Granda Robust and interpretable blind image denoising via bias-free convolutional neural networks. In ICLR, 2020. (cit. onp. 7)
[86] Ame Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blundell, Pietro Lió, Carla Gomes, Max Welling, et al. Structure-based based drug design with equivariant diffusion models. preprint 221.1:2210..100000 2022. (cit. on p.
14

---

[87] Noel M O'Boyle. Michael Banck, Craig A James, Chris Morley, Tim Vandermeersch,, and Geoffrey R Hutchison. Open babel: An open chemical toolbox. Journal of cheminformatics, 2011. (cit. p. 8)
[88]
Greg Landrum. Rdkit: Open-source cheminformatics software, 2016. (cit. pp. and 18)
[89] Charles Harris, Kieran Didi, Arian R Jamasb, Chaitanya K Joshi, Simon V Mathis, Pietro Lio, and Tom Blundell. Benchmarking generated poses: How rational is structure-based drug design with generative models? arXiv preprint arXiv:2308. 074413 2023. (cit. on p.
[90] Anthony K Rappé, Carla J Casewit, KS Colwell, William A Goddard II, and W Mason Skiff. Uff, a full periodic table force field for molecular mechanics and molecular dynamics simulations. Journal of the American chemical society, 1992. (cit. on p. 8)
[91] G Richard Bickerton, Gaia Paolini, Jérémy Besnard, Sorel Muresan, and Andrew I. Hopkins. Quantifying chemical beauty of drugs. Nature chemistry, 2012. (cit. on p. 8)
[92] Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of cheminformatics, 2009. (cit. on p. 8)
[93] Clement Vignac and Pascal Frossard. Top-n: Equivariant set and graph generation without exchangeability. ICLR, 2022. (cit. on p. 8)
[94] GN Ramachandran and V Sasisekharan. Conformation of polypeptides and proteins. Advances protein chemistry, 23:283-437, 1968. (cit. on p. 10)
[95] Xingang Peng, Jiaqi Guan, Qiang Liu, and Jianzhu Ma. MolDiff: Addressing the atom-bond inconsisteney problem molecule diffusion generation. Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 27611-27623 PMLR, 23-29 Jul 2023. (cit. on p. 10)
[96] Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015, (cit. p. 17)
[97] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. (cit. p. 18)
[98] David Schnal, Schastian Bittrich, Mandar Deshpande, Radka Svobodová, Karel Berka, Václav Bazgier, Sameer Velankar. Stephen K Burley, Jaroslav Koča. and Alexander S Rose. Mol* Viewer: modern web app for 3D visualization and analysis of large biomolecular structures. Nucleic Acids Research, 2021. (cit.
on p. 18)
15

---

Appendices
This supplementary material is organized follows:
1. Appendix A includes a broader impact statement.
2. Appendix B includes extra implementation details.
3. Appendix C shows some additional analysis of latent space including some downstream task evaluation.
4. Appendix D presents results of diffusion baseline QM9.
5. Appendix E provides some ablation studies for the model.
6. Appendix F shows additional quantitative results.
7. Appendix H shows additional qualitative results.
8. Appendix G provides some comparison to a bond-diffusion baseline.
9. Appendix includes the NeurlPS checklist.
A Broader Impact Statement
This work introduces some technical advancements in unconditional 3D molecule generation, an important component of molecular design and pharmaceutical research. A key advantage of our model is that it scales to larger molecules unlike existing models and has at least one order magnitude faster sampling time. Although extensive validation through wet-lab experiments and clinical trials necessary, successful developments in this area have the potential to enhance human health, impacting a wide number of fields such as drug discovery, biology, materials science to few. As with any technology, ensuring safe, ethical, and accountable deployment of these models is necessary to guarantee positive impact on society.
B Implementation details
3D CNN
f$
Figure 4: Auto-encoding approach for neural field representation. A voxelized representation of molecule is encoded int the latent space z with 3D CNN. This representation is then decoded with conditional MFN for any point in space.
Here, we provide some more implementation details that complement Section 5.
Conditional neural field. The codes z are computed with an encoder that takes as input a low- resolution voxelized representation of the molecular field with grid dimension of N 163. We use N 5 for QM9, N 8 for GEOM-drugs and N 6 for CREMP. We resolution of 5Á to generate the wowrresolution grid QM9, IÂ on GEOM-drigs and 1.667À for CREMP. Before
16

---

voxelizing the molecules, we first center the atoms around the tightest bounding box encapsulating the molecule, apply random rotation the atoms (each Euler angle rotated randomly between (0,2 and normalize their coordinates to the range of [-1,1] 1]. 4 residual blocks (number of hidden units 256, 512, 1024, 2048 for each block), where each block contains 3 convolutional layers followed by BatchNorm, ReLU and pooling layers (we use max pooling on the first three blocks and average pooling on the last one). The encoder has 145M on QM9 and 229M on GEOM-drugs and CREMP. In the case of FuncMoldec. we do not use any encoder and directly optimize the codes 2, one for each molecule in the dataset.
The encoder is 3D CNN containing
The neural field and codes are optimized over free-form discretization grid A, that changes at each iteration. For each training step, we sample a random training molecule and randomly pick N = 4000 points, half of the points are taken out of an uniform discretization grid of resolution 25A, and the remaining points are sampled equally across cubes of size 3 x33 and resolution 225,, centered on each atom in the molecule. We found that this choice helped speed up training. For each point, we compute the atomic occupancy value for each atom using Equation (1).
The parameters of the conditional neural field are optimized with Adam. For FuncMol, we use learning rate of 10 -4 for the encoder and x 10 for the decoder using a node of 2 A100 GPUs with batch size of 96 per GPU. For FuncMolaces optimizing the codes with Adarse using a learning rate 10-3 The decoder optimizer is Adam with learning rate of 10-3, We train the models for 900 epochs QM9, 300 epochs on GEOM- drugs and 1000 epochs on CREMP. Algorithm I and Algorithm 2 provide pscudocodes for learning the conditional neural field decoder and the latent codes (FuncMolaee) or the encoder (FuncMol).
efficiently scaled auto-decoding large datasets by
The
Algorithm 1: Auto Auto-decoding conditional neural field training pseudo-code Equation (2)
Input: D dataset of molecular fields, {Ze, <- OlvED codes, o < 6o conditional MFN parameters; N number of points to sample
while not converged do
for batch B DD do
Sample a discretization grid X' and compute occupancy ((x), E ( {2e) X) LeeB,ee f2, z.) v(x)]
{EvLEEB + [Zo]uEB - ny lo, (aoLuEB, X); Φ + Φ- no Volorlo (zoJoeB, A);;
Update codes */
/+ Update decoder weights +/
Algorithm 2: Auto-encoding conditional neural field training gseudo-code-Fquueee (3)
Input: D :D dataset of molecular fields, eh + the voxel encoder parameters: & < du conditional
MFN parameters; N number of points to sample, uniform "low-resolution" voxel grid G while not converged do for batch B C 'D do
Sample discretization grid X' and compute occupancy v(x), Vo E B, Vo E X and low-resolution voxel grid G,, Vo E B.
lac v X, B)- = EvEBaEe [fo( - v(x)1 @ <- Φ no V o las (s. N, B);
/* Update decoder weights */ /* Update encoder weights +/
< 2a Thp V feec (Ø) 14, B);
Modulation code denoiser وq. Once the modulation codes and the conditional neural field are learned, we pre-process the codes to have zero mean and unit variance, then learn a denoiser in normalized space using a 1.2 on GEOM-drugs and CREMP and
for QM9, following Algorithm 3.
Our denoiser has a projection linear layer (that embed the 1024 2048 code into a 6144 space) followed by several residual blocks, where each block contains (in this order): group normalization layer, ReLU non-linearity, fully-connected layer, normalization layer, ReLU lineaiira,,,t, drop-out with rate 0.3 for FuncMollde or none for FuncMol and another fully-connected layer. We then add one final layer to go back to the original 1024 2048 code space. We use similar "skip-connections"" as in the MLP denoiser of [33], adapted from 2D U-Net architectures 196]. For GEOM- and CREMP, we consider model with 1.9B parameters (12 residual blocks, 6144 hidden units). For QM9, we train model of size 445M parameters (6 residual blocks, 4096 hidden units). The models
17

---

are trained with batch size 2048 on single A100 GPU for 2500 epochs with AdamW [97] (learning
rate 10-3, weight decay 10 3) and exponential moving average (EMA) with decay of. of.9999. As [36], we use the following learning rate schedule: we warm-up the learning rate linearly from 0 to 3e-4 for the first 4000 iterations, then decay it proportionally to the square root of the iteration count. The pseudo-code is given in Algorithm 3.
Algorithm 3: Denoiser training pseudo-code- Equation (6)
Input: :D: = & FeED normalized codes, denoiser 20
while not converged do
for batch B C D2 do
y x 2 E. E E~ N/ o² d2 Edcooiser (0, B) E3EB ||2 0 0- Valdenoiser (), B),
20 (v)
OEMA f EMA0.9990 (EEMA. 0)
Sampling. The walk-jump sampling approach is very flexible and allows us configure sampling in different ways. For example, we can choose the number of walk steps between jumps, the maximum number of walk steps per chain the number of chains run in parallel. Different sampling hyperparameters can change the statistics of samples, e.g., samples that are close to each other on sample chain will likely be similar molecules. Therefore, we decided to fix some sampling hyperparameterrs for benchmarking purposes. In all our quantitative experiments, we generate samples in the following way: (i) we initialize all the chains Y/ parallel, (ii) we "walk" K steps with Langevin MCMC to sample smoothed codes yk. and (iii) we "jump" with the denoiser (in a single step) to get the clean codes åg. In practice, we sampled 10000 molecules using 1000 MCMC steps for both QM9 and GEOM-drugs, and 10000 steps for CREMP on single A100 GPU.
Algorithm 4: Sampling pseudo-code the For loop corresponds to walk steps Input 5 (step size), y (friction), K (steps), denoiser &S trained at noise level a. yo Ui DEiin..,.,EE1..d 2i, max;ED,,e((--d 2:) + (0 (2)) MAD +- 0
for k- 0, 0..... K- do
Uk+1/2 + yk uk
g g (3 _ (80(U)+1/2)= - yk+ 21+1122/00
/* score Equation (5) */
U2+1/2 t + 29
uk+1 f- exp( -7 uk+1/2 + 5g + v(1 (11-ex((-220) exp(-270))e,
8 ~~ N~0
yk+1 + 1k++1/2 + ux+1
Output 2K - (y()
/* jump step (denoising) +/
From codes to molecules. After generating modulation codes, we need to extract the atom types and coordinates from them. This is constrained optimization problem, and we provide simple algorithm to find its solution: (i) render a voxel grid representation of the molecule at resolution of .25A (tensors 96 x 96 x 96 on QM9, GEOM-drugs
of dimensions x 32 32 x 32, 8 x 64 x 64 x 64 and and CREMP, respectively), (ii) find the peaks of the voxel grids- they correspond to discretized version of the atomic coordinates with with a simple 3 x 3 3 kernel, and (iii) find the local optima of the atomic coordinates with the approach described in Section 3.3. Our continuous refinement approach leverages L-BFGS with learning rate 1.0 and is batched across 100 molecules of same size.
Assets used in this work. Our code is available at https: /tttps://giie //github. com/ coo/ppesciente design/ funcmol. Our neural field code is based on the open source implementation of MFN from [31] and the conditional version from [27]. Our code for walk-jum sampling is based on the open source implementation VoxMol from [5]. Our metrics are computed using code from [54] and RDKit [88]. Our datasets GEOM-drugs [13]. CREMP [12] and QM9 [81] are downloaded from the corresponding webpages. We use the protein visualization tool of [98]. All these assets are available publicly and to our knowledge have a CC-BY 4.0 license.
18

---

C Analysis of the latent space
We perform three experiments to qualitatively explore the learned manifold and show empirically that it is well structured.
First, we pick several pairs molecules and show the interpolation trajectory in latent modulation space. We project the interpolated codes back to the learned manifold of molecules via noise/denoise operation. Figure 5 illustrates six trajectories, where we observe that molecules close in latent space share similar structure.
Figure 5: Interpolation in the latent modulation space for different pairs of molecules from GEOM- drugs. Each interpolated codes is protected back to the learned manifold of molecules via noise/denoise operation. FuncMol produces semantically meaningful patterns the interpolated space and we observe that molecules close in latent space share similar structure.
Second, we show t-SNE plots to demonstrate that the modulation space encodes molecular properties of QM9, For four different properties, we use I-SNE to embed 400 molecules divided equally between those with the highest and those with the lowest property values. Figure shows that molecules with similar property values cluster together.
Finally, evaluate the latent codes downstream tasks. We train linear regression model on frozen latent codes (a.k.a. linear probing) to see how the learned modulations correlate with different properties. Figure shows the scatter plots and Spearman correlation for four different properties. We observe that the codes are highly predictive of the considered properties, despite being trained in unsupervised fashion.
D Diffusion baseline
We consider one additional model, FuncMoldec, dif for the auto-decoding setting. This model is similar to FuncMoldes but we sample codes with a diffusion model instead of walk-jump sampling. We use the same neural field and modulation codes as in FuncMolare and we train multi-level denoiser (with 1000 levels of noise) instead of single-level one. The modulation codes are sampled like in standard diffusion models: we start from Gaussian noise and iteratively apply the denoiser until we arrive clean codes. We tried to train the diffusion variant of the model on GEOM-drugs,
19

---

(a) capacity (c.)
(b) Internal energy (1))
12
(0] Isotropic polartzation (x)
(d) Dipole moment
Figure 6: t-SNE plots of latent modulations codes of QM9 molecules for different molecular properties. For each plot, we pick 200 molecules from validation set with high value of a property (blue) and 200 with low value (red). We show results for four properties: (a) heat capacity (C.), (b) internal energy (Un). (c) isotropic polarization (a) and (d) dipole moment (p)).
(9) Heal capacity (C)
(b) Internal energy (Dg)
0.00
1c) lootropic polancation (c)
(3) Uppole moment ())
Figure 7: Performance linear regression model (a.k.a linear probing) trained on modulation codes to predict molecular properties QM9, We show the scatter plots and Spearman correlation for four different properties: (a) heat capacity (C+), (b) internal energy (Un), (c) isotropic polarization (a) and (d) dipole moment (μ4).
but we were not successful (the generated molecules had very low stability). Table shows the performance of the diffusion model relative to our other models. It performed worse than walk-jump sampling but was still able to generate good quality molecules.
Table 4: Comparing walk-jump sampling and diffusion model QM9 results (10000 samples per model). 144 indicate that higher/lower numbers are better. The row data are randomly sampled molecules from the validation set.
stable mol % atom
stable
valid %,
unique %2
valency
atom
bond
bond
bond
time
W11
TV;
TV₃
len W1y
ang W1,
s/mol.
FuncMol
99.1
100.
93.1
.021
.012
.006
.004
1.49
FuncMolite
88.6
99.2
100.
81.1
.022
.066
.032
.006
1.21
0.05
FuncMolacr. diff
97.3
95.8
81.1
.007
.034
.021
.006
1.25
0.07
20

---

E
Ablations
E.1 Reconstruction quality
We analyze the reconstruction quality of the fields and molecules we compressed with latent codes.
Field reconstruction. We report in Table the reconstruction performance of the molecular fields using our conditional MFN architecture and learned codes.
Table 5: Ablation: field reconstruction (whole training set),
dset
MSE
PSNR 1
GEOM- -rugs
2.8 10-6
55.5 2.9 10-6
CREMP
55.4
QM9
6.1 10-6
52.1
Molecule reconstruction. To make more sense out of these raw reconstruction metrics, we show that the 3D molecules decoded from learned training codes are valid and stable molecules. This means that we successfully compressed the training data into low-dimensional latent vectors via our field- based representation. Table reports metrics molecules decoded from the learned training codes for GEOM-drugs and QM9. For each dataset, display the metrics of the training molecules, the molecules decoded from our training codes and the molecules derived from a voxelized representation of the field resolution of 25.. We observe that the molecules rendered from codes have better metrics than those derived from voxels showing the validity of our new representation. These results are an upper bound the results Table
Table 6: Ablation: molecule reconstruction (sample of 4k).
dset
type
stable mol %4
stable atom%
valid %-
valency Way
atom
bond TV₁
bond
bond
TV1
len W
ang Wi1
GEOM-
data
99.9
99.9
100.
.001
.001
.025
.000
.05
drugs
code
83.1
99.5
100.
188
.007
.026
.004
.19
voxel
83.7
99.4
93.8
.252
.006
.026
.001
.43
QM9
data 98.7
99.8
98.9
.001
.003
.000
.000
.12
code
95.5
99.7
100.
.010
.009
.003
.001
.16
voxel
92.5
99.4
98.8
.017
.009
.002
.002
.30
E.2 Atomic coordinate refinement
A big advantage of using neural fields is that represent signals, here molecules, in continuous space rather than in discrete space as in voxel representations. The continuous refinement introduced in Section 3.3 improves the quality of the molecules by finding more precise atomic coordinates. Table 7 shows the improvement of this continuous refinement over the refinement used in [5] that operates in discrete space: by going to continuous space, overcome the limitations of discrete grids and substantially improve the stability of the molecules and the angle between bonds.
E.3 Neural field robustness to noise
Here, we analyze how the neural field is robust to noise on the modulation code space. Figure illustrates how molecular stability and the distance between the distribution of bond angles change as we increasingly add noise to codes. Each point consists of the average of the metric over 4000 random modulation codes from the validation set. Interestingly, the neural field is quite robust to noise as we see that the metrics unchanged even at reasonable amount of noise. We believe that this code robustness to noise helps better Iearn the denoiser.
21

---

Table Ablation: continuous refinement improvement on code reconstruction performance. Metries computed with 4000 generated samples on validation reference set.
dset
FuncMol
stable mol %
stable atom%6
valid
valency
atom
bond
bond
bond
coord refine
%6
W1j
TV;
TV
len Wiy
ang W1
GROM
83.1
99.5
.188
.007
.026
.004
.189
drugs
X
78.8
96.2
100.
.090
.007
.018
.011
2.71
QM9
V
95.5
99.7
100.
.010
.009
.003
.001
.158
79.1
96.4
.002
.011
2.76
100
GEOM.drugs
GEOM:drugs
stable mol (16)1)
band ong 1V (1)
100
60
0.00
0.50
1.00
1.50
2.00
0.00
0.50
1.00
2,00
(a)
Figure Ablation: code robustness noise on GEOM-drugs (blue) and QM9 (red). Stable molecule (a) and bond angle distance (b) metrics as we increasingly add noise to the codes. Metrics are computed with 4000 generated samples on validation reference set.
E.4 Number of walk steps
Table shows how the quality of molecules changes as we increase number of walk steps K walk-jump sampling for FuncMol GEOM-drugs. In this experiment, sample 2000 samples and compute metrics w.r. the validation set. First, we observe that some metrics improve (e.g.. molecular stability) while other get worse (e.g., uniqueness). Second, extremely stable, allowing us to perform as much as 50000 MCMC steps in the chain without breaking it. Finally, sampling time does not increase significantly: going from 500 50000 steps only results in a 10x increase in sampling time (this is because the sampling bottleneck on finding the atomic coordinates).
observe that the walk-jump chain
Table 8: Ablation on the number of walk steps K on GEOM-drugs. Metries computed with 2000 generated samples on test reference set.
K
stable
stable
unique
valency
atom
bond
bond
bond ang W11
(n steps)
mol %6-
%
%6
Wi1
TV:
TV1
len Wi.
s/mol.|
500
52.8
98.1
99.8
.235
.116
.031
.003
2.58
.279
1000
68.8
98.8
97.4
.246
.109
.051
.003
2.51
.298
2000
77.1
99.0
93.7
.247
.108
.068
.003
2.86
.337
5000
80.6
99.0
85.6
.247
.150
.091
.003
3.37
.456
10000
82.4
99.0
77.9
.247
.162
.109
.003
3.52
.654
20000
83.8
98.9
73.6
.252
.154
.130
.004
3.52
1.05
50000
84.9
99.0
66.6
.259
.166
.158
.003
3.44
2.24
22

---

E,5 Impact of resolution at decoding time
Finally, we measure the impact of resolution on sampling time and quality. As expected, sampling time becomes slower as we have finer resolution. We also notice that finer resolution has better results than coarse ones (although the results stop improving). We chose resolution 0.25A as provides good trade-off between performance and speed. The table below shows the results function of resolution.
Table 9: Ablation the impact of resolution sampling quality GEOM-drugs. Metrics computed with 2000 generated samples on test reference set.
resolution
stable mol %
stable
valid
unique
valency
atom
bond
bond
avg.
A
atom &
%
%
Wai
TV:
TV;
len W
ang Wi,
s/mol.
0.167
68.6
98.8
100.
97.4
.246
.109
.052
.003
2.52
.89
0.25
68.8
98.8
100.
97.4
.250
.109
.003
.29
0.5
58.8
97.7
100.
98.0
.247
.066
.051
.003
2.46
.08
F Additional quantitative results
We report some additional plots for evaluation on GEOM-drugs. See Section 5 for more details.
GEOM-drugs
1.0
0.8
Cuiveaatiee
0.6
Reference
EDM
0.2
VoxMel
RuncMol
200
400
600
800
1000
Strain Energy (UFF, kcal/mol)
Figure 9: Cumulative distribution function of strain energy of generated molecules on GEOM-drugs based on 10000 molecules.
1.0
Refevence
Beference GeOLDM
Aefeinace
Refermoc
.0.0
Soetell
PuncMM dec
30
202
0.8g
2.4
of fragments molesule
Seference
Reference Pursted dec
Saference
LBM
GeoLoN
20.
202
601
0.6
15
10
Size
0.00
Axforencc
Roforacce
0.00
CDN
20
40
60
80
100
20
40
40
40
350
130 XL
atoms molecule
60
60
800
42
40
60
Figure 10: Histograms (over 10000 samples) showing (first row) distribution of number of fragments, (second) distribution of ring size, and (third) distribution of number of atoms per molecule.
23

---

2.6
Nefrence
Refernnc
Reforenco
0.32
20.10
2000
goob
2ode
0002
000D
05
0.5
QED
05
0.55
-GOLLM
Raferrncc
Parssho dec
nuncMol
20.10
20.00
b.05
D.RgD
S4
Deferrtce
Teleeasce
Ralurree
Telerance
2:03
600
0.0.
0.0
02
0.4
06
0.6
1.0
0.000
24.00
02
06
0.
1.0
logo
Figure 11: Histograms (over 10000 samples) showing (top) distribution of QED, (mid) SA and
(bottom) log
G Comparison to bond-diffusion baselines
Some recent papers e-g., MolDiff [56], MiDi [54], show that incorporating extra information such as bonds and formal charges in point cloud-base approaches (e.g. EDM) improves the quality of the generated samples. These contributions are orthogonal to ours and are an interesting future addition to FuncMol, e.g. via additional channels in the molecular field.
We compare FuncMol to MolDiff despite different training assumptions, for completeness. MolDiff only incorporates bond information into the diffusion process, making it simple representative baseline for this class of model. Since the weights for MolDiff with hydrogens were unavailable, compared FuncMol using MolDiff's metrics and the MolDiff performance reported in their Appendix D.1. Table 8. The results are reported in Table 10. We observe that FuncMol achieves competitive results in most metrics despite not leveraging bond information.
Table 10: Comparison of MolDiff with H and FuncMol
MolDiff with H
FuncMol
Validity f
0.957
1,000
Connectivity 1
0.772
0.739
Succ. Rate f
0.739
0.739
Novelty 1
1.000
0.992
Uniqueness 1
1.000
0.977
Diversity
0.427
0,810
Sim. Val. 1
0.695
0.554
QED t
0,688
0.715
SA f
0.806
0.815
Lipinski
4.868
5,000
RMSD +
1.032
1,088
JS bond lengths +
0.414
0.529
IS bond angles +
0.182
0.217
IS dihedral angles 4
0.244
0.232

---

H
Additional qualitative results
We display some generated molecules in Figures 12 and 13 and display a MCMC chain in Figure 14.
Figure 12: Generated samples from FuncMol trained on GEOM-drugs.
25

---

Figure 13: Generated samples from FuncMol trained on CREMP.
Figure 14:
Two single MCMC chains generated by FuncMol, initialized randomly with different seeds (seen from left to right, top to bottom). Molecules are generated after each 200 "walk" steps.
26

---

I NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: we clearly state in the introduction in Section our contributions and scope. Our paper does not make any important assumptions and we provide discussion in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide discussion model including limitations in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer:
[NA]
Justification:
we do not provide theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification: We detail all the hyperparameters, architectures and model training details in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
28

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
coomppreccientdesiiie funcmol. We provide all the hyperparameters used to run our experiments in Appendix B.
Justification: We will provide our code at
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (uttps://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //ipps.ccppublleee wwtpccccpppccccccccccccccccccccc gwwgguuieeeeeeeeeo CoodeSummmsscc ssi assionPoicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we provide a succinet description of the experimental setting in Section 5 and provide more details in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: we report 1-sigma error bars after repeating 3 times sampling in our main Tables 2 and 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report in Appendix B the compute resources required for for the experi- mental runs. In addition, had access to around 20 A100 GPUs internal cluster for prototyping over the last 5 months.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics tttps://neuripe: ctcp/ubbic/thhiccsuuudellllll
Answer: [Yes]
Justification: we reviewed the Code Of Ethics and confirm that the research in the paper conform to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: of Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses such risks. the considered datasets commonly used to benchmark generative models the field.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
we included discussion of all assets used in Appendix B.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide all the required information to reproduce models and the assets we used in Appendix B. Our license is CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "xlcnjmgiy4", "reviewer": "Reviewer_yg8b", "summary": "This study proposes a neural field model that treats molecular data as continuous atomic occupancy fields. The model learns a latent code that can be used to predict the atomic occupancy in discretized grids. The authors then perform score-based generative modeling using neural empirical Bayes and show higher generation quality than point cloud-based and voxel-based models.", "questions": "1\\. It would be interesting to see whether the latent space has any meaningful manifold. For example, does the latent code capture patterns such as basic molecular fragments or similarity between molecules, etc.?\n\n2\\. Also, though the sampling is fast, the peak finding and iterative refinement could be time-consuming. Could the authors comment on this?\n\n3\\. Some recent methods [1,2] explicitly model bonds, leading to high quality generations. How does the proposed method compare with these methods? Is it also possible to incorporate bond information into the molecular field modeling, e.g. as additional channels?\n\n[1] Peng, Xingang, et al. \"Moldiff: Addressing the atom-bond inconsistency problem in 3d molecule diffusion generation.\" arXiv preprint arXiv:2305.07508 (2023).\n\n[2] Vignac, Clement, et al. \"Midi: Mixed graph and 3d denoising diffusion for molecule generation.\" Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Cham: Springer Nature Switzerland, 2023.", "limitations": "The authors have properly addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This study utilizes a novel representation of 3D molecular structures. The proposed model does not require pre-defined atom numbers, which is a common limitation of point cloud representations, has higher expressivity than GNN-based methods, and also scales better than voxel grid representations. It could also scale to larger molecules such as cyclic peptides.", "weaknesses": "I only have some minor concerns and questions. See the \"Questions\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive and valuable feedback. A general rebuttal is posted above. Below we address the reviewer's concerns.\n\n**1. Meaningful representation of latent space.** See \"Meaningful representation of latent space [yg8b]\" on the main Rebuttal. Overall, the additional experiments show that the latent space is well structured.\n\n**2. Peak finding + iterative refinement can be time consuming.** To clarify, the average sampling time reported on Table 2 is the time required to fully generate the molecules (\"from noise to set of atom types/coordinates\"). It includes four steps: (i) the walk-jump sampling steps to sample latent codes, (ii) rendering (going from the code to the voxel grid at resolution .25A), (iii) the peak finding and (iv) the iterative refinement. The sampling time for all steps together is an order of magnitude faster than alternative baselines. \nThe bottleneck in sampling time is the rendering phase (ii). The sampling time can be reduced at the cost of performance by rendering codes at a lower resolution (see \"Ablation study\" on the main Rebuttal).\n\n**3. Explicitly modeling bonds between atoms.** Some recent works e.g., MolDiff, MiDi, show that incorporating extra information such as bonds and formal charges in point cloud-based approaches (e.g. EDM) improves the quality of the generated samples. These contributions are orthogonal to ours and can potentially be incorporated into our generative model, e.g. via additional channels as suggested by the reviewer. This is, however, not the focus of our work (nor that of the baselines we considered). Here, we aim at proposing for the first time neural fields as a new representation for 3D molecules (a non-trivial task). \n\nFor completeness (despite different training assumptions), we compare FuncMol to MolDiff. MolDiff only incorporates bond information into the diffusion process, making it a simple representative baseline for this class of model. Since the weights for MolDiff with hydrogens were unavailable, we compared FuncMol using MolDiff’s metrics and the MolDiff performance reported in their Appendix D.1, Table 8. We observe that FuncMol achieves competitive results in most metrics despite not leveraging bond information during training.\n\n| Metric               | MolDiff with H | FuncMol | \n|----------------|---------------|----------------|\n| Validity ↑                   | 0.957        | 1.000     | \n| Connectivity ↑           | 0.772        | 0.739     | \n| Succ. Rate ↑             | 0.739        | 0.739     | \n| Novelty ↑                   | 1.000        | 0.992    | \n| Uniqueness ↑            | 1.000        | 0.977    | \n| Diversity ↑                 | 0.427        | 0.810    | \n| Sim. Val. ↑                 | 0.695        | 0.554    | \n| QED ↑                       | 0.688        | 0.715    | \n| SA ↑                          | 0.806        | 0.815    | \n| Lipinski ↑                   | 4.868        | 5.000   | \n| RMSD ↓                    | 1.032        | 1.088    |\n| JS bond lengths ↓     | 0.414        | 0.529    | \n| JS bond angles ↓      | 0.182        | 0.217    | \n| JS dihedral angles ↓ | 0.244        | 0.232    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z5kltSsoMq", "reviewer": "Reviewer_R2Qo", "summary": "This paper introduces FuncMol, a method that leverages recent work on implicit neural representations for unconditional 3D molecule generation. The key idea is to (1) parametrize continuous atomic densities through a neural field network and molecule-specific modulation codes and (2) use a score-based generative model in modulation code space to efficiently generate new molecules. The authors demonstrate competitive performance on the GEOM-drugs dataset, showing that the generated molecules are roughly on par with existing techniques at significantly faster sampling speeds.", "questions": "* What is the rationale behind switching from the meta-learning approach in [4] to optimizing the modulation codes and neural field parameters jointly? Could this cause some of the issues with generating novel/unique molecules?\n* Tables 1 and 2 report metrics for 10000 samples per model. However, in lines 257-263, the authors mention that the model without dropout needs to generate twice as many samples as the model with dropout. Are the models generating more than 10000 samples and are non-novel molecules filtered out before reporting the results? If not, what is the percentage of novel molecules for each model?\n* [2] defines *atom stability* as the proportion of atoms that have the correct valency and *molecule stability* as the proportion of generated molecules for which all atoms are stable. How can the percentage of valid molecules be higher than the percentage of stable molecules, since RDKit’s sanitization filter checks if a molecule has correct valencies?\n\n---\n\n[1] Grambow, Colin A., et al. \"RINGER: Rapid Conformer Generation for Macrocycles with Sequence-Conditioned Internal Coordinate Diffusion.\" arXiv preprint arXiv:2305.19800 (2023).\n\n[2] Hoogeboom, Emiel, et al. \"Equivariant diffusion for molecule generation in 3d.\" International conference on machine learning. PMLR, 2022.\n\n[3] Vignac, Clement, and Pascal Frossard. \"Top-N: Equivariant Set and Graph Generation without Exchangeability.\" International Conference on Learning Representations.\n\n[4] Dupont, Emilien, et al. \"From data to functa: Your data point is a function and you can treat it like one.\" International Conference on Machine Learning. PMLR, 2022.", "limitations": "The authors provide a brief discussion of the lack of equivariance as a limitation in Section 6.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Using implicit neural representations to represent 3D point clouds and atomic densities is an innovative approach to 3D molecular generation that could complement the current graph- and voxel-based state-of-the-art.\n* The score-based generative model and walk-jump sampling process in latent space afford significantly faster sampling speeds than existing approaches and enable the method to scale favorably to larger systems.\n* The paper is well-written and the authors clearly describe their method and the motivation behind it, as well as the experimental details used for training and evaluation.", "weaknesses": "* The approach seems to have some issues with overfitting/memorization, which the authors mitigate by applying dropout between the MFNs multiplicative blocks. However, the model still seems to generate significantly fewer unique molecules than the baselines. It would be good if the authors could comment on whether this is caused by e.g. limited capacity of the neural field, a degenerate modulation code space, or the walk-jump sampling scheme, and if there are any ways to address this.\n* The method performs strictly worse than existing approaches on the QM9 dataset. The authors argue that QM9 is not a suitable dataset to evaluate unconditional generative models, since it consists of an exhaustively enumerated set of small molecules. However [2] and [3] only state that novelty is not a meaningful performance metric in this setting. The ability to capture this relatively well-behaved training distribution and generate stable and unique molecules should still be a reasonable test of generative modeling performance.\n* The paper only provides qualitative results for the CREMP dataset. Since the scalability of the proposed method to larger systems is listed as one of its main advantages, it would be helpful to include a quantitative comparison to other methods from [1]. Most of the metrics in Table 1 should be directly applicable to macro-cyclic peptides, in addition to e.g. the percentage of closed rings and canonical side chains. Similarly, it would be good to include the sampling times in this setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. A general rebuttal is posted above. Below we address the reviewer's specific questions.  \n\n**Issues with overfitting/memorization.** By further investigating this issue, we realized that memorization is likely caused by a degenerate latent space—as alluded to by the reviewer. Data augmentation helped mitigate this issue and learn a more \"semantically meaningful\" latent space. This is reflected in the results of Rebuttal Table 1 and 2 (see main Rebuttal \"Issues with memorization\"), where uniqueness is significantly improved and we no longer observe memorization. This is a consequence of the model overfitting less, which also allowed us to train denoisers with higher capacity. \n\n**Worse performance on QM9.**  We agree that QM9 is a good task to test generative models. FuncMol and VoxMol perform comparably on QM9 and both are outperformed by point cloud approaches. FuncMol, however, has an order of magnitude faster sampling time. QM9 has limited relevance for drug discovery: the number of stable and unique molecules is limited as QM9 is an enumeration of all molecules up to 9 heavy atoms satisfying some constraint. Achieving a high performance on QM9 does not mean the model can handle more complex distributions. In fact, the best models we reported on QM9 (GeoLDM, EDM) perform the worst on GEOM-drugs. \n\n**Quantitative results on CREMP.** To our knowledge, we are the first to report results on CREMP in the unconditional all-atom 3D molecule generation setting. This experiment is more for qualitative purposes as CREMP is very recent and not a standard benchmark for this task. RINGER [1] also considered CREMP but tackled conformer generation, a different problem from ours: it assumes knowledge of the molecular sequence/graph during training and sampling. This simplifies generation, since the model knows a priori the number of atoms, their types, the bonds between them and the approximate atom locations. RINGER only parametrizes angles and torsions, while FuncMol and the baselines perform all-atom generation. \nWe tried to extend our baselines to CREMP but did not succeed, mainly due to the high memory consumption (e.g. VoxMol took 40 GPU-hours per epoch, while FuncMol took 2.7 GPU-hours). To encourage comparisons to FuncMol, we include some quantitative metrics used in [1] that measure the distance between test and generated distributions using KL divergence. The KL divergence for bond angles are 0.1615 ($\\theta_1$), 0.1345 ($\\theta_2$) and 0.2197 ($\\theta_3$); the ones for the dihedral angles are 0.1127 ($\\phi$), 0.1178 ($\\psi$) and 0.1813 ($\\omega$). The percentage of valid generated peptides (for which we can extract the sequence of amino acids from their SMILES) is 82.7%. \n\n**Sampling time on CREMP.** We reported the sampling time on L327: \"our model takes around 1.4s to generate a molecule. [..] should VoxMol be trained successfully, it would take over a minute [per] molecule\".\n\n**Why switch from the meta-learning approach in [6]?** Joint optimization, a.k.a. auto-decoding has been applied in several works [2, 5] to learn signed-distance functions. Auto-decoding is attractive as it does not require the memory-expensive double loop optimizations in meta-learning and is simpler to optimize. We attribute the novelty/uniqueness issue to the lack of data augmentation during training (see \"Issues with memorization\" in the main Rebuttal).\n\n**Filtering samples due to memorization.**  As written on L261: \"we only consider the novel molecules of FuncMol$ _{drop=0}$ and exclude those pathological chains when benchmarking this model\"; the total number of molecules that are evaluated is 10,000. The other reported FuncMol model (with dropout) considers both novel and non novel molecules. Our initial novelty scores are 56.3\\% for FuncMol$ _{drop=0}$ and 77.3\\% for FuncMol (with dropout). With data augmentation, the novelty score is over 99\\%.\n\n**How can the percentage of valid molecules be higher than the percentage of stable molecules?** We use the same metrics as previous work for an apples to apples comparison to other work. In fact, all baselines that report these two metrics have higher validity than mol stability (EDM, GeoLDM, VoxMol). _Validity_ is measured as the success rate of RDKit sanitization and is computed only on the largest fragment of the generated molecules (when they consist of disjoint fragments). _Atom_ and _molecule stability_ are defined as the reviewer described and were introduced in [4]. As explained by [3]: \"[mol stability] is similar to validity but, in contrast to RDKit sanitization, they do not allow for adding implicit hydrogens to satisfy the valency constraints.\"\n\n[1] Grambow et al. \"RINGER\" Arxiv23\n\n[2] Park et al. “DeepSDF” CVPR19\n\n[3] Vignac et al. \"MiDi\" ECML23\n\n[4] Satorras et al. \"E(n) equivariant normalizing flows\" NeurIPS21\n\n[5] Chou et al. \"Diffusion-SDF\" ICCV23\n\n[6] Dupont et al. \"From data to functa\" ICML22", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FYayPAffMP", "reviewer": "Reviewer_eB4P", "summary": "This paper proposes a new representation for 3d molecules, and the representations are low-dimensional, compact, and scalable. Based on the representation, this paper proposes a new score-based generative model, FuncMol, which shows competitive reuslts on GEOM-drug and scales up to CREMP. Besides, FuncMol adopts the recently proposed Walk-Jump sampling and enjoys fast sampling.", "questions": "1. Why FuncMol cannot outperform GeoLDM and VoxMol in many metrics?\n2. Why use walk-jump sampling? It seems unnecessary to \"jump\" before the last step.\n3. How to guarantee the generalization of learned \"codes\"? The manifold for valid molecules may not be continuous, or support the embedding space. Thus, the reviewer suspects that random initialization and Langevin MCMC may produce invalid molecules.\n\nI will raise my score if the questions are properly answered.", "limitations": "See above", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. New representation. This paper proposes atom occupancy fields as Gaussian-like 3D shapes, enabling \"density\" estimation for any points in the space.\n2. Feasible decoding. Decoding a molecule (atom types and coordinates) directly from an embedding vector is extremely difficult, yet decoding a molecular field from an embedding vector seems more feasible.\n3. Competitive results. Based on the proposed molecular neural field, this paper proposes a generative model and achieves competitive results with SOTA models, making this representation potential in downstream applications.\n4. Fast sampling.", "weaknesses": "1. From the perspective of 3d molecule generation, FuncMol doesn't outperform VoxMol and GeoLDM in almost all metrics. It could be a trade-off between generative ability and representation quality.\n2. The authors claim FuncMol can scale up to larger molecules, yet the more fundamental problem is, can we have a scaling law for molecules like LLM. Scaling law is essential for molecular foundation models, and should have a much larger impact.\n3. Molecule generation may not be a perfect scenario for this representation, which seems more like a generative pretraining framework. Besides, the reviewer is interested in the application of such representations on downstream applications, which are not tested, nor discussed.\n\nI will raise my score if the authors present more evidence for weakness 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and positive feedback. A general rebuttal is posted above. Below we address the specific issues raised by the reviewer.\n\n**“Why does FuncMol not outperform GeoLDM and VoxMol?\"** It is challenging to explain why these models perform differently, as they make different choices w.r.t. data representation, NN architectures and generative modeling. FuncMol outperforms GeoLDM in most of the metrics of Rebuttal Table 2 and some metrics of Rebuttal Table 1 (c.f. main rebuttal). VoxMol outperforms FuncMol on GEOM-drugs (the performance gap is smaller when adding data augmentation to FuncMol). VoxMol uses 3D UNets, an expressive and well-established NN architecture tailored to discrete data, while FuncMol uses relatively new neural field architectures; neural fields tend to underperform against discrete-data architectures in many computer vision applications [1]. FuncMol, however, generates 3D molecules an order of magnitude faster than VoxMol and GeoLDM. Moreover, unlike them, FuncMol easily scales to larger molecules (such as macro-cyclic peptides).\n\n**Scaling laws for molecules.** We agree that coming up with scaling laws for 3D molecules is an interesting research direction. However, this lies beyond the scope of this work, especially when taking into account the large amount of computation resources required for this type of study. That being said, we observed that at fixed dataset, increasing the parameter count of our models (both the neural field and our denoiser) significantly improved our results. Moreover, increasing the dataset e.g. with data augmentation leads to better performance. \n\n**Downstream applications: \"Molecule generation may not be a perfect scenario for this representation\".**  We thank the reviewer for this comment that helped improve the paper. Generative modeling for molecules is an important task with many practical applications. It also serves as a testbed to demonstrate that neural fields provide a very expressive/efficient/scalable decoder for 3D molecules, unlike point clouds or voxel grids. \n\nAt the reviewer’s request, we show that neural field representations can also be used in other downstream tasks, e.g. discriminative tasks related to property prediction. We consider some properties from QM9 ($\\mu$ dipole moment, $u_0$ internal energy, $\\alpha$ isotropic polarizability, $C_v$ heat capacity) and report the Spearman correlation performance (ranging from -1 to 1) in the linear probing setting (i.e. training a linear regression on top of frozen codes). We observe high ranking scores and correlations between predictions and ground truth labels as shown on Rebuttal Fig 4 (c.f. main rebuttal pdf) and the table below.\n\n|                | $\\mu$ | $u_0$ | $\\alpha$ | $C_v$ |\n|----------------|---------------|----------------|----------|-----------|\n| Spearman_rho  ↑       | 0.632          | 0.939           | 0.968     | 0.950      |\n\n**Why use walk-jump sampling?** We explain why we use walk-jump sampling/neural empirical Bayes on L40-41: \"[it] enjoys many properties such as fast-mixing, simplicity for training and fast sampling speed\" as it relies on a single noise level. It has been successfully applied in 3D molecule generation (e.g., VoxMol) and antibody sequence generation [2]. Despite choosing walk-jump sampling for our main results, we stress that our framework is agnostic to the generative model used; we reported results with diffusion models on Appendix C. Compared to diffusion, we found that walk-jump sampling on this task is simpler/faster to train, faster to sample from (less sampling steps are required) and achieves slightly better empirical results. \n\n**How to guarantee the generalization of learned \"codes\"? Random initialization and Langevin MCMC may produce invalid molecules.** We agree with the reviewer that the manifold of valid molecules is very complicated (low-dimensional in a large ambient space and potentially non-smooth). However, the manifold of codes after _gaussian-smoothing_ is much simpler, especially with large noise. This allows us to perform Langevin MCMC from random initialization. This is precisely the intuition of neural Empirical Bayes [3], the generative framework we used in this work. We observed empirically that Langevin MCMC on smoothed codes mixes well. For example, Rebuttal Fig. 5 (c.f. main rebuttal pdf) shows two single MCMC chains initialized randomly with different seeds, where molecules are generated after each 200 \"walk\" steps. This phenomenon has also been observed on different data modalities, e.g., images [4], biological sequences [2] and voxelized molecules (VoxMol).\n\n[1] Dupont et al. \"From data to functa: Your data point is a function and you can treat it like one.\" ICML2022.\n\n[2] Frey et al. \"Protein Discovery with Discrete Walk-Jump Sampling\". ICLR24\n\n[3] Saremi et al.. \"Neural empirical Bayes\". JMLR19\n\n[4] Saremi et al.. “Multi-measurement Generative Models”. ICLR22", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cI8OegAPsy", "reviewer": "Reviewer_NVMn", "summary": "In this study the authors propose a novel FuncMol model for molecular generation. The proposed approach is based on continuous space neural fields and involves the joint training of a molecular neural field along with molecular modulation codes. The implementation of a neural empirical Bayes technique allows the generation of new molecular structures during the inference phase. The performance of the proposed model is compared against established neural baseline models on the GEOM-DRUGS dataset, which comprises small drug-like molecules. Additionally, the model is examined on the CREMP dataset, containing macrocyclic peptides, to provide a comprehensive analysis of its capabilities.", "questions": "1. (major) The authors have incorporated several diffusion models as baselines, such as EDM and GeoLDM, to benchmark their model's performance on the GEOM-DRUGS dataset. However, they have missed the important MolDiff (a) baseline, which mitigates the problems of generated spatial structures including the atom-bond inconsistency problem by introducing guidance from auxiliary bond predictors. Moreover, MolDiff enhances the evaluation framework by incorporating additional Jensen-Shannon divergence metrics to assess structural and spatial aspects more comprehensively.\n2. (major) The section on macrocycle peptide generation does not take into account other relevant baselines, one of which is RINGER (b), whose authors released code for reproduction.\n3. (moderate) The experimental setup in this study focuses only on unconditional molecule generation. In the practice, spatial molecular structures generation is mainly applied given some conditions, like conformation generation (c) and pocket-conditioned generation (d), which respectively aim to generate spatial molecular structures based on a given graph or a protein pocket. The paper would benefit from extending its analysis to demonstrate how the model could be adapted to integrate external conditions and discussing potential avenues for this adaptation.\n4. (minor) The current model is not SE(3) equivariant, which implies that molecular rotations and translations could affect the modulation codes. The authors note this as a limitation in line 346 but do not specify whether any techniques were utilized, such as coordinate canonicalization, to mitigate this issue during model training. Further clarification on the methods used to address this challenge would be valuable.\n5.(minor) One of the model bottlenecks is reconstructing atom positions from the continuous occupancy field, which necessitates calculations on a uniform discretization grid with cubic scaling. The authors select a discretization step of 0.25Å, followed by L-BFGS optimization for refining atom positions. However, the manuscript does not include an ablation study examining the impact of different discretization steps on the balance between model performance and computational speed, which would be critical for justifying the chosen discretization step.\n\n(a) MolDiff: Addressing the Atom-Bond Inconsistency Problem in 3D Molecule Diffusion Generation\n\n(b) RINGER: Rapid Conformer Generation for Macrocycles with Sequence-Conditioned Internal Coordinate Diffusion\n\n(c) Torsional Diffusion for Molecular Conformer Generation\n\n(d) 3D Equivariant Diffusion for Target-Aware Molecule Generation and Affinity Prediction", "limitations": "In the \"Discussion\" section, the authors address several model limitations; however, they do not explore the significant issue of integrating external conditions into the model.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "To the best of my knowledge, the idea of using continuous space neural fields for the generation of molecules is novel and previously unexplored. Moreover, in addition to training on the popular GEOM dataset, the authors also train the model on the CREMP dataset, which contributes to the advancement of peptide generation domain.", "weaknesses": "While the approach presented in the paper is novel in the context of molecular generation, the experimental analysis is incomplete. Several baselines and thorough ablation studies to justify the selection of hyperparameters are absent, as is an in-depth examination of the model's limitations. Further details can be found in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and questions. A general rebuttal is posted above. Below we address the reviewer's individual questions. \n\n**1. MolDiff Baseline.** MolDiff shows that incorporating bond information in point cloud-based approaches improves the quality of the generated samples. This contribution is orthogonal to ours and can potentially be incorporated into our generative model, e.g. via additional channels. This, however, is not the focus of our work, nor that of the baselines we considered. Here, we aim at proposing for the first time neural fields as a new representation for 3D molecules (a non-trivial task). \n\nFor completeness (despite different training assumptions), we compare FuncMol to MolDiff. Since the weights for MolDiff with hydrogens were unavailable, we compared FuncMol using MolDiff’s metrics and the MolDiff performance reported in their Appendix D.1, Table 8. We observe that FuncMol achieves competitive results in most metrics despite not using bond information.\n\n|                | MolDiff with H | FuncMol | \n|----------------|---------------|----------------|\n| Validity ↑                   | 0.957        | 1.000     | \n| Connectivity ↑           | 0.772        | 0.739     | \n| Succ. Rate ↑             | 0.739        | 0.739     | \n| Novelty ↑                   | 1.000        | 0.992    | \n| Uniqueness ↑            | 1.000        | 0.977    | \n| Diversity ↑                 | 0.427        | 0.810    | \n| Sim. Val. ↑                 | 0.695        | 0.554    | \n| QED ↑                       | 0.688        | 0.715    | \n| SA ↑                          | 0.806        | 0.815    | \n| Lipinski ↑                   | 4.868        | 5.000   | \n| RMSD ↓                    | 1.032        | 1.088    |\n| JS bond lengths ↓     | 0.414        | 0.529    | \n| JS bond angles ↓      | 0.182        | 0.217    | \n| JS dihedral angles ↓ | 0.244        | 0.232    | \n\n**2. RINGER Baseline.** To our knowledge, we are the first to report results on CREMP in the unconditional all-atom 3D molecule generation setting. This experiment is more for qualitative purposes as CREMP is very recent and not a standard benchmark for this task. RINGER [1] also considered CREMP but tackled conformer generation, a different problem from ours: it assumes knowledge of the molecular sequence/graph during training and sampling. This simplifies generation, since the model knows a priori the number of atoms, their types, the bonds between them and the approximate atom locations. RINGER only parametrizes angles and torsions, while FuncMol and its baselines perform all-atom generation. \nWe tried to extend our baselines to CREMP but did not succeed, mainly due to the high memory consumption (e.g. VoxMol took 40 GPU-hours per epoch, while FuncMol took 2.7 GPU-hours). To encourage comparisons to FuncMol, we include some quantitative metrics used in [1] that measure the distance between test and generated distributions using KL divergence. The KL divergence for bond angles are 0.1615 ($\\theta_1$), 0.1345 ($\\theta_2$) and 0.2197 ($\\theta_3$); the ones for the dihedral angles are 0.1127 ($\\phi$), 0.1178 ($\\psi$) and 0.1813 ($\\omega$). The percentage of valid generated peptides (for which we can extract the sequence of amino acids from their SMILES) is 82.7%. \n\n**3. Focus on unconditional generation, which is not as good as conditional generation.** We agree that conditional generation presents more practical value than unconditional generation. We note, however, that our approach is the first to introduce neural fields as a representation of 3D molecules. The focus of our submission is to show that (i) it is a feasible approach (this is non-trivial), (ii) it scales well with data and molecule size, and (iii) it achieves competitive results with fast sampling time. We will include a discussion about this topic on the appendix and we will leave extensions to conditional generation for future work (similar to how TargetDiff [1]/DiffSBDD [2] extend EDM or VoxBind [3] extends VoxMol).\n\n**4. The model is not SE(3) equivariant. How to leverage rotation/translation?** We do not leverage any SE(3) equivariance on the approach described in the original manuscript. We acknowledge that this is a good point and address it in the rebuttal by performing data augmentation during training (see main Rebuttal comment \"Issues with memorization\"). Data augmentation improves the quality of the representations as it helps learn a \"more semantic\" latent space z. This is reflected in the empirical results of Rebuttal Tables 1 and 2, where the uniqueness score significantly improves.\n\n**5. Ablation study missing: Impact of different discretization steps.** Please, see the main Rebuttal, where we address this comment. Appendix D of the manuscript contains four other thorough ablation studies that justify other design choices in FuncMol.\n\n[1] Guan et al. \"3D Equivariant Diffusion for Target-Aware Molecule Generation and Affinity Prediction\". ICLR23\n\n[2] Schneuing et al. \"Structure-based Drug Design with Equivariant Diffusion Models\". Arxiv22\n\n[3] Pinheiro et al. \"Structure-based drug design by denoising voxel grids\" ICML24", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for the helpful comments. The reviewers agree that the paper has good soundness, presentation and contributions. They also acknowledge that our approach is novel, addresses limitations of point-cloud and voxel representations, is scalable, has faster sampling and is competitive. 

Next, we highlight the new experiments of the rebuttal (*see Rebuttal Figures in the attached pdf and Rebuttal Tables in the responses*). We address each review in their respective section.

**Issues with memorization [R2Qo]. How to leverage data augmentation [NVMn]?** We trained a new model with data augmentation. Instead of using *auto-decoding* [1] to learn the modulation codes (c.f. manuscript), we use an *auto-encoding* approach [2],  which allows us to leverage data augmentation on the fly. In this setting, z is computed with a (trainable) 3D CNN encoder that takes as input a voxelized molecule (Rebuttal pdf Fig 1). Once the encoder is learned, we use it to generate samples z with random data augmentation and train a denoiser as described in Section 4.1. This approach leads to a more structured latent space z as reflected in Rebuttal Table 1 and 2. We no longer observe memorization and uniqueness is significantly improved. 

**Rebuttal Table 1:**
| GEOM-drugs         | stable mol %↑ | stable atom %↑ | valid %↑ | unique %↑ | valency W₁↓ | atom TV↓ | bond TV↓ | bond len W₁↓ | bond ang W₁↓ |
|----------------|---------------|----------------|----------|-----------|--------------|----------|----------|---------------|--------------|
| **data**                                            | 99.9 | 99.9 | 99.8 | 100.0  | .001 | .001 | .025 | .000 | 0.05 |
| **EDM**                                           | 40.3 | 97.8 | 87.8 | 99.9 | .285 | .212 | .048 | .002 | 6.42 |
| **GeoLDM**                                    | 57.9 | 98.7 | 100. | 100. | .197 | .099 | .024 | .009 | 2.96 |
| **VoxMol**                                       | 75.0 | 98.1 | 93.4 | 99.6 | .254 | .033 | .024 | .002 | 0.64 |
| **FuncMol$ _{autodec}$**               | 60.6 | 98.2 | 100. | 86.6 | .244 | .079 | .044 | .003 | 2.05 |
| **FuncMol$_{autodec,drop}$**       | 69.7 | 95.3 | 100.  | 77.5 | .268 | .035 | .028 | .003 | 2.13 |
| **FuncMol$ _{autoenc}$**               | 72.6 | 99.1 | 100. | 95.6 | .250 | .107 | .046 | .003 | 2.31 |

**Rebuttal Table 2:**
| GEOM-drugs  | single frag %↑ | median energy↓ | ring sz TV↓ | atms/mol TV↓ | QED ↑ | SA ↑ | logp ↑ | time s/mol.↓ |
|------------------|----------------|----------------|-------------|--------------|-------|------|--------|--------------|
| **data**                                       | 100.0 | 54.5  | .011 | .000 | .658 | .832 | 2.95 | -    |
| **EDM**                                      | 42.2  | 951.3 | .976 | .604 | .472 | .514 | 1.11 | 9.35 |
| **GeoLDM**                               | 51.6  | 461.5 | .644 | .469 | .497 | .593 | 1.05 | 8.96 |
| **VoxMol**                                 | 82.6  | 69.2  | .264 | .636 | .659 | .762 | 2.73 | 7.55 |
| **FuncMol$ _{autodec}$**         | 86.0 | 105.0   | .263 | .677 | .713 | .862 | 2.87 | 0.69  |
| **FuncMol$ _{autodec,drop}$** | 80.2 | 96.4 | .324 | .970 | .677 | .788 | 2.87 | 0.53  |
| **FuncMol$ _{autoenc}$**         | 73.9  | 111.7 | .434 | .878 | .715 | .816 | 3.03 | 0.86 |

**Ablation study missing: impact of different discretization steps [NVMn].** As suggested by the reviewer, we include a new ablation study that measures the impact of resolution on sampling time and quality. We observe that finer resolutions improve performance but also increase sampling time. We chose resolution 0.25A as it provides a good trade-off between performance and speed. The table below shows the results as a function of resolution: 

| resolution     | stable mol %↑ | stable atom %↑ | valid %↑ | unique %↑ | valency W₁↓ | atom TV↓ | bond TV↓ | bond len W₁↓ | bond ang W₁↓ | time per mol (s)↓ |
|----------------|---------------|----------------|----------|-----------|--------------|----------|----------|---------------|--------------|--------------|
| **0.167A** | 72.8 | 99.1 | 100. | 95.8 | .248 | .127 | .042 | .003 | 2.36 | 2.8 |
| **0.25A**   | 72.6 | 99.1 | 100. | 95.6 | .250 | .127 | .046 | .003 | 2.41 | .86 |
| **0.5A**     | 51.7 | 97.6 | 100. | 96.2 | .246 | .126 | .046 | .003 | 2.29 | .15 |

For reference, we had already included four ablation studies in Appendix D.4.

**Meaningful representation of latent space [yg8b]; downstream task and “generalization of learned codes” [eB4P].**  We perform three experiments to qualitatively explore the learned manifold and show empirically that it is well structured. 
* First, we pick several pairs of molecules and show the interpolation trajectory in latent modulation space. We project the interpolated codes back to the learned manifold of molecules via a noise/denoise operation. Rebuttal pdf Fig 2 illustrates six trajectories, where we observe that molecules close in latent space share similar structure. 
* Second, we show t-SNE plots to demonstrate that the modulation space z encodes molecular properties of QM9. For four different properties, we use t-SNE to embed 400 molecules divided equally between those with the highest and those with the lowest property values. Rebuttal pdf Fig 3 shows that molecules with similar property values cluster together. 
* Finally, we evaluate the latent codes on downstream tasks. We train a linear regression model on frozen latent codes (a.k.a. linear probing) to see how the learned modulations correlate with different properties. Rebuttal pdf Fig 4 shows the scatter plots and Spearman correlation for four different properties. We observe that the codes are highly predictive of the considered properties, despite being trained in an unsupervised fashion.

[1] Park et al. "DeepSDF Learning continuous signed distance functions for shape representation"CVPR19

[2] Mescheder et al. "Occupancy networks Learning 3d reconstruction in function space"CVPR19

Meta-review: A breath of fresh air for molecular generation, implemented and presented well. The key idea is to learn an atom position density instead of learning atomic positions explicitly (or some roto-translational invariant featurization), using an implicit neural representation. The authors parameterize the atomic densities to depend on a 'modulation code.' At inference time, new molecules are sampled from a smoothed distribution of the modulation code using a combination of Langevin MCMC and walk-jump sampling. A gap remains when compared to particle-based methods in terms of sampling time. No reviewers have highlighted this limitation, which might be due to their limited experience within this domain. However, the authors show potential for scaling to larger systems, as illustrated here using macrocycles. 

Nevertheless, the model generates a density as its output, suggesting that an inverse problem needs to be solved post hoc to get a molecular structure, not unlike fitting of experimental data such as CryoEM densities. This fact may prove to be a significant computational overhead for larger systems.
Decision: Accept (poster)