Pocket-specific 3D Molecule Generation by Fragment-based Autoregressive Diffusion Models
Authors: Xingang Peng
Jiaqi Guan
Jian Peng
Jianzhu Ma
Venue: Submitted to ICLR 2023 (2023)
Keywords: 3D molecule generation
drug design
protein binding pocket
generative model
diffusion model

Abstract: Autoregressive model is widely adopted to generate 3D molecules which can fit any protein binding pocket. Current autoregressive model suffers from two major drawbacks. First, it is hard to  capture local geometric patterns as only one atom is generated at each step. Second, most of the autoregressive models generate atoms and chemical bonds in two separate processes, which causes a number of problems such as incorrect counts of rings, a bias distribution of bond lengths, and inaccurate 3D molecular structures. To tackle this problem, we designed a model, named FragDiff, to generate 3D molecules fragment-by-fragment for pockets. In each generation step, FragDiff places a molecular fragment around the pocket by using E(3)-equivariant diffusion generative models to simultaneously predict the atom types, atom coordinates and the chemical bonds of the fragment. Extensive experimental results confirm our assumption that unifying the atoms and bonds generations could significantly improve the quality of the sampled 3D molecules in terms of more accurate distributions of 2D subgraphs and 3D substructures.

Under review as a conference paper at ICLR 2023
POCKET-SPECIFIC 3D MOLECULE GENERATION BY FRAGMENT-BASED AUTOREGRESSIVE DIFFUSION MODELS
Anonymous authors Paper under double-bling review
ABSTRACT
Autoregressive model is widely adopted to generate 3D molecules which can fit any protein binding pocket. Current autoregressive model suffers from two major drawbacks, First, it is hard to capture local geometric patterns only one atom is generated at each step. Second, most of the autoregressive models generate atoms and chemical bonds in two separate processes, which causes number of problems such as incorrect counts of rings, a bias distribution of bond lengths, and inaccurate 3D molecular structures. To tackle this problem, we designed model, named FragDiff, to generate 3D molecules fragment-by by-fragment for pockets. In each generation step, Frag places a molecular fragment around the pocket by using E(3)-equivarian) diffusion generative models to simultaneously predict the atom types, atom coordinates and the chemical bonds of the fragment. Extensive experimental results confirm our assumption that unifying the atoms and bonds generations could significantly improve the quality of the sampled 3D molecules in terms of more accurate distributions of 2D subgraphs and 3D substructures.
INTRODUCTION
Drug design has been greatly improved with the assistance of AI (Stokes al. 2020 Zhavoronkov ett al. 2019). Insilico Medicine recently announces the world's first drug designed by AT has entered Phase I clinical trial (Zhavoronkov al., 2019). Al-based drug design has experienced several important stages. The first generation of methods focus on generating molecule graphs by leveraging multiple graph representation techniques (Tin et al. 2018a b). Later, researchers realized that the biochemical functions of a small molecule is partially determined by its 3D structure so new models are proposed to directly sample molecular drugs in the 3D space (Hoogeboom et al.] [0022] Wu let al. 2022). Recently, an increasing amount of generative models have been developed to generate molecules which can bind to the target protein based on the 3D structures of the binding pockets, A straightforward approach is to encode the geometric features of amino acids the protein pockets and then translate them to a molecule (Skalic et al. 2019 Xu et al. 2021a). The central problem of this end-to-end approach is that it does not explicitly characterize the interactions of atoms between molecules and pockets. Although involving the pockets, the structure complex of the target protein and molecules are missing so it is hard quantify whether these molecules can dock into the desired pocket. To solve this problem, new models have been proposed to capture the atom-level interactions between molecules and pockets by directly sampling 3D molecules inside the 3D pockets (Masuda et al. 2020; Luo et al., 2021a la Liu et al. 2022, Peng et al. 2022). However, in comparison to the pocket-free generation, pocket-specifte models are still an early stage and suffers from quite a few problems.
Most pocket-specific models rely on the autoregressive process to generate molecule. The atoms are placed one by one in the pocket and the chemical bonds are predicted by a separate model. This procedure often leads to inaccurate bond predictions and unrealistic 3D structures. For instance, needs six steps generate a benzene ring, which is unnecessary and error-prone. A natural solution is to adopt fragment- based generation approach. However, generating fragments is hard because the model has to simultancously capture the relationship of more atoms and bonds. The diffusion models have achieved the state-of-the-ar performance in various tasks (Ho et al. al,(00 including 3D molecule generation (Luo et al. 2022 Xu et al. 2021b Hoogeboom al. 2022 Wu et al. 2022

---

Under review as a conference paper at ICLR 2023
Diffusion models learn the data distributions by randomly adding noise and recovering the denoised data points. Empirically, it is nontrivial to directly apply the diffusion process to the pocket-spectie molecule generation. It is hard to keep the local geometric constraints of a sampled molecules (e.g., carhon atoms in a ring on the same plane) as the complex data is still limited.
To address all these problems, design new paradigm, named FragDiff, to design pocket-specific 3D molecules by integrating both the autoregressive and diffusion generative processes. One key ob- servation is that a molecule usually contains multiple 3D fragments with higher frequency to appear inside a particular 3D pocket compared to other fragments. Therefore, we adopt the autogressive process to generate molecules at the fragment level instead of the atom level. Fragment is generated based on the diffusion models instead of extracting from manually-curatee the autoregressive process only need to learn to place relatively small amount of elements and the diffusion models only need to learn how to generate relative small amount of atoms such that local geometric and chemical constraints are easily captured. It also helps the diffusion model to capture the atom interactions the interactions within each fragment are much denser and stronger than outside.
databases. In this way,
2 RELATED WORK
Pocket-specific molecule generation Early pocket-specific molecule generation method is to en- code the pockets as latent embeddings and translate them into new molecules (Skalic et al.) 2019) Xu et al. 2021a). Such methods usually represent the output molecules as ID strings or 2D graphs, which cannot explicitly capture the interactions between atoms. Advanced approaches focus on si- multaneously generating 2D graphs and 3D structures of the molecules given the pockets. et et al. (2021) designs a ligand network for this task by using docking scores and Monte Carlo Tree search algorithm. The model infers the interaction between pockets and molecules using Auto Vina (Eberhardt al., 202T) rather than from the observed data, and its performance strongly relies on the accuracy of Dock. MMsuda et al. (2020) utilized 3D Convolutional Neural Network (CNN) to capture the spatial information and leveraged CVAE to sample 3D molecules. The major draw- back is the weak expressivenees and the generated molecules are found to possess poor chemical and drug- properties (Peng et al. 2022). Autoregressive models have been proposed to solve this problem and achieve SOTA performance. [uo et al. (2021a) first propose to learn the probability densities of different atoms the 3D space inside the pockets and place atoms based on the learned distributions. Liu et al. (2022) construct local frame to place atoms around the pockets. |Peng et al.) (2022) further utilized E(3) E33)cqqivariant GNN and more efficient sampling scheme to predict the atoms and chemical bonds. However, these methods all adopt the atom- by-o-my-ttom generation strategy, which may lead to unrealistic subgraphs and inaccurate local 3D structures.
Fragment-basee molecule generation Molecules usually contain multiple frequent functional groups. There are quite few works using fragments for pocket-free molecule generation. Jin et al. (2020) design hierarchical generation models which first generated structural motifs and then decode to atom sets. [Podda et al.] (2020) utilize GRU to generate SMILES of fragments, and Xie et al. (2021) employ MCMC algorithm to iteratively edit these fragments. Recently, [Powers] jet al. (2022) designs a fragment- -based generation model for pocket binding. However, they rely on prior domain knowledge and over-simplified assumptions. Their model can only select from 28 manually-curaae fragments which are connected by single bonds. Generating fragments from a pre- defined fragment vocabulary significantly limits the generation abilities. Another limitation is that these methods do not consider the 3D) structures of the generated fragments. These two drawbacks motivate us develop new models which not only generates arbitrary fragments leamed from data but also determines the 3D coordinates of the fragments.
Diffusion model for small molecule Although there is no diffusion-based model for pocket- specific molecule generation, it has been already applied to multiple pocket- free molecule design tasks (Xu et al. 2021b; Wu et al. 2022; Hoogeboom et al. 2022 Jing et al.) 2022). For instance, GeoDiff applied a model trained based on the diffusion technique to predict the 3D conformations of molecules (Xu et al. 202 Hoogeboom et al. (2022) utilizes equivariant diffusion model to gen- erate small 3D molecules, and Wu et al. (2022) integrates physical prior bridges into the diffusion process for molecule generation. In comparison to pocket-free setting, adding the pocket informa-
2

---

Under review as a conference paper at ICLR 2023
Autoregressive generation for molecule
p(Fml D(FmlPP,oe Mm
Diffusion generation fragment
t=T
[=T-11
c=T-2
Pa 1|, P. P,M)
Figure 1: The molecule generation process
tion usually significantly decreases the search space of small molecules and training data. Besides, previous diffusion-basse based molecule generation models ignored bond generation, which may lead to biased atom connections in the molecules.
3 METHODS
3.1 DEFINITIONS AND NOTATIONS
protein pocket represented as a set of atoms and denoted as {(a, (10)) r (1(00))) N where (mae) and (pro) are the ith atom's element type and coordinate, respectively. The molecules and frag- ments represented by the atoms and bonds, denoted as M ((a!mm))) , (mma)) b(mal) (frag) (frag)
) ) and
F = {(a r b, (frra) )
gafing where the symbol a; and r; are the ith atom type and its coordi- nate, and b; is the bonds between the ithe atom and other atoms. Note that (b M Ninhh represents Jim1 represent both the bonds among fragment
the bonds among the molecular atoms while {6,th (fing) wh atoms and the bonds between the fragment atoms and the molecular atoms that the fragment nects to. The pocket-specifie molecule generation task to generate 3D molecules that can bind any protein pocket. Formally, given pocket P, the model generate 3D molecules M from the learned distributions (e(M)) where θ is the parameter sets the model has to learn.
3.2 MOLECULE GENERATION
3.2. GENERATION PROCESS
The molecule generation process is in an autoregressive manner, i.e., by placing fragments around the protein pocket one by one (Fig. [ In each step, conditioned pockets and the previously generated molecule, the model generates fragment by predicting its atom types, atom coordinates, the bonds within the fragment, and the bonds that connect the fragment and the molecules. Formally, the generation process with M steps is denoted as
p(MP) =
(1)
In the mith step. the model predicts
p(FmlPP,M
(2)
where Mm-1 is defined as Mm-1 UF-m- This probability is predicted by diffusion proba- bilistic model and focal atom predictor. The focal atom predictor determines which atom of the molecule F&m protein P that the fragment will be generated around, and the diffusion model generated the fragment F., around the focal atom.
3.2.2 DIFFUSION FOR FRAGMENT GENERATION
we omit We used diffusion beetusioo--beedd generative model to learn the distribution P(F(P. (Eq. 2
the subscript of autoregressive step in the section). The diffusion probabilistic model defines two
3

---

Under review as a conference paper at ICLR 2023
Markov processes, i.e., forward diffusion process which gradually adds noise to the fragment, and an reverse generative process that denoises the fragment. The fragment at step of the Markov
process is F = ={a,, r b Mammg
At 0, the fragment is the ground-truth, i.e., F0 F
{a (at (frag)
b (ffgg) 1 Nilm
Forward process In the forward process, the Markov transition at step is defined as:
q(F1|r111) d(taf, r/, b |
II Tulafl="-)a(rr )0(())=""))((00110" )a(b/(=(-1)
II alabla (r((lr)) ()b(()) II
(3)
Here we make an assumption that atom types, bonds and coordinates step & only depend on their own state at step - 1. More complicated diffusion process could be defined by letting atom types depend bonds and coordinates in the previous step. We leave this generalization in the further work. In the diffusion step, we add Gaussian noise with variance 8 to the atom coordinate (Ho al. 2020) and define the categorical distributions with probabilities of .3 and Food uniformly resample from element types and bond types, respectively (Hoogeboom et al.) (00T). If we represent the element types and the bond types as one-hot vectors, the diffusion step can be defined as:
q(at]a) = C(a))( - )a + / Keee 1)
=
= =N(ril B 1 1 166,,)017
(4)
+ Krond 1)
where N and C represent Gaussian distribution and categorical distribution, respectively, Kiete) and K(bond) are the numbers of element types and bond types respectively. and I are the all-one vector and identity matrix respectively. Note that the forward process has no learnable parameters. According to Eq. 3 and Eq. 4 as > co, the atom coordinates converge to the standard Gaussian distribution and the element types and bond types converge to the uniform distributions.
Reverse process In the reverse process, the Markov transition at step t is defined as:
PF
Ft, F P,M) po po(1a -, b -- (F.P.
II pe. (a 1|,,, P, P.M)po (r) - MMIL 1
PFM,
(5)
where pe, (Ftt P.M), (r. FF P.M), and po, ( are all parameterized through neural networks, whose architecture and training loss will be described in Sec. B33 and 3.4 We choose large T and used the reverse Markov process generate the fragment follows,
pa Pa (M, =Po -M
(6)
D P.MM)
where P(F'IP,,M .M) is defined as a standard Gaussian distribution for fragment atom coordinates and categorical distributions for atom element types and bond types. Technically, we move the focal atom to the origin of the coordinate system the beginning so that p(F IP, M) E(3)-equivariant We also set maximal number of atoms of a fragment can contain and define a dummy element type for the fragment atom, so that the model can generate fragments with variant sizes.
3.3 MODEL ARCHITECTURE
In this section, introduce how define equivvaina graph neural network parame- terize the reverse generation model pa (F [ P. (Eq. 5) in Sec. 3.3.1 and other auxiliary
models in Sec. 3.3.2
4

---

Under review as a conference paper at ICLR 2023
3.3.1 REVERSE GENERATION MODEL
Inputs The input of the model is the fragment step t, the molecule and the pocket, i.e., {F', P,M.. The initial features of the protein atom are the element type, the amino acid type it belongs to, and an indicator representing whether it is on the protein backbone. The initial features of atoms are their element types. The atom features of the fragment, the molecule and the pocket are individually embedded to the same dimension using three embedding layers and are composed together as vertex Then we build two sets of edges for the vertices, denoted as E and C. The set E. contains edges of nearest neighbors for all vertices, and the edges are initially represented by the edge distances and the types of the two ends, denoted eij. The set C contains potential bonds where one ends are fragment atoms and the other ends are the fragment atoms or molecule atoms. The bonds are initially represented by their lengths, atom types of two ends, and bond types (including single bond, double bond, triple bond and none), and passed through an embedding layer get the hidden vectors denoted Cij- Note that the hidden vectors Cij will also be updated simultaneously with vt as described in the following, because they will be utilized for bond type prediction.
Message passing We use message passing to update the hidden vectors of vertices v, and bonds Ciy. Intuitively, the vertex feature is updated using messages calculated from the vertices connects to, and the bond feature is updated using messages from the other bonds that share the same vertices with it. The updating algorithm is defined as:
0. < (1:) +
Σ (K)EE
(e((
+
Σ (ij)cC
Es cis)
Cij c wh (Cij. Vi, 03)) +
Σ
(C 1x)) +
Σ (a(C)))
(K,))
(LA)EC
where & and &a represent different fully-connected neural networks
Outputs After L layers of message passing, the model finally used the updated hidden vectors v and Cij to predict the denoised fragment, including the categorical distributions of atom element types à; and bond types b.g and the mean of the Gaussian distribution of the atom coordinates A, as follows,
à, = (v()))
by = 6يt (2(cy + cjs);
= AA:
AA; = Σ (L.))C
pa(u; U. cij)(r; - r() + Σ (L,))E
( Uj ;y (r)) - rs)
(8)
where the symbol & represents neural networks and r; are the input coordinates of vertex i. The outputs of (P1 and (P2 are the probabilities of element types and bond types, respectively, while the outputs of (0 and 12 are scalars which work like "force" that drag the atom i towards atom (Luo al. (202Tb] Guan et al. 202T). Such a design also keeps the E(3)-qquivariance, which approved in the Appendix A.3
3.3.2 AUXILIARY MODELS
Focal predictor Before the diffusion model generates the fragment, we use a focal predictor to determine around which atom the new fragment should be placed. And all atoms should be translated so that the focal atom is located at the origin before the diffusion begins. The focal predictor is trained together with the diffusion model. After masking, the unmasked atoms that are connected to the fragments are defined as focal atoms. If all molecular atoms are masked, the protein atoms that are the closed to the fragment atoms are defined as focal atoms. The focal predictor uses the message passing defined in Eq. to learn the features of atoms and use two MLPs to classify the focal atoms for protein atoms and molecular atoms.
Molecule discriminator To obtain more information about the molecule structures, we build and train a molecule discriminator on a pocket-free 3D molecule dataset to classify whether a molecule a real one. After the generation of new fragments, the molecule discriminator is used to score the generated molccules, and the molecules with low scores are dropped. Once an unrealistic interme- diate molecule occur, it will be filtered out immediately. The molecule discriminator is trained in
5

---

Under review as a conference paper at ICLR 2023
the contrastive manner: for a real molecule, create fake counterpart by adding noise. Then train the discriminator to discriminate the two molecules. The molecule discriminator uses the same message passing defined in Eq. (8 learn the features of all atoms and bonds, two max pooling lay- ers to aggregate atom-level and bond-level features, and a final MLP to output a score representing the probability of being real molecule.
3.4 TRAINING
3.4.1
TRAINING LOSS
The overall training process is to mask some parts of the molecules and train the model to recover the masked molecules seameent-y-fragment More specifically, given the structure of a pocket- molecule complex, first decompose the molecule into molecular fragments (described in Ap- pendix A. and then masked fraction of fragments. The ratio of masked fragments is randomly sampled from uniform distribution (40, 1) and is different for different training epochs. Next we choose the masked fragment that originally connected to the unmasked atoms for the model to recover.
The diffusion model learned the reverse generative process (o P, M) by approximating the true posterior F defined in the forward diffusion process, i.e. by minimizing the KL divergence between the two distributions:
Lail = F,g [Dx
(F PPMM
which can be decomposed into three terms:
Lait = Lele + Leoor + Prond
Lele = Eg
Σ DKrlula, Mat aa)t,,))lze llo,, (ap (at- P. MI]
Looor = Ey
P "re (r: 1 P, Mj]
(10)
Lrond = Ey
DKIla 69)) lpp,, (b P, 1j
For the loss of coordinates, of the predicted mean the Gaussian distributions and the true coordinates at step 1, i.c., Laver = Eng lr! 12.
the simplified version Ho et al. 2020), i.c., the MSE loss
The focal predictor uses the binary enssopy loss Liee and is trained together with the diffusion model. During training, randomly sample time step and optimized the total loss Lair + Lixe
The molecule discriminator is trained using contrastive loss. More specifically, let Dreal and Treal be the predictions for the real and noisy molecules, the loss is defined Lmot [maxx Dreat + 0.5, 0)].
3.4.2 PRETRAINING ON THE POCKET-FREE DATASET
The number of mocket-molecule complex pairs in the pocket-specific data is limited compared to numerous 3D molecules (i.e., pocket-free molecule data). To enable the model to learn more infor- mation about the distribution of the 2D and 3D structures of molecules, we pretrain FragDiff on the freket-free dataset and then finetune on the pocket-specifie dataset.
4 RESULTS
4.1 SETUP
Following previous work (Luo et al. 202 Liu et al. 2022 Peng et al., 2022),, we use the protein-
molecule structure dataset Cross ked Francoeur al. (0000) and the same training and test

---

Under review as a conference paper at ICLR 2023
(a)
KL divergence with training set
(b)
ARings per
#Rings bond
#HHu aloms
#Rings one stom exists
Ring types
0.08
CVAE
1.0236
0.8072
0,2961
0.1606
1.6167
0.05
5000
0.04
SBOD
0.9085
0,5138
0.2017
0.0423
0.4163
Graph@P
3.0896
2,5693
1.4876
0.1740
2,6646
0.03
Pockel2Mail
0,8219
0,4348
1,0753
0,1488
0:4309
0.02
FragDin
0.3545
0.3480
0.0155
0.0699
0.1894
0.00
Training sot
0,0000
0.0000
0.0030
Geom set
0.1595
0.1384
0,0201
0,0109
0.2152
0.00
6
10
12
14
1.6
20
(c)
(d)
#Rings per moleccule
10-1
Mocel
10"4
Moddtt/ SMt
CVAC
Log of densky
Log of densty
10n
10
to
20
Fagghn
30a
10
11
#Hub atoms per molecule
#Rings one atom exists
The analysis of distributions of rings. (a) The KL divergence of different ring-related metrics between the generated molecules and training set. (b) The distributions of counts of rings per molecule. (c) The distributions of counts of hub atoms per molecule. (d) The distributions of counts of rings that an atom exists in.
Figure 2:
splicting. For the pocket-free dataset used for pretraining and molecule discriminator, we use the Geom-Drug dataset (Axeirod & Komez--Bobareed (022) which contains around 300,000 drug- like 3D molecules. We benchmark FragDiff with CVAE(Masuda al. 2020), SBDD et al.) 202 GraphBP(Liu et al. 2022), and Pocket2Mol(Peng et al., 2022). CVAE uses 3D CNN to en- coded the pocket and conditional VAE generate 3D molecules. SBDD, GraphBP and Pocket2Mol are autoregressive models. The
and training details of FragDiff could be found in Appendix A.22] We choose ten pockets in the test set and sample 100 molecules for each pocket. In the following section, we analyze the 2D and 3D substructures of the generated molecules and com- pare the distributions with the molecules in the training set. We also conduct the same comparison between the Geom-Drug dataset and the training set for reference.
4.2 ANALYSIS OF SUBSTRUCTURES
We show visualization of the generation process of FragDiff in Fig. in Appendix where the molecule is generated by sequentially adding fragments. As discussed previously, current models cannot fully capture the local substructures and were inclined to erroneously connect atoms. Since all molecules are connected graphs, incorrect chemical bonds usually lead to the biased ring distri- butions and form unrealistic rings. Therefore, to evaluate the different approaches, we analyze the rings in the sampled molecules. We count the number of rings in each molecule. Since the number of rings is related to the molecule size, we normalize the number of rings by the numbers of atoms and bonds and compare the distributions of generated molecules with the training set. As shown in Fig. a the distributions of the number of rings of our method are more consistent with the training set than other baselines. As shown in Fig, [b),, GraphBP and Pocket2Mol tend to generate excess rings, which caused by adding bonds in a greedy way to connect atoms.
In addition, we also find such erroneous generation may result in incorrect stack of rings. For instance, too many predicted bonds can form fishing-net like subgraphs. To check this issue, we conduct two experiments: First, if the molecule contains fishing-net like subgraphs, there must be at least one atom existing in three different rings. We define such atoms as "hub atoms" and calculate how many hub atoms each molecule contains. Second, to measure density of fishing-net subgraph, we calculate how many rings an atom exists in, i.e., counts of rings one atom exists in. We showed the distributions of the above two metrics in Fig. 2 c-dd) and calculate the KL divergence with the training set (Fig. Aa)). Overall, FragDiff shows the most similar distributions with the training set, indicating that molecules generated by FragDiff share more realistic substructures than baselines.
7

---

Under review as a conference paper at ICLR 2023
(8)
Length
Angle
Dh.Agg
Langh
Angle
Arr
Length
Angle
Dih.Agg
Lungh
Angle
ChiAng
Length
Angle
CVVE
3.81
3.28
4.14
6.74
3.03
0.41
3.81
3.28
4.14
5.70
1.93
2.64
7.52
4.60
2:13
S0DD
0.90
0.97
9:24
2:14
0.84
0.21
0.90
0.97
9.24
2.79
0.78
5.37
GraphBP
0.34
0.19
0.63
2.18
4.28
2.45
0.34
0:18
0.63
Poc
2.11
1.19
0.32
1.26
FregDif
0.24
0.04
0.78
0.54
0.09
0.11
0.24
0.04
0.78
0.35
0.25
0.61
1.04
0.21
1.27
Train
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Geom set
0.36
0.01
0.00
0.82
0.02
0.09
0.36
0.01
0.00
0.47
0.02
0.00
0.25
0.08
0.00
Bond Lengths
Bond Angles
Dinedral Angles
1.75
150
1.5
125
1.25
100
75
CAAE
Graphep
PockattMol
Priegate
Thai sot
Gauor sot
CURE
5800
Grantop
PorkettMMa
Froptar
Irans sat
CHAS
s800
Grapistr
Pach Pacher2Mol
Ragtor
Geem ses
Grom
Rain
Model /Sot
Model /Sat
Modal /Ber
Figure 3: The analysis of 3D distributions of rings. (a) The KI. divergence of bond lengths, bond
angles, and dihedral angles for each ring type. (b) The distributions of bond angles, bond angles and dihedral angles of benzene rings.
On the contrary, the baselines GraphBP and Pocker2Mol generate molecules that contain more than four shared atoms, which is rare in both the training set and the Geom set.
We next analyze different types of rings and their 3D structures. We retrieve the top 20 frequent ring types in the training set and then calculated the ratios of the 20 ring types the generated molecules of different methods (Fig. in Appendix A..4). CVAE and GraphBP generate a large number of rings with only three carbon atoms but extremely small number of benzene rings. We calculate the KL divergence of the distributions of the 20 types of rings to the training set (the last columns in Fig. a)), and find FragDiff has the most similar distributions with the training set. For top ten frequent rings, we analyze their 3D structures by calculating the bond lengths, bond angles and dihedral angles and compare their distributions with training set. We show the KL divergence of their distributions in Fig. 3 and Fig. 8 in Appendix A.4 and plot the distributions of benzene rings as an example in Fig. 3 Overall, the distributions of Frag are the most similar with the training set, indicating that the molecules generated by FragDiff have more accurate 3D substructures. In the end, we expand the 3D structure analysis to chemical bonds. We analyze the bond lengths of the most frequent bond types (Fig. 9.. which also demonstrate that FragDiff generated more accurate bond lengths.
4.3 ABLATION STUDY
There are not too many components in our framework. We conduct the ablation study to find whether the pocket-free data contributes to the generation. Apart from the original FragDiff pretrained on the geom dataset, we further construct three ablation models: 1) No pretraining: the model directly trained on the pocket-specific dataset; 2) No pretrain & no discriminator: the model directly trained on the pocket-specific dataset and does not use the molecule discriminator to filter during sampling; 3) Guidance: a model that used different methods incorporate the pocket-free information. More specifically, apart from diffusion model directly trained on the pocket- -specffcc dataset, we train an additional diffusion model on the pocket-free dataset. During sampling, the probabilities of the fragments (i.e, the probabilities of the atom element types, atom coordinates and the bond types) predicted by the two diffusion models at each step are weighted averaged, which is inspired by the classifier-free free diffusion models (Ho & Salimans 2022). The weights for the pocket-specific models and the pocket-free models are set as 0.8 and 0.2, respectively. The sampling process also adopts the molecule classifier to filter molecules.
Our conclusion is that a proper use of pocket-free data can enhance the quality of 3D substructures. Fig. A shows four pairwise comparisons where the scatters in the figures represent the KL diver- gence of the 3D features (i.e., bond lengths, bond angles, and dihedral angles) of different rings and the bond lengths of different bond types. First of all, both pretraining the pocket-free dataset and
8

---

Under review as a conference paper at ICLR 2023
No. pretraining
No protraining
Guisdannce
4 ablation
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
abistion
pratraining
pretraining
Figure 4: Comparison of KI. divergence between different ablation models.
Wha Soason to
Whe Wre
g 9.5
Tregrr
haggr
5800
Goppar
CIE
S000
Paggle
Tee set
Tett
Mood
Model
Figure 5: The distributions of QED, SA, Vina score and Vina rmsd of molecules
using molecule discriminator for sampling filtering beneficial for the 3D structures of the gener- ated molecules, shown in the first and the last subfigures. The second subfigure also indicates that pretraining the model on the pocket-free dataset more effective than training a separate diffusion model the pocket-free dataset. The third subfigure exhibited that the guidance model showed almost the same performance as the model without pretraining, indicating that training a separate diffusion model the pocket-free dataset might not provide extra information for better molecule generation.
4.4 EVALUATION OF OTHER PROPERTIES MOLECULES
Following (Peng et al. 2022). also calculated the QED (quantitative estimation of likenli) and SA (synthetic accessibility score) of the generated molccules and used AutoDock Vina (Eber- found all these metrics have
hardt et al. 2021) to dock the molecules to the pockets. However, specific bias. We combined the generated molecules of all these models and those in the test set and found that the SA score the molecule is correlated the count of atoms and the count of bonds (correlation coefficients are -0.553 and -0.546). The Vina score is highly related to the counts of rings, counts of bonds and counts of atoms (correlation coefficients are 0.713, 0.678 and 0.628). As unalyzed above, some baselines tend to generate molecules with excess rings, and thus the Vina scores will high. Therefore. these metrics are actually not suitable for molecule evaluation real applications and can only be used as reference. As shown in Fig. 5 we showed the distributions of the these metrics, and found that the distributions of FragDiff was similar to the test set.
5 CONCLUSIONS AND DISCUSSION
Generating 3D molecule for protein binding pocket important for drug design. We designed fragment-based autoregressive diffusion model for this task. The model generates the molecule by sequentially placing the molecular fragments in the pocket, and for each fragment generation the model leverages diffuuion probabilistic model to simultancously predict the atom types, atom coor- dinates and bond types. We then detailedly analyzed the 2D and 3D substructures of the generated molecules, and found that FragDiff generated more accurate substructures than baselines.
REFERENCES
Simon Axelrod and Rafael Bomarerlll...lli.. Geom, energy-annotand molecular conformations for property prediction and molecular generation. Scientific Data, 9(1):1-14, 2022.

---

Under review as a conference paper at ICLR 2023
Jerome Eberhardt, Diogo Santos-Martins, Andreas F Tillack, and Stefano Forli. Autodock vina 1.2. O: New docking methods, expanded force field, and python bindings. Journal of Chemical Information and Modeling, 61(8):3891-3892 2021.
Paul G Francoeur, Tomohide Masuda, Jocelyn Sunseri, Andrew Jia, Richard B lovanisci, Ian Snyder, and David R Koes. Threc-dimensional convolutional neural networks and cross-docked data set for pruccure-based drug design. Journal of chemical information and modeling, 60(9))4200- 4215, 2020.
Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGilinchey, David Michalovich, Bissan Al-Lazikani, al. Chembl: large-scale bioactivity database for drug discovery. Nucleic acids research, 40(D1):D1 100-D1
1000)) 107, 2012.
Jiaqi Guan, Wesley Wei Qian, Wei- Ying Ma, Jianzhu Ma, Jian Peng, al. Energy-inspiree molecu- lar conformation optimization. In International Conference on Learning Representations, 2021.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 12454-12465. Curran Asso-
ciates, Inc., 2021. URL htttp:t///o:occeeeee wewpeediiinnnnneniiieeeeee neurips wwtppapper/222//11ee.... Paper.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant dif- fusion for molecule generation in 3d. In International Conference on Machine Learning, pp- 8867-8887. PMLR, 2022.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference machine learning, pp. 2323-2332. PMLR, 2018a
Wengong Jin, Kevin Yang. Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to- graph translation for molecule optimization. In International Conference on Learning Represen- lations, 2018b.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International conference on machine learning, pp. 4839-4848. PMLR, 2020.
Bowen Jing, Gabriele Corso, Regina Barzilay, and Tommi S Jaakkola. Torsional diffusion for molec- ular conformer generation. In JCLR2022 Machine Learning for Drug Discovery, 2022.
Yibo Li, Jianfeng Pei, and Luhua Lai. Learning to design drug molecules in dice-dimmnsiooal space using deep generative models. arXiv preprint arXiv:2 100.20444404444 2021.
Meng Liu, Youzhi Luo, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji. Generating 3d molecules for target protein binding. arXiv preprint 2XXi:22244009100, 2022.
Shitong Luo, Jiaqi Guan, Jianzhu Ma, and Jian Peng. A 3d generative model for structure-based drug design. Advances in Neural Information Processing Systems, 34, 2021a.
Shitong Luo, Chence Shi, Minkai Xu, and Jian Tang. Predicting molecular conformation via dy- namic graph score matching. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 19784-19795. Curran Associates, Inc., 2021b. URL ht / mew.eeereddinnn..nns............
10

---

Under review as a conference paper at ICLR 2023
Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang, Jian Peng, and Jianzhu Ma. antibody design and optimization with diffusion based generative models. bioRxiv, 2022. doi: 10. 1101/2022.07110449951 10. URL https: www. www.:iiiiiiivv.... org/ http///eeeeeeee///222222 07/11/2022.07. 10. 499510
Tomohide Masuda, Matthew Ragoza, and David Ryan Koes. Generating 3d molecular struc- tures conditional on receptor binding site with deep generative models. arXiv preprint 1xXiv:2010.14442. 2020.
Xingang Peng, Shitong Luo, Jiaqi Guan, Qi Xie, Jian Peng, and Jianzhu Ma. Pocket2Mol: Effi cient molecular sampling based on 3D protein pockets. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning .mlr.
Research, pp. 17644-177777 17655. PMLR, 17-23 Jul 2022. URL
press/ vl 62 wwt.p/2ppppb....mm html
Marco Podda, Davide Bacciu, and Alessio Micheli. A deep generative model for fragment-based molecule generation. In International Conference on Artificial Intelligence and Statistics, pp- 2240-2250 PMLR, 2020.
Alexander Powers, Helen Yu, Patricia Suriana, and Ron Dror. Fragment-based ligand generation guided by geometric deep learning on protein-ligand structure. bioRxiv, 2022.
Miha Skalic, Davide Sabbadin, Boris Sattarov, Simone Sciabola, and Gianni De Fabritiis. From target to drug: generative modeling for the multimodal structure- based ligand design. Molecular pharmaceutics, 16(10):4282 4291, 2019.
Jonathan M. Stokes, Kevin Yang. Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz, Nina M. Donghia, Craig R. MacNair, Shawn French, Lindsey A. Carfrae, Zohar Aoomm-Acermmnnn, Vic- toria M. Tran, Anush Chiappino-Pepe. Ahmed H. Badran, Ian W. Andrews, Emma J. Chory, George M. Church, Eric D. Brown, Tommi Jaakkola, Regina Barzilay, and James J. Collins. A deep learning approach to antibiotic discovery. Cell, (0804)0688-702220)) feb 2020. doi: 10. 1016//.cell.2020.00 202.0.00...20...0.22... /tLhhttts::///tt /ttp://////001111110111111111110
1 2020 01. 021
Lemeng Wu, Chengyue Gong, Xingchao Liu, Mao Ye, and Qiang Liu. Diffusion-based molecule generation with informative prior bridges, 2022. URL httttp:://axxi...tttt http://///xxxv..oo//bb.ss.. wbwp:/ggaab/////0// 00665
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. arXiv preprint arXiv:2103. 10432, 2021.
Mingyuan Xu, Ting Ran, and Hongming Chen. De novo molecule design through the molecular generative model conditioned by 3d information of protein binding sites. Journal of Chemical Information and Modeling, 61(7):3240-322 2021a.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geo- metric diffusion model for molecular conformation generation. In International Conference on Learning Representations, 2021b.
Alex Zhavoronkov, Yan A. Ivanenkov, Alex Aliper, Mark S. Veselov, Vladimir A. Aladinskiy, Anas- tasiya V. Aladinskaya. Victor A. Terentiev, Daniil A. Polykovskiy, Maksim D. Kuznetsov, Arip Asadulaev, Yury Volkov, Artem Zholus, Rim R. Shayakhmetov, Alexander Zhebrak, Lidiya I. Mi- naeva, Bogdan A. Zagribelnyy, Lennart H. Lee, Richard Soll, David Madge, Li Xing, Tao Guo, and Alán Aspuru-Guzik. Deep learning enables rapid identification of potent DDR1 kinase in- hibitors. Nature Biotechnology, 3709
sep 2019. doi: 10. 10336641587-009-02224 0ttp:pppp///ooooooo//////0222111

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 FRAGMENTATION OF MOLECULES
We introduce how we break a molecule into fragments.
First of all, we pre-defined fragment vocabulary set. Note that this fragment vocabulary is only used for fragmentation and training but not for generation. For all molecules in the Chembl dataset Gaulton et al.. (2012). we extract all the rings and add them into the fragment vocabulary. Then for each connected graphs the remaining acyelic molecular graphs, we enumerate all trees given a fragment size and add them into the vocabulary. Finally, we count the frequency of unique fragments in the vocabulary sets and drop out the rare fragments.
After constructing the fragment vocabulary set, can decompose molecule into fragment and the maximum fragment size. First, we extract all rings out. Second, for the rings and the remaining connected graphs that are larger than the maximum fragment size, we recursively break the bond so that the two fragments after breaking the bond have the maximum total frequency in the vocabulary. In this way, the molecule is composed into many fragments.
A.2 HYPERPARAMETERS THE MODEL AND TRAINING
In the diffusion model, the number of message passing layers is 6. The hidden dimensions of vertices and bonds are both 256. The hidden dimension of edges is 64. The number of nearest neighbor for edge calculation is 32. For the focal predictor, the difference is that the number of message passing layers is and the hidden dimensions of edges are 32. For the molecule discriminator, the difference from the diffusion model is that the hidden dimension of edge is 128.
For the diffusion parameters, we choose the number of time steps as 3000. The schedulers of all element types, coordinates and bond types are sigmoid Hoogeboom et al. 202 and the 3s at the start and end steps are x 10-7 and 2 x 10-3
We train FragDiff with learning rate 1x using Adam optimizer. The batch size is set 32. We pretrain FragDiff on the pocket-free dataset for 100,000 iterations and finetune on pocket-specific dataset for 120,000 iterations.
A.3 E())-EQUTVARIANCE OF THE DIFFUSION MODEI.
It is straightforward to prove the E(3) ((3-evvvvvae of the generation model pa Assume T is some transformation E(3) group and are applied to (F', P.M. The input features and the updated hidden vectors in Eq. remain invariant, thus the predicted element types and bond types are invariant. The predicted coordinates under the transformation are defined as:
=Trr+ + Σ U, (K.))C
Ug ci) -(rr))+ + Σ pa(u,, U, (T(r) - -[(r:))
(NE)
= -T(r:)+ + Σ
pa Co T(r, - -ri]+ Σ
p Uj eij Tir - rr:
(LJ)EC
(t.))R
(11)
= =T(r: + Σ
u,. -r)) + Σ
p4(0; Uj enj)(r r:)l
(L))
(iJ)ER
= -T(A
which indicates that po equivariant to the inputs.
P, = Po P, P,M), i.e., the outputs are E(3)-
A.4 MORE ANALYSIS OF MOLECULES
12

---

Under review as a conference paper at ICLR 2023
Figure 6: The generation process of molecule generated by Frfg
Model Set
CVAE
S6DD
GraphBP
Ratio
Pocket2Mel
FragDiff
Train set
0.0
clocccel
cIccocl
CICCCCC1
clenencl
ciclnHlen1
clsenecl
ClsccCC=CI
CICCNCCI
clccCC1
CICCNCI
ClCCCCCC1
clcnfnHHjJc
CICC1
C1=CNCNC1
cICCOCci
deacc
clcmnHkc1
CI=CONC=C1 CI#NCON1
Smiles of rings
Figure 7: The ratio of the 20 ring types of different models sets.
Length
Angle
Dih.Ang
Length
Angle
Dih.Ang
Length
Andle
Dih.Ang
Length
Angle
Dhh,ng
Length
Angle
DhhAng
CVAE
7.29
7.59
3.83
8.10
9.47
3.21
13.69
11.16
3.61
10.05
9.03
3.66
10.60
10.38
2.86
SBDD
2.25
1.97
6.19
1.63
2.49
4.43
4.12
2.96
1.76
7.12
2.35
0.45
2.84
0.72
0.27
GraphBP
3.48
1.73
3.86
2.91
7.09
4.51
2.95
7.97
4.16
2.06
6.58
3.47
Pocket2Mol
FragDiff
0.78
0.86
1.16
1.32
0.37
0.43
3.26
0.02
Train set
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Geom set
0.19
0.00
0.00
1.05
0.05
0.03
0.52
0.03
0.22
0.99
0.02
0.06
Figure 8: The KL divergence of bond lengths, bond angles, and dihedral angles for more ring types.
13

---

Under review as a conference paper at ICLR 2023
(8)
cc
CN
00
C20
QP
C-C
CS
OsP
C-N
NO
7.10
10.81
$600
1.40
1.02
0.97
1.24
0.96
4,00
3.78
GraphaP
9.22
MockertMal
1.51
1.35
1.07
1.30
2,50
3.45
3.38
3.89
FregDif
Train set
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.19
(b)
250
2:5
230
200-
2:00
2.00
1.75-
15
1.79
1.5
1.2
1.5
0.15
9.79
frusthr
Taar wt
Gostop
deems
2.50
2.50
220
2.5
1.79
17
175
125
2.25
0.7
0.75
0.2
1000
Podertted
Frosiia
Ran Ad
Grem
9800
Ghppdd
Hostlal
Baln
Grom
1000
Rotariat
Proginr
Glake
Ceg
Figure 9: The distributions of bond lengths of different bond types. (a) The KI. divergence of bond lengths of different bond types with the training set. (b) The distributions of bond lengths of the most frequent six bond types.
14


Reviews:
[{"review_id": "bGloxzv-Dgp", "reviewer": "Reviewer_nh1t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to address the problem of generating 3D molecules for a specific protein binding site by integrating both the autoregressive and diffusion generative processes. Specifically, this paper proposes a model FragDiff, which generates 3D molecules fragment-by-fragment auto-regressively. In each time step, the model generates a fragment around the pocket by an E(3)-equivariant diffusion generative model, predicting the atom types, atom coordinates, and the chemical bonds of the fragment. ", "summary_of_the_review": "Overall, this paper looks not ready for now. As a protein-target-based generative model, its experiment did not focus on protein-ligand binding position/affinity, but focused on ligand-based generation (pocket-free). So I recommend rejecting the paper.", "strength_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow.\n\nWeaknesses:\n1. My most concerning part is the experience section and the performance of the proposed method.\n    - There are many metrics for the ligand itself (pocket-free, unconditional), and mostly for the fragments. It is quite confusing as the paper aims to propose a method for protein-specific generation, in which the binding affinity is the most important metric. \n    - The selected baselines in the paper are mostly optimized for the binding position/affinity. It is not fair to compare with them in these metrics, especially the proposed method in the paper is optimized for the ring/fragment generation.\n    - To compare these ligand-based metrics, it is better to use the ligand-base baseline models. \n2. The Vina score from Figure 5 seems poor. I agree that the Vina is not accurate for the binding affinity assessment, but it is a reference. Besides, authors said \"the distributions of FragDiff was similar to the test set.\", while the distribution of Pocket2Mol is more close to the test set in Vina. And I don't believe close to test-set is better. In short, the current experimental results cannot prove the proposed method is better than previous work in binding affinity assessment. \n3. What is \"Vina rmsd\" ?\n4. The inference efficiency. Since the autoregressive model and diffusion model are not efficient during inference, can the authors compare the generation efficiency of the method proposed in this paper with that of other models?\n5. Is the proposed a from-scratch protein-target based generation? From Sec. 3.4.1, it seems the training is based on fragment masking, and part of the molecule is retaining. Is part of the molecule is used when performing protein-target based generation?\n6. The training is still based on a fragment vocabulary. Did the authors check whether the generative fragments are in the vocabulary or not?\n7. The details of \"dummy element type\" is not provided. How do you define the dummy atoms during training?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EOR2E86sSUU", "reviewer": "Reviewer_5r3H", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "FragDiff is a new diffusion model that generates compounds for a given binding pocket. The generation is a mix of autoregressive modeling and diffusion that creates molecular fragments. E(3)-equivariant graph neural networks are used to embed molecular graph information. At each diffusion step, the atom types and bond types are represented with discrete distributions and atom coordinates are represented with a continuous distribution. To select the attachment point for the new fragment, a focal predictor is trained. Additionally, a molecule discriminator is trained and used for filtering unrealistic compounds after generation. FragDiff is compared against several recent generative models in terms of structure similarity to the reference set, QED, SA, and Vina scores.", "summary_of_the_review": "Based on the comments above, I am leaning towards the rejection of the paper.", "strength_and_weaknesses": "Strengths:\n- The notation used to explain the model is clear and easy to follow.\n- The architectural choices are reasonable, e.g. defining the diffusion of atom types and bond types using a discrete distribution and coordinates with a Gaussian distribution, the use of E(3)-equivariant networks.\n- The model equivariance is proven.\n- A filtering method based on a discriminator network is prepared and proven to improve the quality of generated compounds.\n- The results show that FragDiff can generate realistic 3D structures, at least as well as other recent pocket-based generative models.\n- The example of compound generation in Figure 6 shows that the model produces reasonable fragments.\n\nWeaknesses:\n- In the abstract, it is argued that current autoregressive models generate one atom at each step. This is certainly not true for all the current models, e.g. see [1, 2]. In the second paper, the fragments are generated using VQ-VAE.\n- The statement that “the world’s first drug designed by AI” was recently proposed is exaggerated given the following criticism about the structural similarity to the existing compounds. Also, “drug candidate” would be a better word choice.\n- The motivation about using diffusion models to generate fragments is unclear for more. What do we gain exactly?\n- For the molecule discriminator, it is said that fake counterparts of molecules are created by adding noise. What kind of noise is added to the molecules? The discriminator training could be described in more detail.\n- I am wondering if the filtering procedure based on the discriminator is not too heavy. How many compounds on average are filtered? Could this post-processing method improve the results of other generative models in the benchmark? \n- I am confused about the guidance ablation. What was the goal of implementing this ablation study? Were other weights of the pocket-specific and pocket-free models tested as well?\n- Some figures are illegible. More specifically, Figure 2 compares the statistics of the generated compounds against the training set, but the columns corresponding to each model are very narrow and difficult to compare with the baseline. It would be far more clear if each model distribution was compared against the training set on a separate plot, or, for the sake of paper length, the baseline distribution could be marked at least in a different way (e.g. a line) to make the other bars wider and easier to distinguish.\n- In Figure 2, there are zeros in the row corresponding to the training set. How should I interpret this information, or is it a mistake?\n\n[1] Yang, Soojung, et al. \"Hit and lead discovery with explorative RL and fragment-based molecule generation.\" Advances in Neural Information Processing Systems 34 (2021): 7924-7936.\n[2] Chen, Benson, et al. \"Fragment-based Sequential Translation for Molecular Optimization.\" arXiv preprint arXiv:2111.01009 (2021).\n\nMinor points:\n- The paper needs proofreading to eliminate grammar errors and typos. Some of the typos: “amount of elements” -> “number of elements”, “around the pocket” -> “inside the pocket”, “the protein atoms that are the closed to the fragment atoms” -> “the protein atoms that are the closest to the fragment atoms”, “bond angles, bond angles and dihedral angles” ->  “bond lengths, bond angles and dihedral angles” \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "43ZKCCE0gCN", "reviewer": "Reviewer_ia3Y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A general framework called FragDiff for pocket-specific 3D molecule generation is introduced. In particular, the generation process is executed in a local-to-global style. Namely, the diffusion model is adopted to generate the local fragment from scratch, while the autoregressive model is used to assemble the fragment into molecules. ", "summary_of_the_review": "Overall, the idea of this paper is average, but it is hard to read due to the wrong equations. Moreover, the organization of this paper is poor. Some important contents are not discussed. ", "strength_and_weaknesses": "Strength:\n\n(1) It is reasonable to leverage the advantages of two kinds of generative models for molecule generation.\n\n(2) The experiment results are impressive. \n\nWeaknesses: \n\n(1) The organization of this paper is not clear. As claimed in the paper, the contribution is the fusion of two kinds of generation models into the same framework. However, the whole technique part, Section 3, is about the construction of diffusion for fragment generation.  \n\n(2) How to combine the diffusion model and autoregressive model is not detailed. More content should be added. \n\n(3) Some equations are not correct or confusing. \n\n3.1: Eq.6 is wrong. As the Intermediate variables, $F^{1:T}$, are not integrated out, $F^0$ should be replaced with $F^{0:T}$ in the left part of Eq.6 as well.  \n\n3.2: Eq.7 and Eq.8 is wrong. The summation operator over the set of edges, i.e., $\\mathbb{E}$ and $\\mathbb{C}$, should be normalized. \n\n3.3:  $\\psi_1$, $\\psi_2$ and $\\psi_3$ in Eq.7 and Eq.8 should be different, as they have different types of input.  \n\n(4) It seems Eq.9 is the loss for the diffusion model only. Does it mean the autoregressive model is fixed in the whole training process? \n\n(5) Too many grammatical errors, the paper should be double-checked.\nDifferent tenses are mix-used in related work. Singular and plural are mix-used in Section 3.3.2. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject