FREED++: Improving RL Agents for Fragment-Based Molecule Generation by Thorough Reproduction
Authors: Alexander Telepov
Artem Tsypin
Kuzma Khrabrov
Sergey Yakukhnov
Pavel Strashnov
Petr Zhilyaev
Egor Rumiantsev
Daniel Ezhov
Manvel Avetisian
Olga Popova
Artur Kadurin
Venue: Accepted by TMLR ()

Abstract: A rational design of new therapeutic drugs aims to find a molecular structure with desired biological functionality, e.g., an ability to activate or suppress a specific protein via binding to it.
Molecular docking is a common technique for evaluating protein-molecule interactions.
Recently, Reinforcement Learning (RL) has emerged as a promising approach to generating molecules with the docking score (DS) as a reward.
In this work, we reproduce, scrutinize and improve the recent RL model for molecule generation called FREED (Yang et al., 2021).
Extensive evaluation of the proposed method reveals several limitations and challenges despite the outstanding results reported for three target proteins.
Our contributions include fixing numerous implementation bugs and simplifying the model while increasing its quality, significantly extending experiments, and conducting an accurate comparison with current state-of-the-art methods for protein-conditioned molecule generation.
We show that the resulting fixed model is capable of producing molecules with superior docking scores compared to alternative approaches.

Published in Transactions on Machine Learning Research (12/2023)
FREED++: Improving RL Agents for Fragment-Based Molecule Generation by Thorough Reproduction
Alexander Telepov* Artem Tsypin", Kuzma Khrabrov", Sergey Yakukhnov Pavel Strashnov", Petr Zhilyaev", Egor Rumiantsev", Daniel Ezhov", Manvel Avetisian",
Olga Popova", Artur Kadurin*a.
Corresponding authors. Contacts: Telepov@airi.nn net, KadurindWurri.re net
AIRI, Kutuzouskiy prospect house 32 building K.1, Moscow, 121170, Russia.
Kuban State University, Stovropolskaya Street, 149, Krasnodar 350040, Russia.
Sirius University of Science and Technology, Olimpiyskiy ave. b.1, Sirius, Krasnodar 354340, Russia. Independent researcher.
Reviewed on OpenReview: https: openreview. net/ formn? id= YVPb6tyRJu
Abstract
A rational design of new therapeutic drugs aims to find molecular structure with desired biological functionality, c.g., ability to activate suppress specific protein via binding to it. Molecular docking is a common technique for evaluating protein-molecule interactions. Recently, Reinforcement Learning (RL) has emerged as a promising approach to generating molecules with the docking score (DS) as a reward. In this work, we reproduce, scrutinize and improve the recent RL model for molecule generation called FREED (Yang al. 2021). Extensive evaluation of the proposed method reveals several limitations and challenges de- spite the outstanding results reported for three target proteins. Our contributions include fixing numerous implementation bugs and simplifying the model while increasing its quality, significantly extending experiments, and conducting accurate comparison with current state-of-the-art methods for protein-conditioned molecule generation. We show that the re- sulting fixed model is capable producing molecules with superior docking scores compared to alternative approaches.
1 Introduction
The traditional drug discovery process notoriously long and expensive. Modern drug discovery pipelines utilize computational methods to decrease the amount of in vitro experiments reducing the time and cost of the whole process. One of the crucial early steps of computer-aides drug discovery (CADD) is the virtual screening(VS)) (Lyne] 2002) of vast databases of chemical compounds. VS identifies potential drug candidates possessing desired chemical properties (e.g., stability, low toxicity, synthetic accessibility, etc.). Despite being cheaper and faster than the traditional web-lab approach, VS has another major drawback: is limited to the databases of already known drug candidates and not suitable for de novo drug design. Moreover, is estimated that there are more than 1080 (Reymond & Awale 2012) potential drug-like molccules, making it impossible to screen all candidates even if such a database was available.
Recently, deep learning (DL) has been used to overcome the limitations of traditional approaches. Early works focus on speeding up the virtual screening process by using neural networks (Soichet 2004 Dahl
eet al.
2014
Ma et al., 2015 Wallach et al.) 2015; Ramsundar et al., 2015 Unterthiner et al.) 2014 Gawehn
et al.
2015
Mayr et al. 2016 Baskin et al. 2016 Koutsoukas et al. 2017p. However, despite the existence
of large number of chemical databases, including drug-like molecules (Gaulton al. 2012 Irwin ct al.
2012
Polykovskiy al.
2020). pre-computed chemical properties (Chmiela et al. [2017 Isert al.) 2022
Khrabrov et al. 2022
Fastman et. al. 2023), and protein-ligand pairs (Francoeur et al. 2020 Hu et. al.

---

Published in Transactions on Machine Learning Research (12/2023)
State,
Agent
Action
State 1+1
Environment
An overview of RL.-based sequential generation methods. The agent takes the current state (section 3.2) and selects an action (section 3.3). The action is usually molecular fragment appended to the current state. Individual atoms or atom bonds are considered trivial cases of fragments. The transition dynamic is straightforward: the new state is assembled from the previous state by attaching
Figure 1:
fragment to it. Note that in some frameworks (Zhou et al. 2019 Jeon & Kim 2020 removal of the fragment is also considered an action. The specific lask defines the reward. Some examples include cLogP, QED, and various binding affinity proxies. If J is the optimization objective, the reward can be chosen as T1+1 = (s (-++1)+01 - (s;) if 8 is non-terminal:
0
or T++1
(8t+1)
if 8141 is terminal.
2005 the size of the drug-like molecule space suggests that generative and other semi-supervised methods may be more promising.
Generative models are trained to map drug-like molecules into a low-dimensional hidden space and restore the original molecules from this low-dimensional representation. This approach allowed the sampling of new drug candidates by the perturbation of dov-dimensional representations of existing molecules. Generative models operate several different molecule representations, such SMILES Klómez-Bombarelll et al. 2018).
2D graphs (Jin et al. 2018 De Cao & Kipf 2018), hyper-graphs (Kajino 2019), and 3D structures (Gebauer et al. 2019 Simmm el al. 2021). Guacamol Brown et al. 2019 provides benchmark for such models.
Despite several notable achievements (Zhvoronkov et al. 2019), such models do not consider the target protein the molecule binds to. This is suboptimal, as the therapeutic effect of the molocule is largely determined by its binding affinity to the target protein. Various groups have proposed an approach to the protein-conditioned generation of molecules based on an autoregressive generative model that sequentially assembles molecules by adding atoms and atom bonds Li et al. (2021); Drotár et al. (2021): 0 Peng et al.
20022; Liu et al. (2022). Diffusion models (Hoogeboom al., 2022) Igashov et al. 2022 Schneuing et al.
2023 Guan et al. 2023) have been proposed an alternative to autoregressive sequential generation to speed the generation process significantly. Both autoregressive and diffusion generation require a dataset of protein-ligand pairs, and the amount of training data limits their quality.
An alternative approach protein-conddtionel molecule generation is to use Reinforcement Learning (RL). RL-based approaches can be roughly divided into two categories. The first category, briefly described in
Fig. [ is the sequential generation (You et al. 2018 Zhou et al. 2019 Jeon & Kim 2020 Yang et al. 2021 Such models do not require training datasets and can utilize desired molecular properties such docking score, drug-likeness (Lipinski et al.) 1997), epitope score (Shashkova et al. 2022). or chemical constraints into the training objective. In cont rast,, most of graph generative models such as VAEs, Diffusson models or GANs cannot directly incorporate such properties into their training objective. However, different strategies, such as active learning or evolutionary algorithms, can be used to optimize desired propertics (Segler et al.
2018
Schneuing al. 2023). The second category (Popova et al. 20188 Blaschke et al., 2020 Thomas al.
2021
Ghugare al. 2023 Mazuz al. 2023) uses RT. to fine-tune pre-trained generative models large
2

---

Published in Transactions on Machine Learning Research (12/2023)
datasets to produce molecules with desired properties, such as cLogP and QED. Unlike the first category, these models require two-step training and a dataset to pre-train the generative model on.
The binding affinity of a molecule to the target protein is a perfect reward for protein-conditoneed molecule generation. However, since the real binding affinity is intractable, recent research has utilized its proxy, Docking Score (DS), as a reward in RL setting (Jeon & Kim 2020 Cieplinski et al. 2020 Thomas et al., 2021 Yang et al. 2021). In this work, we focus on FREED pipeline (Yang et al.) 2021), which sequentially generates molecules using fragments as building blocks. This allows to ensure the validity of generated molecules that have high DS for various target proteins.
We strongly believe in the approach used in FREED, so in this work, we conducted a thorough analysis of its implementation and experimental setup. Our findings are as follows: i) the implementation contains multiple bugs, ii) the evaluation setup is inconsistent between the proposed model and baselines, iii the amount of target protcins selected for model evaluation is insufficient, and iv) the model is overcomplicatod and lacks ablations.
In order to address these limitations, we meticulously inspected the code, fixed the bugs, conducted abla- tion studies, and simplified the proposed model. We have named the resulting fixed and simplified model FREED++. Moreover, we have substantially broadened the scope of our experiments. To avoid cherry- picking of target proteins, we have tested FREED++ on a large number of proteins from the DUD-E database (Mysinger et al. 2012), and additionally, on the ubiquitin-specific protease 7 (USP7) protein (Leger et al. 2020a). We have experimented with fragment librarics and additional structural constraints to provide further insight into the model. Finally, we have fixed the evaluation protocol and compared FREED++ with alternative approaches to accurately compare it with current SOTA protein-conditioned molccule gencration methods. We find that FREED++ exhibits great generalization ability while producing valid molecules with docking scores superior to those of alternative approaches. The code is accessible by link
2 Related work
Current state of protein-conditioned molecule generation Our work is concerned with protein- conditioned molecule generation. This area of research has received increased attention lately. REIN- VENT Oliveeronna et al., 2017 Thomas et al., 2021), ChemRLformer (Ghugare et al.) 2023) and Taiga (Mazuz et al.) 2023) utilize RL to fine-tune an NN that was pro-trained to generate SMILES strings. RL guides the generation towards molecules with high value of the property of interest (e.g., QED, DS).
MolDQN (Zhou et al. 2019) belongs to the family of sequential generation methods based on RL. It uses a modification of the Deep Q-Learning algorithm @Mnih et al. 2013) to sequentially select atoms and bonds to add to or remove from the molecular graph. A new promising alternative to RL-based generation are Generative Flow Networks (Bengio et al.) 2021 Jain et al.) 2023). These models are trained to sample objects with probabilitics proportional to the reward function. Unlike RL-based methods, Generative Flow Networks are capable of sampling from different modes of the distribution which is especially useful in drug design applications. Pocket2Mol (Peng et al. 2022) is trained in an autoregressive fashion to predict masked atoms of drug molecules in protein-ligand pairs. The trained model sequentially generates the atom's posi- tion, type, and bonds. The generation process is conditioned on both the protein target and the previously generated atoms. These methods have demonstrated their ability to generate potential drug candidates that selectively target specific proteins. We compare FREED++ with all the approaches described in this section except DifISBDD, which has been shown (Schncuing et al. 2023 to produce molecules inferior to basclines in terms of various metries.
3 FREED++
This section provides detailed description of the proposed FREED++ method. We highlight the differences between the original FREED and the proposed FREED++ with blue boxes. In section ! we explain the bugs in the original implementation of the FREED model and discuss their effects on the model's performance.
https://github... AAIRI-IITstittte/FRREEE
3

---

Published in Transactions on Machine Learning Research (12/2023)
Molecule
Fragments
s
GCN
a
Critic
Actor
Q (s,a)
π (a)
Figure 2: Overview of fragment-based molecule generation frameworks. At each step, fragment. is selected from pre-defined fragment library F and attached to the current state s. In general, different encoding methods may be used to handle and the selected fragment f, but in this work, both are processed with the same GCN.
tragment-based based molecule generation framework; its overview is shown in Fig. 2 It operates on fragments instead of single atoms to speed up the generation process and ensure the validity of generated molecules. The differences between FREED++ and the original FREED are described in sections 4.2
FREED++ is a fragment-
3
3.1
MDP
The state space S is defined set of all possible extended molecular graphs describing stable molecules (see section 3.2 The action a consists selecting a fragment, f and concatenating it with the molecular graph s. The action comprises three steps: "selecting where to attach a new fragment (az)", "choosing which fragment attach (a2)"; and "determining where on a new fragment to form a chemical bond (ag)'. More details about the action can be found in section 3.3 The initial state so is benzene ring with 3 attachment points. Transition dynamics is straightforward: given graph reprosentation s of the current state of the molecule and action a₂ (at, af, af), the new graph is assembled from and the new fragment via RDKit (Landrum) et al. 2022). The length of the episodes is fixed and equals T. Tn the original implementation. the same length T = 4 of episodes is used for all proteins (see section for discussion). The reward r, is calculated as follows:
S0. if <T; re(St, a, S(41) = max DS) if T.
(1)
We define the DS as the negative binding energy estimated by the docking software. DS(s) calculated in three steps. First, initial 3D conformation for docking is generated for a given molecular graph with OpenBabel O'Booyle al. 2011). Then, the binding affinity of the molccule to the target protein is estimated with QuickVina2 [Alhossary al., 2015). Details on reward computation are in Appendix H Lastly, take the negative value of the estimated binding affinity to get the DS.

---

Published in Transactions on Machine Learning Research (12/2023)
Step1
Step
Step
Attachnent
Fragmann
Atbechmnt
Concatenation
points
MLP
point
Fragments
Sample&Fuse
Sample&Fuse
Sample&Fuse
MLP
a
a2
ag
Q (s,a)
(a) Action selection
(b) Critic architecture
Figure 3: A schematic overview of the actor and the critic. The act selection process depicted in Fig, 3a Step 1: the current state 8 is embedded with a GCN Gg, and the resulting embedding is fused with the embeddings its attachment points. Then, one the attachment points is selected as 01, and its embedding âj is used in the next step. Step 2: âj passed through an MLP to get distribution over the available fragments. One of the fragments is selected d2; and its embedding ãa used in the next step. Step 3: the selected fragment d2 is processed by the same GCN Go to obtain the embeddings of its attachment points, After fusing these embeddings with 02, one of the attachment points of the fragment is selected as ag. Critic: The embeddings of all actions are concatenated with the state embedding and processed by the critic (Fig. 3bb.
3.2 State
FREED operates on extended molecular graphs - graphs that contain a special type of node called an 'attachment point'. These points indicate the locations where the atomic bonds were severed during the fragmentation process. Consequently, the state is representod molecular graph with attachment points. When the generation process is completed, all attachment points (if present) are replaced with hydrogen atoms.
To be processed by Graph Convolution Network (GCN) graph needs to be converted to a tensor. This is achieved by representing the nodes and edges with a set of feature vectors and an adjacency matrix. The node features encompass onc-hot encodings of the atom's type, valency, degree, the count of hydrogen atoms, and a flag indicating whether the atom a part of au aromatic ring. Edge features consist of one-hot encoding of the bond type.
3.3 Action
a2, The action = (01, a2, d3) is composite of three elements. The first component 01 is the index of an attachment point to which the new fragment is attached. To obtain categorical distribution over the attachment points of state s, we first embed s with a GCN (Kipf & Welling, 2017): V= Gs(s). We then use the resulting matrix node embeddings V RVV xd two ways: 1) an aggregated graph embedding Š == =Vgg(V), Š Rd is derived from V. where agg is an aggregating function (normally, a sum or an average of rows); 2) a submatrix patt e g/v"'|xdd of node embeddings corresponding to attachment point is formed. V and | are the total number of nodes the graph and the number attachment points, respectively, and is the embedding size. 5 and past are then combined via trainable fusing (i.e., Multiplicative interactions (Jayakumar et al.] 2020 layer a concatenation layer; refer to section 4.3 for in-depth
5

---

Published in Transactions on Machine Learning Research (12/2023)
discussion) function for, resulting in (s) R xxd;
(s(s) fig (S, Vatt).
(2)
After that, we use an MLP fig : RIv RIV that maps (s) to logits that define calegori- cal distribution over attachment points. The Gumbel-Softmax (Jang al., 2017) operator is used reparameterize the resulting distribution. Categorical reparameterization is essential because ag and ag are sampled autoregressively and depend on previonsly sampled discrete actions.
(fg (A)(s)).
The second component 02 the index of the fragment to be attached to s. First, we sample action ai from Pi(- () (Eq. 3) and select the astt row â of the matrix (s). To obtain the logits of the categorical distribution over the fragment candidates, we process â1 with an MLP fØ, Rd R1 where F set of all available fragments, and is their total number.:
P2(. 92(-18,01) (ffo2(ā)))
(4)
In FREED, the categorical distribution over fragment candidates is computed similarly the distribu- tion over attachment points (see Section 4.3) First, ECFP embeddings (Rogers & Hahn 2010) PECPP are calculated for all the fragments. Then, F ECCP and â are combined via fusing funtion:
Åo føz (ā), FECPP).
(5)
After that, an MLP for : Flll + R²x maps Ag (āz) to logits which define a categorical distribution over fragments.
p2(. 8, 8,01) = (fss (A2)).
(6)
Note that if BRICS fragmentation (see section 5.2 is used, logits associated with fragments that lead to invalid molecules are replaced with -oc, effectively preventing the sampling of such fragments. Then, we sample an index 02 ~ p2(-1 8, a1) and the corresponding fragment, faz and process along with all available fragments (to reuse in case batch of fragments sampled) with the same GCN Go get matrix F R RFFx of aggrogated embeddings:
F = (agg
(7)
Then, the ag-nd row of matrix F' selected and combined with ãj using a fusing function fzz:
02(8, a1) = fio (ā,, Fag)
(8)
Instead of fusing ă with the GCN embedding the selected fragment a2 the original implementations uses fixed ECFP emdding:
(2(8, 01) = fag (ā1, BECFP).
(9)
Unlike in 'Step 1", where we first fuse the state and emboddings of attachment points and then sample a in 'Step 2", we first sample ag and then fuse its embedding Fa, with âz. This simplification (see section 4.3) allows to speed up the training process significantly.

---

Published in Transactions on Machine Learning Research (12/2023)
The third component 0s is the index of attachment point on the selected fragment fag- To obtain categorical distribution over attachment points of the selected fragment fag: we first retrieve the matrix of node embeddings of the selected fragment Fa, = Go(fuz). We then select node embeddings corresponding IR Ma Ixd where |saz is the number of attachment points on the selected fragment fag- A fusing function fØ, is utilized to combine this embedding with 0z:
attachment points Fatt
a2
Ag(s, a1 02) = fos (ā2, F
(10)
After that, we employ an MLP fØg Rlayied 7 R Sag that maps Az 01 a2) to logits that define a categorical distribution over the attachment points of the selected fragment. Similarly to 'Step 2', if BRICS fragmentation used, logits corresponding to invalid attachment points are replaced with -00.
ps(: 02) = fas a1 (1,)))
(11)
Lastly, sample as Pa(- 8, a2) and select the a3-rd row äs of the matrix Ag(s, (1s,,a1,a)) 02) The whole action selection process is illustrated in Fig. 3a
3.4 Critic
The critic architecture is illustrated Fig. 3b It consists two components: an encoder GCN Go, which is also used for action selection, and an MLP fa Rid IR. To ensure training stability, the encoder Go is frozen in the actor and is only trained part of the critic. In FREED, the MLP fw takes the concatenation
of (S, Vett Foz. (Sst) )os) and outputs the Q-value.
There are two scenarios in which the critic is used. The first scenario involves applying the critic to the actions saved in the replay buffer. Since the saved action (41, 02 (2,0g) is just set of indices, the critic needs to reprocess the saved state and fragment with the current version of the GCN and then pass to f., This is necessary because the GCN parameters may have been updated since the transition was saved in the replay buffer. The second scenario involves applying the critic to actions generated with up-to-date Go (i.e., when calculating actor loss in SAC). In this case, there is need for reprocessing, and the concatenation of (8, Vatt Fozs (Faat)) )as) is passed directly to f.
In the original FREED paper, crtitic architecture was not covered. For the description of the critic used FREED refer to the section
4 Fixing FREED
This section summarizes all the changes made to the original FREED model. This section is divided into three subsections. In the first subsection, we describe the issues and bugs found in the original implementation of FREED, along with their potential effects on the model's performance. In the second subsection, we describe minor changes introduced to simplify the model, reduce the number of hyperparameters, or improve the the training stability. By implementing all the changes described FREED, which we dub FFREED".
4.1
4.2
we arrive at a fixed version of
In 4.3 describe various components of FFREED that can be removed simplified to speed up the model and reduce the number of trainable parameters. The resulting model after all the simplifications is called "FREED++".
4.1 Major implementation issues
Critic architecture In the original implementation, the critic is parameterized as an MLP, which takes input concatenation of four vectors: 1) embedding of the molecule generated by Go; 2) probabilities
7

---

Published in Transactions on Machine Learning Research (12/2023)
associated with attachment points (see 3q 3) one-hot encoding of the selected fragment fazi and 4) probabilities associated with attachment points on Saz (see eq. 11
First, the critic receives no information about the selected attachment points, as it operates on probabilities. Furthermore, the order in which the probabilities for the attachment points are passed to the critic is determined by the inner representation of the current state RDKit (Landrum et al. 2022). This order may change as new fragments are added throughout the trajectory. Due these two factors, the critic cannot attribute high rewards to selecting specific attachment points. Consequently, learning any meaningful policy for selecting a1 and 0g becomes impossible. Refer to 3.4 for details of our proposed implementation of the critic architecture.
Critic update The agent is trained with the SAC (Haarnoja et al., 2018a algorithm that operates inside the Maximum Entropy Framework (Ziebart et al. 2008 Haarnoja et al. 2017), which augments the standard maximum reward reinforcement learning objective with an entropy maximization term. The target for the Q" function includes terms responsible for both the future discounted reward and the entropy of the policy:
(s), = (1.1 [r(8), un, 8441) + ())+1,)) a aologTi ((ā 8++1))]
(12)
In the original implementation, the entropy term is omitted from the equation. This way, at the policy improvement step, the actor is forced to maximize the cumulative return and ignore the term responsible for the entropy of the policy in the succeeding states.
Target networks usage Target networks are used to stabilize the training of Q functions (Mnih et al, 2015 A common practice for the SAC algorithm to update the larget network throughout training smool hly: Vuarg = ( =Q"; where T is a smoothing constant that usually set to a small value between 0.01 and 0.001 (Haarnoja et al. (0118)). Instcad, in the original implementation, is set 1, which effectively means that no target network was used. This leads to the moving target problem and destabilizes the training.
Gumbel-Softmax Recall (see section 3.3 that the action proposed in FREED consists of three dis- crete random variables which are sequentially sampled in an autoregressive fashion. The authors employed Gumbel-Softmax (Jang et al. 2017 to propagate the gradients through the discrete sampling process.
We found two issues the original implementation. First, probabilities were used instead of logits. Second, a parameter was introduced without a clear motivation. We show that sampling from the Gumbel-Softmax distribution with these two issues is equivalent to sampling y & exp I with a temperature 7, where is the desired discrete distribution, and use exp docs not necessarily define distribution. Refer to Appendix
B lor the derivation and in-depth discussion of the effects.
To evaluate the significance of each particular major implementation issue, we perform an ablation study in Appedinx K
4.2 Minor issues and simplifications
Message passing As states and fragments represent extended molecular graphs, we use GCN obtain embeddings. GCN iteratively updates node representations by aggregating the information from the node's local neighborhood. In the original implementation, directed graphs used instead of undirected ones, The direction of the edge is determined by the inner representation of the graph in RDKit (Landrum et al. 2022). Such molecular representation differs from the undirected one previously used in various applications of ML and DL in CADD (Sun et al. 2020 Wieder et al. 2020). The intuition behind the usage of undirected molecular graphs lies in the fact that chemical bonds do not have naturally imposod directions. Morcover, Wu et al. (2018) have previously shown that treating molecules as undirected graphs improves performance in predieting various molecular properties. We use undirected molecular graphs.
8

---

Published in Transactions on Machine Learning Research (12/2023)
Learning rate schedulers In the original implementation, the learning rate scheduler is used. The actor learning rate is decreased by a factor of 10 if the training actor loss has not improved for a certain number of steps. This is not an optimal choice for actor-critic RL algorithms, as actor loss is poorly correlated with agent performance. While studying the original implementation, we noticed that for some target proteins, such a scheduler caused the learning rate to converge quickly at the beginning of training, preventing the agent from improving further. We use constant learning rate throughout the training to mitigate this issue and simplify the selection of hyperparameters.
Reward shaping In the original implementation, reward shaping is used. While investigating the proposed reward shaping approach, we found it to be meaningless (sec A.7 and removed it. Instead, we only assign non-zero rewards to the terminal states (see 3.1
Temperature hyperparameter in SAC In the original implementation, the temperature a is only trained for a fraction of the whole training time, and, furthermore, a is clipped to be in the range [0.05, 20] (details in A.5 We follow the original implementation, remove clipping, and train throughout training.
Architecture In the original implementation, the sizes of latent spaces are decreased quickly and deeper network for fragment selection is used compared to the selection of attachment points. We operate on bigger embeddings and unify the structure of logits projectors for all actions. The exact architecture changes and their effects are discussed in section A.9).
Other minor changes and simplificatious can be found in the Appendix A
4.3 From fixed FREED to FREED++
In this section, we describe how to significantly speed up the FFREED model and reduce the number of trainable parameters. We investigate several components of the original FREED framework and show that they can be removed or simplified without performance drop while significantly speeding up the model and reducing the number of trainable parameters.
Prioritization To encourage the agent to generale diverse molecules, the original paper's authors propose their variant. of prioritized experience replay (Schaul et al. 2015). Instead of using the TD-error to determine the probability of sampling a transition from the replay buffer, the authors suggest prioritizing novel transi- tions. The novelty in terminal states is defined as the L2 error between the reward in the terminal state and the predicted reward. In non-terminal states, the non-existent. reward is approximated with the Q function, and the Le error between its output and the predicted reward is used novelty. Apart from introducing additional computational load, this approach has multiple issues, which we discuss in Appendix C We train the agent without PER and sample transitions uniformly.
Fusing functions As mentioned in 3.3 fusing functions fØr føz, and føg are used when selecting a1 and 0g to combine the embedding a state or fragment with the embedding of an attachment point (see equations 10p. In the original paper, multiplicative interactions (Jayakunar et al.) (2200) are used fusing functions. Let Rdi,2 Rd represent the embeddings to be combined. Then, the MI of x and is:
M c 2) = =" W; (x,
i + U,2 + V + bb,
(13)
where We E Rdaxdsxdi is a learnable 3D tensor, U E do V E xd are learnable matrices, and b E Rd is a bias term. Note that if not stated otherwise, d1 d2 d2 d. Notice that the MT layer contains bilinear form, which is considered to be computationally and memory expensive. To overcome this drawback, we simplify the fusing function and replace the MI layer with a concatenation layer:
PCAT (x, 2) Q Q1(1; b;.
(14)
where :1 denotes the concatenation operator, Q R (d)+ is a learnable matrix, and b Rd is a bias term.
9

---

Published in Transactions on Machine Learning Research (12/2023)
Fragment selection In the original implementation of FREED, the fragment selection protocol resembles the procedure of attachment point selection. First, the ECFP representations of the fragments FECPP (ECEP(A) FEF are built. ECFP is a function that computes d2 = 1024-bit Morgan fingerprint of radius (Morgan 1965). Then, the embeddings of all fragments are fused with ā1: A = (ā,, FFCFP)) and the resulting mairix Ag is processed with an MLP fag I2d - R to obtain the logits of the distribution on the fragments. Notice that when d2 large, the bilinear form equation 13 becomes expensive to compute. To avoid that, we replace the procedure described in the original paper with the one described in equation |
In summary, by applying these three changes to FFREED, get FREED++. We carefully compare FREED++ with different variants of FFREED in section F
5 Experiments
In this section, we describe the experiments conducted and our evaluation protocol. We thoroughly tested the improved FREED++ framework on various tasks related to generating molecules with desired properties. As binding affinity is a key property of molccule that defines the therapeutic effect of drug, we consider docking score optimization as the main objective in our experiments.
Comparison with baselines First, we compare FREED++ with existing molecular generative baselines on the task of generating molecules with high affinity against particular biological targets 5.1 We consider this to be the main experiment.
Fragment library collection Existing tools for de novo drug design use various librarics of building blocks with sizes ranging from dozens to several thousands of fragments (Rotstein Murcko 1993 Pierce el al. 2004 Douguet et al., 2005) Spiegel & Durrant 2020 Yuan al.) 2020). The main reasons behind the choice of fragment library can be summarized follows: 1) the building block library should be reasonably small to reduce the chemical space efficiently, and 2) if known inhibitors for a particular target are available, it is better to use their fragments for assembling. [Lin 2000). While the used fragment library essentially defines the set of attainable molecules, research papers on de novo drug design focus mainly on scoring functions, search algorithms, and assembling strategies (Schneider & Fechner 2005). To improve the understanding of the problem, we conducted experiments with several fragment libraries obtained from two molecular datasets with diffcrent fragmentation techniques 5.2
Development of USP7 inhibitors Finally, we have tested FREED++ in the practical scenario of gen- erating inhibitors for the USP7 protein and provided qualitative analysis of the generated molecules 5.3
Protein targets In the original paper, three protein targets are considered: fa7 Zbiindnn et al. parpl (Penning et al., 2010), and 5ht1b (Wang et al., 2013). To show an improved generalization ion, we experiment with three additional proteins: abll (Cowan-Jacob et al., 2007) and fkbla (Sun et al. from DUDE (Mysinger et al. 2012) and usp7 [Leger et al.) 2020a) from PDB (Berman et al. Binding pockets of interest for proteins were computed as follows: first, we extracted the 3D structure of the ligand from the corresponding pdb file; then, we computed the center of the bounding box as the average of all atoms' coordinates. The size of the bounding box along each axis is estimated by adding the maximum difference between the coordinates of the atoms and 4A for the corresponding axis.
2005
2003
2000
Evaluation protocol To evaluate the generation quality, we generate 1000 molecules with a fully-trained model, remove invalid compounds and duplicates and score the filtered molecules with QVina 2. We repeat the generation 3 times with different random seeds. Since the combinatorial generator is a simple baseline, we allow it to generate 30000 molecules instead of 1000.
It is important to note that the evaluation protocol differs from the one used in the original paper. In the original work, all models except FREED are compared on the first 3000 molecules generated during the training. The FREED is evaluated on the 3000 molecules generated after the exploration phase (see section A.3). Such an approach does not reflect the actual performance of models, as the evaluated models are
10

---

Published in Transactions on Machine Learning Research (12/2023)
underfitted. For example, the default REINVENT model generates 1.8 x 102 molecules throughout the training, requiring approximately twice much computational time as FREED.
Metrics To evaluate the performance of different models, we report the uniqueness of valid molecules, the average docking score of unique and valid molecules (Avg DS), the maximum docking score (Max DS) and the average docking score of 5% of top-scoring molecules (Top-5 DS). We report DS metrics in KCal/Mol units. In section G. we also consider PAINS, SureChEMBL and Glaxo metrics, which denote the fraction of valid unique molecules that successfully pass the corresponding structural filters.
5.1 Comparison with baselines
In this section, we compare the model corresponding to the original impleementationf] which we call FREED fixed model FFREED 4.3 and our simplified model FREED++ on the task of generating high-affintyy molecules.
Baselines We take the same objective-orieneed baselines as the original paper: REINVENT and MolDQN. Additionally, we consider two methods: the combinatorial generator (random walk), which selects fragments proportional to their frequences al each step, and one of the current SOTA protein-conditioned generative methods Pocket2Mol (Peng et al. 2022). Moreover, we report metric values computed over sets of known inhibitors (KI rows in tables 6). For fa7, parpl, 5ht1b, abl1, and fkbla targets we take inhibitors from the DUDE datasel, and for the usp7 protein we take inhibitors reported in the paper (eger el al. 2020b).
For the combinatorial generator (CombGen) we used the implementation from MOSE For Pocket2Mol we used a pretrained model provided by the authors
We train REINVENT with DS as the optimization objective. For non-valid molecules, DS is considered to be O. We train REINVENT with the default parameters of the original implementation except for the batch size (we take 32 instead of 64). As the reward we take 0.1 DS(ss) for valid molecules and 0 otherwise, to keep a return approximately in the range [0, 1]. We search for the optimal a E [60, 100, 200] (see table 5) and pick the best in the evaluation.
Regarding MolDQN, we explore two setups. The first setup is similar to the original implementation except we replace QED with DS. In this version, gradient steps are performed every 20 iterations, and the reward at each step is calculated as DD((s1)) The second version employs sparse rewards, with rewards set to 0 everywhere except for preterminal states, where the reward is DS($++1). In the 'sparse" setup, we do 24 gradient steps every 480 iterations maintaining the same update-to-data ratio. From the training plots (sce (). we observe that the version from the original implementation performs better but is approximately three limes slower. We use the "sparse" version setup in the Docking score optimization section. Overall, changing the "sparse" version to the dense one does not alter the results.
To sum it up, the corrections to the FREED model (FFREED and FREED++) perform superior to other methods in all metrics except uniqueness. However, we agree with (Peng et al. (022) that uniqueness and diversity are not very important metrics for the packet-based generation task because protein pocket is known to have strong specificity, Morcover, FREED++ allows controlling the trado-off between diversity and generation quality via the target entropy paraneter.
Results The results of the experiment are presented in tables 1
6
While Pocket2Mol is trained to recover the masked atoms of the known inhibitors, ubjective-orienter methods REINVENT, MolDQN, FFREED, and FREED++ are directly aimed to optimize the docking score through the design of the reward function. Although known inhibitors commonly form strong interactions with their
4 httpps: /tt.:///www.//TTTRTS//TEEE.
hitpp:/ http://ttbbbb/mmm/e/ http://commmmmmclarreeetttt moses
hitpp:/ http:///ubboommcgggggggge/mmmeee wowkannngnnooMMtMMo Poockct2Mol
VENT
11

---

Published in Transactions on Machine Learning Research (12/2023)
target proteins, there guarantees that they have the highest possible docking scores, which also observe in our experiments. Therefore, one can expect that RI-based methods will be more successful in the DS optimization task. We partially observed such an effect in our experiments, except MolDQN, which failed to generate molecules with higher docking scores than Pocket2Mol. Possible reasons for this could be the underfitting of models, short episode length, poor choice of hyperparameterss, or the use non-pre-trained encodor. While REINVENT succeeded in outperforming Pocket2Mol, uses a backbone pre-trained on a large database compared to MolDQN, which learns from scratch.
We observe that FREED++ and FFREED outperform MolDQN and REINVENT. We hypothesize that this behavior can be explained by the fact that FREED methods work in the paradigm of fragment generation. Fragment generation exponentially reduces the search space compared to atom-based assembling strategies, allowing the RL agent to learn efficiently.
We found that the original FREED model performs significantly worse than other modcls. The learning process of FREED lends to diverge (see ligure G) while FFREED and FREED++ stably outperform their competitors. This confirms the importance of correcting implementation issues (sec. 4 We compared
FREED and FREED++ on the first 10 proteins from the DUDE database to provide more evidence (see figure Once again, FREED poorly optimizes the docking score and works unstably.
method
unique (1)
Avg DS (t)
Max DS (t)
Top-5 DS (1)
CombGen
0.98 + 0.00
8.03 = 0.00
13.00 + 0.26
10.53 + 0.00
MolDQN
0.21 + 0.03
7.92 + 0.23
9.93 = 0.41
9.43 + 0.34
REINVENT
0.99 + 0.00
9.69 + 0.39
13.13 + 0.89
12.05 + 0.73
Pocket2Mol
1.00 + 0.00
8.71 + 0.37
12.36 + 0.40
11.30 + 0.65
FREED
0.10 + 0.07
8.58 + 1.02
11.36 + 0.98
11.12 + 0.88
FFREED
0.66 + 0.01
10.91 + 0.15
14.03 + 0.46
13.35 + 0.27
FREED++
0.64 + 0.07
9.66 + 3.28
14.16 + 0.47
13.30 + 0.13
KI
9.40
11.90
11.90
Table 1: Comparison of FREED, FFREED and FREED++ with basclines on DS optimization task for USP7 target. See full table in appendix
5.2 Fragment library collection
An appropriate fragment library is an essential part of a successful molecular generation. Such a set of fragments can be handcrafted by chemist or obtained via a fragmentation procedure applied to a given set of molecules.
To obtain fragments for generation, the authors of FREED use CReM fragmentation PPolishhhk 2020 to fragment 250k drug-like molecules from the ZINC [Irwin et al.) 2012) database. Additionally, the authors filter fragments according to the number of atoms, radius, and frequency. Fragments that contain fewer than 12 atoms or appear less than 3 times in the ZINC are excluded. Fragments that cause RDKit parsing errors are also removed. When two fragments have the same graph structure but different attachment sites (almost duplicates), the one with fewer attachment sites is removed. Finally, the authors choose 91 fragments that appear most frequently from the filtered dataset.
CReM fragmentation is not widely used, and formally, the given datasct cannot be reconstructed with fragments obtained by CReM. BRICS fragmentation (Degen et al. 2008) a set of rules for breaking rethesynthetilallly interesting chemical substructures commonly used for de novo design.
We fragment MOSES (Polykovskiy et al. 2020)- cleaned version of ZINC) and ZINC datascts by CRcM and BRICS fragmentation procedures and compare different combinations of fragmentation and dataset. The same fragment dictionary in the previous section used for CReM-ZINC setup. For other setups, we do
12

---

Published in Transactions on Machine Learning Research (12/2023)
the following steps: extract fragments from the dataset, exclude charged fragments, remove fragments that appear only once, remove fragments with more than 16 heavy atoms, remove almost duplicate fragments, and sample 100 fragments uniformly.
In + the optimal number of generation steps is unknown beforehand and needs to be tuned (see section J for an in depth discussion). Because of that, we run the generation for and steps and use the one which produces molecules with higher docking scores on evaluation.
fragmentation
dataset
unique (t)
Avg DS (t)
Max DS (t)
Top-5 DS (t)
BRICS
MOSES
0.58 + 0.17
5.82 + 4.32
13.16 = 0.51
12.22 + 0.67
ZINC
0.63 + 0.13
9.07 + 0.70
13.39 + 0.88
12.42 + 0.46
CReM
MOSES
0.74 + 0.07
1.21 + 1.06
12.00 + 1.81
7.77 + 6.27
ZINC
0.64 0.07
9.66 + 3.28
14.16 + 0.47
13.30 + 0.13
Table 2: Comparison of different fragments libraries used in PREED++ on DS optimization task for USP7 target. See full table in appendix
Results Tables 2 and 7 demonstrate that generation performance significantly inffuenced by the frag- ment library used. We explain these results by drawing an analogy between molecular fragmentation and tokenization in the NLP field. Both procedures break unstructured data into set of meaningful elements, which are used as atomic units that embed contextual information. It is known that tokenization has a major impact on overall pipeline performance in NLP tasks (Chirkova & Troshin 2023 Zhang & Li 2021 Tay et al. 2022 Park et. al. (220)). Based on our empirical results, we conclude that the design of a fragment library is an extremely significant step in molecular generation.
5.3 Development of USP7 inhibitors
USP7 is a deubiquitinating enzyme (DUB) which takes part in regulating of a plethora of biological processes. The up-regulated function of the protein is related to the development of several types of cancer, thus, inhibition of USP7 is considered a promising strategy for the treatment of these malignancies. During the last decade, numerous series of USP7 inhibitors were developed, however, none of them progressed into clinical trials due to their insufficient efficacy and selectivity. For that reason, it is still highly desired to develop novel structural types of USP7 inhibitors which possess improved profile of pharmacodynamiie and pharmacokinetic properties. Known USP7 (see figure 9) inhibitors bind (Li & Liu 2020 Oliveira et al. 2022]) Koreney et al., 2022) to the protein in three diffcrent locations: inside the catalytic center (some irreversible inhibitors form a covalent bond with sulfur atom for catalytic Cys223); in the ubiquitin-binding cleft near the catalytic site and in the allosteric binding site (4-ethylpyridine series of inhíbitors).
A highly promising structural type of USP7 inhibitors was reported in 2020 (Leger al. 2020b Although these molecules bind to the familiar USP7 pocket, the outstanding selectivity relative to other DUBs was demonstrated together with notable efficacy in vivo. Those inhibitors fit the targeted pocket well and form four well-defined hydrogen bonds. In this work, we utilize FREED++ to generate novel plausible inhibitors of USP7 which bind to the same binding site.
We perform the generation procedure for several fragment librarics: with the basic set of fragments (CReM- ZINC), with a relatively big set of fragments (1000 fragments) from the MOSES dataset (BRICS-MOSESS and the set of fragments obtained via fragmentation of known inhibitors of usp7 (BRICS-USP7). We con- struct the reward function in way that takes into account several commonly desired properties of drug candidates. We search for molecules 1) with octanol-water partition cocfficient LogP E [o, 5]; 2) with a number of heavy aloms < 40 3) with a number of hydrogen acceptors < 10; 4) a number of hydrogen donors < 5; 5) which pass PAINS, SureChEMBL, Glaxo filters. We introduce penalties P. in the following way: the penalty for molccule equals if the corresponding property Pr, lies in the acceptable range [L Iymaa] and RReLU
We consider the final reward to be a weighted sum of penalties and docking score: r(st, at, 81+1) = max(O, (8(++1)) + Σe w. (s(++1))
linearly grows outside P. (MM = ReLU(Lmmn
13

---

Published in Transactions on Machine Learning Research (12/2023)
gayo
12A 04
221,05
119
114,05 05
130,04
127,
120 66
118,05
117.07
116.06
124.04
124
(a) BRICS-MOSES
(b) BRICS-USP7
(c) CReM-ZINC
Figure 4: Selected representative molecules which generated by FREED++ with USP7 as target pro- lein. Docking score and maximum Tanimoto similarity to set of known inhibitors depicted below molecules. Fragment libraries: BRICS-MOSES (A); BRICS-USPT (B); CReM-ZINC (C).
Results The results of the FREED++ generation are highly dependent on the chosen fragment library. the case the BRICS USP7 fragment library, is notable that FREED+++-genertted molecules are closely related to the previously reported USP7 inhibitors. The molecules generated using other librarics do not exhibit explicit structural analogy of the previously reported USP7 inhibitors while having higher docking scores. Such results indicate that FREED++ can design novel potential drug candidates and can serve motivation to test the inhibition activity of designed molecules experimentally.
6 Conclusions
In this paper, we present detailed ablation study on the recent state-of-the-art objective-oriented molec- ular generation method FREED. Throngh extensive evaluation and deep analysis of FREED, we identify numerous implementation issues and inconsistencies the evaluation protocol. We and simplify the orig- inal FREED model, provide an accurate comparison with an extended set of baselines, perform additional analysis on different fragment libraries, and test the approach in the practical scenario of USP7 inhibitors generation. We show that FREED++ framework can produce molecules with superior docking scores compared to allernative approaches and can be valuable tool for drug discovery.
Code and Reproducibility
The code for FFREED and FREED++ is accessible by link | The code for other methods and all configs are available in supplementary material.
Broader Impact Statement
As our framework a reimplementation of FREED, our "Broader Impact Statement" " similar to the original work. We provide slightly edited version below.
FREED++ is powerful tool (hat can generate various chemical compounds with desired properties. How- ever, it is important to note that if this tool is used for malicious purposes, it can result in the creation of harmful substances such as biochemical weapons. The potential for abuse of this framework underscores the need for strict regulations and ethical guidelines to be put in place to prevent any misuse, Furthermore, it is crucial that users this framework exercise caution and responsibility in their actions to ensure that it is only used for legitimate purposes that don't harm individuals or society as a whole.
https://githhb... com/AIRI---- AtA.AARRT-TattittterRREEEa
14

---

Published in Transactions on Machine Learning Research (12/2023)
References
Amr Alhossary, Stephanus Daniel Handoko, Yuguang Mu, and Chee-Keong Kwoh. Fast, accurate, and reliable molecular docking with QuickVina 2. Bioinformatics, 31(13):221-2221, 02 2015. ISSN 1367-4803. doi: 10.1093. 1000//boinfoommaiccooooo bioinformaaiccs/bi0088. URL hhttps://doo http://....0//.boooooooaaaaaaoop
Jonathan B. Baell and Georgina A. Holloway. New substructure filters for removal of pan assay interference compounds (pains) from screening libraries and for their exclusion in bioassays. Journal of Medicinal Chemistry, (3))2271 2740, 2010. doi: 10.1021/im901137) URL httLhtttp://doo http:////ooooog//...o 11111/991001///111133133 PMID: 20131845.
Igor I Baskin, David Winkler, and Igor V Tetko. A renaissance of ncural networks in drug discovery. Expert Opin Drug Discou, 11(8):785 795, July 2016.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-27391 2021.
Helen M. Berman, John Westbrook; Zukang Feng, Gary Gilliland, T. N. Bhat, Helge Weissig, Nya N. Shindyalov, and Philip E. Bourne. The Protein Data Bank. Nucleic Acids Research, 28(1):235 242, 01 2000. ISSN 0305-1048. doi: 1//09/nar//8..
CRL http://///oi..rrr111111/////////
235
Thomas Blaschke, Ola Engkvist, Jürgen Bajorath, and Hongming Chen. Memory-assisted reinforcement learning for diverse molecular de novo design. Journal of cheminformalics, 12(1):1 17, 2020.
Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3):1096-111 1108, 2019.
Nadezhda Chirkova and Sergey Troshin. CodeBPE:
Investigating subtokenization options for large language model pretraining source code. In The Fleventh International Conference on Learning Represenlations, 2023. URL httpps:/opeerevvice wwt.eeewwweeeeeeeeee doorumiddhttLAA2344m
Stefan Chmiela, Alexandre Tkatchenko, Huziel F. Sauceda, Tgor Poltavsky, Kristof T Schütt, and Klaus- Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances, 366333133 2017.
Tobiasz Cicplinski, Tomasz Danel, Sabina Podlewska, and Stanislaw Jastrzebski. We should at least be able design molecules that dock well. arXie preprint arxww:0006. 16955, 2020.
Sandra W. Cowan-Jacob, Gabriele Fendrich, Andreas Floersheimer, Pascal Furet, Janis Liebetanz, Gabriele Rummel, Paul Rheinberger, Mario Centeleghe, Doriano Fabbro, and Paul W. Manley. Structural biology contributions to the discovery of drugs to treat chronic myelogenous leukaemia. Acta Crystallographica Section D, 63(1):80 93, Jan 2007, doi: 10.1077/500774499004777777 URL https://dol: attoi.org//0.1100 509074449050472877
George E. Dahl, Navdeep Jaitly, and Ruslan Salakhutdinov. Multi-lask neural networks for qsar predictions, 2014.
Nicola, De Cao and Thomas Kipf. MolGAN: An implicit generative model for small molecular graphs. JCML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018
Jörg Degen, Christof Brgscheid-Gerlach, Andrea Zaliani, and Matthias Rarey. On the art of compiling and using 'drug-like' chemical fragment spaces. ChemMeddChem, 3:1503 7, 10 2008. doi: 10.1002/cmde cmdcc 200800178.
Dominique Douguet, Hélène Munier-Lehmann, Gilles Labesse, and Sylvie Pochet. LEA3D: computer-aided ligand design for struct ure-based drug design. J Med Chem, 48(7):2457-244) April 2005.
Pavol Drotár, Arian Rokkum Jamasb, Ben Day, Cätälina Cangea, and Pietro Liò. Structure-aware generation of drug-like molecules. Xixie preprint or Xiv:21 11.04107, 2021.
15

---

Published in Transactions on Machine Learning Research (12/2023)
Peter Eastman, Pavan Kumar Behara, David L Dotson, Raimondas Galvelis, John E Herr, Josh T Horton, Yuezhi Mao, John D Chodera, Benjamin P Pritchard, Yuanqing Wang, et al. Spice, dataset of drug-like molecules and peptides for training machine learning potentials. Scientific Data, 10(1):11, 2023.
Paul G Francocur, Tomohide Masuda, Jocelyn Sunscri, Andrew Jia, Richard B Iovanisci, Ian Snyder, and David R Koes. Three-dinenniooall convolutional neural networks and a cross-docked data set for structure- based drug design. Journal of chemical information and modeling, 60(9):4200 42150422---2 2020.
Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: large-scale bioac- tivity database for drug discovery. Nucleic acids research, 40(D1):D1100 1110)--110--11777 2012.
Erik Gawehn, Jan Hiss, and Gisbert Schneider. Deep learning drug discovery. Mol Inform, 35(1):3 3-14, December 2015.
Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmelry-adapted generation of 3d point sets for the targeted discovery of molecules. Advances in nearal information processing systems, 32, 2019.
Raj Ghugare, Santiago Miret, Adriana Hugessen, Mariano Phiclipp, and Glen Berseth. Searching for high- value molecules using reinforcement learning and transformers, 2023.
Rafael
Jennifer N Wei, David Duvenaud, José Miguel HernándezzLobato, Benjamín Sánchez-Tengeling Dennis Sheberla, Jorge Aguilera-Tparragune Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 4(2):268 276, 2018.
Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma. 3d equivariant diffusion for target-aware molecule generation and affinity prediction. arXiv preprint orXiv:2303.035. 2023.
Tuomas Haarnoja, Haoran Tang, Picter Abbecl, and Sergey Levine. Reinforcement lcarning with deep energy-based policies. In International conference on machine learning, pp. 1352 1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861 1870. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint ar 181: 05905, 2018b.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling, Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pp. 8867-8887. PMI.R,
2022.
Liegi Hu, Mark I. Benson, Richard D Smith, Michael G Lerner, and Heather A Carlson. Binding moad (mother of all databases). Proteins: Structure, Fanction, and Bioinformalics, 60(3):333 340, 2005.
Ilia Igashov, Hannes Stärk, Clément Vignac, Victor Garcia Satorras, Pascal Frossard, Max Welling, Michael Bronstein, and Bruno Corrcia. Equivariant 3d-conditional diffusion models for molecular linker design. arXiv preprint N.V::221000577, 2022.
John J. Irwin, Teague Sterling, Michael M. Mysinger, Erin S. Bolstad, and Ryan G. Coleman. Zinc: free tool to discover chemistry for biology. Journal of Chemical Information and Modeling, 52(7):1757-1768, 2012. doi: c0.1027/ URL
101110111111/00 PMID: 22587354.
Clemens Isert, Kenneth Atz, José Jiménez-Luna, and Gisbert Schneider. Qmugs, quantum mechanical properties of drug-like molecules. Scientifie Data, 9(1):273, 2022.
16

---

Published in Transactions on Machine Learning Research (12/2023)
Moksh Jain, Sharath Chandra Raparthy, Alex Gernández-Gariia, Jarrid Rector-Brooke Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective GFlowNets. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 14631 14653. PMLR, 23 29 Jul 2023. URL httts::/ppocceedingg mw...g......pppr....... ppress/222//jan2222) htmml
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In In- ternational Conference on Learning Representations, 2017. URL https::/openreviex: wtt.://wwww.eeeeoomnniniti
TkE3y8bee
Siddhant M. Jayakumar, Wojeiech M. Czarnocki, Jacob Menick, Jonathan Schwarz, Jack Rac, Simon Osin- dero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where to find them. In International Conference on Learning Representations, 2020. URL https://openrrieew.nttt Torum?id=ryInKKSvDDI
Woosung Jeon and Dongsup Kim. Autonomous molecule generation using reinforcement learning and docking develop potential novel inhibitors. Scientific reports, 10(1):22104, 2020.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autocncoder for molccular graph generation. In International conference on machine learning, pp. 2323 2332. PMLR, 2018.
Hiroshi Kajino. Molecular hypergraph grammar with its application molecular optimization. In Interna- tional Conference on Machine Learning, pp- 3183 3191. PMLR, 2019.
Kuzma Khrabrov, Πya Shenbin, Alexander Ryabov, Artem Tsypin, Alexander Telepov, Anton Alekseev, Alexander Grishin, Pavel Strashnov, Petr Zhilyaev, Sergey Nikolenko, et al. nabladft: Large-scale con- formational energy and hamiltonian prediction benchmark and dataset. Physical Chemistry Chemical Physics, 24(42)2558332255 -28633, 2022.
Thomas N. Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL
SJU4ayYg1
Georgiy Korenev, Sergey Yakukhnov, Anastasia Druk, Anastasia Golovina, Vitaly Chasov, Regina Mirgaya- zova, Roman Ivanov, and Emil Bulatov. USP7 inhibitors in cancer immunotherapy: Current status and perspective. Cancers (Basel), 14(22), November 2022.
Alexios Koutsoukas, Keith J. Monaghan, Xiaoli Li, and Jun Huan. Deep-learning: investigating deep neural networks byper-parameters and comparison of performance to shallow methods for modeling bioactivity data. Journal of Cheminformaticc, 9(1):42, Jun 2017. ISSN 1758-2946. doi: 10.116//13331-001-00226
URL httpp:://bbi.orr//..111181 210_0-6666612222--11--0000011110
Greg Landrum, Paolo Tosco, Brian Kelley, Ric, sriniker, gedeck, Riccardo Vianello, NadineSchneider, Eisuke Kawashima, Andrew Dalke, Dan N, David Cosgrove, Brian Cole, Matt Swain, Samo Turk, Alexander- Savelyev, Gareth Jones, Alain Vaucher, Maciej Wójeikowski, Ichiru Take, Daniel Probst, Kazuya Ujihara, Vincent F. Scalfani, guillaume godin, Axel Pahl, Francois Berenger, JLVarjo, strets123, JP, and Doliath- Gavid. / 2022 03 1 [ql 2022) release, March 2022. URL https:/:::///o org/ 2tt.:/i....r/01..22222//e./ 122.ee/e00....... 6388425
Stephen J Lane, Drake Eggleston, Keith Brinded, John Hollerton, Nicholas I, Taylor, and Simon A Readshaw. Defining and maintaining a high quality screening collection: the GSK experience. Drug Discov Today, (1(66)):267 2727772272222 March 2006.
Paul R. Leger, Dennis X. Hu, Berenger Biannic, Minna Bui, Xinping Han, Emily Karbarz, Jack Maung, Akinori Okano, Maksim Osipov, Grant M. Shibuya, Kyle Young, Christopher Higgs, Betty Abraham, Delia Bradford, Cynthia Cho, Christophe Colas, Scott Jacobson, Yamini M. Ohol, Deepa Pookot, Payal Rana, Jerick Sanchez, Niket Shah, Michael Sun, Steve Wong, Dirk G. Brockstedt, Paul D. Kassner, Jacob B. Schwarz, and David J. Wustrow. Discovery of potent, selective, and orally bioavailable inhibitors of usp7
17

---

Published in Transactions on Machine Learning Research (12/2023)
with in vivo antitumor activity. Journal of Medicinal Chemistry, 63(10):5398 54200 2623. doi: 10.1021 www.sjjjecceem..0000000 URL https:://di..rrgg 10..////..rr//1..00//////o w
5420, May 2020a. ISSN 0022-
Paul R. Leger, Dennis X. Hu, Berenger Biannic, Minna Bui, Xinping Han, Emily Karbarz, Jack Maung, Akinori Okano, Maksim Osipov, Grant M. Shibuya, Kyle Young, Christopher Higgs, Betty Abraham, Delia Bradford, Cynthia Cho, Christophe Colas, Scott Jacobson, Yamini M. Ohol, Deepa Pookot, Payal Rana, Jerick Sanchez, Niket Shah, Michael Sun, Steve Wong, Dirk G. Brockstedt, Paul D. Kassner, Jacob B. Schwarz, and David J. Wustrow, Discovery of potent, selective, and orally bioavailable inhibitors of usp7 with in vivo antilumor aclivity. Journal of Medicinal Chemistry, 63 63(10))5398 5420, 2020b. doi: 10.1021/ acs. ros.jmedcchem.0000
URL https:///oi...oo http://oo...rr//0..000////aaaa.o www.:essejjeccee.oocooo
PMID: 32302140.
Peng Li and Min Liu. Recent advances in the development of ubiquiinn-pociiii-proossig protease 7 (USP7) inhibitors. Eur J Med Chem, 191:112107, February 2020.
Yibo Li, Jianfeng Pei, and Luhua Lai. Structure- based de novo drug design using 3d deep generative models. Chemical science, 12(41):13664-13677 2021.
Shu-Kun Lin. Pharmacophore perception, development and use in drug design. edited by osman f. güner. Molecules, 577):987-989, 2000. ISSN 1420-3049. doi: 10.3390/50700987. URL htttps//www. http:://wwwmmpp...... com/
1420-3049/5/7/987
Christopher A. Lipinski, Franco Lombardo, Beryl W. Dominy, and Paul J. Feeney. Experimental and computational approaches to estimate solubility and permeability in drug discovery and development settings. Advanced Drug Delivery Reviews, 23(1):3 25, 1997. ISSN 0169-409X. doi: https://doi.
org/ 0eee/10.011/// 100.000011/01194000000000 /00/0/00000000000000000022---213 URL S0169409X966042311
http://wwwsccoooodreetttttt wotpcct..cmm/// woom///ciene//rreeeeeeeee /atte///aticlepp///eo Tn Vitro Models for Selection of Development Candidates.
Meng Liu, Youzhi Luo, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji. Generating 3d molecules for target protein binding. International Conference on Machine Learning, 2022.
Paul Lyne. Structure-based virtual screening: an overview. Drug discovery loday, 7(20):1047 1055, 2002.
Junshui Ma, Robert Sheridan, Andy Liaw, George E. Dahl, and Vladimir Svetnik. Deep neural nets as a method for quantitative structure activity relationships. Journal of Chemical Information and Modeling, 25(2):263-274, Feb 2015. ISSN 1549-9596. c1500747m
doi: 10.1021/ci500747n. URL https://doli ftttss://ao..org/10.100 org/10.101///
Andreas Mayr, Günter Klambauer, Thomas Unterthiner, and Sepp Hochreiter. Deeptox: Toxicity prediction using deep learning. Frontiers in Environmental Science, 3, 2016. ISSN 2296-665X. doi: 10.3389 2015.00080. URL
www.:/frorttrrrss...rrrrrrraaaae 330.:////.0033/////////// 2015.00080
Eyal Mazuz, Guy Shtar, Bracha Shapira, and Lior Rokach. Molecule generation using transformers and policy gradient reinforcement learning. Scientific Reports, 13(1):8799, May 2023. ISSN 2045-2322. doi:
10.1038/s4159880023 119-35648-w. URL https://doi. ott.://........o....///.///00--0
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/ 1612.222, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540))529- -33, Feb 2015. ISSN 1476-4687. doi: 10.1038/ 14236. URL https: http:///........11////11. /tt.//00..008//aauue1442225
H. L. Morgan, The generation of a unique machine description for chemical structurcs-a technique developed chemical abstracts service. Journal of Chemical Documentation, 5(2):107-113, May 1965. ISSN 0021- 9576. doi: 10.1021 (002)cc66017701s URL httpp://oaaiarrr//00000////11110
18

---

Published in Transactions on Machine Learning Research (12/2023)
Michael M. Mysinger, Michael Carchia, John. J. Irwin, and Brian K. Shoichet. Directory of useful decoys, enhanced (dud-e): Better ligands and decoys for better benchmarking. Journal of Medicinal Chemistry, 55(14)66582 6594, 2012. 22716043.
doi: 10.1021/jm300667. URL http::://dii.org10...00 /00g///00.0022/00000067 PMID:
Noel M. O'Boyle, Michael Banck, Craig A. James, Chris Morley, Tim Vandermeersch, and Geoffrey R. Hutchison. Open babel: An open chemical toolbox. Journal of Cheminformatiies, 3(1):33, Oct 2011. ISSN 1758-2946. doi: 10.1186
33. URL https://doi: org/ 1186/117 3-36-33-33
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformualics, 9(1 ):48, Sep 2017. ISSN 1758-2946. doi: 10. 10.1186/113211-01--0233-- URL https: /tt/://..0......111/11/111111110
Rita I. Oliveira, Romina A. Guedes, and Jorge A. R. Salvador. Highlights in usp7 inhibitors for cancer trcatment. Frontiers in Chemistry, 10, 2022. ISSN 2296-2646. doi: 10.3389 10.3389/fchenm http//fhh...22// 2022.1005777. URL
https://wwxt www.tiieiiiiiinn..mommmmmm http://www..raaieeaaiiaarrrratt/ www./rtiileee///. 3t19//0..389ffhf...mm/o 2022. 1005727
George Papadatos, Mark Davies, Nathan Dedman, Jon Chambers, Anna Gaulton, James Siddle, Richard Koks, Sean A. Irvine, Joe Pettersson, Nicko Goncharoff, Anne Hersey, and John P. Overington. SureChEMBL: large-scale, chemically annotated patent document database. Nucleic Acids Research, 44 (D1):D1220-DD) D1228, 11 2015. ISSN 0305-1048. doi: 10.1093/ 10.1093/nar/gkvv nar/gkv12533 URL http:://dooiooo///// 1093/nar/gkvi28)
Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. Time limits in reinforcement learning. International Conference on Machine Learning, pp. 4045 4054. PMLR, 2018.
Kyubyong Park, Joohong Lee, Scongbo Jang, and Dawoon Jung. An empirical study of tokenization strategies for various korean nlp tasks. CoRR, abs/2010.02534 2020. URL https://arxiiv.rrg
02534
Xingang Peng, Shitong Luo, Jiaqi Guan, Qi Xie, Jian Peng, and Jianzhu Ma. Pocket2mol: Efficientt molecular sampling based on 3d protein pockets. In International Conference on Machine Learning, pp. 17644 17655. PMLR, 2022.
Thomas D. Penning, Gui-Dong Zhu, Jianchun Gong, Sheela Thomas, Viraj B. Gandhi, Xuesong Liu, Yan Shi, Vered Klinghofer, Eric F. Johnson, Chang H. Park, Elizabeth H. Fry, Cherrie K. Donawho, David J. Frost, Fritz G. Buchanan, Gail T. Bukofzer, Luis E. Rodriguez, Velitchka Bontcheva-Diaz. Jennifer Bouska, Donald J. Osterling, Amanda M. Olson, Kennan C. Marsh, Yan Luo, and Vincent L. Giranda. Optimization of phenyl-substituted benzimidazole carboxamide poly(adp-ribose)) polymerase inhibitors: Identification of highly potent and efficacious inhibitor. Journal of Medicinal Chemistry, :112)3311233111 Apr 2010. ISSN 0022-261 2023-2623. doi: 10.1021/jm9011755y. URL https: //opii org//o. wtt.://......///000011111111111.
(a-966492), a
Albert C Pierce, Govinda Rao, and Guy W Bemis. BREED: Generating novel inhibitors through hybridiza- tion of known ligands, application to CDK2, p38, and HIV protease. Med Chem, 47(11):2768 2775, May 2004.
Pavel Polishchuk. Crem: chemically reasonable mutations framework for structure generation. Journal of Cheminformatics, 12(1):28, Apr 2020. ISSN 1758-2946. doi: 10.1188/413311-00000085 //doi. org/1000000 0100011100001111000000000000
URL https:
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling. Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, et al. Molec- ular sets (moses): a benchmarking platform for molecular generation models. Frontiers in pharmacology, 11:565644, 2020.
Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science advances, 4(7):eaap7885, 2018.
19

---

Published in Transactions on Machine Learning Research (12/2023)
Bharath Ramsundar, Steven Kearnes, Patrick Riley, Dale Webster, David Konerding, and Vijay Pande. 2015.
Massively multitask networks for drug discovery. orXiv preprinl
Jean-Louis Reymond and Mahendra Awale. Exploring chemical space for drug discovery using the chemical universe database. ACS chemical neuroscience, 399):649 657, 2012.
David Rogers and Mathew Hahn. Extended fingerprints. Journal of chemical information and modeling, 50(5):742 754, 2010.
S H Rotstein and M A Murcko. GroupBuild: a. fragment- method for de novo drug design. J Med Chem, 36(12):1700-1110, June 1993.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05555 2015.
Gisbert Schneider and Uli Fechner. Computer-based de novo design of drug-like molecules. Nat Rev Drug Discov, 4(8))649-60) 663, August 2005.
Arne Schneuing, Yuanqi Du, Charles Harris, Arian Rokkum Jamasb, Tlia Tgashov, weitao Du, Tom Leon Blundell, Pietro Lio, Carla P Gomes, Max Welling, Michael M. Bronstein, and Bruno Correia. Structure- based drug design with equivariant diffusion models, 2023. URL httpps://oppmreiiie.no/////eeees http://iwww..toooonioii.aa uKmuzIull IuV18z
Marwin H. S. Segler, Thierry Kogej, Christian Tyrchan, and Mark P. Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS Central Science, (1):110-111,, 2018. doi: 10.1021 10.1021/acccenntee crtscentscii7b005512 URL htttps://aii.rrg// 10211/0..1012/1 wttp:////....1...aaaatttt.......
PMID: 29392184.
Tatiana 1. Shashkova, Dmitriy Umerenkov, Mikhail Salnikov, Pavel V. Strashnov, Alina V. Konstanti- nova, Ivan Lebed, Dmitriy N. Shcherbinin, Marina N. Asatryan, Olga L. Kardymon, and Nikita V. Ivanisenko. Sema: Antigen b-cell conformational epitope prediction using deep transfer learning. Fron- tiers in Immunology, 13, 2022. ISSN 1664-3224. doi: 10.3389 10.3339//fmmm.2022..... /ttm:mmmm022..000055.. URL https:
7/ww. httpantiersiin.orrrrtttillleeeee http///....99/ffmmmmm 202.. 960985
Brian K Shoichet. Virtual screening of chemical libraries. Nature, 432(7019)::622 865, 2004.
Gregor N. C. Simm, Robert Pinsler, Gábor Csányi, and José Miguel Hernández-Lobatoo. Symmetry-aware actor-critic for 3d molecular design. In International Conference on Learning Represendations, 2021. URL https://openreeview httpe/vwwwwwwwwefoeeeee (oo
Jacob O. Spiegel and Jacob D. Durrant. Autogrow4: an opun-source genetic algorithm for de novo drug design and lead optimization. Journal of Cheminformattiss, 12(1):25, Apr 2020. ISSN 1758-2946. doi: URL
10.1186/133211-000004443
Fci Sun, Pengyun Li, Yi Ding, Liwci Wang, Mark Bartlam, Cuilin Shu, Beifen Shen, Hualiang Jiang, Song Li, and Zihe Rao. Design and structure-based study of new potential FKBP12 inhibitors. Biophys J, 85 0-8194--201,, November 2003.
Mengying Sun, Sendong Zhao, Coryandar Gilvary, Olivier Elemento, Jiayu Zhou, and Fei Wang. Graph convolutional networks for computational drug development and discovery. Brief. Bioinform., 21(:)9919- 935, May 2020.
Yi Tay, Vinh Q. Tran, Sobastian Rudor, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. In International Conference Learning Representations, 2022. URL https: //openreview.net.o hwt.
Morgan Thomas, Robert T Smith, Noel M O'Boyle, Chris de Graaf, and Andreas Bender. Comparison of structure-and ligand-based scoring functions for deep generative models: gper case study. Journal of Cheminformatics, 13(1):1- (011):1-20, 2021.
20

---

Published in Transactions on Machine Learning Research (12/2023)
Thomas Unterthiner, Andreas Mayr, Günter Klambauer, Marvin Steijaert, Jörg K Wegner, Hugo Ceulemans, and Sepp Hochreiter. Deep learning as an opportunity in virtual screening. In Proceedings of the deep learning workshop at NIPS, volume 27, pp. 1 9, 2014.
Tzhar Wallach, Michael Dzamba, and Abraham Heifets. Atomnet: A deep convolutional neural network for bioactivity prediction in structure-based drug discovery. CoRR, abs/1510.02855, 2015. URL http: 7/dblp. trier
cttp///rrccccr11110// html#WallachDHH5
Chong Wang, Yi Jiang, Jinming Ma, Huixian Wu, Daniel Wacker, Vsevolod Katritch, Gye Won Han, Wei Liu, Xi-Ping Huang, Eyal Vardy, John D. McCorvy, Xiang Gao, X. Edward Zhou, Karsten Melcher, Chenghai Zhang, Fang Bai, Huaiyu Yang, Linlin Yang, Hualiang Jiang, Bryan L. Roth, Vadim Cherezov, Raymond C. Stevens, and H. Eric Xu. Structural basis for molecular recognition at serotonin receptors. Science, 340(6132):610-614 614, 2013. doi: 2..116//cciene..225 abs/10. 1tte/...112/sccinnee 1232807
URL https::/ //www. science. org.eoiroooolloo
Steven D. Whitehead and Dana H. Ballard. Learning to perceive and act by trial and error. Machine Learning, 7(1):45-833 83, Jul 1991. ISSN 1573-0565. doi: 10.1023/ A:1022619109594 URL https://dol org/
10.
Oliver Wieder, Stefan Kohlbacher, Mélaine Kuenemann, Arthur Garon, Pierre Ducrot, Thomas Seidel, and Thierry Langer. A compact review of molecular property prediction with graph neural networks. Drug
Discovery Today: Technologies, 37:1-12, 2020. ISSN 1740-6749. doi: hupps///ooiorg//00000///// 2020.11.009. URL
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chem. Sci., doi: 10.1039 10.1039/C7SC026611 C7SC02664A. URL http: doi. www.dddooooo.......ooooo///o/o.. 100.00011199970082688000
9:513-530 530, 2018.
Soojung Yang. Doycong Hwang, Scul Lee, Scongok Ryu, and Sung Ju Hwang. Hit and lead discovery with explorative rl and fragment-basee molecule generation. Advances in Neural Information Processing Systems, $4:7924-7938 2021.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in newral information processing systems, 31, 2018.
Yaxia Yuan, Jianfeng Pei, and Lubua Lai. LigBuilder v3: A Multi-Target de novo drug design approach. Front Chem, 8:142, February 2020.
Katrin Groebke Zbinden, Ulike Obst-Sander, Kurt Hilpert,, Holger Kühne, David W. Banner, Hans-Joachim Böhm, Martin Stahl, Jean Ackermann, Leo Alig, Lutz Weber, Hans Peter Wessel, Markus A. Riederer, Thomas B. Tschopp, and Thierry Lavé. Selective and orally bioavailable phenylglycine tissue factor ceatoorfaacce /factor viia inhibitors. Biooryanic & Medicinal Chemástry Letters, 15(23) 5334-552,33.. 2005. ISSN 0960-894X. doi: bww..mm0...0...mmmmmmm 04.099.
pi1/509908894000011668
Xinsong Zhang and Hang Li. (AMBERT): A pro-trained language model with multi-grained tokenization, 2021. URL htttps://ppenreviiii htt.:///ee/.oouuriodd//aooonooo
Alex Zhavoronkov, Yan A Ivanenkov, Alex Aliper, Mark S Veselov, Vladimir A Aladinskiy, Anastasiya V Aladinskaya, Victor A Terentiev, Daniil A Polykovskiy, Maksim D Kunnetsov, Arip Asadulaev, el. al. Deep learning enables rapid identification of potent ddrl kinase inhibitors. Nature biotechnology, 37( 1038-1040, 2019.
Zhenpeng Zhou, Steven Kearnes, Li Li, Richard N Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning. Scientific reports, 9(1):1-10, 2019.
Brian D. Zicbart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse rein- forcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence Volume 3, A AA pp. 1433-1438. AAAT Press, 2008. ISBN 9781577353683.
21

---

Published in Transactions on Machine Learning Research (12/2023)
A Other changes
A.1 Reproducibility
The docking score computation consists 2 steps: first, a molecular conformation is generated from a molecular graph with OpenBabel software; then, the docking score estimated with the QVina02 tool. Both steps include random sampling, thus requiring setting random seeds to ensure the reproducibility of the experiments. In the original implementation, seeds are set only for torch, random, numpy, and dgl packages while ignored for QVina02 and OpenBabel libraries. In our experiments, we fix that and set seeds for all the librarics.
A.2 Fragment library preparation
For reward computation in FREED, molecular conformation is generated with OpenBabel software. In some cases, OpenBabel may fail to generate coordinates for charged molecules. We exclude charged fragments from the fragment library during the preprocessing step to avoid such situations. For the fragment library obtained with the CReM fragmentation method from the ZLNC dataset, there are five charged fragments out of 91 total fragments. We use a filtered library of 86 uncharged fragmonts in the main experiments.
A.3 Exploration
One commonly used strategy to increase agent exploration of the environment is to select actions uniformly. Such enviromment exploration can be done periodically with some probability (e-greedy strategies) or once before the actor and the critic updates. In the original implementation, the mumber of steps for uniform- random action selection was set to 3k, after which the real policy was run. While this technique helps exploration, we remove from our framework for debugging purposes and to save computational time.
Additionally, start update of all networks' parameters soon the replay buffer is filled with batch- size observations. In the original implementation, the number of environment interactions to collect before starting to do gradient descent updates was set to 2k. Such a strategy is used to ensure that the replay buffer is complete enough for useful updates. However, we believe this stage should not change performance significantly, and we drop it for the simplicity of debugging.
We test the importance of described modifications on the DS optimization task. From table that additional exploration phase does not improve the performance of the model.
WC can see
A.4 Backbone update
To cope with the overestination bias, the SAC algorithm uses an ensemble two Q-lunctions (Harnojj et al. 2018b). In the original FREED framework, the actor and both Q-functions share GCN backbone, and the gradients are propagated to the backbone only through the critic loss. In the original implementation propagated only through one Q-function (a "stop gradient" is applied to the second one). We follow the original implementation of the SAC algorithm (Haarnoja et al. 2018b), and allow the gradients to propagate to the GCN through both Q-functions.
the gradients
A.5 Temperature parameter in SAC
FREED uses version of the SAC algorithm with learnable temperature parameter a. Automatic tuning of a during training allows to avoid costly manual grid-scarch. In the original implementation, is tuned only between interaction steps 3000 and 33000. We do not see any reason for such a setup and follow the procedure described in (Haarnoja el al. 2018b), which suggests that alpha should be tuned throughout the whole training.
Morcover, the a parameter clipped to be in the range [0.05, 20 during the computation of the entropy loss term in the actor update. Here, again follow (Haarnoja al. 2018b) and discard the clipping of
22

---

Published in Transactions on Machine Learning Research (12/2023)
A.6 Terminal states handling
In the FREED framework, trajectories are lerminated either when the timelimit is reached or when there are no attachment points in the extended molecular graph representing the current state. Adding terminal states to the replay buffer is crucial, as the docking score can only be calculated in such states. In the original implementation, terminal states with no attachment points are not added to the replay buffer. This approach negatively influences the training of the critic as other states from such trajectories do not carry any information about the docking score.
A.7 Reward shaping
In the original implementation, the authors use reward shaping (sec eq- 15 cach step except for the terminal one, the agent received constant rewards fi 0.05. Typically, such constant reward is used when want to increase episode length artificially. In FREED framework, the episode's length is fixed, there is point in such increment.
Moreover, when the number of explicit atoms NEA (including attachment points) in the molecular graph increases, an additional reward 0.005 is given. Note hydrogen atoms considered to be presented in the graph implicitly, as in REINVENT, MolDQN, and GCPN, The only scenario in which N EA((++1) = EE(ss;) is when an attachment point is replaced with a single atom-like CL.
0.05,
N NEA()(+1) S NEA(s)), t<T NEA($ > NEA(s)); t<T
a(, 81+1)
0.055,
(15)
max(00 DS())), t
In our experiments, we get rid of the reward shaping and only assign a non-zcro reward to the terminal states, where it possible to calculate the docking score:
0, if <; re(St, (1,
(16)
max(0, (8++))))), ift=
A.8 Entropy estimation
As the SAC algorithm designed to solve the Maximum Entropy RL task, the entropy regularization is crucial. To update both actor and critic SAC, an estimate of the entropy H[m)] used. standart implementation of the algorithm (Haarnoja al. 2018b) uses the following estimate:
H [MT[s)] - log (ā R2, @ils), ã. no. i= 1,2,3
FREED implementation, a non-default estimate of entropy is used:
Ta 02 an Tū,, ō1, 8) The (aslāz, āi, 8) T (a2) lū s) ne, (ails)
[s) = Σ (a a2 a1 ā2 āi, 8) log (a a2 a T0 ăn 8), âi ~ -T, āg ~ Tzaz
01.02.03
Despite the fact that such an estimate is unbiased, we use the vanilla version it is easier implement and the implementation is more readable.
A.9 Arhitecture
Architectures of FREED, FFREED and FREED++ are presented in table g In fusing functions fz, for, E for FFREED and FREED + output size 12= instead dy=64 in FREED. We use 3-layer ML.Ss
23

---

Published in Transactions on Machine Learning Research (12/2023)
with a number of hidden neurons (128, 128) as projectors fur fus: while in FREED, 2-layer MLPs with 32 neurons are used. Additionally, use 4-layer MLPs as critic and auxiliary reward predictor heads with layer normalization. Moreover, in FREED, on the third step of action selection, embedding of fragment ag is used to condition the attachment point selection:
As 01, 01) fog (Fag. Fast)
(17)
We instead use fused representation of state, attachment point and fragment embeddings:
As a1 a2) fōs (02, Fatt)
(18)
To sum it up, use wider and deeper architecture for FFREED than the original FREED uses. In the case of FREED++, even with increased embedding sizes, the resulting architecture appears to be smaller (in terms of the total number of trainable parameters) because of the simplified mechanism fragment selection (see section 4.3) To test the importance the increased number of layers and neurons, we compare the FFREED model and its version FFREED(ORIC) with same module sizes as in FREED. We observe that this change does affect the model performance (see table 4 Anyway, the resulting FREED++ model is faster and requires fewer paramcters than FREED (scc table 10
module
FREED
FFREED
FREED ++
Go
GCN(29, (128, 128, 128))
GCN(29, (128, 128, 128))
GCN(29, (128, 128, 128))
fo.
MI(128, 128, 64)
MI(128, 128, 128)
CAT((18,, 128, 128)
f
MLP(64, (32, 1))
MLP(128, (128, 128, 1))
MLP(118, (128, 128, 1))
MI(1024, 64, 64)
MI(1024, 128, 128)
CAT(118, 128, 128)
MLP(64, (64, 64, 1))
MLP((18, (128, 128, 1))
MLP(118, (128, 128, 86))
MI(128, 128, 64)
MI(128, 128, 128)
CAT(128, 128, 128)
MLP(G6, (32, 1))
MLP((18,, (128, 128, 1))
MLP((18,, (128, 128, 1))
f.
M299, (149, 1))
MLP(512, (256, 128, 128, 1))
MLP(512, (256, 128, 128, 1))
r
G2N(29, (128, 128, 128))
GCN (128, 128, 128))
M12( (64, 1))
M12( (128, 128, 128, 1))
Table 3: Comparison of architecures of FREED, FFREED and FREED++.
B
Gumbel-Softmax Issues
When sampling from categorical distribution x defined by class probabilitics Ti, Tke the following reparametrization is used (Jang et al. 2017):
exp ((log n. + 91)TT)
y/ Σイ= exp (llgg n, 9;)/T) -.,.....9. 9b i.i.d. from Gumbel(O, 1).
(19)
The authors of the original paper use the following reparametrization in their implementation:
exp ((Ti pg:)/T)
V: Σラ exp ((T, "gz)/T)
exp (1 + +9:)/E)
(20)
= exp (1 + 9;)/
exp (log(eexp 4) + 9:)/
Σ5 exp (log(exx +) + 9;)/3)
As can be seen from equation 20 sampling from such distribution is equivalent to sampling x exp (P) with temperature g. In the original implementation, v was set x 10-3 3 and r was set to x 10-1, making the temperature parameter equal to I 100. The described issue had thus the following effect on the action distribution:
24

---

Published in Transactions on Machine Learning Research (12/2023)
protein
name
unique (1)
Avg DS (1)
Max DS (1)
Top-5 DS (t)
fa.7
FFREED(ORIG)
0.60 + 0.18
9.52 + 0.53
13.33 + 0.25
12.40 + 0.07
FFREED
0.53 + 0.12
8.39 = 0.82
12.86 0.05
12.37 = 0.01
FREED
0.43 + 0.40
7.89 + 0.46
10.00
1.12
9.53 + 0.85
parpl
FFREED(ORIC)
0.58 + 0.17
10.09 = 2.29
17.50 + 0.91
15.93 + 0.27
FFREED
0.58 + 0.09
12.16 + 1.58
17.90 + 0.98
16.13 + 0.15
FREED
0.33 + 0.32
10.57 = 0.81
12.80 1.91
12.29 + 1.51
5htlb
(ORIED(ORIG)
0.50 + 0.20
11.49 = 0.04
14.60 + 0.79
13.45 + 0.07
FFREED*
0.45 + 0.16
11.97 + 0.20
14.85 + 0.21
13.99 + 0.45
FREED
0.21 + 0.21
6.43 = 5.57
8.83 + 7.65
7.97 + 6.90
fkbla
FFREED(ORIG)
0.42 + 0.09
8.19 + 0.13
11.50 + 0.00
10.65 = 0.26
FFREED
0.36 + 0.14
8.26 + 0.05
11.50 + 0.00
10.49 + 0.29
FREED
0.44 + 0.32
5.71 = 0.92
8.60 + 0.91
8.00 + 0.69
abll
(ORIED(ORIG)
0.37 + 0.13
10.74 = 0.35
13.00 0.36
12.39 0.19
FFREED
0.39 + 0.12
10.77 + 0.35
13.03 + 0.23
12.18 + 0.06
FREED
0.24 + 0.23
2.17 + 1.68
7.83 + 3.75
6.36 + 5.28
usp7
POREED(ORIG)
0.49 + 0.14
10.79 = 0.50
14.13 + 0.15
13.37 = 0.23
FFREED
0.66 + 0.01
10.91 + 0.15
14.03 0.46
13.35 + 0.27
FREED
0.10 + 0.07
8.58 = 1.02
11.36 + 0.98
11.12 + 0.88
Table 4: Comparison of small and big FFREED-s. denotes the model has seed with Max DS S on evaluation, while on the last epoch of training Min DS > 0. We exclude such runs.
The initial distribution scaled by 1000, making some of the unnormalized probabilities >> 1.
The exponent was applied to unnormalized probabilities, effectively making the distribution degen- erate as the largest unnormalized probability dominated all the probability mass.
An extremely large temperature т = 100 was applied to somewhat counter the effect the two previous steps.
Together with the apparent effect of making the action distribution deterministic and thus hindering the exploration, such reparametrization inconsistent. with the entropy optimization, as the entropy of the different distribution is maximized.
C Prioritized Experience Replay lssues
In this section, we discuss the PER proposed in the original paper. As mentioned in section 4.3 the probability of transition being sampled from the replay buffer is defined by its novelty. Suppose that rę is the auxiliary reward model. Then, the unnormalized probabilities for transactions in the replay buffer are computed as follows:
|r<(se) (1)) (r((st) rel. t =
t<t T
p( 0t 8141,
(21)
where fo denotes the critic. The auxiliary reward model is trained to minimize the following objective:
(( [llr: (+))11 + min,
(22)
where Df denotes the set of all terminal states where reward rt + 0.
Estimating the con-available reward in non-terminal states with critic has several issues. The critic is trained to optimize the TD loss (see equation 12 The target for the critic includes discounting factor 7
25

---

Published in Transactions on Machine Learning Research (12/2023)
well entropy term arising from the Maximum Entropy framework. Even if consider 1, as operate with fixed-length episodes, the entropy term in the critic target makes it incorrect to use the critic's output as a reward estimate. We hypothesize that for this exact reason, the entropy term was omitted from the critic target in the original implementation (see section 4.1)
Comparison with baselines
protein
unique (1)
Avg DS (1)
Max DS (t)
Top-5 DS (t)
fa7
60
0.99 + 0.00
5.98 + 4.73
10.96 + 0.72
9.80 + 0.88
100
0.99 + 0.00
8.79 + 0.70
11.23 + 0.45
10.49 + 0.48
200
0.90 + 0.04
2.86 + 3.41
10.23 + 0.80
6.71 + 5.46
usp7
60
0.99 + 0.00
9.69 + 0.39
13.13 + 0.89
12.05 + 0.73
100
0.97 + 0.01
9.70 + 0.23
11.56 + 0.20
10.99 + 0.05
200
0.79 + 0.04
10.26 + 1.31
11.93 + 1.17
11.50 + 1.35
parpl
60
0.98 + 0.02
10.04 + 1.42
15.93 + 0.15
14.06 + 0.81
100
0.96 + 0.02
9.28 + 1.62
14.33 + 0.46
13.35 + 0.75
200
1.14
8.90 + 3.59
14.19 + 1.90
13.20 + 1.18
5htlb
60
0.99 0.01
10.27 + 0.16
13.33 + 0.11
12.20 + 0.27
100
0.99 0.00
8.71 + 3.28
13.06 + 0.90
12.08 + 0.91
200
0.90 0.13
10.43 + 2.12
12.13 + 2.55
11.58 + 2.47
abl1
60
0.98 0.00
10.08 + 0.95
12.40 - 0.95
11.66 + 1.21
100
+.00
10.86 + 0.66
12.83 + 0.60
12.21 + 0.47
200
0.85 0.03
11.61 + 0.57
13.16 + 0.20
12.56 + 0.42
fkbla
60
0.99
0.00
7.16 + 0.23
9.03 + 0.37
8.37 + 0.36
100
0.98 + 0.01
7.20 + 0.12
8.90 + 0.36
8.35 + 0.13
200
0.93 + 0.03
7.83 + 0.21
9.33 + 0.11
8.88 + 0.05
Table 5: Comparison of different a parameters for REINVENT on DS optimization task.
26

---

Published in Transactions on Machine Learning Research (12/2023)
MolDQN
fa 7
usp7
8.0
7.5
7.0
DS. Acal mo)
DS. Acal ml)
7.0
6.5
6.5
sparse
sparse
dense
dense
20
40
60
80
100
0
20
40
60
80
100
Itoration
Iteration
parp1
5ht1b b
8.5
9.5
M0
9.0
7.5
8.5
DS. Rtal
mol
Acal D5.
mal
6.5
7.0
6.5
sparse dense
6.0
5.0
abll
fkbla
M MMMM
DS, kcal
2011
DS.
sparse dense
sparse
dense
4.5
20
40
60
100
Itaration
Iteratioon
Figure 5: Average docking score over batch during training for MolDQN. Shaded regions denole 95% confi- dence interval.
27

---

Published in Transactions on Machine Learning Research (12/2023)
fa7
parp1
11
10
DS. Ical
mg
mm1
McIDON
McIDON
8
REINVENT
DS.
REINVENT
FREED
FREED
FFREED FREED +++
FFREED
6
FREED+
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
Epoch
Epoch
5ht1b
fkbla
12.5
10
10.0
Ical
DS, mal
DS. kcal 1R0
7.5
MoIDON REINVENT
MoIDON REINVENT
5.0
FREED
FREED
2.5
FFREED FREED 4-4
FFREED
FREED
0.0
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
Epech
Epoch
abll
usp7
12
12
DS, Koal
mal
8
DS. kiai
10
6
McIDON
FREED
FREED
2
FFREED
FFREED
FREED ++
FREED+
75
100
200
25
150
175
Epoch
Epoch
Figure 6: Average docking score over batch during training. Training curves aligned on x-axis. Shaded regions denote 95% confidence interval.
28

---

Published in Transactions on Machine Learning Research (12/2023)
protein
method
unique (1)
Avg DS (1)
Max DS (t)
Top-5 DS (t)
la7
CombGen
0.98 0.00
7.15 + 0.01
12.00 0.30
9.40 0.01
MolDQN
0.19 0.01
8.00 + 0.27
9.93 + 0.15
9.57 + 0.18
REINVENT
0.99 = 0.00
8.79 0.70
11.23 + 0.45
10.49 + 0.48
Mocket2Mol
1.00 = 0.00
8.26 + 0.01
11.16 + 0.20
10.24 + 0.26
FREED
0.43 = 0.40
1.46
10.00 1.12
9.53 - 0.85
0.85
FFREED
0.53 0.12
8.39 + 0.82
12.86 0.05
12.37 0.01
FREED
0.76 0.05
7.78 + 1.80
13.23 + 0.41
12.37 + 0.07
KI
8.85
10.30
10.10
parpl
CombGen
0.98 = 0.00
8.50 + 0.01
15.26 + 0.57
11.82 + 0.00
MolDQN
0.22 + 0.02
9.51 + 0.49
11.89 0.20
11.35 + 0..00
REINVENT
0.98 = 0.02
1 1.42
15.93 0.15
14.06 0.81
Pocket2Mol
1.00 0.00
11.16 0.13
15.66 0.98
14.09 + 0.54
FREED
0.33 0.32
10.57 0.81
12.80 + 1.91
12.29 + 1.51
FFREED
0.58 = 0.09
12.161..
17.90 + 0.98
16.13 + 0.15
FREED
0.71 = 0.06
12.98 + 0.15
17.73 0.83
16.26 - 0.19
KI
10.13
13.60
12.51
5htlb
CombGen
0.98 0.00
7.63 + 0.04
13.13 0.20
10.85 0.01
MolDQN
0.20 0.03
7.97 + 0.10
10.36 0.64
9.61 + 0.17 0.17
REINVENT
0.99 = 0.01
10.7=:0..
13.33 0.11
12.20 + 0.27
2ocket2Mol
1.00 + 0.00
13.26
11.72 + 0.33
FREED
0.21 0.21
5.57
8.83 +83+7.65
7.97 6.90
FFREED*
0.45 0.16
11.97 + 0.20
14.85 + 0.21
13.99 0.45
FREED+4
0.45 = 0.07
11.78 + 0.12
14.50 + 0.79
13.50 + 0.15
KI
8.07
10.60
10.52
ikbla
CombGen
0.98 = 0.00
3.99 + 0.10
9.56 0.15
7.94 0.01
MolDQN
0.18 0.03
6.75 + 0.16
8.79 0.17
8.20 + 0.12
REINVENT
0.93 0.03
7.83 0.21
9.33 + 0.11
8.88 0.05
Pocket2Mol
1.00 0.00
5.75 0.03
9.00 + 0.45
8.22 + 0.30
FREED
0.44 = 0.32
5.71 + 0.92
8.60 + 0.91
8.00 + 0.69
FFREED
0.36 = 0.14
8.26 + 0.05
11.50 + 0.00
10.49 + 0.29
FREED++
0.30 0.03
8.50 0.06
11.50 + 0.00
10.92 0.02
KI
6.99
8.90
8.76
abll
CombGen
0.98 0.00
3.77 + 0.17
12.83 0.25
10.57 + 0.00
MolDQN
0.16 + 0.02
6.99 + 0.43
9.50 + 0.36
9.05 + 0.49
REINVENT
0.85 + 0.03
11.61 + 0.57
13.16 0.20
12.56 + 0.42
Pocket2Mol
1.00 = 0.00
3.27 - 0.14
13.26 + 0.72
11.37 + 0.28
FREED
0.24 0.23
2.17 1 1.68
7.83 + 3.75
6.36 5.28
FFREED
0.39 0.12
10.77 + 0.35
13.03 + 0.23
12.18 + 0.06
FREED
0.34 + 0.02
11.00 + 0.10
12.86 + 0.05
12.36 + 0.21
3.02
11.70
10.36
usp7
CombGen
0.98 = 0.00
8.03 + 0.00
13.00 0.26
10.53 + 0.00
MolDQN
0.21 0.03
7.92 + 0.23
9.93 + 0.41
9.43 0.34
REINVENT
0.99 0.00
9.69 + 0.39
13.13 0.89
12.05 + 0.73
Pooket2Mol
1.00 = 0.00
8.71 + 0.37
12.36 + 0.40
11.30 + 0.65
FREED
0.10 + 0.07
8.58 + 1.02
11.36 + 0.98
11.12 + 0.88
FFREED
0.66 0.01
10.91 + 0.15
14.03 + 0.46
13.35 + 0.27
FREED++
0.64 0.07
9.66 + 3.28
14.16 + 0.47
13.30 + 0.13
KI
9.40
11.90
11.90
Table 6: Comparison of FREED, FFREED and FREED++ with basclines on DS optimization task. denotes the model has seed with Max DS < 0 on evaluation, while on the last epoch of training Min DS > We exclude such runs.
29

---

Published in Transactions on Machine Learning Research (12/2023)
aa2ar
abll
12
10.0
DS. kcal ma
DS. kcal mol
7.5
+REED++
FREED++
8
2.5
6
50
75
100
125
150
175
200
50
75
100
125
150
175
200
Epoch
ace
aces
12
12
10
DS, Kcal
mo!
10
maa
FREED FREED
FREED FREED++
+
DS.
50
75
100
125
175
200
25
50
125
150
175
200
Epoch
Epach
ada
ada17
11
10
DS. kcal
DS, Ical
mo)
FREED +
FREED FREED++
8
Mmml
w MMM
6
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
Epoch
Epoch
adrbl
adrb2
12
12
DS, kcal
mal
moi
FREED+
DS,
+REED++
mm
6
MMA
WM Vm
4
75
100
125
150
175
200
25
50
100
125
150
175
200
Epoch
a akt1
a akt
10
Kcal DS. mnc)
Anm
FREED FREED++ +
FREED
DS.
FREED+
w
25
75
100 Epach
125
150
175
200
25
50
125
150
175
200
Epach
Figure 7: Comparison of FREED and FREED++ on DS optimization task. 10 first proteins from DUDE considered as targets.
30

---

Published in Transactions on Machine Learning Research (12/2023)
E
Fragments library experiments
fa7
usp7
12.5
DS. kcal
mo
D5. Icall
mol
10
7.5
8
5.0
GReM
MOSES
MOSES
ZNC
ZINC
200
Epoch
Epech
parp1
5ht1b
15.0
10
DS. Rcal
ma
12.5
DS. kcal
mol
10.0
BRICS
ZINC
ZINC
25
50
125
150
25
125
150
200
Epoch
Epoch
abl1
fkbla
10.0
10.0
DS. Neal
mol
7.5
DS. Newl mol
7.5
5.0
BRICS
2.5
2.5
0.0
ZNC
0.0
2IC
25
50
100
125
150
175
200
25
50
100 Epach
125
150
175
200
Epoch
Figure 8: Average docking score over batch during training. Shaded regions denote 95% confidence interval.
F Simplifications of FFREED
In this section, we compare FFREED, FREED++ models and their intermediate versions. As described in FREED++ differs from FFREED 1) the absence of prioritization, 2) the simplified mechanism of fragment selection, and 3) the usage of "lightweight" fusing functions. All proposed modifications aimed to decrease the total amount of parameters in the model and speed up the training process.
We perform ablation studies on the DS optimization task to test the importance of described modifications. We dub FREED++ with a prioritization mechanism FREED++ (PER), FREED++ with mechanism of fragment selection taken from FFREED as FREED+ (AM),, and FREED with multiplicative interac- tions as fusing functions as FREED++ (FUSE). As it can seen in table g the overall performance of different models is approximately the same, while it can differ significantly for particular targets. For ex- ample, FREED++ stably outperforms FREED++ (PER) for fa7 protein, while the situation inversed for parpl target. The same effect can be seen for FREED++ and FREED++(FUSE) models on fa7 and parpl proteins.
To compare the speed of different FFREED versions, we collect 10 Transactions with a uniform policy, split them into 100 batches, and measure the time of gradient updates for each method. From table [0] we can see that for all modifications of FREED++ in the direction of FFREED number of trainable parameters is increased. The most significant increase (~7.3x) occurs when MI is used fusing function. Gradient update times increase when adding modifications to FREED++ only for PER and FUSE settings; for AM
31

---

Published in Transactions on Machine Learning Research (12/2023)
protein fragmentation
dataset
unique (t)
Avg DS (1)
Max DS (t)
Top-5 DS (t)
5htlb
BRICS
MOSES
0.56 1 0.16
9.28 2.22
13.40 1 0.65
12.44 1 0.32
ZINC
0.47 = 0.10
9.47
1.65
13.03 = 0.20
12.23 + 0.31
CReM
MOSES
0.69 = 0.18
7.07 1.90
11.86 + 0.61
10.79 + 0.60
ZINC
0.45 1: 0.07
11.78
0.12
14.50 0.79
13.50 + 0.15
abll
BRICS
MOSES
0.52 = 0.41
1.29
1.31
7.06 6.23
6.33 = 5.58
ZINC
0.61 0.13
5.02 3.45
9.56 2.18
8.78 + 2.50
CReM
MOSES
1.00 + 0.00
0.00
0.00
0.00 0.00
0.00 + 0.00
ZINC
0.34 0.02
11.00 0.10
12.86 0.05
12.36 + 0.21
fa7
BRICS
MOSES
0.61 = 0.09
3.10 0.26
14.20 0.34
13.09 - 0.07
ZINC
0.66 = 0.14
2.82
0.81
13.66 = 0.94
12.48 + 0.89
CReM
MOSES
0.64 + 0.13
1.29 0.95
12.56 1 0.40
9.26 3.78
ZINC
0.76 + 0.05
7.78 1.80
13.23 + 0.41
12.37 1 0.07
kkbla
BRICS
MOSES
0.57 = 0.13
5.79
1.65
9.03 0.35
8.35 + 0.37
ZINC
0.47 1 0.36
6.55 0.86
9.86 0.46
9.20 1 0.33
CReM
MOSES
0.57 + 0.15
4.93 0.68
8.26 0.68
7.67 + 0.64
ZINC
0.30 = 0.03
8.50 0.06
11.50 0.00
10.92 + 0.02
parpl
BRICS
MOSES
0.51 = 0.13
5.89 1.83
16.96 = 0.49
16.07 + 0.31
ZINC
0.60 = 0.27
6.82 1.60
16.70 + 0.36
15.51 + 0.53
CReM
MOSES
0.70 1 0.03
1.51 0.86
15.69 1 0.50
13.82 1 0.57
ZINC
0.71 + 0.06
12.98 + 0.15
17.73 0.83
16.26 + 0.19
usp7
BRICS
MOSES
0.58 + 0.17
5.82
4.32
13.16 1 0.51
12.22 + 0.67
ZINC
0.63 1 0.13
9.07 0.70
13.39 1 0.88
12.42 1 0.46
CReM
MOSES
0.74 + 0.07
1.21
1.06
12.00 + 1.81
7.77 + 6.27
ZINC
0.64 = 0.07
9.66 = 3.28
14.16 0.47
13.30 + 0.13
Table Comparison different fragments librarics used in FREED ++ on DS optimization task.
protein
method
unique (f)
Avg DS (t)
Max DS (t)
Top-ā DS (t)
fa7
FREED++
0.76 = 0.05
7.78
1.80
13.23 + 0.41
12.37 + 0.07
F
0.62 + 0.05
8.84 + 1.53
13.03 + 0.11
12.36 + 0.11
parpl
FREED++
0.71 + 0.06
12.98 + 0.15
17.73 + 0.83
16.26 + 0.19
FREED + (EXPL)
0.67 + 0.17
11.37 + 1.72
17.26 + 0.73
16.04 + 0.11
5htlb
FREED++
0.45 = 0.07
11.78 0.12
14.50 0.79
13.50 + 0.15
FREED + (EXPI.) (EXPLI)
0.48 + 0.04
11.98 + 0.14
14.80 + 0.34
13.68 + 0.15
fkbla
FREED
0.30 + 0.03
8.50 + 0.06
11.50 = 0.00
10.92 = 0.02
FREED + (EXPL)
0.39 - 0.18
8.24 + 0.20
11.50 + 0.00
10.29 + 0.55
abll
FREED++
0.34 = 0.02
11.00 = 0.10
12.86 = 0.05
12.36 + 0.21
FREED+
0.31 = 0.03
11.09 + 0.03
12.93 + 0.32
12.19 + 0.15
usp7
FREED++
0.64 = 0.07
9.66 = 3.28
14.16 0.47
13.30 + 0.13
(EXPL++(EXPL)
0.61 + 0.16
11.15 + 0.42
14.26 + 0.30
13.19 + 0.07
Table 8: Comparison of FREED++ run with additional exploration stage and without.
version, update time stays approximately the same. FREED has the worst performance in terms of size and time required for a gradient update- FFREED is around ~8.5x slower on average and ~22x bigger than FREED++
32

---

Published in Transactions on Machine Learning Research (12/2023)
protein
version
unique (1)
Avg DS (1)
Max DS (t)
Top-5 DS (t)
la7
FREED+
0.76 = 0.05
7.78 1.80
13.23 + 0.41
12.37 + 0.07
FFREED
0.53 = 0.12
8.39 0.82
12.86 + 0.05
12.37 + 0.01
FREED++(PER))
0.65 = 0.05
6.24 0.85
12.90 + 0.00
12.19 + 0.18
FREED++(AM)
0.69 = 0.19
8.14 0.90
12.96 + 0.20
12.28 + 0.04
FREED++(FUSE))
0.48 + 0.02
8.31 + 0.98
13.26 + 0.30
12.66 + 0.18
parpl
FREED
0.71 = 0.06
12.98 1 0.15
17.73 + 0.83
16.26 + 0.19
FFREED
0.58 = 0.09
12.16 + 1.58
17.90 + 0.98
16.13 + 0.15
FREED (PER)
0.61 = 0.07
13.25 + 0.53
18.16 + 0.75
16.46 + 0.03
FAMED++(AM)
0.59 = 0.21
12.36 + 1.09
18.20 = 0.86
16.27 + 0.12
FREED B (FUSE)
0.73 = 0.06
9.63 + 3.01
16.89 + 0.43
15.81 + 0.31
5htlb
FREED++
0.45 = 0.07
11.78 + 0.12
14.50 + 0.79
13.50 + 0.15
FFREED*
0.45 = 0.16
11.97 + 0.20
14.85 = 0.21
13.99 + 0.45
FREED+
0.56 = 0.10
11.78 + 0.21
14.73 = 0.55
13.65 + 0.30
FREED++AME
0.45 = 0.08
11.94 + 0.13
14.00 + 0.00
13.42 + 0.16
(FUSE)+(UUEE))
0.55 + 0.05
11.54 + 0.14
14.46 + 0.72
13.38 + 0.08
Table 9: Comparison of different version of FFREED on DS optimization lask. * denotes the model has a
seed with Max DS < on evaluation, while on the last epoch of training Min DS > We exclude such runs.
method
size, MB
time,
FREED
24.12
1.21 + 0.30
FREED+
3.77
0.38 + 0.13
FFREED
84.60
3.28 + 0.47
FREED +++ (PER)
4.17
0.52 + 0.21
FREED ++ +A(M))
4.18
0.36 + 0.12
FREED ++ (FUUSE
27.80
0.48 + 0.11
Table 10: Comparison of the speed of different versions of FFREED, Time denotes gradient update time measured over a batch with size 100.
33

---

Published in Transactions on Machine Learning Research (12/2023)
G Fragments library preprocessing
One valuable property of FREED on which authors of the original paper were focused is the ability to incorporate prior knowledge into generation via filtration of used fragments. For example, fragments which not passed commonly used medicinal chemistry filters can be excluded during the preprocessing of fragments collection.
We compare two generation setups: with and without filtraioon of the fragments dictionary. In experiments. we use the same set of fragments as in section 5.1 CReM/ZING Fragments were filtered with Glaxo (Lane
ct al. 2006); SureChEMBL (Papadatos et al. 2015), and PAINS (Baell & Holloway 2010 filters.
protein
method
library
unique (1)
Aug DS (t)
Max DS (f)
Top-5 DS (1)
PAINS (1)
SureChEMBL (f) Glaso (1)
6x7
FREED++ +
fitered
0.62 1 0.10
10.49 = 0.13
13.16 1 0.35
12.45 = 0.15
0.99 + 0.00
0.96 1 0.01
0.99 1 0.00
vanilla
0.76 + 0.05
7.78 + 1.80
13.23 + 0.41
12.37 0.07
0.33 + 0.21
0.97 + 0.00
0.99 + 0.00
FREED
fitured
0.24
8.91
0.26
0.34
9.45
1.19
0.99 0.00
0.95
0.00
vanilla
0.43 = 0.10
7.89 + 0:46
10.00 1.12
9.53 + 0.85
0.97 0.02
0.77 = 0.21
0.71 = 0.23
purp1
FREED++
Eitered
0,56 = 0.09
12:23 = 1.19
18.00 + 1.10
16.50 = 0.50
0.99 = 0.00
0.97 + 0.00
0.99 1 0.00
vanilla
0.71 0.06
12.98 = 0.15
17.73 0.83
16.26 0.19
0.45 = 0.27
0.97 0.00
0.98 + 0.01
FREED
fiteree
0.04 0.06
12.35 0.91
15.20
0.70
13.18
2.27
1.00 + 0,00
0.97
1.00 0.00
vanilla
0,33 1 0.32
10.57
1 0.81
12,80 1: 1.91
12.29
1: 1.51
0.95 = 0.02
0.89 1 0.12
0.91 1 0.09
5hilb
FREED++
+
fitered
0.31 0.01
11.46 1 0.10
13.36 0.28
12.95 0.26
0.99 1 0.00
0.88 1 0.05
0.99 1: 0.00
vanilla 0.45
11.78
0.12
14.50
13.50
0.00
0.04
0.00
FREED
fitered
0,28 + 0.27
7,50 + 4.36
10.06 4.82
8.90 + 5.74
0.99 + 0.00
0.83 + 0.18
0.99 = 0.00
sunilla
0.21 + 0.21
6.43 + 5.57
8.83
7.65
7.97 + 6.90
0.63 + 0.54
0.58 + 0.50
0.60 + 0.53
1abla
FREED4+
fitered
0.32 0.04
8.30
0.36
10.30 1.05
9.81 1.00
0.99 + 0.00
0.86 0.05
0.96 +
vanilla
0.30 1 0.03
8.50 0.06
11.50 0.00
10.92 0.02
0.98 0.00
0.78 1 0.00
0.93 1 0.03
FREED
fittred
0.17 = 0.20
6.18 + 1.03
9.00 = 0.36
8.42 + 0.31
0.98 = 0.00
0,81 + 0.06
0.99 = 0.00
vanilla
0.44 0.32
5.71
0.92
8.00 0.91
8.00 + 0.69
0.80 = 0.03
0.55 + 0.09
0.67 + 0.38
abIl
FREED++
Bitered 0.33 0.07
11.16 0.09
13.13 0.37
0.19
1.00 = 0.00
0.73
0.99 1 0.00
vanilla
0.34 + 0.02
11.00 + 0.10
12.86 0.05
12.36 + 0.21
0.90 + 0.00
0.73 + 0.08
0.97 + 0.02
FREED
filtered
0.02 = 0.03
2.92 + 4.50
5.70 = 5.0G
5.12 + 4.45
1.00 + 0.00
0.00 + 0.53
1.00 + 0.00
vanilla
0.24
2.17
7.83 3.75
6.36 5.28
0.92 0.07
0.83 0.16
0.89 +
usp7
FREED++
fitered
0.64 0.07
11.37 0.23
13.76 0.40
13.27 0:23
0.90 0.00
0.92 + 0.03
0.99 + 0.00
vanilla
0.64 1 0.07
9.66 1 3.28
14.16 1: 0.47
13.30 = 0.13
0.94 L 0.04
0.91 1 0.04
0.90 1 0.00
FREED
filtered
0.14 = 0.23
3.22 + 5.14
5.13 = 6.76
4.05 + 6.37
1.00 - 0.00
0.99
+ 0.01
1.00 - 0.00
vanilla
0.10 0.07
8.58
1.02
11:36 0.98
11.12 0.88
0.99 0.00
0.90 0.06
0.93 0.06
Table 11: Effect of fragment library filtration for FREED++ and FREED.
The metrics are shown in table 11 Using filtered fragments library results in similar docking score metrics during generation. However, filtering the fragments library increases the proportion of molecules that pass chemical filters. Since the filtration preprocessing step does not require additional computational resources and generally improves the generation quality, we conclude that is helpful to include it in the generation pipcline.
Results
The same effect be seen for FREED: filtration procedure increases the percentage of valid molecules in terms of considered filters. As in previous experiments with FREED, it commonly tends to diverge. Because of original implementation issues analyzing filtration impact on DS metrics is difficult.
H Reward computation
Following FREED paper, we estimate binding affinity with OpenBabel and QuickVina2 tools. First, we generate initial molecular conformation with OpenBabel Igen3d operation, which consists of several steps: the generation of an initial approximation using a predefined set of rules and 3D templates, energy opti- mization with an empirical forcefield, the search for conformation with low energy in rotor space with a genetic algorithm, and the energy optimization with conjugate gradient method. Further, the optimal pose and energy of the protein-ligand complex are searched with QuickVina2. Conceptually it uses a form of Memetic Algorithm, which interleaves the Monte Carlo algorithm with a restart for global search and BFGS method for local optimization. QuickVina2 provides a parameter called exhaustiveness, which defines the computational effort for docking simulation. Simulations with high exhaustiveness sample more candidates, which improves the performance of search, but requires more computational time. Following FREED we use
34

---

Published in Transactions on Machine Learning Research (12/2023)
exhaustiveness=1 during training to save computational time. During evaluation (see 5 generate 3 different initial conformations with OpenBabel, dock them with QuickVina2 with exhaustiveness= and take the minimum over obtained estimates, to obtain a reliable estimate of binding affinity.
USP7 structure
Figure 9: Structure of USP7 catalytic domain in complex with molecule 23 (PDB ID: 6VN3). The molecular surface was added. The orange surface depicts the catalytic center; the green surface depicts the cleft wet the Thumb and Palm domains; the blue surface depicts the allosteric binding site. The molecule 23 shown by stick model inside the cleft.
J Time Limits
Following FREED, we start generation with benzene ring with three attachment points and restrict our- selves to 4 assembling steps. After the molecule is assembled, we perform docking simulation and assign a reward for the episode. Hence we state generation as a time-limited task. Note that we define the state as a partially assembled molecule, described by extended molecular graphs (see sec. 3.2 ignoring any notion of time. This problem formulation results in 2 major drawbacks.
First, needs to be clarified how to select a time limit properly. Authors of FREED motivate the choice of four assembling steps by the fact that active compounds from DUDE are most commonly fragmented into 4-6 blocks. However, the number of fragments in a molecule can vary significantly depending on the protein of interest (see figure 10
Second, in the described setup, the agent faces a state aliasing problem (Whitehead Ballard 1991) that ariscs when an extended graph can be assembled from several sets of fragments different sizes. As described in (Pardo et al. (2018), the timo-mawar agent cannot accurately estimate the value function because an estimate for the same state will differ depending whether the time limit reached.
35

---

Published in Transactions on Machine Learning Research (12/2023)
One possible solution is to include notion of the remaining time as part of the agent's imput (Pardo et al, 2018), which was utilized in MolDQN; however, it is still unclear how to select time limit. A more promising solution would be the controllable generation termination when the termination signal is predicted with an auxiliary neural network based on the current state representation, similar to the one used GCPN and Pockeu2Mol. We leave this for future work.
0.8
0.8
0.8
MOSES
KI USP7
KI
0.7
0.7
0.7
0.6
0.6
0.6
0.5
0.5
0.5
Probability
Probability
Probability
0.4
0.4
0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
0.0
2.5
5.0
7.5
10.0
12.5
0.0
2.5
5.0
7.5
10.0
12.5
0.0
2.5
5.0
7.5
10.0
12.5
Number of fragments
Number of fragments
Number of fragments
Figure 10: Distribution of number of fragments in molecules according to BRICS fragmentation. Kl stands for "Known Inhibitors"!
K Major implementation issues
this section, we evaluate the significance of each major implementation issue from Section 4.1 First, we take the FFREED and add bugs one at a time (see Table 12
Then, we take the original FREED and
remove bugs one a time (see Table 13
protein
method
unique (t)
Avg DS (t)
Max DS (t)
Top-5 DS (t)
5htlb
FFREED*
0.45 + 0.16
11.97 + 0.20
14.85 + 0.21
13.99 + 0.45
FFREED (CRITIC __RC)
0.50 + 0.08
10.25 + 2.14
14.46 + 0.50
13.44 + 0.44
FFREED CEEDIICRTIUUUU _UDD
0.62 + 0.20
11.03 + 0.55
14.33 + 0.45
13.24 0.17
(UMMED)GUMBEL)
1.00 + 0.00
7.82 + 1.35
12.70 0.26
11.21 0.05
FTRRED(TARGET NЕT)*
0.43 + 0.13
11.46 + 0.19
14.70 = 0.42
13.63 0.20
usp7
FFREED
0.66 + 0.01
10.91 = 0.15
14.03 + 0.46
13.35 - 0.27
FFREED/CRITIC _RCC)
0.66 + 0.06
9.34 + 0.21
13.50 + 0.62
12.58 + 0.25
"FREED(CRITIC.UU
0.46 + 0.24
8.91 + 0.07
13.80 + 0.60
12.84 + 0.11
FFREED(GUMBEL))
1.00 + 0.00
7.58 1.36
11.96 + 0.50
10.64 + 0.19
PTREED(TARGET NET)
0.52 + 0.06
10.11 + 0.57
14.03 + 0.45
13.28 0.23
akt
FFREED
0.44 + 0.05
8.87 + 0.24
10.66 + 0.40
10.17 + 0.17
FFREED(CRITIC ARC)
0.60 + 0.13
8.09 - 0.91
10.20 + 0.34
9.64 + 0.21
FFREED(CRITIC UPD)
0.51 + 0.04
8.58 0.16
10.36 + 0.20
9.78 + 0.23
FUUEED(GUMBEL) (TAREE NET)
1.00 + 0.00
4.96 2.62
9.96 + 0.73
8.60 + 0.26
0.55 + 0.19
8.38 0.06
10.03 + 0.25
9.58 + 0.07
Table 12: Adding bugs to FFREED. Asterix (*) denotes models with seeds with a positive Avg DS during training but negative Max DS on evaluation. We exclude such runs.
36

---

Published in Transactions on Machine Learning Research (12/2023)
The main takeaway from Table 12 is that adding all major bugs leads to performance degradation. The Gumbel-Softmax issue is the most significant one. Next, we take the original FREED and remove bugs one at time (see Table 13
protein
method
unique (t)
Avg DS (1)
Max DS (t)
Top-5 DS (t)
5htlb
FREED
0.21 + 0.21
6.43 = 5.57
8.83 = 7.65
7.97 = 6.90
FREED
TREED(CRITIC CRITIC *RC)*
0.00 + 0.00
4.95 = 7.00
4.95 = 7.00
4.95 + 7.00
FREED FREED(CRITIC (CRITIC (CRITIC_UPD) UPD)
0.19 + 0.33
4.66 = 4.13
6.43 = 6.08
5.78 + 5.61
FREED (GUMBEL)
0.98 + 0.02
7.41 + 2.48
13.40 + 0.55
12.16 + 0.19
FREED (TARGETT
0.67 + 0.44
6.78 + 1.11
12.56 + 0.77
11.74 = 0.38
usp7
FREED
0.10 + 0.07
8.58 + 1.02
11.36 + 0.98
11.12 0.88
FREED CRITIC ARC)
0.06 0.05
6.19 1.84
10.76 + 2.48
10.37 = 2.14
FREED SEUTIC_UPD
0.27 + 0.46
6.94
10.50 1.17
7.71 +.65
FREED GUMBEL)
0.99 = 0.00
7.71 1.37
12.30 0.39
11.33 = 0.34
FREED (EAAGGTTTTTOA NET)
0.82 + 0.26
8.35 = 0.22
12.86 + 0.30
11.41 0.56
akt1
FREED
0.70 + 0.45
3.98 = 0.21
9.26 = 0.30
8.47 0.26
FREED CRITIC CRITIC_ARC)
0.30 + 0.52
3.62 = 2.72
6.33 = 4.06
5.35 3.92
FREED (REED(CRITIC CRITIC_UPD)
0.14 0.20
1.30 + 2.25
3.06 + 5.31
2.89 5.01
FREED (GUMBEL)
0.99 0.00
5.29 = 1.38
9.83 + 0.41
8.77 0.07
FREED(TARGIS NET)
1.00 0.00
5.84 - 0.97
9.73 = 0.23
8.73 0.13
Table 13: Removing bugs from FREED. Asterix (*) denotes models with seeds with positive Avg DS during training but negative Max DS on evaluation. We exclude such runs.
This ablation shows that removing any particular bug does not lead to significant performance improvements. All the bugs need to be fixed simultaneously.
37


Reviews:
[{"review_id": "CnKUiAdEUU", "reviewer": "Reviewer_4UNS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper provides significant detail on describing various parts of the problem setting, for both the original FREED implementation and the improvements made in FREED++.\n* The paper provides additional experiments in creating fragment and USP7 inhibitors.\n\nWeaknesses:\n* It is unclear of the the original FREED is an implementation of the authors. It would be good to clarify since the authors claim that the original implementation contains bugs.\n* The papers spend a lot real estate explaining the prior method (FREED) and less on the new contributions. I think the paper would be stronger by focusing more of the new contributions and why they would be of interest to TMLR readers.\n* Some important experimental details, such as the reward functions for all the experiments, are only briefly described in the appendix. It would be good to have them defined in the main part of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces FREED++, which is a re-implementation of the FREED algorithm for fragment based molecular generation. The authors claim that the original implementation had various bug and issues and suggest additional improvements that are captured in FREED++, including:\n* Updates and critic architecture and training\n* Using undirected graphs for message\n\nThe authors run various experiments on FREED++, as well as the original baselines in the FREED paper, and generally find that their of FFREED and FREED++ outperform the original FREED. The papers also shows additional results on the effects of fragment creation through CReM and BRICS and a case study for the design of USP7 inhibitors.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the following to increase chances for acceptance:\n* Clarify if the experimental results (other than FFREED and FREED++) are run based on your implementation or taken from other papers.\n* Provide greater clarity on your contributions and why they would be interesting to TMLR readers.\n   * I think the contributions could be further strengthened by running additional experiments that change network architectures (e.g. different GNNs) or training mechanisms (e.g. different RL algorithms).\n* Place the reward function definitions for each experiment in the main paper.\n* Add additional related work:\n   * RL for molecular design using text-based methods [1] [2]\n   * Other methods for molecular design in graph-based settings, such GFlowNets [3] [4]\n\n[1] Raj Ghugare and Santiago Miret and Adriana Hugessen and Mariano Phielipp and Glen Berseth. \"Searching for High-Value Molecules Using Reinforcement Learning and Transformers.\" arXiv preprint arXiv:2310.02902 (2023).\n[2] Mazuz, E., Shtar, G., Shapira, B., & Rokach, L. (2023). Molecule generation using transformers and policy gradient reinforcement learning. Scientific Reports, 13(1), 8799.\n[3] Bengio, E., Jain, M., Korablyov, M., Precup, D., & Bengio, Y. (2021). Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34, 27381-27394.\n[4] Jain, M., Raparthy, S. C., Hernández-Garcı́a, A., Rector-Brooks, J., Bengio, Y., Miret, S., & Bengio, E. (2023, July). Multi-objective gflownets. In International Conference on Machine Learning (pp. 14631-14653). PMLR.", "broader_impact_concerns": "No major concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gsm2FvrT18", "reviewer": "Reviewer_PMcY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nGenerally, I think it is good to publish when another algorithm is lacking, and explain why it does or doesn’t work. I think the reimplementation is convincing and helpful to show their contribution.\n\nInteresting to break down the two different kind of RL-based algorithms: sequential vs pretrained generative.\n\nI think the “Current state of protein-conditioned molecule generation” is thorough and up-to-date.\n\nWriting out the steps in the action is very helpful.\n\nThe description of the different fragment library collections is helpful.\n\nWeaknesses\n\nThe message passage section in 4.1 is confusing. “The direction of the edge is determined by the inner representation of the graph in RDKit” – what does this mean? Moreover, in practice, does updating that part of the network improve performance? If so, where do you show this?\n\nFor the USP7 molecule generation task, how many different ligands are available, and what is the diversity of molecule poses? It’d be great to see how well this framework generalizes across a wide variety of ligands and binding pocket poses. Are docking scores validated by in vitro binding?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors benchmark the recently published FREED algorithm, providing additional experimental details and improvements. The authors then compare their improved version, FFREED and FREED++, to additional benchmark algorithms and tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "“Despite being cheaper and faster than the traditional wet-lab approach, VS has another major drawback: it is limited to the databases of already known drug candidates and is not suitable for de novo drug design.” Why is this true? Why cannot novel molecular candidates be used for CADD?\n\n“It uses a modification of the DQN algorithm…” - Please don’t use acronym\n \n“Research that reproduces and improves previous work” - I don’t think this section is necessary. I don’t think you need to justify review and critique of previously published work, especially if you are making improvements therein.\n\nSection header: “3.1 MDP” - This could have a much better title. What do you mean by MDP? It isn’t defined anywhere else in the work.\n\nIn Figure 4, what is the null distribution of Tanimoto similarity? What would the similarity be at random, when using that same fragment library and assembly rules? I.e. What is the null distribution?\n\nTerminology:\n\nSection 3.1: \n\\overline{DS(s_{t+1})} = -DS(s_{t+1})  : Why not just keep it negative? This makes it look like the average, which is confusing.\n\nWhat are the units for the Docking Score?\n\nSection 3.3 : What is MI? I think this is good to define, especially because you refer to it later in section 4.3.\n\nSection 3.4 : “The first scenario involves applying the critic to the actions saved in the replay buffer.” Can you better describe the replay buffer? Why is it called that?\n\nSection 4.1 : Can the “target network” be defined before when discussing the FREED algorithm? It is unclear what the motivation of such a network is. If it wasn’t used in the original algorithm, did it have a large effect when the right hyperparameter was chosen?\n\nFusing function : What are z and x? Have these variable been defined anywhere yet, and how do they refer to either the action or the state? Second, isn’t the concatenation layer, and the associated matrix Q, equivalent to simply removing the W learnable tensor and using just U and V?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hI974e5DPj", "reviewer": "Reviewer_rFQy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The performance of the fixed FREED, FFREED, is superior to the original FREED, hence it can be practically valuable.\n- The proposed FREED++ is more efficient than FREED while keeping the competitive (or superior) performance, hence it can be also valuable contribution for TMLR's audience.\n### Weaknesses \n\n1. The effectiveness of one of the main contributions of this paper, FFREED, is restricted to the original implementation of FREED, which significantly limits the audience  of this paper.\n3. The procedure of message passing is discussed as one of major issues in Section 4.1, where the authors mention that graphs are treated as directed in the original implementation and it is not appropriate. However, if a graph is directed by definition in a dataset, considering the direction in message passing should be more appropriate than treating it as an undirected graph. So it is not correct to always treat graphs as undirected and this point should be more carefully discussed.\n4. More detailed empirical evaluation would be desirable. In FFREED, there are five major modifications as stated in Section 4.1. Thus each of them should be empirically evaluated separately via ablation study. \n5. In experiments, I recommend using the GuacaMol benchmark, which has been commonly used in molecular generation. \n6. Presentation should be improved.\n\t1. Since the authors mention about the \"original\" implementation of FREED, I guess the authors discuss about https://github.com/AITRICS/FREED , which has been provided in the original FREED paper. However, it is better to be explicitly refer the link in the paper.\n\t2. The section title of Section 3 is \"FREED++\", but it seems that Section 3 just introduces the existing FREED. So the title should be modified as \"FREED\".\n\t3. Similarly, in the beginning of Section 3, there is a sentence \"This section provides a detailed description of the proposed FREED++ method.\" However, this section is not about the proposal but the existing FREED method. So this should be also revised. If there is some delta that is not contained in the original FREED in this section, it should be clarified further. \n\t4. I do not fully understand the \"Fragment selection\" improvement in FREED++ in Section 4.3 because the process discussed here and Equation (4) do not have one-to-one correspondence. Please clarify this point.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies FREED, which is a reinforcement learning-based molecular generation method. After introducing FREED, this paper points out bugs and issues of the original implementation of FREED and addresses these issues, resulting in the fixed version called FFREED. Moreover, this paper proposes to replace a fusing function for fragment embeddings with a simplified function and called the modified version FREED++. The performance of FREED, FFREED, and FREED++ is empirically examined on real-world molecular generation benchmark datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please resolve all the above weaknesses.", "broader_impact_concerns": "I do not have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is a thorough study of reproducing and finally improving FREED, a fragment-based method for molecule generation. Reviewers have questioned the novelty of the work and raised the concern that the potential audience might be limited to those interested specifically in the algorithm. However, the Authors show through a thorough experimental evaluation a strong empirical performance, which suggests that the results are likely to be relevant as a reference point for other molecule generation methods. In the end, all reviewers were supportive of accepting the paper. The paper also serves as an important reminder that implementation details are extremely important to get right. Thank you for submitting your work to TMLR. All in all, it is my pleasure to recommend accepting the paper as is.
Decision: Accept (poster)