Beam Enumeration: Probabilistic Explainability For Sample Efficient Self-conditioned Molecular Design
Authors: Jeff Guo
Philippe Schwaller
Venue: ICLR 2024 poster (2024)
Keywords: Molecular generative models
reinforcement learning
natural language processing
drug discovery
sample-efficiency
explainability
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: novel algorithm to extract molecular substructures from language-based molecular generative models to jointly address explainability and sample efficiency.

Abstract: Generative molecular design has moved from proof-of-concept to real-world applicability, as marked by the surge in very recent papers reporting experimental validation. Key challenges in explainability and sample efficiency present opportunities to enhance generative design to directly optimize expensive high-fidelity oracles and provide actionable insights to domain experts. Here, we propose Beam Enumeration to exhaustively enumerate the most probable sub-sequences from language-based molecular generative models and show that molecular substructures can be extracted. When coupled with reinforcement learning, extracted substructures become meaningful, providing a source of explainability and improving sample efficiency through self-conditioned generation. Beam Enumeration is generally applicable to any language-based molecular generative model and notably further improves the performance of the recently reported Augmented Memory algorithm, which achieved the new state-of-the-art on the Practical Molecular Optimization benchmark for sample efficiency. The combined algorithm generates more high reward molecules and faster, given a fixed oracle budget. Beam Enumeration shows that improvements to explainability and sample efficiency for molecular design can be made synergistic.

Published as a conference paper at ICLR 2024
BEAM ENUMERATION: PROBABILISTIC EXPLAIN- ABILITY FOR SAMPLE EFFICIENT SELF-CONDITIONED
MOLECULAR DESIGN
Jeff Guo Philippe Schwaller
Laboratory of Artificial Chemical Intelligence (LIAC), Institut des Sciences et Ingénierie Chimiques National Centre of Competence in Research (NCCR) Catalysis École Polytechnique Fédérale de Lausanne (EPFL) Lausanne, Switzerland {jeff. guo, philippppp schwaller)@epfl.co
ABSTRACT
Generative molecular design has moved from proof-of-concepp to real-world ap- plicability, as marked by the surge in very recent papers reporting experimental validation. Key challenges in explainability and sample efficiency present oppor- tunities to enhance generative design to directly optimize expensive high-fidelity oracles and provide actionable insights to domain experts. Here, we propose Beam Enumeration to exhaustively enumerate the most probable sub-sequences from language-baset molecular generative models and show that molecular substruc- tures can be extracted. When coupled with reinforcement learning, extracted sub- structures become meaningful, providing source of explainability and improv- ing sample efficiency through self-conditioned generation. Beam Enumeration is generally applicable any language-bar based molecular generative model and no- tably further improves the performance of the recently reported Augmented Mem- ory algorithm, which achieved the new state-of-the ahe-art on the Practical Molec- ular Optimization benchmark for sample efficiency. The combined algorithm generates more high reward molecules and faster, given fixed oracle budget. Beam Enumeration shows that improvements to explainability and sample effi- ciency for molecular design can be made synergistic. The code available at httpppes/ggggaattccc//////eaella wttprr/rrr.opoouuumeeeeeee.dedee aegg/aatemmeeeeneeeeeee memory
INTRODUCTION
Molecular discovery requires identifying candidate molecules possessing desired properties amidst an enormous chemical space (Sanchez-Lengeling & Aspuru-Guzity (2018). Generative molecular design has become a popular paradigm in drug discovery, offering the potential to navigate chemical space more efficiently with promise for accelerated discovery. Very recently, efforts have come to fruition and a large number of works have reported experimental validation of generated inhibitors, notably for both distribution learning (Merk et al. (2018); Moret et al. (202T): Grisoni et al. (2021);
Yu et al. (2021) 1); Eguida et al. (2022): Liet al. (022); : Tan et al. (202) 1) Jang et al. (2022)) |Chen
Chen et al. (2022); Hua al. (2022) Song al. (2023); Moret al. (2023); Ballarotto al. (2023) and goal-
directed generation (Korshunova et al. (2022); Yoshimori et al. (202)) Zhavoronkov et al. (2019); al. (2023) approaches. Perhaps now more than ever, existing challenges in explainailiity and sample efficincyy offer an avenue to propel generative molecular design towards outcomes that are not yet possible. Specifically, if one can elucidate why certain substructures or molecules satisfy target objective, the model's knowledge can be made actionable, for example, an interplay with domain experts. Moreover, sample efficiency concerns with how many experiments, i.e., oracle calls, are required for model to optimize the target objec- tive. This is a pressing problem as the most informative high-fidelity oracles are computationally expensive, e.g., molecular dynamics (MD) for binding energy prediction (Wang et al. (2015); Moore ct al. (2023). If generative model can directly optimize these expensive oracles, the capabilities of generative design can be vastly advanced.
Ren et al. (2023); Li et al. (2023);

---

Published as a conference paper at ICLR 2024
In this work, propose Beam Enumeration to exhaustively enumerate the most probable token sequennesssss in language- molecular generative models and show that valid molecular sub- structures can be extracted from these partial trajectories. We demonstrate that the extracted sub- structures are informative when coupled with reinforcement learning (RI.) and show that this infor- mation can he made actionable self-condition the model's generation by only evaluating sampled molecules containing these substructures with the oracle. The results show significantly enhanced sample efficiency with an expected small trade-off in diversity. Beam Enumeration jointly addresses explainability and sample efficiency. Our contribution is as follows:
1. We propose Beam Enumeration task-agnostic method exhaustively enumerate sub- sequences and show that molecular substructures can be extracted.
2. We demonstrate that during the course of RI., extracted substructures are on track to yield high rewards and can be used for self-conditioned molecular generation. We extract struc- tural insights from these substructures, thereby providing a source of explainability.
3. We perform exhaustive hyperparameter investigations (2,224 experiments and 144 with molecular docking) and provide insights on the predictable behavior of Beam Enumeration and recommend default hyperparameters for out-of- wofftth-bbor box applications.
4. We introduce new metric: Oracle Burden, which measures how many oracle calls are required to generate unique molecules above reward threshold as one is often interested in identifying a small set of excellent candidate molecules amongst many good ones.
5. We combine Beam Enumeration with the recently reported Augmented Memory (Guo & Schwwalerr (203)) optimization algorithm and show that the sample efficiency becomes suf- ficient (up a 29-fold increase) find high reward molecules satisfying docking objec- tive with only 2,000 oracle calls in three drug discovery case studies.
2 RELATED WORK
Sample Efficiency in Molecular Design. Tailored molecular generation is vital for practical ap- plications as every use case requires optimizing for a bespoke property profile. Over the past sev- eral years, so-called goal-directed generation has been achieved using a variety of architectures, in- cluding Simplified molecular-input line-entry system (SMIESS) (Weininger) (188)-basee recurrent neural networks (RNNs) (Olivecrona al. (2017); Popova et al. (2018); SSegler al. (2018); Goel ct al. 12021), generative ad adversarial networks (GANs) GGodffello et al. [2m14)]aa
ct al. (2017); Guimaraes et al. (2018), variational autoencoders (VAEs) (Kingma & Welling (2022); ;
Liómez-Bombaree et (201 8 Zhavoronkov et al. (2019). graph-based models (You et al. 2019
Jin et (2020); Mercado et al. [02lc);/Atance et al. (2022). GFlowNets (Bengio et al. (202 a). and
genetic algorithms (Fu et al. (2022a). However, while all methods can be successful in optimizing for various properties, the oracle budget, i.e., how many oracle calls (computational calculations)
were required to do so, is rarely reported. To address this, Gao et al. (Gao et al. (2022) proposed the Practical Molecular Optimization (PMO) benchmark, which assesses 25 models across 23 tasks and enforces a budget of 10,000 oracle calls. Recently, Guo et al. proposed Augmented Memory (Guo & Schwaller 2003), which uses a language-based molecular generative model and achieves the new state-of-the-at on the PMO benchmark. In this work, Beam Enumeration is proposed an addi- tion to language-basee molecular generative models, and we show that coupling it with Augmented Memory drastically improves the sample efficiency.
Explainability for Molecules. Explainable Al (XAI) (Gohel et al. (02T) to interpret and explain model predictions is a vital component for decision- weighted Class Activation Mapping (Grad-CAM) (Selvaraju et al. (2017). which uses gradient-base heat maps for convolutional layers and Local Interpretable Model-agnostic Explanations (LIME) (Ribeiro et (2016). which uses a locally interpretable model. Other methods include permutation importance (Altmann et al. (2010), which measures the performance change when shuffling feature values, and SHAP values (Shapley (1953), which measure the average contribution of each feature to the model's prediction. For molecules, the Molecular Model Agnostic Counterfactual Expla- nations (MMACE) (Wellawatte et al. (2022) method was proposed to search for the most similar counterfactual (model predicts the opposite label) molecule. Recently, the pBRICS (Vangala et al. (023) algorithm was proposed to decompose molecule into functional groups and then applying
Existing methods include Gradient-
2

---

Published as a conference paper at ICLR 2024
Pook Extrocted Substructures
b. Enumeration
N Bearn Steps
(HH O
0.5
CC
Substructure Presence
0.05
C
6:75
0.03
Score
Self
8:08
Ofjective Function
0.35
6.02
0.8
0.08
OC
Pool
Beam Enumeration
Ooddl Roward
O
Probable Substructures
0.01
Model Updats
Sample Maleculea
Scaffold
clccocci[[=O|
clccocc((-O)
Up to 29-fold Sample Efficiency Increase
cIccocci([=O)
Figure I: Beam Enumeration overview. a. The proposed method proceeds via steps: 1. generate batch of molecules. 2. filter molecules based on pool to enforce substructure presence, discarding the rest. 3. compute reward 4. update the model. After updating the model, if the reward has improved for consecutive epochs, execute Beam Enumeration. b. Beam Enumeration sequentially enumerates the top tokens by probability for N beam steps, resulting in an exhaustive set of token sebbsequences. c. All valid substructures (either by the Structure or Scaffold criterion) are extracted from the sub-sequences. The most frequent substructures are used for self-conditioned generation.
Grad-CAM to explain matched molecular pairs (MMPs), i.e., pairs of molecules differing by single chemical group. In the context of generative models, previous works have explicitly addressed ex- plainability (Guo et al. (2022); Fu et al. (2022b) and jointly with sample efficiency (Fu et al. (20226). In this work, we aim to make explainability actionable during generative design experiment.
To achieve this, we introduce Beam Enumeration, which extracts molecular substructures directly from the model's token sampling probabilities and derives explainability from a generative proba- bilistic perspective that is modulated by reward feedback. Our approach is based on the fact that during a successful optimization trajectory, must become increasingly likely to generate desir- able molecules. It is thus reasonable to hypothesize that the most probable substructures track to receiving high reward. We verify this statement in the Results section and show that Beam Enumeration can also jointly address explainability and sample efficiency.
PROPOSED METHOD: BEAM ENUMERATION
In this section, each component of Beam Enumeration (Fig. is described: the base molecular gen- erative model, the Beam Enumeration algorithm, and how Beam Enumeration harnesses the model's built-in explainability which can be used to improve sample efficiency through self-conditioned gen- eration (further details on Beam Enumeration are presented in Appendix A).
Autoregressive Language-based Molecular Generative Model. The starting point of Beam Enu- meration is any autoregressive language-based molecular generative model. The specific model used in this work is Augmented Memory (Guo & Schwaller (2023) which recently achieved the new slate-of-the-at performance on the PMO (Gao et al. (2022) benchmark for sample efficiency, out- performing modern graph neural network-based approaches (Fu et al. (2020); Xie et al. (202T) and GFlowNets (Bengio et al. (2021b). Augmented Memory builds on REINVENT (Olivecrona et al. y Blaschke et al. (2020) which a SMILES based (Weininger) (188) RNN using long-short- term memory (LSTM) cells (Hochreiter & Schmidhuber 1997). The optimization process is cast as an on-policy RL problem. We define the state space, S, as all intermediate token sequences and the action space, (st), the token sampling probabilities (conditioned on given state). A given by the policy, Ty. which is parameterized by the RNN. The objective is to iteratively update the policy such that token sampling, (st),. yields trajectories (SMILES) with increasing reward.
3

---

Published as a conference paper at ICLR 2024
Formally, sampling SMILES, x, is given by the product of conditional state probabilities (Equation I]. and the token sampling is Markovian:
T P(x) II P(sc 81-1,
The Augmented Likelihood is defined, (Equation 2 where the Prior is the pre-trained model and is the objective function which yields a reward given SMILES, x.
log
log R Prior
+ S
Augmented
The policy is directly optimized by minimizing the squared difference between the Augmented Likelihood and the Agent Likelihood given a sampled batch, B, of SMILES constructed following the actions, A (Equation B:
1
L(0)
(log ne
log To
[B|
Augmented
Agent
Minimizing L(0) is equivalent to maximizing the expected reward as shown previously (Guo & Schwaller (2023); Fialková al. 20022).
Beam Enumeration. Beam Enumeration is proposed based on two facts: firstly, on successful optimization trajectory, the model's weights must change such that it becomes increasingly likely to generate high reward molecules. Secondly, generation involves sampling from conditional prob- ability distributions. is therefore reasonablc to assume that the highest probability trajectories are more likely to yield high reward. Correspondingly, Beam Enumeration (Fig. D enumerates the top tokens (by probability) sequentially for N beam steps (as it infeasible to sample the full trajec- tories due combinatorial explosion), resulting in exhaustive of token sub-sequences. We show that meaningful molecular substructures can be extracted from these sub-sequences, which harness and demonstrate how it can be made actionable. We note the closest work ours is the application of Beam Search (Graves (0112); DoullaggrrLewanddonn
et al. (2013) for molecu- lar design (Moret al. (202I) find the highest probability trajectories. Our work differs as the objective is not to find a small set of the most probable sequences. Rather, we exhaustively enu- merate the highest probability sub-sequences to extract molecular substructures for self-conditioned generation. We further detail the differences to Beam Search in Appendix G
Probabilistic Explainability. Here, we describe how probabilistic explainability can be extracted from the exhaustive set of token sub-sequences. Usually, token sequences are only translated into SMILES once the sequence complete, i.e., the "end" token has been sampled. We hypothesized that molecular substructures can be extracted from given sub-sequence by iteratively consider- ing every wuub)-uub-equenee (Fig. []. For example, given the sub-sequence "ABCD", the set of (sub)-sub-sequeences are: "A", "AB". "ABC", and "ABCD". In practice, only consider (sub)-sub- sequences with at least three characters ("ABC" and "ABCD") since each character loosely maps to one atom and three approximately the minimum for meaningful functional groups, e.g., "C=O", a carbonyl. It is expected that not every sub-sequence possesses (sub)-sub-sequenccs mapping to valid molecular substructures. Still, we show that sufficient signal can be extracted (Appendix C Finally, we implement two types of substructures: Scaffold, which extracts the Bemis-Murch (Be-) mis & Murcko (1996) scaffold and Structure, which extracts any valid substructure. In the Results section, we discuss the predictable difference in behavior.
Self-conditioned Generation. The sub-sequences were enumerated by taking the most probable tokens, and the model's 's weights should be updated such that high reward molecules are increasingly likely to be generated. Correspondingly, it is reasonable to posit that the most frequent molecular substructures are track to becoming high reward full molecules and that the substructures them- selves possess properties aligned with the target objective. Beam Enumeration saves a Pool of these substructures and filters future sampled batches of molecules to contain them, discarding those that do not. Effectively, the generative process is selffconditioned as the model will only be updated by generated molecules containing the extracted substructures (Fig.
4

---

Published as a conference paper at ICLR 2024
4 METRICS
Sample Efficiency Metries. We define two metrics assess sample efficiency: Generative Yield (referred to as Yield from now on) and Oracle Burden. Yield (Equation 4 is defined the number of unique generated molecules above a reward threshold, where 9 E G are the molecules in the generated set, is the indicator function which returns 1 if the reward, R(g), is above threshold, T. Yield is useful metric for drug discovery as is ubiquitous to triage the generated set (thus, a higher Yield is desirable) and prioritize molecules, e.g., based synthetic feasibility, for experimental validation or more expensive computational oracles.
Generative Yield
Σ [[(())
Oracle Burden (Equation(5) is defined the number of oracle calls (c) required to generate N unique molecules above a reward threshold. This a direct measure of sample efficiency as high reward molecules satisfy the target objective, and the metric becomes increasingly important with expensive high-fidelity oracles. In this work, all Oracle Burden metrics are computed by not allowing more than 10 molecules to possess the same Bemis-Murcke (Bemis & Murcko (199)) scaffold, thus also explicitly considering diversity in the generated set.
Oracle Burden
Σ 9=1
[[
(5)
RESULTS AND DISCUSSION
b.
Beam Enumeration: Probabilistic Explainability
1.0
Epoch 40
Epoch 80
N
NH
NH
0.7 Epoch
N
HN
HN
HgN
M Mwn
3.4
0.891 Reward
HyM
HN
NH
N
NH
HRN
N
NH
3.1
NHH,
NH
20
40 60
80 100
Epoch
Figure 2: Illustrative experiment with the following multi- parameeerreeer optimization objective: max- imize tPSA, molecular weight <350 Da, number of rings 2 2. Augmented Memory (Guo & Schwaller (2023) reward trajectory with annotated top-4 (excluding benzene) most frequent molec- ular substructure scaffolds varying epochs using Beam Enumeration. Examples of molecules with high reward.
We first design an illustrative experiment to demonstrate the feasibility of Beam Enumeration to extract meaningful substructures and, in turn, enable self-conditioned generation. Next, three drug discovery case studies to design inhibitors against dopamine type receptor (DRD2) (Wang et al. 2018), MK2 kinase (Argiriadi et al. (2010), and acetylcholinesterase (AChE) (Kryger et al. (1999) were performed to demonstrate real-world application. The key result we convey is that Beam Enumeration be added directly existing algorithms, and it both provides structural insights into why certain molecules receive high rewards and improves sample efficiency to not only generate more high reward molecules, but also faster, given fixed oracle budget.
5

---

Published as a conference paper at ICLR 2024
Table I: Illustrative experiment: Beam Enumeration improves the sample efficiency of Augmented Memory. All experiments were run for 100 replicates with an oracle budget of 5,000 calls, and reported values are the mean and standard deviation. Scaffold and Structure indicate the type of substructure, and the number after is the Structure Minimum Size. Parentheses after Oracle Burden denote the cut-off number of molecules. Parentheses after values represent the number of unsuc- cessful replicates (for achieving the metric).
Metric
Augmented Memory
Beam Seaffold
Structure
Scaffold
Beam Structure
Baseline
Generative xi (1)
1757 + 305
1669 + 389
1117 + 278
864 + 202
496 + 108
Generative yoa (7)
1 389
+ 256
199 122
85
Oracle yor (1) (
577 +: 310
616 1: 230
1037 +: 414
897 +: 347
1085 1: 483
Oracle Burden (10) (.)
947 350
926 + 332
1881
1745 = 292
2392 + 216
Oracle Burdensso (100) (L)
1530 1: 468
1547 513
2736 335
2713 402
3672 1: 197
Oracle Burdeayos (1) (1)
1311 + 628
1401 695
2423 + 487
2295 482
3164 + 492
Oracle Rurden (10) (.)
1794 617 (1)
2009 804
3124
3241 492
4146 + 326
Oracle Burden (100) (1)
2704 = 689 (1)
2943 811 (6)
3973 = 522 (6)
4415 # 437 (20)
4827 = 170 (69)
5.1 ILLUSTRATIVE EXPERIMENT
Extracted Substructures are Meaningful. The illustrative experiment aims optimize the follow- ing multi-paraaa parameter optimization (MPO) objective: maximize topological polar surface area (tPSA), molecular weight (MW) <350 Da, and number of rings 2 2. This specific MPO was chosen because it is plausible to predict what structural features would be necessary to optimize the objective: rings saturated with heteroatoms. Due to constraining the molecular weight, the model cannot just learn to generate large molecules that would, on average, possess higher tPSA. Augmented Memory (Guo Schwaller (2023) was used to optimize the MPO objective. The reward trajectory tends towards 1, indicating the model gradually learns to satisfy the target objective, desired (Fig. 2 Next, we investigate the top and N beam steps parameters for Beam Enumeration and show that while the majority of sub-sequences do not possess valid substructures, meaningful signal can still be extracted (Appendix C). We hypothesize that the optimal parameters are using low top k we are interested in the most probable sub-sequences and large N beam steps, which would enable extracting larger (and potentially more meaningful) substructures. Fig. shows the top four substructures from Beam Enumeration at varying epochs. As hypothesized, the substructures are in- formative when considering the MPO objective: the most frequent substructures gradually become rings saturated with heteroatoms, which possess a high tPSA.
Self-conditioned Generation Improves Sample Efficiency. Thus far, the results only show that Beam Enumeration can extract meaningful molecular substructures. To enable self-conditioned generation, criterion is required to decide when execute Beam Enumeration. We consider when extracted substructures would meaningful and propose to execute Beam Enumeration when the reward improves for Patience number of successive epochs (to mitigate sampling stochasticity). We combine Beam Enumeration with Augmented Memory (Guo Schwaller (2023) and perform an exhaustive hyperparameter grid search (with replicates) using Yield and Oracle Burden as the per- formance metrics AA).. The results elucidate the behavior of Beam Enumeration with three key observations: firstly, Structure extraction is much more permissive compared to Scaffold and often leads to small functional groups, e.g., carbonyl. being the most frequent substructures which diminish the sample efficiency benefits (Appendix C). Secondly, enforcing larger substructures to be extracted (Structure Minimum Size) improves performance across all hyperparameter combina- tions. This reinforces that extracted substructures are meaningful as larger substructures heavily bias molecular generation during self-conditioning. If they were not meaningful, sample efficiency would not improve (and would likely be detrimental). Thirdly, Structure extraction while enfore- ing higher Structure Minimum Size prevents small functional group extraction which significantly enhances performance.
We perform five experiments (N=100 replicates each) based on the optimal hyperparameters identi- fied from the grid search: Augmented Memory (Guo & Schwaller (2023) (baseline) and Augmented Memory with Beam Enumeration (Scaffold and Structure with and without Structure Minimum Size 15). Table [ shows that Beam Enumeration drastically improves the Yield and Oracle Burden compared to the baseline at both the 00.7 and 00. reward thresholds, particularly when Structure

---

Published as a conference paper at ICLR 2024
POR ID: Semi
De Doparrine Recopier
Vine
Reference Vina:
DED:
GED: 0.66
Soore:
Score:
Score: 2.59
410.45
Mik kinose
Wie 31.8
GED:
Vine:-233
349
Vine: 32.1
Reference
GEB 0 46
Vina: -sa
FoB
Aomycchalontteree
Mrax -133
Vina:-3444
Ven--13
Reference Vina:
060: 0.90
GEB 0.87
QED 0.87
NWW
SA Score: 2.08
Figure 3: Three drug discovery case studies showing the top generated molecule (triplicate exper- iments) using Augmented Memory (Guo & Schwaller (202)) with Beam Enumeration Structure Minimum Structure Size 15 and the reference ligand. Extracted substructures from Beam Enu- meration are highlighted. The multi-parameter optimization objective is: Minimize Vina score, maximize QED, and molecular weight <550 Da. The values, with the Synthetic Accessibility (SA) score (Ertl & Schuftenhauer (2009) are annotated, a. Dopamine type 2 receptor (Wang et al. 2018). b. MK2 kinase (Argiriadi al. (2010). Acetylcholinesterase (Kryger al. (1999).
Minimum Size 15 is enforced. We highlight that the improved sample efficiency is especially significant as baseline Augmented Memory could not find 100 molecules >0.8 reward in 69/100 replicates. The results are further compared using Welch's t-test, and all p-values significant the 95% confidence level (Appendix C
5.2 DRUG DISCOVERY CASE STUDIES
The positive results from the illustrative experiment suggest that Beam Enumeration can be applied to real-world drug discovery case studies to design inhibitors against DRD2 which is implicated in neurodegenerative diseases (Wang al. (2018). MK2 kinase which is involved pro-inflammmtory responses (Argiriadi et al. (211), and AChE which is a target of interest against Alzheimer's ' disease (Kryger et al. (1999). Following Guo al. (Guo al. (2021b); Guo Schwaller (2023), we for- mulate the following MPO objective: minimize the Autk Vina Trott & Olson (2010) docking score as a proxy for binding affinity, maximize the Quantitative Estimate of Druglikeness (QED) [Bickerton et al. (2012) score, and constrain MW <500 Da. The QED and MW objectives prevent the generative model from exploiting the weaknesses docking algorithms to give inflated docking scores to large, lipophilic molecules, which can be promiscuous binders (Arnott & Planey (2012).
Moreover, an oracle budget of 5,000 Vina calls was enforced which almost half the budget of the original Augmented Memory (Guo & Schwaller (2033) work (9,600).

---

Published as a conference paper at ICLR 2024
Since the observations made from the illustrative experiment hyperparameter grid search may not be generalizable to docking tasks, we perform an additional hyperparameter grid search (with repli- cates). The results (Appendix(D) show that the optimal hyperparameters from the illustrative experi- ment are also the optimal hyperparameters across all three drug discovery case studies. We designate these the default hyperparameters and demonstrate the applicability of Beam Enumeration both Augmented Memory (Guo & Schwaller (2023) and REINVENT (Olivecrona et al. (201)); Blaschke ct al. (0200) which is the second most (behind Augmented Memory) sample efficient model in the
PMO Gao et al. (2022) benchmark.
Qualitative Inspection: Why Were These Molecules Generated? We first show that Augmented Memory with Beam Enumeration generates molecules that satisfy the MPO objective (Fig. 3). We emphasize that results were not cherry-picken and the three generated examples shown are the top (by reward) across triplicate experiments. All molecules possess better Vina scores and higher QED than the reference molecules, as desired. Fig. shows the highlighted substructures extracted using Structure extraction with Structure Minimum Size 15 with three key observations: firstly, "uncommon" " molecular substructures may be extracted such as the bridged cycle against DRD2. The exact substructure extracted was amide bond with long carbon chain which implicitly enforces the bridged cycle, and the Vina pose shows that it fits in the binding cavity with no clashes, despite being bulky group. Secondly, bicylic or double-ring systems are often extracted (for all case studies), forming central scaffolds of the full molecule. Thirdly, scaffolds with branch points, i... a central ring with single carbon bond extensions, are often extracted (for all case studies). These substructures are particularly interesting as they heavily bias what can be generated in the remaining portion of the full molecule. An exemplary example of this is in the first generated molecule against MK2, where the branch points are effectively a part of two other ring systems (Fig. D Beam Enumeration can provide insights into the tolerability and suitability of certain substructures in the context of the full molecules (see Appendix D for more examples of extracted substructures). Finally, we posit that the extreme bias of Structure extraction the reason why it can be more performant than Scaffold. Overall, the extracted substructures are meaningful and act both as a source of generative explainability and can self-direct the generative model into specific regions of chemical space with high reward.
Quantitative Analysis: Sample Efficiency. Next, we reinforce results from previous work show- ing that Augmented Memory (Guo & Schwaller (2023) is significantly more sample efficient than REINVENT (Olivecrona et al. (2017); Blaschke
(lasc): et al. (2020) (Table Notably, the Yield of Aug- both the >0.7 and >0.8 reward thresholds, indicating that more high reward molecules are generated. Moreover, Augmented Memory has a lower Oracle Burden than REINVENT all cases. except for Oracle Burden (1) for DRD2 and AChE where there is essentially no difference. The reason for this is because molecules with >0.8 reward were already generated at epoch 1, indicating the pre-trained model (trained on ChEMBL (Gaulton et al.] (2012a)) is a good Prior for these case studies. By contrast, the MK2 case study is considerably more challenging as extremely few >0.8 reward molecules are generated under a 5,000 oracle calls budget. Augmented Memory significantly outperforms REINVENT as the latter could not find 10 molecules with reward 00. (Ta
mented Memory is much greater than REINVENT
Subsequently, we demonstrate that Beam Enumeration can be applied of on top of Aug- mented Memory and REINVENT. Firstly, the addition of Beam Enumeration improves the sample efficiency of both base algorithms, as evidenced by the Yield and Oracle Burden metrics in Table 2 with a small trade-off in diversity (Appendix D). However, the benefits are more pronounced in Augmented Memory as observed by the Yield>as improving by >4x in all cases (MK2 improves by 29x) and the Oracle Burden >0.8 (10 and 100) over halved in most cases, Notably, for MK2 Oracle Burden >0.8 (100), baseline Augmented Memory could not accomplish the task while Beam Enu- meration is successful in almost under 2,000 oracle calls (Table These findings are in agreement with the original Augmented Memory (Guo & Schwaller (2023) work in that the algorithm is much more data efficient and capitalizes on learning from high reward molecules via experience replay (Lin (1922.. Beam Enumeration decreases the diversity of the generated set (as measured by Int- Div Polykovskiy et al. (2020b), but finds considerably more unique scaffolds above the 0.8 reward threshold (up to 19x) (Appendix D). With many unique scaffolds built around (often) central sub- structures, the generated set could conceivably provide insights into structure-activity relationships. These results demonstrate that the combined algorithm achieves both exploration and exploitation. Overall, the results show that Beam Enumeration is task-agnostic and can be applied on top of ex-
8

---

Published as a conference paper at ICLR 2024
isting algorithms to improve their sample efficiency. The combined algorithms generate more high reward molecules and faster, even in challenging (MK2) scenarios under a limited oracle budget. Furthermore, in reference to all the Oracle Burden metrics (Table 2). Augmented Memory with Beam Enumeration can identify a small set of excellent (high reward) candidate molecules in under 2,000 oracle calls and in some cases, even under 1,000 oracle calls.
Drug discovery case studies: effect of Beam Enumeration on sample efficiency. All exper- iments were run triplicate with an oracle budget of 5,000 calls and reported values are the mean and standard deviation. Scaffold and Structure indicate the type of substructure (Structure Minimum Size = 15) extracted. The Generative Yield and Oracle Burden are reported at varying reward thresh- olds. Parentheses after Oracle Burden denote the cut-off number of molecules. Best performance is bolded with the exception of Oracle Burden (1) (DRD2/AChE) which have essentially identical per- formance due to the pre-trained model. * and ** denote one and two replicates were unsuccessful, respectively.
Table 2:
Metric
Target
Angmented Memory
REINVENT
Beam Structure 15
Beam
Beseline
Beam
Beam Scaflold Is
Baseline
Scaffold 15
Stracture 15
138
95
2513 442
1:
2686 235
1879 1
Generative yad (?)
MK2
3127 + 138
2584 + 443
1446 2: 173
1822 + 444
1853 + 101
879 : 10
3902 189
3288
2684 242
2437
DRD2
1780 E: 439
1607 4: 379
363 :: 195
417 ::: 275
687 4: 306
102 4
Generative yod (1)
211
523
2:1
19:1:
AChB
2059
à 327
2124 +: 326
556 4:
323 4: 58
310 4: 200
147 + 11
DRDZ
126
29
51
+ 149
Oracle ou (3)
MK2
736 :: 166
1221 4: 564
1360 4: 543
1110 -: 268
80s +: 524
1724 4: 502
AChE
105 29
63
62:4
29
83 +
DRD2
582 # 83
571 1 194
711 = 120
1099 : 9:0
883 = 105
Oraele Bardou- (.)
MK2
1525
3833
1778 0
3891 631
Palled
ACME
462 + 25
418 +
380 + 0
441 + 132
421 + 130
481 + 108
DRD2
1120 194
1056 1: 146
2558 = 30"
1928 1 117
2109 1: 1060
4593
Oniele Runden pas (1)
MK2
2189 + 181
2676 403*
Fiiled
3208 0**
Failed
Failed 3931
AChE
1110 1: 265
884 1: 162
2021 +:
3073 1: 427
3596 1: 678
CONCLUSION
In this work, we propose Beam Enumeration to exhaustively enumerate sub-sequences from language-basee molecular generative models and show that substructures can be extracted, provid- ing source of generative explainability. Next, we show that the extracted molecular substructures can be used to self-condition the generative model to only perform oracle evaluation for molecules possessing these substructures (discarding the rest). We show that Beam Enumeration can be cou- pled with existing RL-based algorithms including Augmented Memory (Guo & Schwaller (2023) and REINVENT (Olivecrona et al.] (2017): Blaschke et al.) (2020) to improve their sample efñ- ciency. Moreover, enforcing the extraction of larger substructures improves performance across all hyperparameter combinations. We believe this is a particularly interesting observation as it demon- strates the model's remarkable robustness and tolerability to extreme bias. Subsequently, in three drug discovery case studies to design molecules that dock well, the addition of Beam Enumcration to Augmented Memory and REINVENT substantially improves sample efficiency as assessed by the Yield (number of unique molecules generated above a reward threshold) and Oracle Burden (number of oracle calls required for the model to generate N unique molecules above reward threshold) with small trade-off in diversity (which is expected). The extracted substructures them- selves provide valuable structural insights, often enforcing the generation of specific cyclic systems and scaffolds with branch points which impose an overall molecular geometry that complements the protein binding cavity. Beam Enumeration shows that improvements to explainability and sample efficiency for molecular design can be made synergistic. The improvements in the latter will en- able more expensive high-fidelity oracles to explicitly optimized. We note, however, that sparse reward environments (Korshunova et al. (2022) remain difficult optimization task. Finally, Beam Enumeration is a Lask-agnostic method and can be combined with recent work integrating active learning with molecular generation to further improve sample efficiency (Dodds et al. (2024); Kyro et al. (2023). y. If the benefits can be synergistic, we may approach sufficient sample efficiency to directly optimize expensive state-of-the-arr (in predictive accuracy) physics-based oracles such as MD simulations (Wang et al. (2015); Moore et al. (2023). Excitingly, this would in turn enhance explainability as high-fidelity oracles are inherently more informative.

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY STATEMENT
The code is provided in the GitHub link the Abstract and also provided here: https: //github. cwm/ schwa lergroup awwgenteeeeeeem memory In the repository, there are pre- pared configuration files that can be directly run to reproduce all experiments in this work.
REFERENCES
André Altmann, Laura Toloşi, Oliver Sander, and Thomas Lengauer. Permutation importance: corrected feature importance measure. Bioinformatics, 26(10):1340-1441 2010.
Maria Argiriadi, Anna Ericsson, Christopher Harris, David L Banach, David W Borhani, David J Calderwood, Megan D Demers, Jennifer DiMauro, Richard W Dixon, Jennifer Hardman, et al. 2, d-dimiiooyrimmiiin mk2 inhibitors. part i: observation of an unexpected inhibitor binding mode. Bioorganic & medicinal chemistry letters, 20(1):30--333, 2010.
John A Amott and Sonia Lobo Planey. The influence of lipophilicity in drug discovery and design. Expert opinion on drug discovery, 7(10):863- 875, 2012.
Sara Romeo Atance, Juan Viguera Diez, Ola Engkvist, Simon Olsson, and Rocío Mercado. De novo drug design using reinforcement learning with graph-based deep generative models. Journal of Chemical Information and Modeling, 62(20):4863-487 2022.
Marco Ballarotto, Sabine Willems, Tanja Stiller, Felix Nawa, Julian A Marschner, Francesca Grisoni, and Daniel Merk. novo design of nurrl agonists via fragment-augmentte genera- tive deep learning in low-data regime. Journal of Medicinal Chemistry, 2023.
Guy W Bemis and Mark A Mureko. The properties of known drugs. 1. molecular frameworks. Journal of medicinal chemistry, 39(15))288722893 1996.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-277 -22394, 2021a.
Yoshua Bengio, Tristan Deleu, Edward J. Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. GFlowNet foundations. CoRR, abs/211 1.0926,, 2021b.
G. Richard Bickerton, Gaia V. Paolini, Jérémy Besnard, Sorel Muresan, and Andrew I.. Hopkins. Quantifying the chemical beauty drugs. Nature Chem, 4(2):90-98, February 2012. ISSN 1755- 4349. doi: 10. 1038/nchem. 1243. URL fattps://wer nature wwwm//ariicee//cheeee.
1243
Number: Publisher: Nature Publishing Group.
Thomas Blaschke, Josep Arús-Pous, Hongming Chen, Christian Margreitter, Christian Tyrchan, Ola Engkvist, Kostas Papadopoulos, and Atanas Patronov. REINVENT 2.0: An AI Tool for De Novo Drug Design. Chem. Inf. Model., 60(12).591855922 December 2020. ISSN 1549- jaim.
9596. doi: 10.102/ htt.s..cim..c009911.. URL /ttp:p/://oi..orr110111001110110 0cC0915
Publisher: American Chemical Society.
Nicolas Boulanger-LLewandowss Yoshua Bengio, and Pascal Vincent. High-dimensional sequence transduction. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 3178-3182. JEEE, 2013.
Nannan Chen, Lijuan Yang, Na Ding, Guiwen Li, Jiajing Cai, Xiaoli An, Zhijie Wang, Jie Qin, and Yuzhen Niu. Recurrent neural network (rnn) model accelerates the development of antibacterial metronidazole derivatives. RSC advances, (2355):2293-22201. 2022.
Michael Dodds, Jeff Guo, Thomas Löhr, Alessandro Tibo, Ola Engkvist, and Jon Paul Janet. Sample efficient reinforcement learning with active learning for molecular design. Chem. Sci., 2024. ISSN 2041-6539. doi: 10.1039/d3cc04653b
Peter Eastman, Jason Swails, John D Chodera, Robert T McGibbon, Yutong Zhao Kyle A Beauchamp, Lee-Ping Wang, Andrew C Simmonett, Matthew P Harrigan, Chaya D Stern, et al. Openmm 7: Rapid development of high performance algorithms for molecular dynamics. PLoS computational biology, 13())=e005665, 005659, 2017.
10

---

Published as a conference paper at ICLR 2024
Merveille Eguida, Christel Schmitt- Valeneia, Marcel Hibert, Pascal Villa, and Didier Rognan. Target-focused library design by pocket-applied computer vision and fragment deep generative linking. Journal of Medicinal Chemistry, 13720)31377 2022.
Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of cheminfor- matics, 1:1-11, 2009.
Vendy Fialková, Jiaxi Zhao, Kostas Papadopoulos, Ola Engkvist, Esben Jannik Bjerrum, Thierry Kogej. and Atanas Patronov. LibINVENT: Reaction-based Generative Scaffold Decoration for in Silico Library Design. J. Chem. Inf. Model., 6209))2046-2633, May 2022. ISSN 1549-9596. doi:
10.1021/acs.jcmm.1cl 1c0049. URL https: /ttp:::/ddooooooo org 10tl////...000001aee/ jeim. wtt.m..........................e 1c00469
Publisher: American Chemical Society.
Tianfan Fu, Cao Xiao, Xinhao Li, Lucas M Glass, and Jimeng Sun. MIMOSA: Multi-constraint molecule sampling for molecule optimization. AAAI, 2020.
Tianfan Fu, Wenhao Gao, Connor Coley, and Jimeng Sun. Reinforced genetic algorithm for structure-based drug design. Advances in Neural Information Processing Systems, 35:12325- 12338, 2022a.
Tianfan Fu, Wenhao Gao, Cao Xiao, Jacob Yasonik, Connor W Coley, and Jimeng Sun. Differen- tiable scaffolding tree for molecular optimization. International Conference on Learning Repre- sentations, 2022b.
Wenhao Gao. Tianfan Fu, Jimeng Sun, and Connor W. Coley. Sample Efficiency Matters: A Bench- mark for Practical Molecular Optimization, October 2022. URL. http:/tt/eexxi..rrr////////////e 2206. 11.6..24 12411 arXiv:2206. 12411 [cs, q-bio].
Anna Gaulton, Louisa J. Bellis, A. Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey,, David Michalovich, Bissan Al-Lazikani, and John P. Over- ington. ChEMBL: a large-scale bioactivity database for drug discovery. Nucleic Acids Res, 40 (Database (ssue):D1100DDI10 January 2012a. ISSN 0305-1048 doi: 10. /0.10///arr/k/777 URL httphttps://mii ncbi nlm. nih gov http://mc/aarricceee////22322323
Anna Gaulton, Louisa J Bellis, Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, 100D1))D1000-11107 2012b.
100-D1
Manan Goel, Shampa Raghunathan, Siddhartha Laghuvarapu, and U Deva Priyakumar. Molegular: molecule generation using reinforcement learning with alternating rewards. Journal of Chemical Information and Modeling, 61 61(12):5815-5822 2021.
Prashant Gohel, Priyanka Singh, and Manoranjan Mohanty. Explainable ai: current status and future directions. arXiv preprint arXiv:2107.070: 2021.
Rafael Gomerz-Bom Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobate Benjamín Sänchez-Lengeling. Dennis Sheberla, Jorge aguilera-Iparrauirre Timothy Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven contin- uous representation of molecules. ACS central science, 4(2):268 276, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks, June 2014. URL BtLhttp:
//axxv..rr//ab//11000 2661 arXiv:1406.2661 [cs, stat].
Alex Graves. Sequence transduction with recurrent neural networks. 11X1v:1211.3711,200 2012.
arXiv
preprint
Francesca Grisoni, Berend JH Huisman, Alexander L Button, Michael Moret, Kenneth Atz, Daniel Merk, and Gisbert Schneider. Combining generative artificial intelligence and on-chip synthesis for de novo drug design. Science Advances, 7(24)xeabg3338, 2021.

---

Published as a conference paper at ICLR 2024
Gabriel Lima Guimaraes, Benjamin Sanchez-Lengeling Carlos Outeiral, Pedro Luis Cunha Farias, and Alán Aspuru-Guzik, Objective- Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models, February 2018. URL http::/aax.v.org//aaaaaaa
10843 arXiv: 17055 10843 [es, stat].
Jeff Guo and Philippe Schwaller.
Augmented memory: Capitalizing on experience replay accel- crate de novo molecular design. arXiv preprint wrXi:2335..66160 2023.
Jeff Guo. Jon Paul Janet, Matthias R Bauer, Eva Nittinger, Kathryn A Giblin, Kostas Papadopou- los, Alexey Voronov, Atanas Patronov, Ola Engkvist, and Christian Margreitter. Dockstream: docking wrapper enhance de novo molecular design. Journal of cheminformaties, 13(1):1-21, 2021a.
Jeff Guo, Jon Paul Janet, Matthias R. Bauer, Eva Nittinger, Kathryn A. Giblin, Kostas Papadopou- los. Alexey Voronov, Atanas Patronov, Ola Engkvist, and Christian Margreitter. DockStream: docking wrapper to enhance de novo molecular design. Journal of Cheminformatics, 13(1): doi: 10.1186611332--01-00055 //doi. 11..0011166111111 021-00063-11111--1
89, November 2021b. ISSN 1758- 2946.
URL https:
Minghao Guo, Veronika Thost, Beichen Li, Payel Das, Jie Chen, and Wojeiech Matusik. Data- efficient graph grammar learning for molecular generation. arXiv preprint arXiv:2203.08031, 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Yi Hua, Xiaobao Fang, Guomeng Xing. Yuan Xu, Li Liang Chenglong Deng, Xiaowen Dai, Haichun Liu, Tao Lu, Yanmin Zhang, al. Effective reaction-based de novo strategy for ki- nase targets: a case study on mertk inhibitors. Journal of Chemical Information and Modeling, (27):1654-1668 2022. 2022.
Scong Hun Jang, Dakshinamurthy Sivakumar, Sathish Kumar Mudedla, Jachan Choi, Sungmin Lee, Minjun Jeon, Suneel Kumar Bvs, Jinha Hwang, Minsung Kang, Eun Gyeong Shin, et al. Pew- a1001, ai-assisted de novo design approach to design a selective inhibitor for flt-3 (d835y) in acute myeloid leukemia. Frontiers in Molecular Biosciences, 9:1072028, 2022.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junetion tree variational autoencoder for molecular graph generation. In International conference machine learning, pp. 2323-2332. PMLR, 2018.
Wengong Jin, Dr Regina Barzilay, and Tommi Jaakkola. Multi-Objective Molecule Generation using Interpretable Substructures. In Proceedings of the 37th International Conference on Machine Learning, pp. 4849-4859. PMLR, November 2020. URL ttetts::/proceeeiingg mml.
press/v119 (in.1l//jinbb....m html
ISSN: 2640-3498.
Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes, December 2022. URL
http:://axiv. org/ abs/ 6114 arXiv:1312.6114 [es, stat].
Maria Korshunova, Niles Huang, Stephen Capuzzi, Dmytro S. Radchenko, Olena Savych, Yuriy S. Moroz, Carrow Wells, Timothy M. Willson, Alexander Tropsha, and Olexandr
Isayev. Generative and reinforcement learning approaches for the automated de novo de- sign of bioactive compounds. Commun Chem, 5(1):1-11, October 2022. ISSN 2399- 3669. doi: 10. 000-8//422000022-077733333 URL htthttps::/ 942004-022-00733--
com. articles/ Number: 1 Publisher: Nature Publishing Group.
Gitay Kryger, Israel Silman, and Joel L Sussman. Structure of acetylcholinesterass complexed with e2020 (aricept@): implications for the design of new anti-alzheimer drugs. Structure, 7(3):297- 307, 1999.
Gregory W Kyro, Anton Morgunov, Rafael I Brent, and Victor S Batista. Chemspaceal: An efficient active learning methodology applied to protein-specific molecular generation. arXiv preprint arXiv:2309. 05853, 2023.
12

---

Published as a conference paper at ICLR 2024
Yangguang Li, Yingtao Liu, Jianping Wu, Xiaosong Liu, Lin Wang, Ju Wang, Jiaojiao Yu, Hongyun Qi, Luoheng Qin, Xiao Ding, et al. Discovery of potent, selective, and orally bioavailable small- molecule inhibitors of cdk8 for the treatment of cancer. Journal of Medicinal Chemistry, 2023.
Yueshan Li, Liting Zhang, Yifei Wang. Jun Zou, Ruicheng Yang. Xinling Luo, Chengyong Wu, Wei Yang. Chenyu Tian, Haixing Xu, et al. Generative deep learning enables the discovery of a potent and selective ripk1 inhibitor. Nature Communications, 13(1);6891, 2022.
Long-Ji Ji Lin. Self improving reactive agents based on reinforcement learning planning and teaching. Machine learning, 8:293-321, 1992.
Rocío Mercado, Esben J Bjerrum, and Ola Engkvist. Exploring graph traversal algorithms in graph- based molecular generation. Journal of Chemical Information and Modeling, 62(9):2093-2100, 2021a.
Rocío Mercado, Tobias Rastemo, Edvard Lindelöf, Günter Klambauer, Ola Engkvist, Hongming Chen, and Esben Jannik Bjerrum. Graph networks for molecular design. Machine Learning: Science and Technology, 2(2):025023, 2021b.
Rocío Mercado, Tobias Rastemo, Edvard Lindelöf, Günter Klambauer, Ola Engkvist, Hongming Chen, and Esben Jannik Bjerrum. Graph networks for molecular design. Mach. Learn.: Sci. Technol. 2(2):025023, March 2021c. ISSN 2632-2153. doi: 10. (188/2632-2153/abchh URL Publisher: IOP Publishing.
https: /tttp::/dxxdoo/ org htte//11111188288 20000///6622213//80000
Daniel Merk, Francesca Grisoni, Lukas Friedrich, and Gisbert Schneider. Tuning artificial intelli- gence the de novo design of natural-prosn product-prrr retinoid x receptor modulators. Commu- nications Chemistry, ((1):68, 2018.
J Harry Moore, Christian Margreitter, Jon Paul Janet, Ola Engkvist, Bert L de Groot, and Vytautas Gapsys. Automated relative binding free energy calculations from smiles to dóg. Communications Chemistry, 6(1):82, 2023.
Michael Moret, Moritz Helmstädter, Francesca Grisoni, Gisbert Schneider, and Daniel Merk. Beam search for automated design and scoring of novel ror ligands with machine intelligence. Ange- wandte Chemie International Edition, 19135)::9477119992 2021.
Michael Moret, Irene Pachon Angona, Leandro Cotos, Shen Yan, Kenneth Atz, Cyrill Brunner, Martin Baumgartner, Francesca Grisoni, and Gisbert Schneider. Leveraging molecular structure and bioactivity with chemical language models for de novo drug design. Nature Communications, 14(I): 14,,1114220 2023.
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatics, 9(1) 911,, September 2017. ISSN 1758-2946. doi: 10.1
URL
1186/s13321-01 017- 0235-
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, Artur Kadurin, Simon Johansson, Hongming Chen, Sergey Nikolenko, Alán Aspuru- Guzik, and Alex Zhavoronkov. Molecular Sets (MOSES): A Benchmarking Platform for Molec- ular Generation Models. Frontiers in Pharmacology, 11, 2020a. ISSN 1663-9812. URL https: wwwntiersiinnnnmo org wtt.e/sww.........a.eeeeeeo..... 338./339//80 /ttp9//paaa....mm 2020.565611
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, et al. Molecular sets (moses): a benchmarking platform for molecular generation models. Frontiers in pharmacology, 11:666644, 2020b.
Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science Advances, 4(7)xeaap7885, July 2018. doi: 10.1126/sciadv.aup788 URL https: / www.. science..... org/ /ww//odio1/////// 1126/ 2//66/scian sciadv. aap7885 Publisher: Ameri-
can Association for the Advancement of Science.
13

---

Published as a conference paper at ICLR 2024
Anthony K Rappé. Carla J Casewit, KS Colwell, William A Goddard III, and W Mason Skiff. Uff, a full periodic table force field for molecular mechanics and molecular dynamics simulations. Journal of the American chemical society, 114(25::102441005) 1992.
Feng Ren, Xiao Ding, Min Zheng, Mikhail Korzinkin, Xin Cai, Wei Zhu, Alexey Mantsy- zov, Alex Aliper, Vladimir Aladinskiy, Zhongying Cao. Shanshan Kong, Xi Long, Bonnic Hei Man Liu, Yingtao Liu, Vladimir Naumov, Anastasia Shneyderman, Ivan V. Ozerov, Ju Wang, Frank W. Pun, Danii A. Polykovskiy, Chong Sun, Michael Levitt, Alán Aspuru-Guzik, and Alex Zhavoronkov. AlphaFold accelerates artificial intelligencee powered drug discovery: ef- ficient discovery of a novel CDK20 small molecule inhibitor. Chemical Science, 14(6);1443- 1452, 2023. doi: 10.1039/D2SC05709CC URL https: /pubs. rsc. org http//////onnn//////// artic www.ceelandiin S
Publisher: Royal Society of Chemistry.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Leslic Salas-Estrada, Davide Provasi, Xing Qiu, H Umit Kaniskan, Xi-Ping Huang, Jeffrey DiBerto, Joao Marcelo Lamim Ribeiro, Jian Jin, Bryan L. Roth, and Marta Filizola. De novo design of k-opioid receptor antagonists using a generative deep learning framework. bioRxiv, pp. 2023-04, 2023.
Benjamin Sanchez-Lengeling and Alán Aspuru-Guzik. Inverse molecular design using ma- chine learning: Generative models for matter enginecring. Science, 361(6400):360-36) July 2018. doi: scienceeeatt2666 Publisher: American Association for the Advancement Science.
URL wwww. www.sceieee.ee..............ee.. wttp:///////o1111/6666
Benjamin Sanchez-Lengelingg Carlos Outeiral, Gabriel L Guimaraes, and Alan Aspuru-Guzik. Opti- mizing distributions over molecular space. an objective-reinforreed generative adversarial network for inverse-design chemistry (organic). ChemRxiv, 2017.
Marwin HS Segler, Thierry Kogej, Christian Tyrchan, and Mark P Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS central science, 4(1); 120-131, 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp- 618-626, 2017.
Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(1)) 1095-1100, 1953.
Shukai Song, Haotian Tang, Ting Ran, Feng Fang, Linjiang Tong, Hongming Chen, Hua Xie, and Xiaoyun Lu. Application of deep generative model for design of pyrrolo (2, 3-d] pyrimidine derivatives new selective tank binding kinase (tbk1) inhibitors. European Journal of Medici- nal Chemistry, 247:1 15034, 2023.
Xiaoqin Tan, Chunpu Li, Ruirui Yang. Sen Zhao, Fei Li, Xutong Li, Lifan Chen, Xiaozhe Wan, Xiaohong Liu, Tianbiao Yang, al. Discovery of pyrazolo [3, 4-d] pyridazinone derivatives as selective ddrl inhibitors via deep learning based design, synthesis, and biological evaluation. Journal of Medicinal Chemistry, 65(1):103-119, 2021.
Oleg Trott and Arthur J Olson. Autodock vina: improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading. Journal of computational chemistry, 31(2)455-46), 2010.
Sarveswara Rao Vangala, Sowmya Ramaswamy Krishnan, Navneet Bung, Rajgopal Srinivasan, and Arijit Roy. pbrics: A novel fragmentation method for explainable property prediction of drug- like small molecules. Journal of Chemical Information and Modeling, 2023.
14

---

Published as a conference paper at ICLR 2024
Lingle Wang, Yujie Wu, Yuqing Deng, Byungchan Kim, Levi Pierce, Goran Krilov, Dmitry Lupyan, Shaughnessy Robinson, Markus K Dahlgren, Jeremy Greenwood, et al. Accurate and reliable prediction of relative ligand binding potency in prospective drug discovery by way of modern free-energy calculation protocol and force field. Journal of the American Chemical Society, 137 (T:2669-27033, 2015.
Sheng Wang, Tao Che, Anat Levit, Brian Shoichet, Daniel Wacker, and Bryan Roth. Structure of the d2 dopamine receptor bound to the atypical antipsychotic drug risperidone. Nature, 555 (7695):269-21)) 273,, 2018.
David Weininger. SMILES, a chemical language and information system. 1. Introduction to method- ology and encoding rules. J. Chem. Inf. Comput. Sci., 28(1):11-36, February 1988. ISSN 0095- 2338. doi: 10. (000000/:000057:0055. URL Bttta:///a...orr/L...12////////
Publisher: American Chemical Society.
Geemi P Wellawatte, Aditi Seshadri, and Andrew D White. Model agnostic generation of counter- factual explanations for molecules. Chemical science, 13(11):3697-3705, 2022.
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. MARS:
Markov molecular sampling for multi-objective drug discovery. In ICLR, 2021.
Atsushi Yoshimori, Yasunobu Asawa, Enzo Kawasaki, Tomohiko Tasaka, Seiji Matsuda, Toru Sekikawa, Satoshi Tanabe, Masahiro Neya, Hideaki Natsugari, and Chisato Kanai. Design and synthesis of ddrl inhibitors with a desired pharmacophore using deep generative models.
ChemMedChem, 16(6):955-958, 2021.
Jiaxuan You, Bowen Liu, Rex Ying, Vijay Pande, and Jure Leskovec. Graph Convolutional Pol- icy Network for Goal-Directed Molecular Graph Generation, February 2019. URL http:
//axxiv.rrg/ab///0000
02473 arXiv:1806.02477 [cs, stat].
Yang Yu, Tingyang Xu, Jiawen Li, Yaping Qiu, Yu Rong. Zhen Gong, Xuemin Cheng, Liming Dong, Wei Liu, Jin Li, et al. A novel scalarized scaffoold hopping algorithum with graph-based variational autoencoder for discovery of jak1 inhibitors. ACS omega, 6(35):22945-2299 2021.
Alex Zhavoronkov, Yan A. Ivanenkov, Alex Aliper, Mark S. Veselov, Vladimir A. Aladinskiy, Anastasiya V. Aladinskaya, Victor A. Terentiev, Daniil A. Polykovskiy, Maksim Kuznetsov, Arip Asadulaev, Yury Volkov, Artem Zholus, Rim R. Shayakhmetov, Alexander Zhebrak, Lidiya Minaeva, Bogdan A. Zagribelnyy. Lennart H. Lec, Richard Soll, David Madge, Li Xing, Tao Guo, and Alán Aspuru-Guzik. Deep learning enables rapid identification of po- tent DDRI kinase inhibitors. Nat Biotechnol, 37(9):1038 1040, September 2019. ISSN 1546- 1696. doi: 1300000004 141587-019--220
URL https: /www. nature.com.. wwt.///aa/aee....eeee. Number: 9 Publisher: Nature Publishing Group.
15

---

Published as a conference paper at ICLR 2024
A
APPENDIX
The Appendix contains further experiments, ablation studies, experiment hyperparameters, and al- gorithmic details.
A
BEAM ENUMERATION
This section contains full details on Beam Enumeration including hyperparameters, design deci- sions, and pseudo-code.
A.1
ALGORITHM OVERVIEW
a.
b. Beam Enumeration
N Beam Steps
Enforce
0.5
CC
Top
0.75
0.05
Score
Salf Filter
S
0,03
0.1
Objective Function
0.15
0.08
0.8
OC
Pool
Oraic Reward
Enumeration
(0.29)
Probable Substructures
0.01
Sample Molecules
Update
Substructure Extraction
Structure
Scaffold
cIcccciC[=O)
4
clccccc1(=O)
Up to 29-fold Sample Efficiency Increase
clcccciC[=O)
Figure A4: Beam Enumeration overview. The proposed method proceeds via 4 steps: 1. generate batch of molecules. 2. filter molecules based on pool to enforce substructure presence, discarding the rest. 3. compute reward 4. update the model. After updating the model, if the reward has improved for consecutive epochs, execute Beam Enumeration. b. Beam Enumeration sequentially enumerates the top tokens by probability for N beam steps, resulting in an exhaustive set of token sub-sequences. c. All valid substructures (either by the Structure or Scaffold criterion) are extracted from the sub-sequences. The most frequent substructures are used for ceff-conditioned generation. This overview figure is the same in the main text.
Beam Enumeration (Fig. A4) is an algorithm that extracts molecular substructures from generative model's weights for self-conditioned generation. The problem set-up any molecular design task to optimize for target property profile, e.g., high predicted solubility and binding affinity. When molecular generative models are coupled with an optimization algorithm, should be increasingly likely to generate desirable molecules, i.e., molecules that possess the target property profile.
Beam Enumeration is proposed based two facts:
1. On successful optimization trajectory, the model's weights must change such that desir- able molecules are more likely to be generated, on average.
2. The act of generating molecules in an autoregressive manner involves sequentially sam- pling from conditional probability distributions.
In this work, Beam Enumeration is applied to a language- based autoregressive generative model operating the simplified molecular-input line-entry system (SMILES) (Weininger (1988) rep-
16

---

Published as a conference paper at ICLR 2024
resentation. The optimization algorithm is Augmented Memory (Guo & Schwaller (2023) which builds on REINVENT (Olivecrona et al. (017)) Blaschke et al. (2020) and casts the optimization process on-policy reinforcement learning (RL) problem. Following RL terminology. sampling from the generative model involves sampling trajectories, which in this case, are SMILES, and the desirability of the corresponding molecule is given by the reward.
The underlying hypothesis of Beam Enumeration that during the RL optimization process, partial trajectories provide source of signal that can be exploited. Usually, full trajectories are sampled which map to a complete SMILES sequence that can be translated to a molecule. Our assumption that partial trajectories (partial SMILES sequence) can be mapped to molecular substructures (a part of the full molecule). This statement is not guaranteed as SMILES and molecules are discrete and small perturbations often leads to invalid SMILES. We prove this assumption in Section [ by showing that although the vast number of partial trajectories do not map to valid SMILES, the raw number is sufficient to extract meaningful signal. Correspondingly, Beam Enumeration leverages partial trajectories on the assumption that molecular substructures are track to becoming full molecules that would receive high reward.
A.2 ENUMERATING PARTIAL TRAJECTORIES
In order to extract molccular substructures, a set of partial trajectories must be sampled from the generative model. Recalling the fact that on a successful optimization trajectory, it must become increasingly likely generate desirable molecules, partial trajectories are sampled by enumerating the top k tokens, based on the conditional probability. Therefore, the process of enumerating par- tial trajectories involves sequentially extending each token sequence by their next top k probable tokens, resulting in the total number of partial trajectories 2 where N is the number of beam steps, i.e., how many tokens the partial trajectory. We note that taking the top most proba- ble tokens does not guarantec that the partial trajectories probability penalty early can lead to higher probabilities later. However, our assumption is that on average, this leads to set of partial trajectories that are at the very least, amongst the most probable. Moreover, there is a practical limit to how many partial trajectories are sampled due to exponential growth which makes scaling quickly computationally prohibitive. In the later section, we discuss this thoroughly. Finally, from here, partial trajectories will be referred to as token sub-sequences.
indeed the most probable, as paying
A.3 EXTRACTING MOLECULAR SUBSTRUCTURES
Given a set of token uub-sequences, the goal to extract out the most frequent molecular substruc- tures. This is done by taking each sequence, considering every sub))sub-sequenee and counting the number of valid substructures (Fig. A).. For example, given the sub-sequence "ABCD", the set of sub)-sub--equeenes, are: "A", "AB", "ABC", and "ABCD". In practice, we only consider (sub)-sub- sequences with at least three characters ("ABC" and "ABCD") since each character loosely maps to one atom and three is approximately the minimum for meaningful functional groups, e.g., "C=O", carbonyl. The set of most frequent substructures assumed to be on track to receive a high reward.
A.4 DEFINING MOLECULAR SUBSTRUCTURES: Scaffold vs. Structure
As shown in Fig. molecular substructures can be defined on the Scaffold or Structure level. The former extracts the any valid structure. The any valid structure an important distinction as our experiments find that extracting by Structure leads to the most frequent molecular substructures being small functional groups that do not have corresponding scaffolds. By contrast, extracting the scaffold always leads to ring structures. Moreover, extracting specifically the Bemis-Murcko scaffold is important heavy atoms, e.g., nitrogen, are important for biological activity. Consequently, extracted substructures are also enforced to contain least one heavy atom we find that benzene, perhaps unsurprisingly, is commonly the most frequent substructure. See Section B for more details on the differing behavior of 'Scaffold" vs. 'Structure'.
(Bemis & Murcko (1996) scaffold while the latter extracts
17

---

Published as a conference paper at ICLR 2024
A.5 SELF-CONDITIONED GENERATION
Self-conditioned generation is achieved by filtering sampled batches of molecules from the gener- ative model to only keep the ones that possess least one the most frequent substructures. The effect is that the generative process is self-biased to focus on a narrower chemical space which we show can drastically improve sample efficiency the expense some diversity, which is acceptable when expensive high-fidelity oracles are used: want to identify small set of excellent candidate molecules under minimal oracle calls.
A.6 PROBABILISTIC EXPLAINABILITY
The set of most frequent molecular substructures should be meaningful otherwise, the model's weights would not have been updated such that these substructures have become increasingly likely to be generated. We verify this statement in the illustrative experiment in the main text and in Section C In the drug discovery case studies (Appendix D), the extracted substructures are more subtle in why they satisfy the target objective but certainly must possess meaning, however subtle, as otherwise, they would not receive a high reward. In the main text, we show that extracted sub- structures form scaffolds and structural motifs in the generated molecules that complement the protein binding cavity. Finally, we emphasize that the correctness and usefulness of this explainabil- ity deeply depends on the oracle(s) being optimized for. The extracted substructures do not explain why the generated molecules satisfy the target objectíve. Rather, they explain why the generated molecules satisfy the oracle. The assumption in generative design task is that optimizing the or- acle is a good proxy for the target objective, e.g., generating molecules that dock well increases the likelihood of the molecules being true binders. This observation directly provides additional commentary on why sample efficiency is so important: the ability to directly optimize expensive figh-fidelity oracles would inherently enhance the correctness of the extracted substructures.
B BEAM ENUMERATION FINDINGS FROM HYPERPARAMETES SCREENING
In this section, introduce all seven hyperparameters of Beam Enumeration and then present results on an exhaustive hyperparameter search which elucidates the behavior and interactions of all the hyperparameters. In the end, we present analyses and provide hyperparameter recom- mendations for Beam Enumeration which can serve as default values to promote out-of-the-box application.
B. BEAM ENUMERATION HYPERPARAMETERS
Beam k. This hyperparameter denotes how many tokens to enumerate at each step. Given that our hypothesis is that the most probable sub-sequences yield meaningful substructures, we fix Beam to 2. A larger value would also decrease the number of Beam Steps possible as the total number of sub-sequences is K.N and the exponential growth quickly leads computational infeasibility.
Beam Steps N. This hyperparameter denotes how many token enumeration steps to execute and is the final token length of the enumerate sub-sequences. This parameter leads exponential growth in the number of sequences which can quickly become computationally prohibitive. An important implication of this hyperparameter is that larger Beam Steps means that larger substructures can be extracted. In our experiments, we find that enforcing size in the extracted substructures can dras- tically improve sample efficiency with decreased diversity as the trade-off. We thoroughly discuss this in a later sub-section. Finally, in our experiments, the upper-limit investigated is 18 Beam Steps.
Substructure Type. This hyperparameter has two possible values: Scaffold or Structure. Scaf- fold extracts Bemis-Murcko (Bemis & Murcko 1996) scaffolds while Structure extracts any valid substructure.
Structure Structure Minimum Size. This hyperparameter enforces the partial SMILES to contain at least a certain number of characters. In effect, this enforces extracted molecular substructures to be larger than Structure Minimum Size. From the illustrative experiment in the main text and Section C Structure extraction often leads to small functional groups being the most frequent in the sub-sequences. By enforcing minimum structure size, Structure extraction leads to partial
18

---

Published as a conference paper at ICLR 2024
structures which may carry more meaning. We find that this hyperparameter greatly impacts sample efficiency and we present all our findings in a later sub-section.
Pool Size. This hyperparameter controls how many molccular substructures to keep track of. These pooled substructures are what is used to perform self-conditioning. The hypothesis is that the most frequent ones carry the most meaning and thus, very large pool size may not be desired.
Patience. This hyperparameter controls how many successive reward improvements are required before Beam Enumeration executes and molecular substructures extracted. Recalling the first fact in which Beam Enumeration was proposed on: On successful optimization trajectory, the model's weights must change such that desirable molecules are likely to be generated, average. Patience is effectively an answer to "when would extracted substructures be meaningful?" Too low a patience and stochasticity can lead negative effects while too high patience diminishes the benefits of Beam Enumeration on sample efficiency.
Token Sampling Method. This hyperparameter has two possible values: "topk" "sample" and denotes how tokens sub-sequences are enumerated. "topk" takes the top k most probable tokens at each Beam Step while "sample" samples from the distribution just like during batch generation. Our results show interesting observations surrounding this hyperparameter as "sample" can work just as well and sometimes even better than taking the "topk". These results were unexpected as the under- lying hypothesis is that the most probable sub-sequee sequnnesssses lead to the most useful substructures being extracted. However, our findings are not contradiction as sampling the conditional probability distributions would still lead to sampling the top k tokens, average. Moreover, after extensive experiments, we find that "sample" leads to more variance in performance across replicates which is in agreement with the assumption that sampling the distributions can lead more improbable struc- tures. We thoroughly discuss our findings in a later sub-section where we provide hyperparameter recommendations and analyses the effects of tuning each hyperparameter.
B.2 HYPERPARAMETERS GRID SEARCH
We performed two exhaustive hyperparameter grid searches the illustrative experiment which has the following multi-parameter optimization (MPO) objective: maximize topological polar surface area (tPSA), molecular weight <350 Da, number of rings 2 2 with oracle budget of 5,000. The first grid search investigated the following hyperparameter combinations:
Beam K 2
Beam Steps = [15, 16, 17, 18]
Substructure Type [Scaffold, Structure]
Pool Size = [3, 4, 5]
Patience [3, 5]
Token Sampling Method ['topk', 'samplc']
All hyperparameter combinations (144) were tried and run for 10 replicates each for statistical reproducibility, total of 1,440 experiments. Next, additional grid search was performed with the following hyperparameter combinations:
Beam K 2
Beam Steps = [17, 18]
Substructure Type [Scaffold, Structure]
Structure Structure Minimum Size = [10, 15]
Pool Size = [4, 5]
Patience [4, 5]
Token Sampling Method ['topk', 'sample']
We take the general trends from the first grid search and narrow down the most optimal hyperpa- rameters to further investigate Substructure Type and structure Structure Minimum Size. As from
19

---

Published as a conference paper at ICLR 2024
before, all hyperparameter combinations (64) were tried and for 10 replicates each for statistical reproducibility, total of 640 experiments.
The following heatmaps performance by the Generative Yield and Oracle Burden (10) metrics at the >0.8 reward threshold and under a 5,000 oracle budget. The Generative Yield measures how many unique molecules above 0.8 reward were generated. The Oracle Burden (10) measures how few oracle calls were required to generate 10 molecules above 0.8 reward.
B.3 ANALYSIS OF GRID SEARCH RESULTS
In this section, we summarize our analysis on the grid search experiments. Unless stated, each bullet point means the observation was observed for both Generative Yield and Oracle Burden (10). For example the point: Scaffold SStrcture means Scaffold is generally more performant than Structure across all hyperparameters on both the Generative Yield and Oracle Burden (10).
For Scaffold, higher Pool, higher Patience, and higher Beam Steps improves performance
For Structure, lower pool and lower patience improves performance
Scaffold Structure
Scaffold and Structure become performant with increasing Structure Minimum Size
Scaffold and Structure with Structure Minimum Size: "sample" sampling can be better than "topk" sampling but with more variance
Based on the above analysis, we propose the optimal hyperparameters for the illustrative experiment as:
Scaffold
"topk" sampling ("sample" sampling can be more performant but exhibits higher variance)
Patience
Pool Size
Beam Steps = 18
Finally, we provide more commentary on interesting observations from the grid search results. Structure without Structure Minimum Size enforcing often leads to small functional groups being the most frequent molecular substructures extracted with Beam Enumeration. Enforcing Structure Minimum Size puts it almost on par with Scaffold, suggesting (perhaps not surprisingly) that larger substructures can carry more meaningful information. Moreover, when using "sample" sampling, the generative model undergoes more "filter rounds". Specifically, each epoch, the sampled batch is filtered contain the extracted substructures. When using "sample" sampling, the model is more prone to some epochs containing no molecules with the substructures. sequential as sampling is computationally inexpensive and a next batch of molecules can easily be sampled. However, specifically in the Structure with "sample" sampling and Structure Minimum Size = 15 experiment, "filter round" can be quite extensive, taking up to 100,000 epochs (maxi- mum observed) for an oracle budget of 5,000 (adding about an hour to the wall time which is minor when the oracle is expensive). This means that many epochs contained molecules without the tracted substructures. There are two observations here: firstly, "sample" sampling can lead to more improbable substructures which hence less likely be sampled and secondly, Structure with Structure Minimum Size enforcement leads to extreme biasing (which improves sample efficiency). We believe the remarkable tolerability of the generative model sampling to such bias is an interest- ing observation. By contrast, Scaffold with Structure Minimum Size enforcement is not as prone "filter rounds" because Scaffold "truncates" the substructure to central shape (scaffold). For example, toluene (benzene with methyl group) has a Bemis-Murcko (Bemis & Murcko (1996) scaffold of just benzene. The consequence that Structure leads to more extreme biasing (it is more likely for molecule to contain benzene than specifically toluene) which is in agreement with the general observation that the diversity the generated set decreases when using Structure. Overall, both Scaffold and Structure with Structure Minimum Size enforcing exhibits the best performance and "sample" sampling can be more performant than "topk" sampling but exhibits notably higher variance.
practice, this is incon-
20

---

Published as a conference paper at ICLR 2024
Generative Yield 0.8 Reward
Scaffold
Poticncc
270e174
351m211
46 322
200
229:248
265
423
S19+215
527
402m247
354m2236
405
0.7355.04
334
302
304=2117
431=106
207
407+335
450+300
432=2200
162
0. 766
0.75
7.7200004
0.77:0.04
0.77 00
0,7580.05
0.75e0.03
0.7280 04
0.7420.03
0. 0.76m0
7.75 2.
0.77
0.78:0.03
0.73u0.03
0:3300.05
0.72:9.00
6.74x9. 00
0.77x0.03
0.7520.04
0.75x0.03
0.75:..02
0. 0.74:0.03
6 04::0.04
0.74:9.00
15
16
17
15
17
16
15
16
17
18
Deam Staps
Bearn Staps
Beamn Stups
Structure
Patience
Patience
Patience
252
249-127
167
252-701
223-143
295-133
101
211
0.75:0003
0.79:0.00
0.7910.03
0.7620.02
0.79:0
800
211-001
153162
145157
259-97
210103
227+75
2074134
256-111
120-136
220-00
154:37
290-171
600
400
200
2391162
179:63
144570
211=100
217e115
227:180
179:101
2195151
184:95
176:301
212-100
108::3
0.77-0.03
0.000.03
0.79:0001
0.75
0.72-0.01
0.7410.07
0.75x0.00
0.29-0.03
0.73-003
0.75-0.05
0.79:000
15
18
Stops
Stops
Scaffold "Sample"
Patience
Patfence
517-291
568-155
655-117
583+296
521-275
747+172
091
523-165
0.74:0.04
0:
800
634:248
730:121
602:267
731
577:190
5718154
589:171
6211255
002
610x132
730:178
600
400
200
400=247
2:2
239:97
023:2270
712=104
528:259
B17:158
S19$219
806-224
878
364::101
0.75-5.04
2.00
0.74-0.07
6.77
0.7710.00
0.7510.24
6.75
0.27-0..44
0. 177--000
2.75-55.00
16
Stops
Stops
Structure "Sample"
276-241
235-140
208=133
223:165
200
245
216+170
215+115
176m22
160
210m2211
220
237
150=100
221m113
209
168
168e213
136a00
218e204
237:1104
357m285
301m155
0.7880.04
200
=8
1J6e92
228
205=177
174=:66
183+95
2d6
186m
232
0. 7560.03
0.7620004
0.200.04
0.520.02
0.1080003
6,7580.05
0.7020.3
0,7720.0
0. 0.71.0 0.792000
0.7820.04
2.P629.00
15
16
15
16
17
18
Ream Steps
Beam Steps
Searn Steps
Figure B5: illustrative experiment Generative Yield >0.8. The IntDivl (Polykovskiy et al.
(2020b
annotated.
21

---

Published as a conference paper at ICLR 2024
Generative Yield 0.8 Reward
Scaffold Min. Size
9533-170
0.76
0.74:10:5
0:79:00:02
Pool 522
324-1T3
012-172
0700:19
673:3300
45
17
18
17
Beam Staps
Beam Staps
Scaffold Min. Size 15
0.7260.00 07
0.73x0:04
6.73
0.68:-0.05
Fow Stee
252-
892-304
796-279
40
0.
0:73::0.0
17
18
17
Stapps
Staps
Structure Min. Size 10
Parlence
500m224 0.77
0.7189.20
C.7280 27
0.7450.04
Pool Stzn
400
0.7680.04
0.72m0.05
6.75e0.00
0.7400.03
200
Structure Min. Size 15
Pasience
Patience
8776-450 0.6000.23
731 0.825.033
015 C.P120.C
201:-250
116920.03
aco
Pooi Stc
600
0.6920..03
0.72:0.03
6.7.0.05
D D.P220.04
200
Figure B6: illustrative experiment Generative Yield >0.8 with Structure Minimum Size. The Int- Div1 (Polykovskiy al, (2020b) annotated.
22

---

Published as a conference paper at ICLR 2024
Generative Yield 0.8 Reward
Scaffold "Sample" Min. Size
0.77=0.02
0.77e0.03
0.7880.02
0.7883.04
Post Dioe
264:2155
649
5936229
864-1200
17
38
Beam Steos
Beam Steps
Scaffold "Sample" Min. Size 15
0.7010.03 03
0.71..5.06
6.72 17720000 0.02
0.73::0.4
Poat
477
761
816
18
17
Shaps
Bear: Staps
Structure "Sample" Min. Size 10
0.7x0.33
0.7465.00
0.72x0.04
0/72=3.05
Poa
549-318
303-109
600-505
443-295
Beam Stwae
Deam Sheps
Structure "Sample" Min. Size 15
Potience
Patorce
562=317
5975.445 0.750.05
0768221
580mm196 0.60:20.00
800
Naat Sum
600
400
0.73 0.21-:0.5
0.71-0.01
200
Figure B7: illustrative experiment Generative Yield 00.8 with Structure Minimum Size and "Sam- ple" token sampling. The IntDiv1 (Polykovskly et al. (0205) is annotated.
23

---

Published as a conference paper at ICLR 2024
Oracle Burden (10) Reward
Scaffold
Patience
Patiecce
Patencc
3200:546
3274:446
3162:617
3375249L
35784500
342424622
30404412
29452504
28641557
3018:404
31762447
3147=286
3020
Pool
1205:1062
2910-305
1561:351
3015E151
150235-11
31708-192
324915-19
330267337
32273503
31318-122
35691315
30711.150
2500
3374:508
3135--554
2018:330
3134:441
308P
3139
3071
2900
3230-523
3134-300
2870x3003
3130+614
15
16
18
15
16
18
15
16
17
18
Boams Stops
Bearn Staps
Bearn Staps
Structure
Petience
Patlence
307245355
2507
718D
30401612
3103-404
2051-640
2910H644
3112
3118
31230450
2011-243
Pho Daw
3000
30241552
3238m519
24202574
31142470
30752512
29692208
30512521
2008
34112684
2017=524
25172618
3257
2500
29708-161
3234
138914066
1102=410
3129x040
3014
3225:504
3118:514
34STA509
3285=100
3075e530
3792
15
15
15
Beam Steps
Beam Steps
Beam Steps
Scaffold "Sample"
2555
2304:345
2351
2626.724
2E17
25SRA506
22542414
22971-27
2143:403
2122-370
2323:200
Phat Cae
3000
2491:452
2224:331
23952473
2409a453
27436:448
2626:2264
24492564
2480:484
26628425
2475:4355
23592574
2532
2300
254016670
2147:385
2428&105
2515u537
2230:305
2258:5434
23362472
2604:600
2613:-04
2233:120
240924433
2419:561
15
15
Steps
Beam Steps
Deam Steps
Structure "Sample"
3136:009
3450
3347
3427-510
3227
33504054
3401
3443:003
3570w580
3491
3740
3067
35000
3000
1511-259
51155351
18272527
51508312
57455857
3750#301
54518460
5418
11223544
52008-495
3418:3307
38744-452
3684-508
35T44558
3541+502
3459H-144
3624+G1D
3513+600
3469G4441
34801-400
36654576
3028e572
1500
15
10
15
16
15
16
Stopo
Steps
Figure B8: illustrative experiment Oracle Burden (10) >0.8
24

---

Published as a conference paper at ICLR 2024
Oracle Burden (10) > 0.8 Reward
Scaffold Min. Size 10
Patience
Petiencc
3500
Pool Soz
3000
1500
Scaffold Min. Size 15
Polience
Parience
3500
3000
1500
Steps
Structure Min. Size 10
26542854
2516-633
27534979
2470=575
3500
Podt
2000
24342862
2892m845
20261819
2797=3033
17
Beam Staps
Beam Staps
Structure Min. Size 15
Patience
Petience -
Pool S32
3000
2500
1500
Steps
Stops
Figure B9; illustrative experiment Oracle Burden (10) >0.8 with Structure Minimum Size
25

---

Published as a conference paper at ICLR 2024
Oracle Burden (10) 0.8 Reward
Scaffold "Sample" Min. Size 10
2203::30
2240.279
2250:395
2218:1107
3500
Pood S00
2000
2290s:432
2337:324
2392:430
2354=143
1300
Deam
Scaffold "Sample" Min. Size 15
Pulience
Pubiance 5
1546+721
1796:977
3000
Pool
2500
1890=557
1500
17
18
Staps
Stops
Structure "Sample" Min. Size 10
2520+A10
2051-661
345G:
2774:1003
3500
Pod S20
2500
2000
2713-1083
3151=740
2708.610
2747
Beam Steps
Beam Steps
Structure "Sample" Min. Size 15
Patlerce
Partence
2500
Poot Sice
3300
1500
Figure B10: illustrative experiment Oracle Burden (10) >0. with Structure Minimum Size and "Sample" token sampling
26

---

Published as a conference paper at ICLR 2024
The set of optimal hyperparameters found here were used in drug discovery case studies. In order to be rigorous with our investigation, we only fix the following hyperparameters:
Patience = (lower variance)
Pool Size = (lower variance, higher Yield, lower Oracle Burden)
Beam Steps (lower variance, higher Yield, lower Oracle Burden)
with these hyperparameters, we small grid search (on the drug discovery case studies) by changing the Structure Type, Token Sampling Method, and Structure Minimum Size hyperparam- eters as the optimal hyperparameters in the illustrative experiment are not necessarily the optimal ones in the drug discovery experiments. The purpose of this is not to necessarily report the best performance on the drug discovery case studies but to gain insights into the optimal general parameters such that Beam Enumeration be used out-of-the-box. In real-world expensive oracle settings, tuning hyperparameters is infeasible.
All results from the drug discovery case studies shown in Section D
B.4 BEAM ENUMERATION: RECOMMENDED DEFAULT HYPERPARAMETERS
Taking into consideration all grid search experiments for the illustrative experiment and Drug Dis- covery case studies, the following optimal hyperparameters are recommended: Patience = 5, Pool Size = 4, Beam Steps 18, Structure, Structure Minimum Size = 15, "topk" sampling.
Notable differences between the final recommended hyperparameters compared those found from the illustrative experiment is that Structure and "topk" sampling are more performant than Scaffold and "sample" sampling. the illustrative experiment, "sample" sampling was sometimes more performant than "topk" sampling. We rationalize these observations follows: in MPO objectives that include physics-basee based oracles, structure specificity becomes increasingly important, e.g. spe- eific chemical motif's dock well because they form interactions with the protein. Therefore, "topk" sampling is more robust as there less variance in the extract substructures compared to "sample" sampling. We empirically observe the increased variance when using "sample" sampling measured by the standard deviation between replicate experiments (Appendix [D]. In the illustrative experi- ment where the oracle was more permissive, i.e., any rings saturated with heteroatoms would satisfy the MPO objective, small deviations in the extracted structure do not have as prominent an effect as physics-basee basee oracles which require specificity. Another observation is that Structure sampling often extracts scaffolds with "branch points" which enforces extreme bias that can lead to more fo- cused chemical space exploration. We discuss this in detail in Section DJand believe the insights are generally interesting in the context of molecular optimization landscape.
Finally, we end this section by stating that we cannot try every single hyperparameter combi- nation and the recommended values are from our grid search results which we make effort to be robust, given that we perform 10 replicates each experiment. We find that the optimal hyperparameters in the drug discovery case studies are generally the same as in the illustrative experiment.
27

---

Published as a conference paper at ICLR 2024
B.5
PSEUDO-CODE
This section contains the pseudo-code for Beam Enumeration and Augmented Memory with Beam Enumeration to show the integration.
B.5.1 BEAM ENUMERATION
This subsection contains the Beam Enumeration pseudo-code. The operator denotes every ele- ment on the left being extended by every clement on the right.
Algorithm 1: Beam Enumeration
Input: Generative Agent Amm Top k, N Beam Steps
Output: Enumerated Token Sub-sequences S
Initialization:
Hidden State = None;
Sub-sequences [Top k ;START; Tokens]; Input Vector = top k number of start tokens;
for i . do
Logits, New Hidden State +- Teagu (Input Vector, Hidden State); Tokens top & tokens from Softmax(Logits); if then
Sub-sequences +- Tokenski
Input Vector <- Tokensk:
Hidden State New Hidden State;
else
Create empty list lemp; for each seq in Sub-sequences do
seq +- seq Tokensk;
Append seq to temp;
Sub-sequences +- temp;
Clear lemp;
Input Vector Flatten Tokensk:
Hidden State + (New Hidden State repeat interleave(top k; dim 1));
return Sub-sequences
28

---

Published as a conference paper at ICLR 2024
B.5.2
BEAM ENUMERATION WITH AUGMENTED MEMORY
This subsection contains the Augmented Memory pseudo-code (from Guo al. (Guo Schwaller with Beam Enumeration integrated. All Beam Enumeration operations are bolded.
2023
Algorithm 2: Beam Enumeration Integrated with Augmented Memory
Input:
Prior "rrorr Epochs N, Augmentation Rounds A, Scoring Function S, Sigma o, Replay Buffer Size K, Patience P
Output: Fine-tuned Agent Policy
Generated Molecules G
Ageni"
Initialization:
Generative Agent 7T Agent
Prior
Diversity Filter DF;
Replay Buffer B = [}:
Substructure Pool Pool = (::
for i 4 1 to N do
Sample batch of SMILES X (z1,...;;5b) m6) with T3 no Agent"
if Pool is empty then
L Filter sampled batch of SMILES contain pooled substructures fittered
Compute reward using the scoring function S(X fittered);
Modify reward based on the diversity filter DF(S(X (flttered));
Update replay buffer B, TopK fittered, U Bi-)): if reward has improved for P successive epochs then L Execute Beam Enumeration to update the Pool
(Optionally) purge replay buffer;
Compute Augmented Likelihood log Te Augmented
loog Prior (Xyiltered) + (itteee)
Compute loss J (0) (log म "Augmenied
log T
Agent (Xiitered))):
Update the Agent's policy Ageent
for j to A do
Augment filtered SMILES X fittered Augmented"
Compute Augmented Likelihood of augmented filtered SMILES (reward is unchanged) + oS(( (Xiutered);
log AAgmentted
logg "rrior (Xfittered Augmented
Compute loss J(0), Augmented = (log π "Augmented
log no
(X (Xittereed Augmented
))
Agent
Augment entire replay buffer B Augmneneed'
Compute Augmented Likelihood on the augmented buffer (reward is the buffer stored rewards) log "T Buffer Augmented
log Prior Augmentod
+ oS(B);
Compute augmented buffer loss
[()) Butter Augmented (log Butter Augmented
log
Agent Augmented ))
Concatenate the augmented sampled SMILES loss and the augmented buffer loss (0) Augmented Memory (()), Augmented + J(0) Buffer Augmented"
Update the Agent's policy me Agent'
29

---

Published as a conference paper at ICLR 2024
C ILLUSTRATIVE EXPERIMENT
This section contains additional results from initial investigations into the feasibility of Beam Enu- meration. The illustrative experiment was performed with the following multi-parameter optimiza- tion (MPO) objective: maximize topological polar surface area (tPSA), molecular weight (MW) <350 Da, number of rings 2 2.
C.I SUBSTRUCTURE EXTRACTION
The first experiments investigated whether a sufficient substructures signal could be extracted from enumerated sub-sequences, The two parameters of Beam Enumeration (without self self-conditioning) are top denoting the top number of highest probability tokens enumerate and N number of beam steps denoting how many steps to perform token expansion for (which is also the length of the final sub-sequence). Our hypothesis that a lower top desirable as we are interested the most probably substructures. Thus, the initial experiments were grid-search with top & of 2 and N beam steps of [15, 16. 17, 18). The illustrative experiment was run for 100 epochs (6,400 oracle calls which is different from the 5,000 used in the main text experiments as this set of results is only to demonstrate that meaningful substructures can be extracted) and Beam Enumeration was applied at epochs 20, 40, 60, 80, and 100.
Table 3: Feasibility of Beam Enumeration to extract valid substructures, Top-k 2.
Beam Steps
Epoch
Epoch 20
Epoch 40
Epoch 60
Epoch
Epoch 100
15
2294/32768
3123/32768
5843/32768
5538/32768
5674/32768
8004/32768
(7.00%)
(9.53%)
(17.32%)
(24.43%)
16
4789165536
5890655336
5771/65536
11159965536
7657/65536
9771/65536
(7.31%)
(8.99%)
(8.81%)
(17.03%%)
(11.68%)
(14.91%)
17
9998/131072
15266/131072
26163/131072
24352/131072
21442/131072
31160/131072
(7.63%)
(11.65%)
(19.96%)
(18.58%))
(16.36%)
(23.77%)
18
20747/262144
33969/262144
72126/262144
48417/262144
45349/262144
46994/262144
(7.91%)
(12.96%)
(27.51%)
(18.47%)
(17.30%)
(17.93%)
Table 3 shows the absolute counts and percentage of sub-sequences containing valid substructures. While the percentage may appear low, note the absolute counts is more than enough extract some notion of most probable substructures. We use V beam steps of 18 for all experiments as we hypothesize that larger substructures can carry more information. The reason the max beam steps investigated was 18 is because of the memory overhead required for sequence expansion.
C.2 EXTRACTED SUBSTRUCTURES
To illustrate the capability of Beam Enumeration to extract meaningful substructures, Fig. shows the top most probable substructures epochs 1, 20, 40, 60, 80, and 100 based on Structure (extract any valid structure) and Scaffold (extract valid Bemis-Murcko (Bemis & Murcko (999) scaffold) using top of 2 and 18 beam steps. We make two crucial observations here. Firstly, Structure often extracts small functional groups which makes the self-conditioned filtering much more permissive as it is more likely for a molecule to possess a specific functional group than specific scaffold. Secondly, benzene appears often and perhaps unsurprisingly as it is ubiquitous in nature. Based on these observations, we design Beam Enumeration to only extract substructures containing at least one heteroatom the assumption that heteroatoms are much more informative in forming polar interactions in drug molecules, e.g., a hydrogen- bond cannot form from benzene. Finally, the general observation is that the most probable substructures gradually contain more het- eroatoms, as desired.
C.3 SUPPLEMENTARY MAIN TEXT RESULTS
In this section, we present the same table as the main text illustrative experiment. The only differ- ence is that the IntDiv1 (Polykovskiy et al. (2020a) is also annotated in the table here to show that the sample efficiency improvements of Beam Enumeration come only at a small trade-off in diver- sity (Table 4.. In agreement with observations in the hyperparameters grid search (Appendix
30

---

Published as a conference paper at ICLR 2024
'Structure'
'Scaffold'
Epoch 1
Epoch 1
HN
Epoch 20
Epoch 20
Epoch
Epoch
HN
N
Epoch 60
Epoch 60
HO
NH
NH
N
N
Epoch 80
Epoch 80
NH
NH
HN
NH
NH
H&N
NHz
HN-NH
Epoch 100
Epoch 100
NH
NH
NHz
NH
N HN--HH
NHH
H&N
NHz
Figure CII: Substructures extracted in the illustrative example varying epochs based on Structure and Scaffold.
31

---

Published as a conference paper at ICLR 2024
A). Structure extraction with 'Structure Minimum Size' enforcement leads highly specific sub- structures which decrease diversity relative to Scaffold extraction but with potential gains in sample efficiency evidenced in the drug discovery case studies (Appendix (D. We further perform statis- tical testing using Welch's t-test to compare all metrics for Scaffold with 'Structure Minimum Size' 15 and Baseline Augmented Memory (Guo & Schwaller (2023). For the experiments that had unsuccessful replicates, we use the total number of successful experiments, c.g., Oracle Burden (100), the Baseline was unsuccessful in 69/100 replicates so 31 sample size was used. Overall, all p-values are significant at the 95% confidence level.
Table 4: Illustrative experiment: Beam Enumeration improves the sample efficiency of Augmented Memory. All experiments were run for 100 replicates with an oracle budget of 5,000 calls and reported values the mean and standard deviation. Scaffold and Structure indicate the type of substructure and the number after is the 'Structure Minimum Size'. Parentheses after Oracle Burden denote the cut-off number of molecules. Parentheses after values represent the number of unsuc- cessful replicates (for achieving the metrie). The IntDiv1 (Polykovskiy et al. (2020b) is annotated under each Generative Yield. Welch's 's t-test is used compare the difference between Scaffold with "Structure Minimum Size = 15 and Baseline Augmented Memory (Guo & Schwaller (2023). All
p-values are significant.
Augmented Memory
t-test (95%)
Scaffold
15
ScufTold
Structure
p-value (N=100)
Generative pur (1)
1117 278
2.60 76
Diiversity
0.77 d: 0.11
0.73 :: 0.04
0.79 + 0.05
0.85 0:03
0,85 0.02
Generative yus (1)
819 291
700 +: 389
425 +: 256
199 122
85 56
3.70 x10 48
Diversity
0.75 +: 0.94
0.69 + 0.05
0.75 + 0.04
0.77 0.04
0.78 0.03
Onalle y (1)
577 + 310
616 +: 230
1037 + 414
897 347
1065 + 483
3,96 x10
Onalle Badent-a (10) (1)
926 -:
1881
1745 + 292
2302 + 216
4.99 x10mmt
Oncele g-r (100) (L)
1530 2: 468
1547 513
2736 : 335
2713 :: 402
3672 : 197
2.34 x100-8
Oracle y.s (1 (L)
1311 1: 628
1401 695
2423 1: 487
2295 1: 482
3164 4: 492
6.07 x100-5
Oracle uus (10)
1794 617 (1)
2009 +: 904 (1)
3124 497
3241 4: 692
4146 4: 326
6.48 x10
Onacle Bardas ss (100) (U)
2704 689
2943 811
4: 592
4415
437
4827 120
6.17 x10
C.4 BEAM ENUMERATION WORKS IN EXPLOITATION SCENARIOS
Beam Enumeration works in exploitation scenarios. all experiments were run for 100 repli- cates with an oracle budget of 5,000 calls and reported values are the mean and standard deviation. Parentheses after Oracle Burden denote the cut-off number of molecules. The IntDiv1 (Polykovskiy al. (2020b) is annotated under each Generative Yield. Welch's t-test used to compare the dif- ference between Scaffold with 'Structure Minimum Size' 15 and Baseline Augmented Memory
Table 5:
(Guo & Schwaller (2023). All p-values are significant.
Metric
Angmented Memory
Welch's 1-test (95%)
Beam Scaffold 15
Baseline
p-value (N=100)
Geacrative Yield (f)
1325 468
496 108
1.54 x10 29
Diversity
0,76 0.04
0.85 0.02
Generative Yieldges (f) Diversity
601 :: 298
85 56
1.35 x10
0,70 0.09
0.78 0.03
Orucle Burden (1) (
626 260
1085 1: 483
4.52 x10
Oracle Burden (10) (L)
997 326
2392 + 216
2.26 x10 80
Oracle o7 (100) (1)
Oracle Burdens-os (1)
1415 : 645
3164 + 492
2.21 x10 63
Oracle ou (10)
1794 553 (2)
4146 +: 326
1.14 x10
76
Burden (100) (1)
2490 07.677600
4827
(69)
x10 -25
In the main text illustrative experiment, Augmented Memory (Guo & Schwaller (2023) was used with Selective Memory Purge activated which is the mechanism to promote chemical space explo- ration, as described in the original work. For completeness, we show that Beam Enumeration also works in pure exploitation scenarios where the goal is only to generate high reward molecules even if the same molecule is repeatedly sampled (Table We perform statistical testing using Welch's t-test to compare all metrics for Scaffold with 'Structure Minimum Size' = 15 and Baseline Aug- mented Memory (Guo & Schwaller (2023). For the experiments that had unsuccessful replicates,
32

---

Published as a conference paper at ICLR 2024
Beam Enumeration Self-conditioning: Number of Molecules Kept
70
Structure
Scaffold
50
40
Kept
20
10
10
20
30
40
50
60
70
80
90
Epoch
Figure C12: Behaviour Beam Enumeration using Structure and Scaffold on self-conditioning.
the total number of successful experiments, e.g., Oracle Burden (100), the Baseline was unsuccessful in 69/100 replicates so a 31 sample size was used. Overall, all p-values are significant the 95% confidence level.
C.5 SELF-CONDITIONED FILTERING: Structure vs Scaffold
There is a clear discrepancy in the substructures extracted by Structure and Scaffold. In particular, Structure substructures contain small functional groups which is much more permissive when used as a filter criterion compared to full scaffolds. Therefore, one would expect that many molecules in the sampled batches would be kept when using Structure Beam Enumeration. We plot the av- erage number of molecules kept out of 64 (batch size) across the generative run when using Beam Enumeration. Note that the experiments ran for variable epochs due the stochasticity Enumeration self-filtrring. The number of epochs shown in in is the minimum number of epochs out of 100 replicates. Therefore, the average values shown are averaged over 100 replicates. is evident that Structure is more lenient as many generated molecules make it through the filter compared to Scaffold which maintains relatively strict filter. One interesting observation is that self-conditioning does not lead to obvious mode collapse. Self-conditioning is inherently biased and one would be concerned that the model gets stuck at generating the same molecules repeatedly. The fact that self-conditioning with Scaffold continues to filter throughout the entire generative shows that the model is continually moving to new chemical space, supporting findings from the original Augmented Memory (Guo & Schwaller (2023) work that Selective Memory Purge (built-in diversity mechanism) is capable of preventing mode collapse.
Beam
D DRUG DISCOVERY CASE STUDIES
This section contains information on the Autodock Vina TToott & Olson 2010) docking protocol from receptor grid preparation to docking execution. The Beam Enumeration hyperparameters grid search results presented for all three drug discovery ples of extracted substructures are also shown and commentary provided to their significance and explainability. Finally, the wall times of all experiments are presented.
studies followed by analysis. Exam-
33

---

Published as a conference paper at ICLR 2024
D.1 AUTODOCK VINA RECEPTOR PREPARATION AND DOCKING
All docking grids were prepared using DockStream (Guo et al. 202 la) which uses PDBFixer East- man et al. (2017) to refine receptor structures. The search box for all grids was 15Á 15Á 15A. Docking was also performed through DockStream and followed a two step process: conformer gen- eration using the RDKit Universal Force Field (UFF) (Rappé et al. (1992) with the maximum con- vergence set to 600 iterations and then Vina docking was parallelized over 36 CPU cores (Intel(R) (e) Platinum 8360Y processors).
DRD2 Dopamine Type 2 Receptor. The PDB ID is 6CM4 (Wang et al. (018) and the docking grid was centered at (x, y, 2) (9.93, 5.85, -9.58).
MK2 MK2 Kinase. The PDB ID is 3KC3 (Argiriadi et al. (2010) and one monomer was extracted. The docking grid for the extracted monomer was centered at (x, y. 7) (-61.62, 30.31, -21.9).
AChE Aceetyycholinesterasee The PDB ID is IEVE (Kryger et al. (1999) and the docking grid was centered (x, y, 2) = (2.78, 64.38, 67.97).
D.2 BEAM ENUMERATION HYPERPARAMETERS GRID SEARCH RESULTS
We performed an additional hyperparameter grid search on all three drug discovey case studies based on the insights drawn from the illustrative experiment grid search results. We fix the following hyperparameters:
Beam 2
Beam Steps 18
Pool Size =
Patience 5
and vary the following:
Optimization Algorithm [Augmented Memory (Guo & Schwaller (2023). REINVENT (Olivecrona et al. (2017) 7) Blacchke et al. (002))
Substructure Type [Scaffold, Structure]
Structure Minimum Size [10, 15]
Token Sampling Method ["topk", "sample"]
All hyperparameter combinations (8) were tried and run for 3 replicates each for statistical reproducibility, total of 144 experiments. There are two main results we want to convey: firstly, the optimal hyperparameters are the same for all three drug discovery case studies and only the Substructure Type differs between the optimal hyperparameters here and the illustrative experiment. Secondly, Beam Enumeration is a task-agnostic general method that can be applied to existing algo- rithms including Augmented Memory (Guo & Schwaller (2023) and REINVENT (Olivecrona et al. 2017) Blaschke et al. 2020). At the end of this section, present these hyperparameters and designate these the default values. All grid search results are now presented in following tables:
Based on the results from the hyperparameters grid search in the drug discovery case studies, make two key observations: firstly, Structure extraction with 'Structure Minimum Size' = 15 is now the most performant, average (for both Augmented Memory (Guo & Schwaller (2023) and REIN- VENT (Olivecrona et al. (2017), Blaschke et al. (2020)). This is in contrast to Scaffold extraction in the illustrative experiment which we rationalize through the permissive nature of the experiment compared to the docking experiments which require structure specificity. Previously, small devia- tions in the substructures may not have a significant impact on the reward. In physics-based oracles such as Vina (Trott & Olson (2010) docking used here, small substructure differences can have an enormous impact on the outcome since the pose requires specific complementary to the protein bind- ing site. The second observation
make which is in agreement with the illustrative experiment that "sample" token sampling has more variance and does not perform better than "topk". The rationale is the same in that docking requires specificity and lower probability substructures exhibit more variable performance. Based on all the observations from the illustrative experiment and the drug discovery case studies, designate the following default hyperparameter values:
34

---

Published as a conference paper at ICLR 2024
Table 6: DRI2 Waang et al. (2018) study hyperparameters grid search results for Augmented Memory (uuo & Schwaller (023). All experiments were run in triplicate and the reported values are the mean and standard deviation. "Sample" denotes "sample" token sampling. All metries are for the reward threshold ~0.8. The IntDivl (Polykovskiy et al, (2020b) is annotated under Generative Yield.
and * denote one and two replicates were unsuccessful, respectively.
Experiment
Generative Yield
Unique Scaffolds 322 + 166
Oracle
Oracle Burden (10)
Oracle
Augmented Memory DRD2
Burden (I)
Burden (100)
Raseline
363 + 195
187 + 51
711 + 120
2558 + 30
Diversity
0.019
Scaffold
957 75
749 4: 62
29
668
1818 107
DDiversity
0,765 0.006
Scaffold Size
1607
1023 =
29
571 +
1056 = 146
Diversity
0.724 0.027
Scaffold Sample
948 # 123
776 128
126
503 = 17
1746 = 20
Diversity
Scaffold Sample Size
1552 = 106
1274 154
84 = 29
598 E 110
[511 = 416
Diversity
0.660 0.041
Structure
112
711 133
154
Diversity
0.764 1: 0.008
Structure Size 15
1780 =
1323 =
126 ::
562
1120 194
Diversity
0.699 0,020
Structure Sample
912 86
757 1 30
63 1:0
583 1: 37
2132 1: 148
Diversity
Structure Sample Size Diversity
1752 105
1352 180
188 103
776 1:
1289 193
0.641 0.059
Table DRD2 (Wang al. (2018) case study hyperparameters grid search results for REINVENT (Olivecrona et al. (2017) B Blacckke et al. (2020). All experiments were run in triplicate and the reported values are the mean and standard deviation. "Sample" denotes "sample" token sampling. The IntDiv1 (Polykovskiy et al. (2020b) is annotated under Generative Yield. All metrics are for the reward threshold >0.8. & and * denote one and two replicates were unsuccessful, respectively.
Experiment
Generative Vield
Unique Scaffolds
Oracle
Oracle
Oracle
REINVENT DRD2
Burden (I)
Burden (10)
Burden (100)
1
0
Diversity
0,833 1. 0.001
Scaffold
190
184 : 32
::
836 :: 178
3516 575
Diversity
0.814 1: 0.007
Scaffold Size 15
687 + 366
377 204
127 4: 52
604 + 71
2109 + 1090
Diversity
0.730 = 0.013
Scaffold Sample
176 86
149 1:
105 d:
720 121
3875 883
Diiversity
0.801 = 0.030
Scaffold Sample Size
3170
Diversity
0.704 + 0.044
Structure
184
183 = 14
104 =
897 + 100
3426 2 282
-iiversity
0.006
Structure Size 15
417 = 275
200 178
63:50
1099 - 930
1928 002821 117
Diverriity
0.730 = 0.014
Structure Sample
126
711
Diversity
0.826 1: 0.003
Structure Sample Size
261 = 225
182 = 132
209 128
840 = 107
3690 :: 1266
Diversity
0.734 1: 0.057
Beam K ==2
Beam Steps = 18
Pool Size = 4
Patience =
Substructure Type Structure
Structure Minimum Size 15
Token Sampling Method "topk"
35

---

Published as a conference paper at ICLR 2024
Table 8: MK2 (Argiriadi et al. (2010) case study hyperparameters grid search results for Augmented Memory (uuo & Schwaller (2023). All experiments were run in triplicate and the reported values are the mean and standard deviation. "Sample" denotes" "sample" token sampling. All metries are for the reward threshold 00.8. The IntDivl (Polykovskiy et al, (2020b) is annotated under Generative Yield. and ** denote one and two replicates were unsuccessful, respectively.
Experiment
Generative Yield
Unique Scaffolds
Oracle
Oracle Burden 3833 + 394
Oracle Burden (100) Failed
Augmented Memory MK2
Burden 1360 + 543
Raseline
34 + 13
32 12
Diversity
0.794 + 0.008
Scaffold
179
131 + 16
1163 d: 457
2550 :: 148
4421 344
Diversity
0.743 + 0.038
Scaffold Size 13
523 = 438
330 260
1221 # 564
2426 1325
2676 =: 403
Diiversity
0.676 + 0,016
Scaffold Sample
106 = 71
87 58
1005 + 573
3296 =: 1181
4592 =: 334
Diversity
0.722 0.017
Scaffold Sample Size 15
379 + 357
257 257422 227
983 540
1846 = 680
3244 = T133"
-Diversity
0.653 + 0.026
Siructure
18
59 20
1246 d: 716
2708 ::
Failed
Diversity
0.769 1: 0.029
Structure Size
987 + 211
610 117
736 166
1122 #: 154
2189 = 181
Diversity
0.704 1. 0,030
Stracture Sample
40 + 15
34 11
1119 1183
3516 d: 506
Failed
Diversity
0.784 + 0.024
Structure Sample Size 15
129
117 1: 50
1208 1 660
2799 1: 476
4037 0
Diversity
0.671 + 0.073
Table 9: MK2 (Argiriadi et al. (2010) case study hyperparameters grid search results for REINVENT (Olveecoona et al. (2017): ; Blaschke et al. (2020). All experiments were run triplicate and the reported values are the mean and standard deviation. "Sample" denotes "sample" token sampling. All metrics are for the reward threshold 0.8. The IntDiv1 (Polykovskiy et al. (2020b) is annotated under Generative Yield. & and #* denote one and two replicates were unsuccessful, respectively.
Experiment
Generative Yield
Unique Scaffolds
Oracle
Oracle Burden (10)
Oracle
REINVENT MK2
Burden (1)
Burden (100)
Baseline
2 2 0.424 + 0.031
2
1723 1 802
Failed
Failed
Diversity
Scaffold
7 + 0.704 + 0.051
2
1272 4 884
4948 0
Failed
Diversity
Scaffold Size
7 0.674 + 0.065
1
808
Failed
Diversity
Scatfold Sample
6 6 2
6
1427 = 343
Failed
Failed
Diversity
0.677 + 0,75
Scaffold Sample Size 15
4 2
3
2600 + 1455
Failed
Failed
Diversity
0.653 + 0.026
Structure
3 3=1
3
2571 1155
Failed
Failed
Diversity
0.571 1: 0.112
Structure Size 15
179 + 241
70 87
1110 :: 268
1778 0
3208 0
Diversity
0.670 1 0,020
Structure Sample
1 0
1
1737 =: 1595
Failed
Failed
Diversity
0.192 0.271
Structure Sumple Size 15
8::.5
714
1943 1E 1153
4851 0
Failed
Diversity
0.357 + 0.255
D.3 EXAMPLES OF EXTRACTED SUBSTRUCTURES: Structure EXTRACTION WITH "STRUCTURE MINIMUM SIZE' = 15
In this section, the top substructures at the end of the generative experiments (using Augmented Memory (Guo & Schwaller (2023) are shown for all three drug discovery case studies (3 replicates). All experiments are for Structure extraction with 'Structure Minimum Size' 15. The extracted substructures are commonly scaffolds with "branch points". i.e., a central scaffold with single carbon bond extensions outward, which heavily bias generation. We posit that this may be reason why Structure extraction can be more performant than Scaffold, as observed in the hyperparameters grid search in the previous subsection.
36

---

Published as a conference paper at ICLR 2024
DRD2 'Structure' Extraction with 'Structure Minimum Size' = 15
Replicate 1
Replicate 2
Replicate
Figure D13: Augmented Memory (Guo & Schwaller (2023) DRD2 Wang et al. (2018) substructures
with Structure extraction and 'Structure Minimum Size after 5,000 oracle calls.
MK2 'Structure' Extraction with 'Structure Minimum Size' 15
Replicate
Replicate
Replicate
Figure D14: Augmented Memory (Guo & Schwaller (2023) MK2 (Argiriadi al. 2010 substruc-
tures with Structure extraction and 'Structure Minimum Size' 15 after 5,000 oracle calls.
37

---

Published as a conference paper at ICLR 2024
Table 10: AChE (Kryger et al. | (1999) case study hyperparameters grid search results for Augmented memory (Guo& & Schwaller (2023). All experiments were run in triplicate and the reported values are the mean and standard deviation. "Sample" denotes "sample" token sampling. All metries are for the reward threshold 00.8. The IntDivl (Polykovskiy et al, (2020b) is annotated under Generative Yield. and ** denote one and two replicates were unsuccessful, respectively.
Experiment
Generative Yield
Unique Seaffolds
Oracle Burden (I)
Oraele
Oracle
Augmented Memory AChE
Burden (10)
Burden (100)
Bascline
356 4.
544 50
62
380 =
2021
Diversity
0,838 +: 0.002
Scaffold
1058 + 102
1006 = 113
62 =
430 90
1400 = 56
Diversity
0,823 + 0,005
Scaffold Size 15
2124 + 326
1323 + 200
63 0
418 27
884 = 162
-Diversity
Scallold Sumple
1187 1: 48
1075
84 d:
409 77
1519 1: 141
DDiversity
0,806 1: 0.003
Scallold Sample Size 15
+:
188
602 ::
1440 115
Diversity
0,750 1. 0.021
Structure
992 1: 64
946 +: 52
105 : 59
558 1: 94
1635 =: 81
Diversity
0.823 0.005
Stracture Size 15
2059 1: 327
1552 4: 344
105 d: 29
462 d: 25
1110 1: 265
Diversity
0.735 0.017
Stracture Sumple
831 126
790 130
62 1:
357 29
1617 220
Diversity
0.841 + 0.003
Structure Sample Size Diversity
1277 +
1031 = 421
=
800 = 342
1879 531
0.657 0.070
Table 11: AChE Kryger et al. 1999) case study hyperparameters grid search results for REINVENT (Oliveerona et al. (2017): ; Blaschke et al. (2020). All experiments were run in triplicate and the reported values are the mean and standard deviation. "Sample" denotes "sample" token sampling. The IntDiv1 (Polykovskiy et al. (2020b) annotated under Generative Yield. All metrics are for the reward threshold >0.8. & and denote one and two replicates were unsuccessful, respectively.
Experiment
Generative Yield
Unique Scaffolds
Oracle
Oracle Burden (10)
Oracle
REINVENT AChE
Burden (1)
Burden (100)
Bascline
147 = 11
146 II
83 29
481 = 108
3931 = 280
Diversity
0.852 0.004
Scaflold
245 1. 50
244 1: 50
63 0
566 1. 136
3360 11 164
Diversity
Scallokd Size 15
310 207
227 1 159
84 29
421 1. 120
3596 1: 678
Diversity
0.744 +: 0.038
Scallold Sample
1:
480
Diversity
0.847 1: 0.004
Scaffold Sample Size 15
310 1 92
271 + 70
148 28
673 = 107
2881 = 475
Diversity
0.759 0.039
Stracture
356 +: 22
351 +: 24
63
294 +: 28
2284 d: 238
Diversity
0.841 0.002
Structure Size
323 1: 58
284 71
62
441 :: 132
3073 =:: 427
Diversity
0.795 + 0.009
Structure Sample
213 = 20
206 22
84 = 30
558 - 222
3073 = 279
Diversity
0.844 + 0.005
Structure Sample Size 15
316 + 253
190 # 146
125 = 50
561 # 140
2683 = 320
Diversity
0.721 0.111
D.4
WALL TIMES
The wall times for all drug discovery case studies with every algorithm is presented in Table 12 The reported values are averaged over replicates. In general, adding Beam Enumeration the base Augmented Memory (Guo & Schwaller (2023) and REINVENT (Olveecoona et al. B jt al. 2020) algorithms increased wall times but only slightly and it is negligible when considering expensive oracles. An interesting observation is that "sample" token sampling increases wall time variance. This is because less probable substructures lead to more "filter rounds", i.e., epochs where all the sampled molecules are discarded as none of them contain the Beam Enumeration extracted substructures. In addition, REINVENT generally has longer wall times even though the oracle bud- get is the same. The reason for this is because REINVENT optimizes the structure components of the MPO objective: QED (Bickerton et al. (012) and MW constraint to lesser extent. Conse- quently, REINVENT generates larger molecules, average, which take longer to dock with Vina
Blaschke
38

---

Published as a conference paper at ICLR 2024
AChE 'Sttructure' Extraction with 'Structure Minimum Size' 15
Replicate 1
Replicate 2
Replicate 3
NH
HN
AChE (Kryger et al. (1999) substruc- tures with Structure extraction and 'Structure Minimum Size 15 after 5,000 oracle calls.
Figure D15: Augmented Memory (Guo & Schwaller 22023
Table 12: Wall times for all drug discovery case studies hyperparameters grid search using Aug- mented Memory (Guo Schwaller (2023) and REINVENT (Olivecrona et al. (2017); Blaschke al. (2020). "Sample" denotes "sample" token sampling. All experiments were run in triplicate and the values are the mean and standard deviation.
Target DRD2
Experiment
Augmented Memory Wall Time
REINVENT Wall Time
Baseline
14h 0m 1: 1h 26m
16h 36m 1: 0h 55m
12h 58m
Scaffold Size 15
12h 56m = 0h 46m
16h 51m
1b 58m
Scaffold Sample
12h 11m + 24m
16h 32m :: 3m
Scaffold Sample Size
13h 32m 1: 50m
16h 26m 2h 58m
Semettre
14h 30m + 5 1m
22h 5m :: 52m
Stmeture Sample
13h 58m + Oh 5
20t 5m #: 1h 42m
Structure Sample
14h
32m
3h
MK2
Baseline
10h 46m Oh
15h 19m 0h 34m
Scaffold
1th 0m + Oh 28m
16h 21m = Oh 53m
Scaffold
+
1h
Scaffold Sample
12h 56m : Uh 36m
15h 49m : 0h 36m
Scaffold Sample Size
11h 52m 1h
28m Oh 33m
Structure
12h 29m =
19h 40m 1h 55m
Structure Size 15
11h 22m 1: 17m
18h 39m 1h 33m
Strettre Sample.
12h 22m +: 28m
18h 12m Oh 57m
Scructure Sample Size 15
12h 37m + 0h 47m
16h 6m = 37m
AChE
39m
Scaffold
11h 46m 1: 51m
15h 10m :: 4m
Scaffold Size
11h 10m 1: 44m
15h 52m ::
Scaffold Sample
10h 55m + 44m
27m 0h 57m
Scaffold Sample Size 15
10b 24m + 0h 17m
14h 53m = 0b 53m
Structure
Structure Size 15
11h 26m 1: 0h 51m
18h 30m 1. 0h 20m
Structrre Sample
Structure Sample Size 15
17h 56m 1: 27m
19h 16m
2h 43m
Trott & Olson (2010). This observation agreement with the original Augmented Memory work which compared to REINVENT.
39

---

Published as a conference paper at ICLR 2024
E AUGMENTED MEMORY AND REINVENT MODEL HYPERPARAMETERS
Table 13: LSTM model hyperparameters for Augmented Memory (Guo & Schwaller (2023
and
REINVENT Oliveeronn et al. 2017)cBlaacchke Blaschke et al. (2020)
Cell Type
LSTM
Number of Layers
Embedding Layer Size
256
Dropout
0
Training Batch Size
128
Sampling Batch Size
64
Learning Rate
0.001
The same pre-trained prior ChEMBL (Gaulton et al. (2012b) used for Augmented Memory
(Guo & Schwaller (2023) and REINVENT (Olivecrona et al. (2017): Blaschke et al.) (2020). All shared hyperparameters (sampling batch size and learning rate) are the same. Default additional hyperparameters for Augmented Memory were used based on the original work (Guo & Schwaller (2023): two augmentation rounds and using Selective Memory Purge prevent mode collapse.
Experience replay (Lin (1992); Blaschke et al. (2020) was kept default in REINVENT (randomly sample 10 molecules out of 100 from the replay buffer each epoch).
F BEAM ENUMERATION: BEYOND LANGUAGE-BASER FORMULATIONS
This section discusses potential extensions of Beam Enumeration other classes of molecular gen- erative models beyond language-base formulations.
F.1 DIRECT APPLICATION
Beam Enumeration enumerates from local probability distributions and is thus well-suited for toregressive models. Correspondingly. autoregressive molecular generative models are not limited to language-based formulations. As conerete example, GraphINVENT (Mercado et al.) (2021b a) generates molecular graphs and has been coupled with RL (Atance et al. (2022). During the opti- mization process, instead of token probabilities being updated, graph action probabilities are tuned, which denote discrete actions to construct the molecular graph, such as adding a node or edge. Beam Enumeration can be directly applied by exhaustively enumerating the most probable set of graph actions and extracting substructures (conditional on the molecular graph being valid).
F,2 APPLYING OBSERVED FINDINGS
Many generative models construct a substructure "vocabulary" such that the optimization process is defined by how to combine these building blocks (Jin et (2018); Guo et al. (2022). This similar token sampling, except the "tokens" are now "words" (groups atoms). The empirical results from Beam Enumeration show that substructure biasing is beneficial, even early in a generative experiment when the model has not necessarily learned generate favorable molecules. Given promising seed molecule, existing models such as the JT- (Tin et al. (2018) and the Data Efficient Graph Grammar (DEG) (Guo et al. (2022) models can used to propose local neighbourhood expansions. Scoring these molecules enable the possibility of iteratively performing substructure (of increasing size) biasing, effectively leveraging both the advantages of the learned "vocabulary" of building blocks, and the iterative construction of increasingly favorable substructures. Results in this work suggest that this can be performant as enforcing larger substructure size always improved sample efficiency (up to size of 15 in this work).
40

---

Published as a conference paper at ICLR 2024
G BEAM ENUMERATION VS. BEAM SEARCH
Beam Enumeration is distinct from Beam Search (Graves (2012) Boulanger-Lewanddosk et al. 2013). In this section, contrast the behavior point-by-point
Objective: Beam Search approximates finding the highest probability sequences by maintaining "width" of the top candidates each step. The number of sequences decoded via Beam Search is exactly k. By contrast, Beam Enumeration exhaustively enumerates the top k tokens at each step to extract substructures, such that the total number of sub-sequences is B-Bam Steps
Search Space: Beam Search keeps the top k candidates globally, pruning less probable sequences. Beam Enumeration keeps the top local candidates per sub-sequence.
Termination: Beam Search terminates when sequence ends or a maximum length is reached. Beam enumeration does a pre-defined number of steps, controlled by Beam Steps.
Use of results: Beam Search returns the highest probability candidates for direct use (for instance, as full decoded molecules Moret et al. (202)) Beam Enumeration uses the enumerated sub- sequences to extract molecular substructures for self-conditioned generation.
In summary, Beam Enumeration performs local, exhaustive enumeration of sub-sequences to extract meaningful substructures, while Beam Search approximates finding set of globally optimal (by probability) sequences. The exhaustive sub-sequence enumeration enables probabilistic extraction of knowledge from the model.
41


Reviews:
[{"review_id": "mrLiGRFcBz", "reviewer": "Reviewer_3svX", "summary": "Generative molecular design has transitioned from theoretical validation to practical utility, evidenced by a recent surge in papers featuring experimental confirmation. In this evolving landscape, addressing challenges related to explainability and sample efficiency becomes crucial for optimizing high-fidelity oracles efficiently and providing valuable insights to domain experts. This paper introduces Beam Enumeration as a solution, aiming to exhaustively enumerate the most probable sub-sequences from language-based molecular generative models. The method demonstrates the extraction of meaningful molecular substructures. When integrated with reinforcement learning, these extracted substructures contribute to enhanced explainability and improved sample efficiency through self-conditioned generation. Notably, Beam Enumeration is adaptable to any language-based molecular generative model and significantly enhances the performance of the Augmented Memory algorithm, a recent state-of-the-art achievement in sample efficiency on the Practical Molecular Optimization benchmark. The joint focus on explainability and sample efficiency makes Beam Enumeration a pioneering method in the field of molecular design.", "questions": "Is it possible to incorporate MD-based simulation as oracle? How long does it take for each oracle call? Can you provide more details?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper is well-written, well-organized, and easy to follow. \n- The idea is quite intuitive and straightforward. It is based on the state-of-the-art language model--augmented memory. \n- The paper also emphasizes sample efficiency, which is an essential problem in practical molecular design. It does not only require the optimization ability but also the oracle efficiency. The oracle can be computationally expensive and the bottleneck of the modern molecular design. \n- The whole setup is realistic. Novel Oracle efficiency-based metric is designed. \n- Code is easy to read and is public, guaranteeing the reproducibility. \n- The experimental results are thorough and solid. The proposed beam enumeration significantly outperforms REINVENT (the strongest baseline in the existing benchmark). It also exhibits explainability during the generation.", "weaknesses": "- It would be great if authors could incorporate more baseline methods. \n- Some minor issue, e.g., repetitive statements, like \n\"To address this, Gao et al. (Gao et al. (2022) proposed the Practical Molecular Optimization (PMO) (Gao et al. (2022) benchmark.\" Too many \"Gao et al\". \n- Do you use the same setup with PMO? If not, it seems unfair to compare with PMO's best baseline REINVENT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AHm58kD6cK", "reviewer": "Reviewer_La1F", "summary": "In this paper, the authors propose \"Beam Enumeration\", a technique that can be used to enhance any language-based molecular generation models in terms of sampling efficiency and explainability.\nThe key idea is to exhaustively enumerate the most probable subsequences, and the model's weights are updated such that high-reward molecules become more likely to be generated in future trajectories. \nThis is achieved by \"self-conditioning\" the generative process by screening batches based on the presence or absence of substructures that are likely to be part of full molecules with high reward scores.", "questions": "Please see the questions and suggestions raised in the section Weaknesses above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Overall, this is an excellent paper that is very well written.\nThe proposed idea is relatively simple but well-motivated, intuitive, and generally applicable to various generative molecular design models based on language models.\nBeam Enumeration, proposed in this paper, has been applied to a state-of-the-art generative language model for molecular design - called Augmented Memory - and results demonstrated that the incorporation of Beam Enumeration was able to further enhance the performance of the Augmented Memory algorithm, which was already shown to outperform other existing methods.\nNotably, the proposed scheme enhances the sampling efficiency significantly and the extracted top-k substructures during the generation process are shown to provide additional sources of explainability - by connecting the presence of most likely substructures in full molecules with high rewards.\n\nThroughout the paper, the authors provide ample intuition and novel insights regarding not only \"how\" the proposed Beam Enumeration scheme works but also \"why\" they may lead to improved sample efficiency and also contribute to better explainability.", "weaknesses": "The overall paper is written very clearly and easy to follow.\nThroughout the paper, the authors distill intuitive explanations and derive meaningful insights, conclusions are drawn with sufficient evidence to back them up, and hypotheses/speculations are made in a reasonable and logical manner.\nAs a result, I do not have any major concerns but have a number of mostly minor remarks to improve the presentation even further.\n\n1. There are additional results in the appendix regarding the impact of the proposed Beam Enumeration scheme on reducing diversity.\nWhile the main text briefly mentions such diversity reduction is expected, the paper would benefit from having (at least some) further discussion (in the main text) on this effect and its extent.\n\n2. There should be further investigation of the impact of k (i.e., the number of top substructures) on the performance of Beam Enumeration.\nThe authors hypothesize that a low top k would be optimal, but wouldn't the use of small k potentially lead to molecules with repetitive substructures and undesirably limit the molecular diversity?\n\n3. Similarly, the manuscript could benefit from having further discussion on the optimal choice of the \"Structure Minimum Size\" - currently set at 15 in the experiments - and its impact on the overall performance.\nIt is mentioned that \"larger substructures ... improves performance\" but what is the best minimum size to use?\nIncreasing it beyond some number would certainly degrade the performance of the molecules and also limit the diversity of the generated molecules significantly.\n\n4. The multi-property optimization (MPO) aspects will need to be discussed in further details.\nCurrently, the MPO appears to completely rely on the baseline algorithm (e.g., Augmented Memory or REINVENT) to be extended with the Beamn Enumeration capability.\nBut since different substructures might be associated with different properties, incorporation of Beam Enumeration for MPO would have to consider the impact of self-conditioned molecular generation when used with a specific baseline algorithm and a specific number of properties for optimization.\nFor example, one may expect that when optimizing a large number of properties, then using a very small k may be detrimental as the selected substructures may be associated with only some of the properties of interest.\n\n5. Finally, while the paper focuses on language models for molecular generation and uses Beam Enumeration for self-conditioning the generative process by filtering the most probable substructures, it would be great if the authors could discuss how similar ideas may be used to extend other types of popular generative molecular design models (e.g., latent space models based on VAE, JT-VAE).\nEspecially, models like the JT-VAE optimize molecules by sampling & assembling substructures (in the form of junction trees), which - at least at a conceptual level - *may* be related to the core \"self-conditioning\" idea in Beam Enumeration, to prioritize certain substructures.\nWhile this may be beyond the scope of the current work, having at least some high-level discussion might benefit the readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tbzAz4lPpj", "reviewer": "Reviewer_d4Xh", "summary": "The manuscript discusses the use of beam search enumeration in tandem with RNN-based reinforcement learning for molecular design. The results demonstrate that meaningful substructures can be derived, which provide insights into the properties they confer. Utilizing these substructures as filters, there is a notable increase in the sample efficiency of molecular optimization. Though there are a few flaws, the results and methods seem solid. Overall, I would recommend the paper for acceptance if authors could address my concerns.", "questions": "- Would it be possible to exhibit the SA score, as per reference [1], for the molecules generated in the context of drug discovery (Figure 3)?\n- Have the authors considered employing a more advanced language model, like the transformer, as an alternative to LSTM?\n- Is there a direct comparison between the presented methods and those in PMO using the original metrics?\n\n### Reference\n[1] Ertl, P., & Schuffenhauer, A. (2009). Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of cheminformatics, 1, 1-11.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Importance of the problem\nThe paper delves into oracle-agnostic molecular optimization, which holds significance in the realm of AI-driven de novo drug design.\n\n## Solid empirical performance\nA marked improvement in empirical performance has been observed.\n\n## Open-sourced code\nThe authors have made the code available, facilitating comparisons and further research.", "weaknesses": "## Ambiguity in Methodology\nThe methodology entails several phases, namely training RNN agents, beam enumeration, and resampling based on substructures. From the manuscript, it appears that only beam enumeration does not invoke the oracle while the others do. The allocation of the oracle budget across these stages is unclear. A comprehensive pseudo-code encompassing the entire algorithm, rather than just the beam enumeration, would be beneficial.\n\n## Exaggerated Claims\nThe assertion that this is the \"first method to concurrently address explainability and sample efficiency\" is overstated. Guo et al. [1] have put forth a concept strikingly akin to the one proposed here, where they too derive interpretable substructures and then generate novel molecules. Likewise, Fu et al. [2] have purportedly tackled both interpretability and enhanced sample efficiency.\n\n## Potential nomenclature confusion\nBeam search is a well-established technique within the NLP community. I would recommend revisiting the nomenclature for the proposed method.\n\n\n### Reference\n[1] Guo, M., Thost, V., Li, B., Das, P., Chen, J., & Matusik, W. (2022). Data-efficient graph grammar learning for molecular generation. arXiv preprint arXiv:2203.08031.\n[2] Fu, T., Gao, W., Xiao, C., Yasonik, J., Coley, C. W., & Sun, J. (2021). Differentiable scaffolding tree for molecular optimization. arXiv preprint arXiv:2109.10469.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f0qvSXTECm", "reviewer": "Reviewer_wYf5", "summary": "Through this paper, the authors aim to increase explainability and sample efficiency in drug discovery problems. To accomplish this, the authors proposed Beam Enumeration that exhaustively enumerate the most probable sub-sequences from Augmented Memory.", "questions": "Please see the *Weaknesses* part for my main questions.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The authors provided the codebase.\n- The writing is easy to follow. The concept figure aids the understanding.", "weaknesses": "The main weaknesses of the paper are that the contribution (both conceptual and technical) is marginal and the proposed method is heuristic rather than machine learning-based (and thus seems out of scope for ICLR) and lacks a mathematical basis. The proposed method relies primarily on Augmented Memory [1], and the methodology outside of Augmented Memory (*Beam Enumeration* ~ *Self-conditioned Generation* paragraphs in Section 3) are short and seems like an additional trick.\n\nI will combine the *Weaknesses* section and the *Questions* section. My concerns and questions are as follows:\n\n- What is the difference of the generation scheme in Figure 1a with that of Augmented Memory?\n- The *Autoregressive Language-based Molecular Generative Model* paragraph in *Proposed Method* section should be moved to *Related Work* section as that part is not proposed by this paper.\n- Are there any ablation studies that quantify the effects of Augmented Memory? Are there experiments that quantify the effects of the combination of Beam Enumeration and other generative models?\n- The method description seems a bit vague. I recommend to explain the self-conditionining scheme in paragraph *Self-conditioned Generation* more concretely.\n\n---\n\n**References:**\n\n[1] Guo et al., Augmented memory: capitalizing on experience replay to accelerate de novo molecular design, arXiv 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)